diff --git a/NEWS.md b/NEWS.md index ac73d39..60d4f35 100644 --- a/NEWS.md +++ b/NEWS.md @@ -24,6 +24,8 @@ ### Notes +- Document `?kit` with a package page that explains OpenMP threads, platform support, encoding behavior, and the main functions. Add "OpenMP and threads" and "Platforms" sections to the README, and add `library(kit)` to the first README example (#90). + - Reduce peak memory in `funique`, `fduplicated`, `uniqLen`, `psort`, and `charToFact` by starting duplicate hash tables at 256 slots and growing them on demand (4x, capped at the previous full-size table) instead of allocating the full table up front. Rehashing re-reads the source values, so results are unchanged; inputs with few unique values use much less memory, while peak memory can be higher at growth boundaries for high-cardinality inputs (#74). - Speed up `vswitch` for string and list outputs by scanning `x` once instead of once per value and fusing default/`na` writes into the match pass; single-value lookups also avoid a separate initialization pass. Atomic outputs keep the value-major vectorized pass, with the first value fused into initialization. Last-match-wins behavior for duplicate `values` is preserved, mixed output item types now fail with a clear error, and list-output scratch arrays are heap allocated so large outputs no longer exhaust the C stack (#74). diff --git a/README.md b/README.md index 870b5fe..29ccda7 100644 --- a/README.md +++ b/README.md @@ -21,6 +21,66 @@ install.packages("kit") install.packages("kit", repos = "https://fastverse.r-universe.dev") ``` +## OpenMP and threads + +`kit` uses OpenMP in `iif()`, `nswitch()`, `vswitch()`, `psort()`, and +`charToFact()`. The other functions use one thread. OpenMP is optional. +Without OpenMP, `kit` still works, but these five functions use one thread. + +- **Linux:** Install the OpenMP runtime for your compiler. With GCC, this is + usually `libgomp` (`libgomp1` on Debian and Ubuntu). With Clang, install + `libomp` (`libomp-dev` on Debian and Ubuntu). +- **macOS:** Apple Clang does not include OpenMP. Install it with Homebrew: + `brew install libomp`. On Apple Silicon, Homebrew installs to a path that + clang does not search by default. Add that path to your R Makevars before + you install `kit` from source (`configure` decides with `CFLAGS`, so the + include path must be there): + + ```sh + # ~/.R/Makevars + CFLAGS += -I$(shell brew --prefix libomp)/include -L$(shell brew --prefix libomp)/lib + LDFLAGS += -L$(shell brew --prefix libomp)/lib + ``` + +- **Windows:** OpenMP is included in Rtools. Install the Rtools version that + matches your R version. Then install `kit` from source. + +Set the number of threads once for each session: + +```r +options(kit.nThread = 4L) +``` + +The default is `1`. If you set a value that is larger than the number of +available processors, `kit` uses the number of available processors. `kit` +reads the option each time you call a function, so you can change it between +calls. + +To check whether OpenMP is enabled, run: + +```r +.Call(kit:::CompEnabledR) +``` + +If this call returns `FALSE`, the five functions above use one thread, even +when `kit.nThread` is larger than `1`. + +## Platforms + +`kit` is tested on Linux, macOS, and Windows. OpenMP is optional. See +[OpenMP and threads](#openmp-and-threads) for the installation requirements. + +`shareData()` behaves differently on POSIX systems and Windows: + +- **Linux, macOS, and other POSIX systems:** `getData()` reads the object one + time. A second call with the same name fails. If the sharing session stops + before `clearData()`, the object can stay in shared memory: on Linux in + `/dev/shm` until you remove it or restart the machine, on macOS until the + machine restarts. +- **Windows:** `getData()` can read the object more than one time while the + sharing session is alive. If the sharing session stops, Windows removes the + mapping. + ## Features ### Row-wise Statistical Functions @@ -33,9 +93,10 @@ Vector-valued functions that combine several vectors or data frame columns eleme - **`pcount`, `pcountNA`**: Count occurrences of values or NAs - **`pfirst`, `plast`**: First/last non-missing values -The word "parallel" in their names means they work across several vectors at once, not that they use multiple threads. These functions are single-threaded C code. Only `iif()`, `nswitch()`, `vswitch()`, `psort()`, and `charToFact()` can use OpenMP, and only when you raise `options(kit.nThread)` above its default of 1. +The word "parallel" in their names means they work across several vectors at once, not that they use multiple threads. These functions are single-threaded C code. ```r +library(kit) x <- c(1, 3, NA, 5) y <- c(2, NA, 4, 1) psum(x, y, na.rm = TRUE) # [1] 3 3 4 6 diff --git a/_pkgdown.yml b/_pkgdown.yml index 8991669..0008078 100644 --- a/_pkgdown.yml +++ b/_pkgdown.yml @@ -44,6 +44,10 @@ navbar: reference: +- title: "Overview" + desc: "Package overview, threading, platform support and encodings." +- contents: + - kit-package - title: "Row-wise Statistical Functions" desc: "Vector-valued (statistical) functions that combine several vectors passed as arguments, or a single list of vectors/data frame, element by element." - contents: diff --git a/man/kit-package.Rd b/man/kit-package.Rd new file mode 100644 index 0000000..cdf86c3 --- /dev/null +++ b/man/kit-package.Rd @@ -0,0 +1,39 @@ +\name{kit-package} +\alias{kit-package} +\alias{kit} +\docType{package} +\title{kit: Fast Data Manipulation Functions in C} +\description{ + \pkg{kit} provides fast, vectorised functions for data manipulation. + The compiled code is written in C. The package has no runtime dependencies. +} +\details{ + \strong{Threads.} + Five functions can use OpenMP: \code{\link{iif}}, \code{\link{nswitch}}, + \code{\link{vswitch}}, \code{\link{psort}}, and \code{\link{charToFact}}. + The other functions use one thread. + + Set the number of threads with \code{options(kit.nThread = 4L)}. + The default is \code{1}. If you set a value that is larger than the number + of available processors, \pkg{kit} uses the number of available processors. + If \pkg{kit} was built without OpenMP, all functions use one thread and + this option has no effect. + + \strong{Platforms.} + \pkg{kit} is tested on Linux, macOS, and Windows. OpenMP is optional. + On macOS, OpenMP requires \code{libomp} from Homebrew. On Windows, OpenMP + comes with Rtools. See the README for installation details. + + \strong{Encodings.} + \code{\link{psort}} sorts text in the C locale by default. Set + \code{c.locale = FALSE} to use the session locale. + \code{\link{charToFact}} uses the session locale. Factor levels and sort + order can thus differ between operating systems for non-ASCII text. + \code{\link{vswitch}} and \code{\link{nswitch}} convert mixed encodings to + UTF-8 by default. +} +\seealso{ + \code{\link{psum}}, \code{\link{funique}}, \code{\link{topn}}, + \code{\link{shareData}} +} +\keyword{package}