Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions NEWS.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,8 @@

### Notes

- Document `?kit` with a package page that explains OpenMP threads, platform support, encoding behavior, and the main functions. Add "OpenMP and threads" and "Platforms" sections to the README, and add `library(kit)` to the first README example (#90).

- Reduce peak memory in `funique`, `fduplicated`, `uniqLen`, `psort`, and `charToFact` by starting duplicate hash tables at 256 slots and growing them on demand (4x, capped at the previous full-size table) instead of allocating the full table up front. Rehashing re-reads the source values, so results are unchanged; inputs with few unique values use much less memory, while peak memory can be higher at growth boundaries for high-cardinality inputs (#74).

- Speed up `vswitch` for string and list outputs by scanning `x` once instead of once per value and fusing default/`na` writes into the match pass; single-value lookups also avoid a separate initialization pass. Atomic outputs keep the value-major vectorized pass, with the first value fused into initialization. Last-match-wins behavior for duplicate `values` is preserved, mixed output item types now fail with a clear error, and list-output scratch arrays are heap allocated so large outputs no longer exhaust the C stack (#74).
Expand Down
63 changes: 62 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,66 @@ install.packages("kit")
install.packages("kit", repos = "https://fastverse.r-universe.dev")
```

## OpenMP and threads

`kit` uses OpenMP in `iif()`, `nswitch()`, `vswitch()`, `psort()`, and
`charToFact()`. The other functions use one thread. OpenMP is optional.
Without OpenMP, `kit` still works, but these five functions use one thread.

- **Linux:** Install the OpenMP runtime for your compiler. With GCC, this is
usually `libgomp` (`libgomp1` on Debian and Ubuntu). With Clang, install
`libomp` (`libomp-dev` on Debian and Ubuntu).
- **macOS:** Apple Clang does not include OpenMP. Install it with Homebrew:
`brew install libomp`. On Apple Silicon, Homebrew installs to a path that
clang does not search by default. Add that path to your R Makevars before
you install `kit` from source (`configure` decides with `CFLAGS`, so the
include path must be there):

```sh
# ~/.R/Makevars
CFLAGS += -I$(shell brew --prefix libomp)/include -L$(shell brew --prefix libomp)/lib
LDFLAGS += -L$(shell brew --prefix libomp)/lib
```

- **Windows:** OpenMP is included in Rtools. Install the Rtools version that
matches your R version. Then install `kit` from source.

Set the number of threads once for each session:

```r
options(kit.nThread = 4L)
```

The default is `1`. If you set a value that is larger than the number of
available processors, `kit` uses the number of available processors. `kit`
reads the option each time you call a function, so you can change it between
calls.

To check whether OpenMP is enabled, run:

```r
.Call(kit:::CompEnabledR)
```

If this call returns `FALSE`, the five functions above use one thread, even
when `kit.nThread` is larger than `1`.

## Platforms

`kit` is tested on Linux, macOS, and Windows. OpenMP is optional. See
[OpenMP and threads](#openmp-and-threads) for the installation requirements.

`shareData()` behaves differently on POSIX systems and Windows:

- **Linux, macOS, and other POSIX systems:** `getData()` reads the object one
time. A second call with the same name fails. If the sharing session stops
before `clearData()`, the object can stay in shared memory: on Linux in
`/dev/shm` until you remove it or restart the machine, on macOS until the
machine restarts.
- **Windows:** `getData()` can read the object more than one time while the
sharing session is alive. If the sharing session stops, Windows removes the
mapping.

## Features

### Row-wise Statistical Functions
Expand All @@ -33,9 +93,10 @@ Vector-valued functions that combine several vectors or data frame columns eleme
- **`pcount`, `pcountNA`**: Count occurrences of values or NAs
- **`pfirst`, `plast`**: First/last non-missing values

The word "parallel" in their names means they work across several vectors at once, not that they use multiple threads. These functions are single-threaded C code. Only `iif()`, `nswitch()`, `vswitch()`, `psort()`, and `charToFact()` can use OpenMP, and only when you raise `options(kit.nThread)` above its default of 1.
The word "parallel" in their names means they work across several vectors at once, not that they use multiple threads. These functions are single-threaded C code.

```r
library(kit)
x <- c(1, 3, NA, 5)
y <- c(2, NA, 4, 1)
psum(x, y, na.rm = TRUE) # [1] 3 3 4 6
Expand Down
4 changes: 4 additions & 0 deletions _pkgdown.yml
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,10 @@ navbar:


reference:
- title: "Overview"
desc: "Package overview, threading, platform support and encodings."
- contents:
- kit-package
- title: "Row-wise Statistical Functions"
desc: "Vector-valued (statistical) functions that combine several vectors passed as arguments, or a single list of vectors/data frame, element by element."
- contents:
Expand Down
39 changes: 39 additions & 0 deletions man/kit-package.Rd
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
\name{kit-package}
\alias{kit-package}
\alias{kit}
\docType{package}
\title{kit: Fast Data Manipulation Functions in C}
\description{
\pkg{kit} provides fast, vectorised functions for data manipulation.
The compiled code is written in C. The package has no runtime dependencies.
}
\details{
\strong{Threads.}
Five functions can use OpenMP: \code{\link{iif}}, \code{\link{nswitch}},
\code{\link{vswitch}}, \code{\link{psort}}, and \code{\link{charToFact}}.
The other functions use one thread.

Set the number of threads with \code{options(kit.nThread = 4L)}.
The default is \code{1}. If you set a value that is larger than the number
of available processors, \pkg{kit} uses the number of available processors.
If \pkg{kit} was built without OpenMP, all functions use one thread and
this option has no effect.

\strong{Platforms.}
\pkg{kit} is tested on Linux, macOS, and Windows. OpenMP is optional.
On macOS, OpenMP requires \code{libomp} from Homebrew. On Windows, OpenMP
comes with Rtools. See the README for installation details.

\strong{Encodings.}
\code{\link{psort}} sorts text in the C locale by default. Set
\code{c.locale = FALSE} to use the session locale.
\code{\link{charToFact}} uses the session locale. Factor levels and sort
order can thus differ between operating systems for non-ASCII text.
\code{\link{vswitch}} and \code{\link{nswitch}} convert mixed encodings to
UTF-8 by default.
}
\seealso{
\code{\link{psum}}, \code{\link{funique}}, \code{\link{topn}},
\code{\link{shareData}}
}
\keyword{package}
Loading