Skip to content

Take the other bindings' spelling of local:, and verify interop against Python, Go and real S3 - #96

Merged
ShawnChen-Sirius merged 3 commits into
mainfrom
feat/durable-interop-polish
Sep 17, 2026
Merged

ShawnChen-Sirius merged 3 commits into
mainfrom
feat/durable-interop-polish

Conversation

@ShawnChen-Sirius

@ShawnChen-Sirius ShawnChen-Sirius commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Interop and verification work found while getting ready to release durable on a
stable engine. Two real fixes, two verifications that are now scripts, and the
documentation caught up with what has actually been measured.

local: was refused, and this binding was the only one refusing it

Python spells the local backend local:, Go registers the same backend under
both names, and Go's comment says why: a namespace URL is the sort of thing
that ends up in a config file shared between a Python service and a Node one.
This binding took only file:, so one configuration worked in two of three
bindings.

It now registers both. My first plan was to mirror Python exactly and leave
local: undecoded — reading Go killed that, because Go decodes both, so Python
and Go already disagree past ordinary paths and there is no three-way behaviour
to match. Following Go is one factory instead of two parsers. local://host/p
is refused rather than silently resolved to /p, and a stray % names the URL
that was malformed instead of surfacing a bare URIError.

The AWS SDK is loaded on first use

It is an optional peer dependency, so "I do want S3 and forgot to install it"
is a real path — and a static import made that a bare
Cannot find module '@aws-sdk/client-s3' thrown from require('chdb/durable/s3'),
a file the caller has never heard of. Python answers the same mistake with
MissingDependency: pip install 'chdb[durable]'. Deferring the load moves the
failure to the first S3 operation and lets it name the package; the scheme
registration, key validation and size limit all still work without it.

Two verifications are scripts now, not notes

Both run by hand — each needs something CI does not have, and what they check
moves only when core releases.

npm run test:durable:cross-version — a newer engine restores an older
one's base and WAL, and the raised floor then refuses the older engine. This
was untestable until 26.7.3 shipped, since 26.7.2-rc.2 was the only
Durable-capable engine. It re-execs per stage, which is load-bearing: a libchdb
is dlopened once per process, so a single-process version loads the first
library and answers every later request with it — both versions report the same
number and it passes while proving nothing. I wrote that version first and it
did exactly that.

npm run test:durable:cross-binding — the same four exchanges against
Python and against Go: it writes a base plus a WAL and Node restores it, Node
appends and it reads the append back, and Node checkpoints so it RESTOREs from
that archive with no WAL to fall back on. A pairing whose runtime is absent is
skipped loudly; only having neither fails.

Go finds its engine through CHDB_LIB_PATH, which the script points at this
repository's libchdb.so, so both sides run one engine build and a difference
would be a binding difference rather than a version one. Worth pinning down:
the chdb-go checkout I ran against declared v26.7.3 in update_libchdb.sh
while the library on disk was still 26.7.2-rc.2, which would otherwise have
made this a cross-version test wearing a cross-binding label.

The 12 MiB case gets a timeout a real bucket can meet

0.4s against a local MinIO, about a minute against a real bucket across an
ocean — measured at 20005ms against the suite's 20s budget on eu-central-1.
It carries its own 180s now. The suite-wide 20s is left alone: it is a useful
constraint on the other ten cases, none of which moves more than a few hundred
bytes.

What was measured

Engine 26.7.3 throughout.

Suite Result
test:durable 141 pass
test:durable:node 24 pass
test:v3 486 pass
test:durable:e2e (Bun FFI) 19 pass
test:durable:s3 — MinIO 11/11
test:durable:s3 — real AWS S3 (eu-central-1) 11/11
test:durable:stack — MinIO 4/4
test:durable:stack — real AWS S3 4/4
test:durable:cross-version (26.7.2-rc.2 → 26.7.3) all claims hold
test:durable:cross-binding (Python 4.4.0 + chdb-go) 13/13 exchanges

Cross-binding was also run over real AWS S3, not only a local directory:
Python wrote a base plus WAL and Node restored it; Node checkpointed and Python
RESTOREd from that archive with wal = 0.

npm run typecheck reports the same 13 pre-existing errors under test/v3/**
as on main. npm test (mocha) cannot run on Node 26 locally — yargs fails to
load — unchanged and unrelated.

Deliberately not done

destroy and scan stay absent. Python has both, but the contract removed
delete_prefix/destroy from V1 and deferred GC to its own safety design
(§8.1 item 4: "safe enumeration of references, orphan grace periods,
protection of concurrent readers, recovery windows and explicit confirmation
for dangerous operations"), and Go does not have them either.

Cloudflare R2 is still unmeasured, so the docs say so rather than inferring it
from the shared code path.

🤖 Generated with Claude Code

Note

Add local: scheme to durable namespace and add cross-binding interop tests

  • Registers the local filesystem backend factory under both file: and local: URL schemes in namespace.ts, so durable namespaces accept the spelling used by other bindings. local: URLs are parsed as opaque paths (no host, percent-decoded), while file: URLs keep using fileURLToPath.
  • Adds durable-cross-binding.mjs, a Node script that exchanges durable objects between this binding and Python chdb / Go chdb-go, verifying base restoration, WAL replay, checkpoint-only archives, and shared local: URL resolution.
  • Adds durable-cross-version.ts, a Bun script that writes an archive with an older engine, restores it with a newer engine, and confirms the older engine is rejected after the compatibility floor is raised.
  • Makes @aws-sdk/client-s3 a lazy import in s3.ts; module loading no longer requires the SDK, and S3 operations report an actionable install error when it is missing.
  • Risk: local: URLs with a host component are now rejected with a RangeError via opaquePath in namespace.ts; any caller previously passing a host in a local: URL will get an error.
📊 Macroscope summarized a2afdfb. 6 files reviewed, 3 issues evaluated, 3 issues filtered, 0 comments posted

🗂️ Filtered Issues

scripts/durable-cross-binding.mjs — 0 comments posted, 3 evaluated, 3 filtered
  • line 88: buildGoHelper() creates a separate /tmp/xbind-go-src-* directory with mkdtempSync, but only root is removed in the final cleanup. Every Go-enabled run leaves the generated go.mod, downloaded module state, and helper binary behind indefinitely. [ Out of scope (post-validation triage) ]
  • line 187: namespace() passes a fresh mkdtempSync directory as scratchRoot, but DurableObject creates and removes only a chdb-durable-* child below that root. Thus every Node namespace used by this script leaves its empty xbind-node-*, xbind-nw-*, etc. parent directory in the system temp directory after a successful run. [ Out of scope (post-validation triage) ]
  • line 192: The native-version probe runs before the try/finally cleanup begins. If new ChdbNodeEngine() throws (for example, the addon has not been built or cannot load), the already-created durable-xbind-* work directory is never removed. [ Out of scope (post-validation triage) ]

ShawnChen-Sirius and others added 3 commits September 16, 2026 10:07
Found while verifying interop before a release, against Python `chdb` 4.4.0.

**`local:` was refused.** Python spells the local backend `local:`, Go
registers the same backend under both names, and its comment says why: a
namespace URL is the sort of thing that ends up in a config file shared
between a Python service and a Node one. This binding took only `file:`, which
made it the one place a working configuration stopped working. It now
registers both.

The two forms differ in how a path is written, not where it points — `file:`
is a real URL so a space is `%20`, `local:` is opaque so the path is closer to
literal. My first plan was to mirror Python exactly and leave `local:`
undecoded. Reading Go killed that: Go decodes both, so Python and Go already
disagree past ordinary paths and there is no three-way behaviour to match.
Following Go is one factory instead of two parsers, and it is the shape the
existing `registerBackendScheme` already wanted. `local://data/objects` is
refused rather than silently resolved to `/objects`, and a stray `%` reports
which URL was malformed instead of surfacing a bare `URIError`.

**The AWS SDK is now loaded on first use.** It is an optional peer dependency,
so "I do want S3 and forgot to install it" is a real path — and a static
import made that a bare `Cannot find module '@aws-sdk/client-s3'` thrown from
`require('chdb/durable/s3')`, a file the caller has never heard of. Python
answers the same mistake with `MissingDependency: pip install 'chdb[durable]'`.
Deferring the load moves the failure to the first S3 operation and lets it name
the package; the scheme registration, key validation and size limit all still
work without it.

Two verifications are now scripts rather than notes, both run by hand:

- `test:durable:cross-version` — a newer engine restores an older one's base
  and WAL, and the raised floor then refuses the older engine. This could not
  be tested until `26.7.3` shipped, since `26.7.2-rc.2` was the only
  Durable-capable engine. It re-execs per stage, which is load-bearing: a
  libchdb is `dlopen`ed once per process, so a single-process version loads the
  first library and answers every later request with it — both versions report
  the same number and it passes while proving nothing. I wrote that version
  first and it did exactly that.
- `test:durable:cross-binding` — Python writes a base plus WAL and Node
  restores it; Node appends and Python reads it back; Node checkpoints and
  Python RESTOREs from that archive with no WAL left; and one `local:` URL
  serves both.

`destroy`/`scan` are deliberately still absent. Python has them, but the
contract removed `delete_prefix`/destroy from V1 and deferred GC to its own
safety design (§8.1 item 4), and Go does not have them either.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`chdb-go/chdb/durable` reached V1 too, so the contract's
read-by-two-other-bindings requirement was one pair short rather than two.
`test:durable:cross-binding` now runs the same four exchanges against Go that
it runs against Python: Go writes a base plus a WAL and Node restores it, Node
appends and Go reads the append back, and Node checkpoints so Go RESTOREs from
that archive with no WAL to fall back on.

Both sides load the same engine build — Go finds libchdb through
`CHDB_LIB_PATH`, which the script points at this repository's copy — so a
difference in results would be a difference in bindings rather than in
versions. Worth pinning down: the chdb-go checkout I ran against declared
`v26.7.3` in `update_libchdb.sh` while the library on disk was still
`26.7.2-rc.2`, which would otherwise have quietly made this a cross-version
test wearing a cross-binding label.

A pairing whose runtime is absent is skipped loudly and the rest still runs;
only having neither fails. Go needs its `ScratchRoot` to exist, same as Node.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The large-object test moves 12 MiB up and back down. Against a local MinIO
that is 0.4s; against a real bucket across an ocean it is about a minute, and
the suite's 20s budget made it the one test that fails on AWS while the
implementation is fine — measured, at 20005ms, on `eu-central-1`. With its own
180s it passes there and stays unchanged everywhere else.

The suite-wide 20s is left alone on purpose: it is a useful constraint on the
other ten cases, none of which moves more than a few hundred bytes. A red
suite that means "your network is far away" is one people learn to ignore,
which is worse than a slow test.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@ShawnChen-Sirius
ShawnChen-Sirius merged commit 0c3f66d into main Sep 17, 2026
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant