Repository navigation
Take the other bindings' spelling of local:, and verify interop against Python, Go and real S3 - #96
Merged
Merged
Conversation
Found while verifying interop before a release, against Python `chdb` 4.4.0.
**`local:` was refused.** Python spells the local backend `local:`, Go
registers the same backend under both names, and its comment says why: a
namespace URL is the sort of thing that ends up in a config file shared
between a Python service and a Node one. This binding took only `file:`, which
made it the one place a working configuration stopped working. It now
registers both.
The two forms differ in how a path is written, not where it points — `file:`
is a real URL so a space is `%20`, `local:` is opaque so the path is closer to
literal. My first plan was to mirror Python exactly and leave `local:`
undecoded. Reading Go killed that: Go decodes both, so Python and Go already
disagree past ordinary paths and there is no three-way behaviour to match.
Following Go is one factory instead of two parsers, and it is the shape the
existing `registerBackendScheme` already wanted. `local://data/objects` is
refused rather than silently resolved to `/objects`, and a stray `%` reports
which URL was malformed instead of surfacing a bare `URIError`.
**The AWS SDK is now loaded on first use.** It is an optional peer dependency,
so "I do want S3 and forgot to install it" is a real path — and a static
import made that a bare `Cannot find module '@aws-sdk/client-s3'` thrown from
`require('chdb/durable/s3')`, a file the caller has never heard of. Python
answers the same mistake with `MissingDependency: pip install 'chdb[durable]'`.
Deferring the load moves the failure to the first S3 operation and lets it name
the package; the scheme registration, key validation and size limit all still
work without it.
Two verifications are now scripts rather than notes, both run by hand:
- `test:durable:cross-version` — a newer engine restores an older one's base
and WAL, and the raised floor then refuses the older engine. This could not
be tested until `26.7.3` shipped, since `26.7.2-rc.2` was the only
Durable-capable engine. It re-execs per stage, which is load-bearing: a
libchdb is `dlopen`ed once per process, so a single-process version loads the
first library and answers every later request with it — both versions report
the same number and it passes while proving nothing. I wrote that version
first and it did exactly that.
- `test:durable:cross-binding` — Python writes a base plus WAL and Node
restores it; Node appends and Python reads it back; Node checkpoints and
Python RESTOREs from that archive with no WAL left; and one `local:` URL
serves both.
`destroy`/`scan` are deliberately still absent. Python has them, but the
contract removed `delete_prefix`/destroy from V1 and deferred GC to its own
safety design (§8.1 item 4), and Go does not have them either.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`chdb-go/chdb/durable` reached V1 too, so the contract's read-by-two-other-bindings requirement was one pair short rather than two. `test:durable:cross-binding` now runs the same four exchanges against Go that it runs against Python: Go writes a base plus a WAL and Node restores it, Node appends and Go reads the append back, and Node checkpoints so Go RESTOREs from that archive with no WAL to fall back on. Both sides load the same engine build — Go finds libchdb through `CHDB_LIB_PATH`, which the script points at this repository's copy — so a difference in results would be a difference in bindings rather than in versions. Worth pinning down: the chdb-go checkout I ran against declared `v26.7.3` in `update_libchdb.sh` while the library on disk was still `26.7.2-rc.2`, which would otherwise have quietly made this a cross-version test wearing a cross-binding label. A pairing whose runtime is absent is skipped loudly and the rest still runs; only having neither fails. Go needs its `ScratchRoot` to exist, same as Node. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The large-object test moves 12 MiB up and back down. Against a local MinIO that is 0.4s; against a real bucket across an ocean it is about a minute, and the suite's 20s budget made it the one test that fails on AWS while the implementation is fine — measured, at 20005ms, on `eu-central-1`. With its own 180s it passes there and stays unchanged everywhere else. The suite-wide 20s is left alone on purpose: it is a useful constraint on the other ten cases, none of which moves more than a few hundred bytes. A red suite that means "your network is far away" is one people learn to ignore, which is worse than a slow test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Interop and verification work found while getting ready to release durable on a
stable engine. Two real fixes, two verifications that are now scripts, and the
documentation caught up with what has actually been measured.
local:was refused, and this binding was the only one refusing itPython spells the local backend
local:, Go registers the same backend underboth names, and Go's comment says why: a namespace URL is the sort of thing
that ends up in a config file shared between a Python service and a Node one.
This binding took only
file:, so one configuration worked in two of threebindings.
It now registers both. My first plan was to mirror Python exactly and leave
local:undecoded — reading Go killed that, because Go decodes both, so Pythonand Go already disagree past ordinary paths and there is no three-way behaviour
to match. Following Go is one factory instead of two parsers.
local://host/pis refused rather than silently resolved to
/p, and a stray%names the URLthat was malformed instead of surfacing a bare
URIError.The AWS SDK is loaded on first use
It is an optional peer dependency, so "I do want S3 and forgot to install it"
is a real path — and a static import made that a bare
Cannot find module '@aws-sdk/client-s3'thrown fromrequire('chdb/durable/s3'),a file the caller has never heard of. Python answers the same mistake with
MissingDependency: pip install 'chdb[durable]'. Deferring the load moves thefailure to the first S3 operation and lets it name the package; the scheme
registration, key validation and size limit all still work without it.
Two verifications are scripts now, not notes
Both run by hand — each needs something CI does not have, and what they check
moves only when core releases.
npm run test:durable:cross-version— a newer engine restores an olderone's base and WAL, and the raised floor then refuses the older engine. This
was untestable until
26.7.3shipped, since26.7.2-rc.2was the onlyDurable-capable engine. It re-execs per stage, which is load-bearing: a libchdb
is
dlopened once per process, so a single-process version loads the firstlibrary and answers every later request with it — both versions report the same
number and it passes while proving nothing. I wrote that version first and it
did exactly that.
npm run test:durable:cross-binding— the same four exchanges againstPython and against Go: it writes a base plus a WAL and Node restores it, Node
appends and it reads the append back, and Node checkpoints so it RESTOREs from
that archive with no WAL to fall back on. A pairing whose runtime is absent is
skipped loudly; only having neither fails.
Go finds its engine through
CHDB_LIB_PATH, which the script points at thisrepository's
libchdb.so, so both sides run one engine build and a differencewould be a binding difference rather than a version one. Worth pinning down:
the chdb-go checkout I ran against declared
v26.7.3inupdate_libchdb.shwhile the library on disk was still
26.7.2-rc.2, which would otherwise havemade this a cross-version test wearing a cross-binding label.
The 12 MiB case gets a timeout a real bucket can meet
0.4s against a local MinIO, about a minute against a real bucket across an
ocean — measured at 20005ms against the suite's 20s budget on
eu-central-1.It carries its own 180s now. The suite-wide 20s is left alone: it is a useful
constraint on the other ten cases, none of which moves more than a few hundred
bytes.
What was measured
Engine
26.7.3throughout.test:durabletest:durable:nodetest:v3test:durable:e2e(Bun FFI)test:durable:s3— MinIOtest:durable:s3— real AWS S3 (eu-central-1)test:durable:stack— MinIOtest:durable:stack— real AWS S3test:durable:cross-version(26.7.2-rc.2→26.7.3)test:durable:cross-binding(Python 4.4.0 + chdb-go)Cross-binding was also run over real AWS S3, not only a local directory:
Python wrote a base plus WAL and Node restored it; Node checkpointed and Python
RESTOREd from that archive with
wal = 0.npm run typecheckreports the same 13 pre-existing errors undertest/v3/**as on main.
npm test(mocha) cannot run on Node 26 locally —yargsfails toload — unchanged and unrelated.
Deliberately not done
destroyandscanstay absent. Python has both, but the contract removeddelete_prefix/destroy from V1 and deferred GC to its own safety design(§8.1 item 4: "safe enumeration of references, orphan grace periods,
protection of concurrent readers, recovery windows and explicit confirmation
for dangerous operations"), and Go does not have them either.
Cloudflare R2 is still unmeasured, so the docs say so rather than inferring it
from the shared code path.
🤖 Generated with Claude Code
Note
Add
local:scheme to durable namespace and add cross-binding interop testsfile:andlocal:URL schemes in namespace.ts, so durable namespaces accept the spelling used by other bindings.local:URLs are parsed as opaque paths (no host, percent-decoded), whilefile:URLs keep usingfileURLToPath.chdb/ Gochdb-go, verifying base restoration, WAL replay, checkpoint-only archives, and sharedlocal:URL resolution.@aws-sdk/client-s3a lazy import in s3.ts; module loading no longer requires the SDK, and S3 operations report an actionable install error when it is missing.local:URLs with a host component are now rejected with aRangeErrorviaopaquePathin namespace.ts; any caller previously passing a host in alocal:URL will get an error.📊 Macroscope summarized a2afdfb. 6 files reviewed, 3 issues evaluated, 3 issues filtered, 0 comments posted
🗂️ Filtered Issues
scripts/durable-cross-binding.mjs — 0 comments posted, 3 evaluated, 3 filtered
buildGoHelper()creates a separate/tmp/xbind-go-src-*directory withmkdtempSync, but onlyrootis removed in the final cleanup. Every Go-enabled run leaves the generatedgo.mod, downloaded module state, and helper binary behind indefinitely. [ Out of scope (post-validation triage) ]namespace()passes a freshmkdtempSyncdirectory asscratchRoot, butDurableObjectcreates and removes only achdb-durable-*child below that root. Thus every Node namespace used by this script leaves its emptyxbind-node-*,xbind-nw-*, etc. parent directory in the system temp directory after a successful run. [ Out of scope (post-validation triage) ]try/finallycleanup begins. Ifnew ChdbNodeEngine()throws (for example, the addon has not been built or cannot load), the already-createddurable-xbind-*work directory is never removed. [ Out of scope (post-validation triage) ]