Skip to content

perf(engine): cache Parquet footers on read connections - #497

Merged
vishr merged 1 commit into
mainfrom
perf/parquet-metadata-cache
Nov 16, 2026
Merged

vishr merged 1 commit into
mainfrom
perf/parquet-metadata-cache

Conversation

@vishr

@vishr vishr commented Nov 16, 2026

Copy link
Copy Markdown
Member

Problem

parquet_metadata_cache was never set, so it kept its engine default of false. Every read statement binds an explicit per-file Parquet list, and a panel issues several statements, so DuckDB re-read each file's footer per statement instead of reusing it.

Closes #484.

Measurement first

This issue called for measurement before any change. Engine-internal A/B benchmark, same immutable fixtures, one pinned read transaction, AB/BA alternation to control the OS page cache, 100 samples per arm — median read latency, cache OFF → ON:

files OFF ON delta
128 11.83 ms 10.69 ms −9.6%
512 33.02 ms 31.02 ms −6.1%
1024 62.67 ms 58.61 ms −6.5%

p95/p99 also improve at the larger file counts (1024 p99 70.5 → 62.0 ms). The win is consistent and monotonic and robust to ordering, matching the setting's purpose (reading the same files repeatedly). Memory cost is noted: the footer cache grows to ~150 MB at 1024 fixture files; it is bounded footer caching under DuckDB's memory accounting (not unbounded read-ahead), and active file counts are bounded by compaction.

Scope finding (verified on the pinned DuckDB 2.0.0 engine, not assumed)

parquet_metadata_cache is global-scoped, default false. But on the shared on-disk DSN the engine uses across its read/write/memory/parse pools, a SET on one connection does not propagate to other pooled connections, and the DSN/instance-config route is rejected because the parquet extension loads later. So it must be set per connection, in the same init hook that sets SET TimeZone='UTC', and added to allowed_configs so the SET still succeeds on connections opened after lock_configuration=true.

Change

  • internal/engine/duck.go — the per-connection init hook now runs SET parquet_metadata_cache=true right after the UTC set.
  • internal/duckdb/policy.go — allowed_configs gains parquet_metadata_cache; lock_configuration=true and the file/external-access lockdown are unchanged.

Correctness — no stale-cache risk

The cache keys on path + mtime + size. Batch paths are unique and never reused: ingest IDs are UUIDs, compaction output IDs are compact-<nanos>; files are immutable once published, compaction writes new paths, and retired inputs are deleted, never overwritten in place. So a cached footer can never serve a different file.

Tests

TestReadConnectionsEnableParquetMetadataCache asserts the setting is true on a freshly opened (post-lock) connection of all four pools and that the lock still refuses SET enable_external_access (lockdown intact). A //go:build filebench A/B harness (metadata_cache_bench_test.go) is the measurement tool.

Verification

just test "./internal/engine/..." and "./internal/query/...", just vet, and just check-static (gofmt, deadcode, behavioral test names) all green; public arbitrary SQL still cannot change file or external access.

The engine binds an explicit per-file Parquet list on every read statement and
a panel issues several, so with parquet_metadata_cache off DuckDB re-read each
file's footer on every statement. parquet_metadata_cache is a GLOBAL-scoped
option that the duckdb-go driver applies only to the connection that issues the
SET, so enabling it once at boot does not reach the pooled read connections. It
is now set on each connection in the init hook, next to the UTC setting, and
kept in the policy allowed_configs so it still applies on connections opened
after lock_configuration. The lockdown is unchanged: the cache only holds
immutable Parquet footers and cannot widen file or external access.

Measured with an engine-internal A/B over immutable fixtures (same files, one
pinned read transaction, AB/BA alternation, 100 samples per arm). Median read
latency drops with the cache on:

  128 files:  11.83 -> 10.69 ms  (-9.6%)
  512 files:  33.02 -> 31.02 ms  (-6.1%)
  1024 files: 62.67 -> 58.61 ms  (-6.5%)

p95 and p99 improve similarly at 512 and 1024 files. The footer cache costs
bounded memory (about 150 MB of OBJECT_CACHE at 1024 fixture files). It keys on
path, mtime and size, and our batch paths are unique and immutable (uuid ingest
ids and compact-<nanos> compaction ids; retired paths are deleted, never
overwritten in place), so a cached footer can never be stale.

Closes #484
@vishr
vishr enabled auto-merge November 16, 2026 01:45
@vishr
vishr added this pull request to the merge queue Nov 16, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 16, 2026
@vishr
vishr added this pull request to the merge queue Nov 16, 2026
Merged via the queue into main with commit 80bed19 Nov 16, 2026
31 checks passed
@vishr
vishr deleted the perf/parquet-metadata-cache branch November 16, 2026 02:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf: parquet_metadata_cache is never enabled

1 participant