Repository navigation
duckdb: bump to v1.5.6 - #662
Conversation
in DuckDB v1.5 secrets are lazily initialized on disk by default, so 'create secret' was failing with a LocalFileSystem permission error since file system access is not available on the sandbox 'allow_persistent_secrets=false' needs to be set before loading any extension, because that initializes the secret manager and it's not possble to change it's settings after initialization not relevant for MotherDuck, it already runs without local file system access
since DuckDB v1.5 extension autoloading goes through the filesystem permission check, so all extensions required need to explicitly loaded before the filesystem is locked
DuckDB 1.5 no longer bundle the parquet and json extensions in the default bundled amalgamation, so reading a parquet file now attempts to autoload the extentions, which fails because LocalFileSystem access is disabled.
| match self { | ||
| Self::Iceberg | Self::S3Tables | Self::R2Catalog | Self::Polaris | Self::Lakekeeper => { | ||
| "install iceberg;load iceberg;" | ||
| "install httpfs;load httpfs;install iceberg;load iceberg;" |
There was a problem hiding this comment.
httpfs is the extension required read from those sources (blob storage, any remote files basically).
If not installed explicitly DuckDB will try to autoload it when a query tries to interact with a remote file, and that autoload fails on v1.5 because the filesystem is locked in your sandbox at query time.
There was a problem hiding this comment.
Hm, I've just a quick sanity check with a v1.3 CLI and it also fails to autoload httpfs with the file system disabled. But your testing suite on main passes with v1.3 so I was confused.
$ duckman run v1.3.2 -c "SET allow_persistent_secrets=false; SET disabled_filesystems='LocalFileSystem'; load httpfs;"
Permission Error:
File system LocalFileSystem has been disabled by configuration
$ duckman run v1.5.6 -c "SET allow_persistent_secrets=false; SET disabled_filesystems='LocalFileSystem'; load httpfs;"
Permission Error:
File system LocalFileSystem has been disabled by configuration
I've narrowed it down: on v1.3 the autoloading that happens on create secret was not gated by the filesystem access check, while it is in v1.5.
$ duckman run v1.3.2 -c "SET allow_persistent_secrets=false; SET disabled_filesystems='LocalFileSystem'; CREATE SECRET (TYPE S3, KEY_ID 'a', SECRET 'b', REGION 'us-east-1');"
true
$ duckman run v1.5.6 -c "SET allow_persistent_secrets=false; SET disabled_filesystems='LocalFileSystem'; CREATE SECRET (TYPE S3, KEY_ID 'a', SECRET 'b', REGION 'us-east-1');"
Extension Autoloading Error: An error occurred while trying to automatically
install the required extension 'httpfs':
File system LocalFileSystem has been disabled by configuration
|
@mkaruza thanks for the review! I cannot merge and CI hasn't run yet since it need approval |
DuckDB v1.3 reached end-of-life on November last year.