Repository navigation
feat(remote): add BigQuery data source - #1066
DoubleCore wants to merge 6 commits into
Conversation
2ec231d to
3b4a6e7
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved critical defects cause production queries to return empty results, break ESM loading and profiling, and allow the mock tokenizer to hang indefinitely.
19 open issues
What changed in this PR
Adds Google BigQuery as a remote data source for AVA, extending the SDK and CLI toward the read-only query and analysis workflow requested in #1013.
Changes:
- Introduces a BigQuery engine, production connector, schema discovery, profiling, and GoogleSQL validation.
- Adds an in-memory mock adapter, sales fixtures, and automated tests.
- Registers BigQuery in SDK/CLI configuration and documents connection options and analysis examples.
| File | Description |
|---|---|
src/types.ts |
Defines BigQuery source options and mock seed types. |
src/remote/index.ts |
Exports the BigQuery engine. |
src/remote/bigquery/schema.ts |
Discovers tables, columns, and key metadata. |
src/remote/bigquery/README.md |
Documents architecture, setup, and examples. |
src/remote/bigquery/profile.ts |
Generates profiling queries and parses metrics. |
src/remote/bigquery/mock.ts |
Implements the in-memory query adapter. |
src/remote/bigquery/index.ts |
Exports BigQuery module utilities. |
src/remote/bigquery/examples.ts |
Supplies sample sales and customer data. |
src/remote/bigquery/engine.ts |
Implements the remote analysis engine lifecycle. |
src/remote/bigquery/connector.ts |
Defines connector contracts and adapter selection. |
src/remote/bigquery/bigquery-connector.ts |
Integrates the optional Google Cloud driver. |
src/query/bigquery.ts |
Adds query generation, validation, and row limiting. |
src/index.ts |
Registers BigQuery and exports source types. |
src/index.browser.ts |
Exports BigQuery-related types for browser consumers. |
src/engines.ts |
Updates engine availability guidance. |
src/duckdb/loaders/index.ts |
Rejects BigQuery sources in DuckDB. |
src/cli/session/protocol.ts |
Validates BigQuery configuration and engine selection. |
src/cli/session/datasets.ts |
Automatically selects BigQuery for CLI sources. |
src/cli/commands/source.ts |
Adds BigQuery help and configuration examples. |
skills/ava/references/sources.md |
Documents the CLI source workflow. |
site/public/logos/bigquery.svg |
Adds the BigQuery logo. |
site/content/docs/en/engines/remote.mdx |
Documents BigQuery among remote engines. |
site/content/docs/en/data/sources.mdx |
Adds connection fields and analysis examples. |
__tests__/remote/bigquery.test.ts |
Adds mock execution, schema, validation, and profiling tests. |
Review effort: Balanced
| if (cachedModule !== undefined) return cachedModule; | ||
| try { | ||
| // eslint-disable-next-line @typescript-eslint/no-var-requires | ||
| cachedModule = require('@google-cloud/bigquery') as BigQueryModule; |
| async query<T extends QueryResultRow = QueryResultRow>(sql: string): Promise<QueryResult & { rows: T[] }> { | ||
| await this.connect(); | ||
| const client = this.client!; | ||
| const [, rowsResponse] = await client.createQueryJob({ |
| await this.connect(); | ||
| const client = this.client!; | ||
| const [, rowsResponse] = await client.createQueryJob({ | ||
| query: sql, |
| i = j; | ||
| continue; | ||
| } | ||
| if (/[a-zA-Z_"]/.test(ch)) { |
| import { sqlIdentifier } from '../../util/sql'; | ||
|
|
||
| import type { MetricField } from '../../util/profile'; | ||
| import type { LogicalType, Metric, ParsedProfileOptions, Profile, Schema } from '../../types'; |
| ) | ||
| ) | ||
| return 'numeric'; | ||
| if (/^(string|bytes|json|array|struct)$/.test(base)) return 'string'; |
| if (/^date$/.test(base)) return `UNIX_MILLIS(CAST(${column} AS TIMESTAMP))`; | ||
| if (/^time$/.test(base)) return `EXTRACT(microsecond FROM ${column}) / 1000`; | ||
| return `UNIX_MILLIS(${column})`; |
| const value = `CAST(${column!} AS STRING)`; | ||
| return `CASE WHEN COUNT(DISTINCT ${value}) > COUNT(*) * ${maxDistinctRatio} | ||
| THEN NULL ELSE ( | ||
| SELECT TO_JSON_STRING(ARRAY_AGG(STRUCT(value, count)) ) |
|
|
||
| const [columnRows, keyRows] = await Promise.all([ | ||
| runQuery(`SELECT ${columnsProjection}\nFROM ${ds}.INFORMATION_SCHEMA.COLUMNS\nORDER BY table_name, ordinal_position`) as Promise<unknown>, | ||
| runQuery(`SELECT table_name, column_name, ordinal_position\nFROM ${ds}.INFORMATION_SCHEMA.KEY_COLUMN_USAGE\nORDER BY table_name, ordinal_position`) as Promise<unknown>, |
| /** BigQuery connection/query timeout in milliseconds (default 60000). */ | ||
| timeoutMs?: number; |
|
修复已推送
|
|
@yvonneyx hello，请问可否提供相关账号来进行开发测试？目前google cloud的BigQuery功能使用需要国际信用卡认证，尝试过用虚拟卡visa，但无法认证成功。 |
Add a read-only BigQuery engine and CLI/SDK registration for AVA v4. Uses the @google-cloud/bigquery driver (runtime require) with a built-in in-memory mock adapter that needs no GCP project, covering authentication, dataset/table schema discovery, read-only SELECT with row caps, and error handling. - src/remote/bigquery/: engine + connector (production driver + mock) + schema (dataset INFORMATION_SCHEMA) + per-column profile + seed examples - src/query/bigquery.ts: GoogleSQL dialect, read-only validation, LIMIT wrapper - Register bigquery engine/source across src/index, engines, remote/index, duckdb loader guard, and CLI session protocol/datasets/source - Docs: site sources.mdx + remote.mdx, module README, skill reference with two natural-language analysis examples; types for BigQuerySourceOptions - 31 vitest tests against the mock adapter (schema, profile, read-only, bounds, errors, dispose, AVA integration)
…mantics, ESM-safe driver load and job results Signed-off-by: DoubleCore <doublecore@users.noreply.github.com>
88d0edc to
5a9f717
Compare
|
本轮续修已推送原 PR head
本机没有 GCP 项目，仍是 mock 契约验证，不声称真实 BigQuery live 验证。fork PR 的 Actions 仍需维护者触发/批准。 最终 SHA 核验：同 tick 的元数据 amend 将原 PR head 更新为 |
5a9f717 to
34a83ba
Compare
Preserve Snowflake registrations and current source documentation. Adapt BigQuery to the shared dialect interface and cover CLI engine selection.
|
@DoubleCore，基于提交 评估范围是本分支新增的 BigQuery 实现及其直接影响，仓库既有问题和其他 PR 的实现不计入。首版可以限定一个 project/dataset；完整交付要求这个范围内能正确连接、读取结构、查询和分析，不要求补齐 BigQuery 的所有功能。
技术选型建议保留官方 验收前重点处理这几项：
测试优先补在现有 |
…tQueryResults, enforce dry-run budget and lossless decoding - package.json: declare @google-cloud/bigquery ^9.6.0 in optionalDependencies; the lazy dynamic import and SDK .d.ts types stay (tsc now catches call-site errors) - bigquery-connector.ts: maxResults/wrapIntegers/autoPaginate moved to the result-read interface (Job#getQueryResults), not the job config; first page only, never pull-all-then-truncate - budget: dry-run estimate checked against maxBytesBilled before any billable job; the job itself carries server-side maximumBytesBilled; useLegacySql stays a dialect flag only - lossless decoding: INT64 exact digits via BigQueryInt (regression 9007199254740993->9007199254740992 fixed), DATE/DATETIME/TIME server format strings, TIMESTAMP ISO 8601 UTC, NUMERIC exact decimal string, BYTES base64, STRUCT/ARRAY recursive; column names/types from driver schema; empty results fall back to job-metadata schema - schema: tableReference maps project (catalog) / dataset (schema) / table; catalog/schema read from INFORMATION_SCHEMA rows; mock INFORMATION_SCHEMA reports the project id like the live views - profile: duplicate_count-style metrics only for numeric/string/boolean/date columns (JSON/ARRAY/STRUCT stay unknown, no fabricated metrics) - docs: README + skills/ava/references/sources.md updated (install steps, options, value formats, unsupported views/relationships/complex types) - tests: SDK request/response stubs covering row-bound placement, budget rejection, lossless values incl. bigint regression, empty-result schema, timeout cancel; mock JSON-column profiling
|
已按评审意见修复，head 推进到 1. 依赖与类型： 2. 3. 结果无损转换：INT64 用字符串保留精度（ 4. 元数据/画像：project / dataset / table 三层映射修正；JSON 列的 5. 只读/费用/读取预算：加 6. 测试： 7. 文档：README 与 examples 的配置、导入路径、真实运行步骤已与实现一致；未支持的视图/关系/复杂类型写清限制。 实跑验证：
未覆盖：本机无 GCP 凭证，真实 BigQuery 接入（ADC/服务账号、最小权限、自然语言与直接 SQL 对照）未实跑验证。 |
|
感谢 @yvonneyx 的评审，本轮按你列出的缺口逐条修了（commit 705b45e + b1edc26，只动 BigQuery 相关文件）。实跑结果： 1. 依赖与类型： 2. SDK 调用位置： 3. 结果无损转换： 4. 元数据/画像： 5. 只读 / 费用 / 读取预算分层： 6. 测试：集中在现有 7. 文档/示例： 未覆盖 / 未验证：本机无 GCP 凭证，真实 BigQuery 接入未验证（含 dry run/maximumBytesBilled 的线上行为、真实 INT64/DATE/TIMESTAMP 行返解码只对照了 SDK 9.6.0 源码与 |
Google BigQuery 数据源
为 AVA v4 增加 Google BigQuery 只读数据源（Issue #1013）。
实现要点
BigQueryEngine（src/remote/bigquery/engine.ts）：通过QueryConnector完成 schema 发现、只读 GoogleSQL 执行与 profile。只读双重保障：本地validateReadOnlyBigQuery拒绝多语句/非 SELECT/DDL/DML/EXPORT DATA/scripting；生产连接以useLegacySql: false运行 query job。@google-cloud/bigquery（bigquery-connector.ts）：运行时require，包缺失不影响类型检查与测试；支持 service-accountcredentials（对象或 JSON 字符串）、clientEmail+privateKey、location、maxBytesBilled、defaultRowLimit。mock.ts）：纯内存的只读 GoogleSQL 子集解释器（SELECT/WHERE/GROUP BY/HAVING/ORDER BY/LIMIT/聚合、dataset.INFORMATION_SCHEMA视图），无需真实 GCP 账号即可跑通 schema/查询/profile/错误处理，供测试与文档示例复现。INFORMATION_SCHEMA.TABLES/.COLUMNS/.KEY_COLUMN_USAGE。limitedBigQuerySql用SELECT * FROM (…) LIMIT N包一层限制行数；maxRows默认 200、maxResultBytes默认 1 MiB；查询错误与缺失表透传清晰错误。注册链路（与 sqlserver 同构）
src/index.ts、src/engines.ts、src/remote/index.ts、src/duckdb/loaders/index.ts（guard）、src/cli/session/protocol.ts（zod source + engineTypeSchema）、src/cli/session/datasets.ts（自动选择 engine）、src/cli/commands/source.ts（帮助文本）、src/types.ts（BigQuerySourceOptions、MockTableSeed、SourceType/EngineConfig）。文档
site/content/docs/data/sources.mdx（连接字段表 + adapters + 2 个自然语言分析示例）与engines/remote.mdxsrc/remote/bigquery/README.mdskills/ava/references/sources.mdsite/public/logos/bigquery.svg测试
__tests__/remote/bigquery.test.ts— 31 个 vitest 用例全部通过（schema 发现、profile、只读校验、行数上限/截断、错误传播、dispose、AVA 端到端集成、mock connector）。tsc --noEmit与仓库eslint均通过。全套测试仅有 3 个 Windows 平台既有失败（cli-sessionwin32 guard 与 feishu HOME env），与本 PR 无关。