feat(remote): add SQL Server data source by DoubleCore · Pull Request #1064 · antvis/AVA

DoubleCore · 2026-11-03T02:57:31Z

Summary

Add a read-only Microsoft SQL Server remote data source to AVA (issue #1011). Users can now connect to a SQL Server, discover its schema, run read-only T-SQL queries, and get per-column profiles, through both the SDK and the CLI.

What's included

SqlServerEngine with two adapters:
- tedious (default) — production TDS driver, opened with ApplicationIntent=ReadOnly, loaded via a runtime require so the module has no hard dependency on tedious.
- mock — in-memory connector serving INFORMATION_SCHEMA catalog views and a small read-only T-SQL subset (SELECT / TOP / WHERE / GROUP BY / HAVING / ORDER BY / aggregates). No server or network required (demos, CI, docs).
Schema discovery via INFORMATION_SCHEMA.TABLES/.COLUMNS/.KEY_COLUMN_USAGE, mapping primary keys onto Schema.primaryKey.
Profile metrics compiled per table into a single read-only aggregate query.
Safety: local read-only guard rejects multi-statement / non-SELECT input; result caps via SELECT TOP (N) (T-SQL has no LIMIT) with maxRows (200) / maxResultBytes (1 MiB); idempotent dispose() that close()s the connection.
SDK + CLI wiring: registered for the SDK, added to CLI --engine help, createDataset engine selection, and the session protocol zod schema (plus instanceName/port mutual-exclusion validation).
Tests: vitest suite (__tests__/remote/sqlserver.test.ts, 27 cases) covering schema discovery, queries, error handling, result caps, read-only validation, and dispose — all on the mock adapter.
Docs: site data/sources.mdx + engines/remote.mdx, and a module README, with two natural-language analysis examples over seeded sales data.

Notes

No live SQL Server was available, so end-to-end verification is mock-driven (reproducible, offline). The tedious path is structurally typed and runtime-required; users install tedious for real connections.
tsc --noEmit passes; the only remaining diagnostic is a pre-existing one in src/duckdb/loaders/index.ts present on the base branch.
vitest new tests pass (27/27); existing (pre-existing) failures in feishu/cli-session (home-dir credential tests) and a flaky worker crash are unrelated and reproduced on the base commit.

Closes #1011

Add a read-only Microsoft SQL Server remote engine for AVA. - SqlServerEngine with tedious (production) and in-memory mock adapters - INFORMATION_SCHEMA-based schema discovery (tables, columns, primary keys) - Per-column profile metrics compiled into read-only T-SQL - Read-only SELECT guard, SELECT TOP (N) result caps, maxRows/maxResultBytes - Runtime tedious require so the module has no hard dependency on tedious - Registered for SDK (src/index.ts) and CLI (protocol/datasets/source helpers) - Vitest suite on the mock adapter; docs (site mdx + module README) with two natural-language examples over the seeded sales data Closes antvis#1011

Copilot

🟡 Changes recommended

Unresolved critical defects allow writes to bypass read-only validation, can crash or hang the process, and break SQL Server connectivity and schema discovery.

18 open issues

Read-only guard permits wrapped write statements · New
Reject unmatched closing brackets without hanging · New
Serialize catalog queries on the Tedious connection · New
Load the optional driver in both module formats · New
Handle connection errors to prevent process crashes · New
Decode Tedious rows using column aliases · New
Preserve CTEs and ordering when applying row limits · New
Expose the SQL Server mock seed through a public entry · New
Support generated profiling expressions in the mock · New
Evaluate HAVING clauses or reject unsupported queries · New
Honor SQL AND-before-OR predicate precedence · New
Preserve composite primary keys in mock discovery · New
Map bit columns as booleans, not numbers · New
Preserve time components in timestamp bounds · New
Escape all JSON-special characters in profile values · New
Cast text, ntext, and xml before profiling · New
Filter key metadata to primary-key constraints · New
Preserve schema-qualified table identities · New

What changed in this PR

Adds SQL Server as a remote analysis engine in AVA, extending the SDK and CLI toward the schema discovery, read-only querying, and natural-language analysis requested in #1011.

Changes:

Adds Tedious and in-memory mock adapters, schema discovery, profiling, and connection lifecycle handling.
Introduces T-SQL generation, validation, result limiting, and SDK/CLI registration.
Adds mock-based tests, sales seed data, and connection documentation.

File	Description
`src/types.ts`	Defines SQL Server source options and mock seed types.
`src/remote/sqlserver/tedious.ts`	Implements the optional production driver connector.
`src/remote/sqlserver/tds-config.ts`	Builds connection, authentication, and timeout settings.
`src/remote/sqlserver/schema.ts`	Discovers tables, columns, and key metadata.
`src/remote/sqlserver/README.md`	Documents adapters, setup, examples, and verification.
`src/remote/sqlserver/profile.ts`	Compiles column metrics into aggregate T-SQL.
`src/remote/sqlserver/mock.ts`	Provides seeded catalog views and limited query execution.
`src/remote/sqlserver/index.ts`	Exports engine components and helpers.
`src/remote/sqlserver/examples.ts`	Supplies sales data for examples and tests.
`src/remote/sqlserver/engine.ts`	Connects discovery, profiling, translation, and execution.
`src/remote/sqlserver/connector.ts`	Defines the connector contract and adapter factory.
`src/remote/index.ts`	Exposes the SQL Server engine internally.
`src/query/sqlserver.ts`	Adds T-SQL generation, validation, and row limiting.
`src/index.ts`	Registers the engine and exports configuration types.
`src/engines.ts`	Updates the unavailable-engine error message.
`src/cli/session/protocol.ts`	Validates SQL Server session configuration.
`src/cli/session/datasets.ts`	Selects SQL Server automatically for matching CLI sources.
`src/cli/commands/source.ts`	Updates engine-selection help text.
`site/public/logos/sql-server.svg`	Adds the source-card image asset.
`site/content/docs/en/engines/remote.mdx`	Documents SQL Server within the remote-engine workflow.
`site/content/docs/en/data/sources.mdx`	Adds connection fields, adapter guidance, and examples.
`__tests__/remote/sqlserver.test.ts`	Adds mock-backed engine tests and profiling unit tests.

_{Review effort: Balanced}

+  if (MULTI_STATEMENT.test(` ${stripped}`)) {
+    throw new Error('SQL Server query must contain exactly one read-only SELECT statement');
+  }


+      i = j;
+      continue;
+    }
+    if (/[a-zA-Z_@\[\]"]/.test(ch)) {


+  const [columns, keys] = await Promise.all([
+    runQuery(`
+      SELECT
+        c.TABLE_NAME AS table_name,
+        c.COLUMN_NAME AS name,
+        c.DATA_TYPE AS type,
+        c.IS_NULLABLE AS nullable
+      FROM INFORMATION_SCHEMA.COLUMNS c
+      WHERE c.TABLE_CATALOG = ${db}
+      ORDER BY c.TABLE_NAME, c.ORDINAL_POSITION
+    `),
+    runQuery(`
+      SELECT
+        k.TABLE_NAME AS table_name,
+        k.COLUMN_NAME AS name,
+        k.ORDINAL_POSITION AS position
+      FROM INFORMATION_SCHEMA.KEY_COLUMN_USAGE k
+      WHERE k.TABLE_CATALOG = ${db}
+      ORDER BY k.TABLE_NAME, k.ORDINAL_POSITION
+    `),
+  ]);


+  if (cachedModule !== undefined) return cachedModule;
+  try {
+    // eslint-disable-next-line @typescript-eslint/no-var-requires
+    cachedModule = require('tedious') as TediousModule;


+    const connection = new Connection(config);
+
+    const ok = await new Promise<boolean>((resolve) => {
+      connection.connect((error) => resolve(!error));


+/** Epoch-milliseconds expression for a date column (86400000 ms per day from SQL Server's epoch). */
+function epochMs(column: string): string {
+  return `(DATEDIFF_BIG(day, '1970-01-01', ${column}) * 86400000.0)`;
+}


+      const value = `CAST(${column!} AS nvarchar(4000))`;
+      return `CASE WHEN COUNT(DISTINCT ${value}) > COUNT(*) * ${maxDistinctRatio}
+        THEN NULL ELSE (
+          SELECT '[' + STRING_AGG(CONCAT('{"value":"', REPLACE(CAST(v AS nvarchar(max)), '"', '\\"'), '","count":', CAST(n AS nvarchar(20)), '}'), ',') + ']'


+          output,
+          expression: definition.expression({
+            table,
+            column: field ? sqlIdentifier(field.name) : undefined,


+      FROM INFORMATION_SCHEMA.KEY_COLUMN_USAGE k
+      WHERE k.TABLE_CATALOG = ${db}
+      ORDER BY k.TABLE_NAME, k.ORDINAL_POSITION


+    models: tables.map(({ table_name }) => {
+      const fields = columnRows
+        .filter((column) => column.table_name === table_name)
+        .map((column) => ({ name: column.name, type: column.type, notNull: !bool(column.nullable) }));


…ce (antvis#1011)

DoubleCore · 2026-11-04T05:24:46Z

Addressed the open Copilot findings on the SQL Server data source (head 35394dc):

Mock connector: fixed the tokenizer hang on a standalone ]; implemented HAVING for aggregate conditions (unsupported forms now error instead of being dropped); applied AND-before-OR predicate precedence; preserved the full ordered composite primary key.
Read-only guard: rejects SELECT INTO, unbalanced parentheses / wrapper escapes, and concatenated statements before the row-limit wrapper is applied.
Tedious adapter: decodes row arrays via each cell's metadata.colName; adds a connection-level error handler that rejects the in-flight call and invalidates the connection; lazy driver loading now works in both the CJS and ESM builds.
Schema discovery: filters KEY_COLUMN_USAGE through TABLE_CONSTRAINTS so only primary-key columns are reported, and serializes catalog queries on the single connection.
Profiling: bit is classified as boolean; datetime min/max use millisecond DATEDIFF_BIG; top-values JSON uses STRING_ESCAPE(..., 'json'); legacy text/ntext/xml columns are cast for metrics.

Tests added under __tests__/remote/sqlserver.test.ts and __tests__/remote/sqlserver/. tsc --noEmit passes and all 46 SQL Server tests pass.

…error handling Signed-off-by: DoubleCore <doublecore@users.noreply.github.com>

DoubleCore · 2026-11-04T11:22:26Z

修复已推送 a070589e。按 review 处理：

query/sqlserver.ts：只读守卫在包装前按语句边界拒绝 INTO/CREATE/DROP/ALTER/INSERT/UPDATE/DELETE/MERGE/TRUNCATE/EXEC，覆盖 codex 给出的 SELECT 1 AS a) AS t SELECT 1 AS b INTO dbo.ava_write_probe FROM (SELECT 1 AS c 及邻居 payload；derived-table 包装不再破坏合法 ORDER BY。
tedious.ts：用 columnMetadata 的 colName 解码行（不再数字 key）；挂连接级 error 监听并 reject 挂起请求。
mock.ts：孤立 ] 等不再让 tokenizer 死循环（带 timeout 回归）；默认 profiling 表达式（COUNT(DISTINCT)/CAST/CASE 等）可在 mock 上执行。
remote/index.ts + docs：SQLSERVER_TABLE_SEED 的导出与文档示例对齐。
验证：vitest run __tests__/remote/sqlserver*.test.ts → 53 passed；tsc --noEmit rc=0。
live SQL Server 未验证，仅 mock 驱动 + 单测（本机无实例）。

DoubleCore · 2026-11-04T13:00:42Z

#1011

DoubleCore · 2026-11-04T13:07:54Z

本轮修复已推送（commit 40556102），PR 由 CONFLICTING 恢复 MERGEABLE：

解冲突 — 与 upstream/oss-camp2026 合并，在 src/duckdb/loaders/index.ts、src/index.ts、src/cli/commands/source.ts、src/cli/session/protocol.ts、src/types.ts、site/content/docs/en/data/sources.mdx 这些共享注册点上同时保留上游新增的 BigQuery/Snowflake/Notion 与我方 SQL Server，未丢任何一边；PR diff 仍为本任务 27 个文件，未卷入上游改动。

2 条未解决 review：

src/query/sqlserver.ts — 只读 guard 现在拒绝多语句/括号闭合逃逸与 INTO 写入，row-limit wrapper 之后不会再产生第二条语句（含 SELECT 1 AS a) AS t SELECT 1 AS b INTO ... 这条绕过用例的回归测试）。
src/remote/sqlserver/tedious.ts — 改为按 column metadata 把 cell 数组解码成按列名的行对象，并处理重复列名/别名，修复 live schema 发现、profiling 与查询结果的数字 key 问题，新增 __tests__/remote/sqlserver/tedious.test.ts 解码用例。

验证：sqlserver 相关测试（__tests__/remote/sqlserver*）与 guard 测试通过、git diff --check 通过。完整 tsc --noEmit 仅因现有 node_modules 缺上游新增的 antlr4ts/antlr4ts-sql 依赖而停在无关文件，与本 PR 无关（未擅自 npm install）。

DoubleCore · 2026-11-06T03:11:18Z

本轮续修已推送原 PR head 70d2b1782f7348ffdf98f72e76fd8493c2906c4c，PR 由 CONFLICTING 恢复 MERGEABLE（base=oss-camp2026，changed_files=27）。

在最新 oss-camp2026 上保留原 SQL Server 实现及此前 review 修复，同时保留上游 Tencent Docs 与其他数据源；相对最新 upstream 的 diff 仅为 SQL Server 范围。
实跑 npx vitest run __tests__/remote/sqlserver.test.ts __tests__/remote/sqlserver/schema.test.ts __tests__/remote/sqlserver/tedious.test.ts：3 files / 55 tests passed。
本地 worktree 初次 tsc 暴露基线声明的 mssql/antlr4ts/antlr4ts-sql 依赖未安装；按现有清单恢复后，npx tsc --noEmit 已复跑通过。
git diff upstream/oss-camp2026...HEAD --check：通过。

未执行真实 SQL Server live 端到端测试，以上均为离线 mock 验证；fork PR 的 GitHub Actions 仍需维护者触发/批准。

yvonneyx · 2026-11-08T13:48:42Z

Add Supports for SQL Server #1052

目前主分支已经支持了 SQL Server。此pr不考虑合入。

不过不用担心，这个 pr 会正常评分。

DoubleCore force-pushed the task-1011-sqlserver branch from 4d93315 to 1cc8f89 Compare November 3, 2026 03:02

DoubleCore force-pushed the task-1011-sqlserver branch from 1cc8f89 to 8ca53fa Compare November 3, 2026 10:43

yvonneyx requested a balanced review from Copilot November 3, 2026 11:05

Copilot started reviewing on behalf of yvonneyx November 3, 2026 11:06

View session

Copilot AI reviewed Nov 3, 2026

View reviewed changes

fix(duckdb): return or throw on unknown data source type (TS7030)

e2b106c

DoubleCore mentioned this pull request Nov 4, 2026

【2026 开源轻训营专属 Issue】为 AVA 增加 SQL Server 数据源支持 #1011

Open

3 tasks done

fix(remote): address open review findings on the SQL Server data sour…

35394dc

…ce (antvis#1011)

fix(remote/sqlserver): harden read-only guard and fix row decoding + …

1b3279f

…error handling Signed-off-by: DoubleCore <doublecore@users.noreply.github.com>

DoubleCore force-pushed the task-1011-sqlserver branch from a070589 to 1b3279f Compare November 4, 2026 11:26

fix(remote/sqlserver): merge base and address review findings

70d2b17

DoubleCore force-pushed the task-1011-sqlserver branch from 4055610 to 70d2b17 Compare November 6, 2026 03:01

yvonneyx closed this Nov 8, 2026

Navigation Menu

Repository navigation

feat(remote): add SQL Server data source - #1064

feat(remote): add SQL Server data source#1064
DoubleCore wants to merge 5 commits into
antvis:oss-camp2026from
DoubleCore:task-1011-sqlserver

DoubleCore commented Nov 3, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

DoubleCore commented Nov 4, 2026

Uh oh!

DoubleCore commented Nov 4, 2026

Uh oh!

DoubleCore commented Nov 4, 2026

Uh oh!

DoubleCore commented Nov 4, 2026

Uh oh!

DoubleCore commented Nov 6, 2026

Uh oh!

yvonneyx commented Nov 8, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

DoubleCore commented Nov 3, 2026

Summary

What's included

Notes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

DoubleCore commented Nov 4, 2026

Uh oh!

DoubleCore commented Nov 4, 2026

Uh oh!

DoubleCore commented Nov 4, 2026

Uh oh!

DoubleCore commented Nov 4, 2026

Uh oh!

DoubleCore commented Nov 6, 2026

Uh oh!

yvonneyx commented Nov 8, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants