Skip to content

fix(splitter): skip the SEARCH and CYCLE clauses of a CTE - #1191

Merged
psteinroe merged 28 commits into
mainfrom
fix/splitter-cte-search-cycle
Nov 16, 2026
Merged

psteinroe merged 28 commits into
mainfrom
fix/splitter-cte-search-cycle

Conversation

@psteinroe

@psteinroe psteinroe commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Stacked on #1184. Blocked on #1187, which makes the pretty printer round-trip the SEARCH and CYCLE clauses of a CTE. Until that merges, the printer's postgres_regress test fails on the statements this PR now splits like the parser.

Fix

In pgls_statement_splitter, the SEARCH and CYCLE clauses after a CTE are skipped, up to the column after SET or USING (gram.y). Without the end, e.g. while typing, the clause ends at the next ;. Before, the query after them started a new statement:

WITH RECURSIVE t(n) AS (SELECT 1 UNION ALL SELECT n + 1 FROM t)
  SEARCH DEPTH FIRST BY n SET seq
SELECT * FROM t;

CTEs inside another statement (CREATE VIEW v AS WITH ...) no longer stop at these clauses either; #1184 left that out on purpose.

Cost: one token per iteration, up to the end of the clause or the next ;. No recursion.

Tests

Fixtures

with.sql is re-recorded with just record-regress <major> for every version; nothing else changes. The postgres_regress__typecheck_*, spans and splitter vs parser snapshots follow, with no finding on a statement Postgres accepted.

Expected CI

Only the printer's postgres_regress test fails, on 23 statements of with.sql per version, until #1187 merges.

Update: #1183, #1184 and #1187 are merged. Main is merged in (814909b); the spans snapshot change moved with the snapshot to pgls_resolve. CI is green, including the printer's postgres_regress test, so this is no longer blocked.

Inside a BEGIN ATOMIC body the END of a CASE expression closed the body,
so the real END became a statement of its own, which the formatter
printed as COMMIT. CASE now opens a level like BEGIN, as in psql, and
nested CASE expressions are tracked. END, CASE and BEGIN used as column
labels or names don't count.
Each statement is checked to parse back the same on its own, which
relies on the statement boundaries. A wrong boundary cuts a valid
statement into fragments, and one of them can format into something
else, e.g. the END of a BEGIN ATOMIC body into COMMIT.

When a statement doesn't parse, the text from the statement before it
to the statement after the last one that doesn't parse is now parsed as
a whole before and after formatting. If the result has a syntax error
or different statements, the file is left as written with a diagnostic,
for --write, --stdin-file-path and the editor alike. --write no longer
rewrites a file whose content doesn't change.
functest_S_15 in create_function_sql now splits into one statement, which
Postgres accepts, and pg_get_functiondef('functest_S_15') finds it. With
the rest of the file no longer behind an unparsable statement, the
CREATE FUNCTION test1 that returns text for int is reported as well.
…ction_bodies off

Postgres doesn't validate the body of a SQL function created while
check_function_bodies is off, so its statements may use tables that an
earlier statement of the body creates.
…ges it

A statement that calls a SQL function of the file whose body may change
the catalog, directly or through another such function, taints the
catalog like CALL does.
# Conflicts:
#	crates/pgls_analyser/tests/snapshots/postgres_regress__typecheck_15.snap
#	crates/pgls_analyser/tests/snapshots/postgres_regress__typecheck_16.snap
#	crates/pgls_analyser/tests/snapshots/postgres_regress__typecheck_17.snap
#	crates/pgls_analyser/tests/snapshots/postgres_regress__typecheck_18.snap
The splitter now keeps the CASE ... END of a BEGIN ATOMIC body in one
statement, so more of create_function_sql resolves.
# Conflicts:
#	crates/pgls_analyser/src/lib.rs
#	crates/pgls_walk/src/walk.rs
# Conflicts:
#	crates/pgls_workspace/src/workspace/server/formatting.rs
# Conflicts:
#	crates/pgls_analyser/tests/snapshots/postgres_regress__typecheck_15.snap
#	crates/pgls_analyser/tests/snapshots/postgres_regress__typecheck_16.snap
#	crates/pgls_analyser/tests/snapshots/postgres_regress__typecheck_17.snap
#	crates/pgls_analyser/tests/snapshots/postgres_regress__typecheck_18.snap
# Conflicts:
#	crates/pgls_analyser/tests/snapshots/postgres_regress__typecheck_15.snap
#	crates/pgls_analyser/tests/snapshots/postgres_regress__typecheck_16.snap
#	crates/pgls_analyser/tests/snapshots/postgres_regress__typecheck_17.snap
#	crates/pgls_analyser/tests/snapshots/postgres_regress__typecheck_18.snap
…the regression suite

From each statement of the splitter, the check looks for the statement psql
sends: the shortest text up to a `;` that libpg_query's parser reads as one
statement. Where that isn't one statement of the splitter, the splitter cut it
in pieces or ran past its end. The mismatches are snapshotted per version.
…d set operations like the parser

- `CREATE SCHEMA s CREATE TABLE ...` keeps its schema elements.
- `WITH` starts a statement only when CTEs follow: `WITH RECURSIVE`,
  `WITH name (` or `WITH name AS [NOT] MATERIALIZED (`. `COPY ... WITH (...)`,
  `WITH HOLD`, `WITH TIES`, `WITH NO DATA`, `WITH (FORCE)` and
  `IS JSON WITH UNIQUE KEYS` no longer start one.
- A missing `(` no longer runs to the next `)`, which swallowed the rest of
  a file after such a cut.
- The `SEARCH` and `CYCLE` clauses of a CTE are skipped.
- `SELECT` after `INTERSECT` or `DISTINCT`, `ON COMMIT DELETE ROWS`,
  `SHARE UPDATE EXCLUSIVE` and `PREPARE p AS INSERT/UPDATE/DELETE` stay in
  the statement.
- CTEs after `AS`, `FOR` and in rule actions, and the query of an
  `INSERT INTO` there, belong to the statement.
…plits like the parser

The statements that were cut in pieces now run as one, and the text that a
`WITH` swallowed up to a later `)` is split again: guc, copy2, limit,
matview, portals, rules and rowsecurity were mostly hidden behind one
statement. Files whose split didn't change keep their verdicts.
…of the parser's statement

A cut at a `;` inside a statement, e.g. in a `BEGIN ATOMIC` body, leaves a
fragment that ends with a `;` and doesn't parse. The `;` of the statements
after it are tried too. The parser's statement has to start where the
splitter's does, so a lone `;` doesn't pass for the start of the next
statement.
…tches

# Conflicts:
#	crates/pgls_postgres_regress/tests/snapshots/splitter_vs_parser__splitter_vs_parser_15.snap
#	crates/pgls_postgres_regress/tests/snapshots/splitter_vs_parser__splitter_vs_parser_16.snap
#	crates/pgls_postgres_regress/tests/snapshots/splitter_vs_parser__splitter_vs_parser_17.snap
#	crates/pgls_postgres_regress/tests/snapshots/splitter_vs_parser__splitter_vs_parser_18.snap
The WITH that starts no CTE, the missing `(` and the SEARCH and CYCLE
clauses of a CTE move to their own PRs: they expose bugs of the pretty
printer and a false positive of the analyser, which get fixed first.

CTEs after AS, FOR and in rule actions stay, with the list of WITH
clauses from before. They stop at SEARCH and CYCLE, like a top-level
WITH.
Starting from the verdicts of main, only the files whose statement
starts change are re-recorded. The typecheck, spans and splitter vs
parser snapshots follow.
The SEARCH and CYCLE clauses after a CTE are skipped, up to the column
after SET or USING. Without the end, e.g. while typing, the clause ends
at the next `;`. CTEs in another statement, e.g. after CREATE VIEW v AS,
no longer stop at them either.
psteinroe added a commit that referenced this pull request Nov 16, 2026
…d set operations like the parser (#1184)

Stacked on #1183, which checks the splitter's boundaries against the
parser on the regression suite. This fixes what it found: the mismatches
go from 144–150 per version to 1.

### Fixes

All in `pgls_statement_splitter`. Each follows what psql sends as one
statement (up to the `;`) and Postgres' grammar.

- **`CREATE SCHEMA s CREATE TABLE ...`**: `CREATE` no longer starts a
statement inside `CREATE SCHEMA`, whose schema elements are `CREATE
TABLE/VIEW/INDEX/SEQUENCE/TRIGGER`
([gram.y](https://github.com/postgres/postgres/blob/REL_18_6/src/backend/parser/gram.y#L1517)).
- **`WITH` that starts no CTE**: `WITH` starts a statement only when
CTEs follow, i.e. `WITH RECURSIVE`, `WITH name (` or `WITH name AS [NOT]
MATERIALIZED (`
([gram.y](https://github.com/postgres/postgres/blob/REL_18_6/src/backend/parser/gram.y#L13040)).
`COPY ... WITH (...)`, `COPY ... WITH DELIMITER AS '|'`, `DECLARE ...
WITH HOLD`, `FETCH ... WITH TIES`, `DROP DATABASE ... WITH (FORCE)`,
`REFRESH ... WITH NO DATA`, `IS JSON WITH UNIQUE KEYS` and `CREATE CAST
... WITH INOUT AS IMPLICIT` stay in their statement. This replaces the
list of exceptions (`WITH ORDINALITY`, `WITH CHECK`, `WITH TIME ZONE`,
...). At the end of the text, a half-written `WITH`, `WITH name` or
`WITH name AS` still starts a statement.
- **No `(` to match**: the skip of a parenthesized part (a CTE's body)
ran to the next `)` when the `(` was missing, which could be hundreds of
statements later. It now reports the missing `(` and stops.
- **CTE in another statement**: CTEs after `AS` (`CREATE VIEW v AS`,
`PREPARE p AS`), `FOR` (`DECLARE c CURSOR FOR`) and
`DO`/`INSTEAD`/`ALSO` (rule actions) belong to that statement, and so
does the statement after them.
- **`SEARCH` and `CYCLE`** after a CTE are skipped, up to the column
after `SET` or `USING`
([gram.y](https://github.com/postgres/postgres/blob/REL_18_6/src/backend/parser/gram.y#L13061)).
Without the end, e.g. while typing, the clause ends at the next `;`.
- **`SELECT` after `INTERSECT` and `DISTINCT`** (`UNION DISTINCT
SELECT`) stays in the statement
([gram.y](https://github.com/postgres/postgres/blob/REL_18_6/src/backend/parser/gram.y#L12935)),
like after `UNION` and `EXCEPT`.
- **`DELETE ROWS`** (`ON COMMIT DELETE ROWS`,
[gram.y](https://github.com/postgres/postgres/blob/REL_18_6/src/backend/parser/gram.y#L4674))
starts no statement, like `delete(` before.
- **`INSERT`/`UPDATE`/`DELETE` after `AS`** (`PREPARE p AS UPDATE ...`,
[gram.y](https://github.com/postgres/postgres/blob/REL_18_6/src/backend/parser/gram.y#L12107))
and **`UPDATE` after `SHARE`** (`LOCK t IN SHARE UPDATE EXCLUSIVE MODE`,
[gram.y](https://github.com/postgres/postgres/blob/REL_18_6/src/backend/parser/gram.y#L12469))
stay in the statement.
- **`INSERT INTO ... SELECT` in another statement** (a rule action,
`PREPARE`): after a nested `INSERT INTO`, `SELECT` and `WITH` no longer
start a statement, like in a top-level `INSERT`.

**Cost:** no extra pass and no recursion. A `WITH` looks at most three
tokens ahead instead of one. The CTE and `INSERT INTO` cases are two
booleans in the loop of a statement. `SEARCH`/`CYCLE` move one token per
iteration and stop at the next `;`.

### Tests
- Splitter unit tests: `create_schema_with_elements`,
`with_that_starts_no_cte`, `cte_search_and_cycle`,
`intersect_and_distinct_before_select`, `on_commit_delete_rows`,
`lock_share_update_exclusive`, `queries_of_prepare_declare_and_rules`,
and `half_written_with` for half-written SQL (a missing `;` before a
CTE, `COPY ... WITH` at the end of a line, a CTE without `(`).
- The mismatch list of #1183 shrinks to the one case below.

### Fixtures
The verdicts were recorded with the splitter's boundaries, so the files
it now splits differently are re-recorded with `just record-regress
<major>`, one version at a time: 31–35 verdict files per version. Only
files whose statement starts changed were kept; the rest, and the
catalogs, are unchanged. The text that a `WITH` swallowed is now split
and run, so the corpus grows by 2100–2300 statements per version, e.g.
`rowsecurity` from 111 to 1223 statements and `rules` from 71 to 621.

The `postgres_regress__typecheck_*` snapshots and the spans snapshot of
the regression suite follow from the new verdicts; they're updated from
CI in a follow-up commit.

### Not covered
- `CREATE SCHEMA` with blank lines between its elements
(`namespace.sql:9`) is still cut at the blank lines. Each element parses
on its own, and that's what the blank-line recovery takes for a missing
`;` (`SELECT a FROM t` + blank line + `BEGIN;`). It stays in the
mismatch list.
- A half-written `COPY t FROM stdin WITH DELIMITER AS` at the end of the
text looks like a CTE and starts a statement until the string is typed.

**Update:** #1184 now keeps only the fixes that are green on their own:
`CREATE SCHEMA` elements, `SELECT` after `INTERSECT`/`DISTINCT`, `ON
COMMIT DELETE ROWS`, `LOCK ... SHARE UPDATE EXCLUSIVE`,
`INSERT`/`UPDATE`/`DELETE` after `AS`, `INSERT INTO ... SELECT` nested
in a rule action or `PREPARE`, and CTEs nested after
`AS`/`FOR`/`DO`/`INSTEAD`/`ALSO`. The others move to two draft PRs
stacked on this one, because the statements they now split like the
parser expose bugs that get fixed first, in their own PRs on main:

- `SEARCH`/`CYCLE` → #1191, blocked on the printer's round-trip of those
clauses (#1187).
- `WITH` that starts no CTE, with the fix for a missing `(` → #1192,
blocked on the printer's round-trips of `IS JSON`, `ALTER TABLE ...
RENAME RULE` and `MERGE` actions (#1186, #1188, #1189) and on the
analyser's false positive on `matview.sql:261` (#1190).

The nested CTEs worked without the new `WITH` check, so they stay here:
a `WITH` after `AS`, `FOR`, `DO`, `INSTEAD` or `ALSO` that isn't in the
old list of `WITH` clauses starts the CTEs of the statement's query.
They stop at `SEARCH`/`CYCLE` after a CTE, like a top-level `WITH` did,
so `CREATE VIEW v AS WITH ... SEARCH ... SELECT` is still cut before the
`SELECT` until #1191. `CREATE` keeps its loop over `WITH (...)` options.

Fixtures and snapshots now: 22–24 verdict files per version are
re-recorded from main's verdicts (`alter_table`, `collate*`,
`create_index`, `create_schema`, `create_view`, `event_trigger`,
`foreign_key`, `guc`, `insert`, `lock`, `maintain_every`, `namespace`,
`plancache`, `publication`, `stats_ext`, `temp`, `transactions`,
`union`, `updatable_views`, `with`, plus `prepare` from 16 and
`partition_prune` in 18); the catalogs are unchanged. Cuts that are
merged back shrink the corpus by about 150 statements per version (39966
→ 39817 for 15, 47288 → 47134 for 18). The mismatch list goes from
144–150 to 48–49 per version; the entries left are the ones #1191 and
#1192 fix, and `namespace.sql:9`. The `postgres_regress__typecheck_*`
snapshots (1622–1914 findings on rejected statements, none on accepted
ones) and the spans snapshot are in this PR now, so the sentence above
about updating them from CI in a follow-up commit is outdated.
@psteinroe
psteinroe marked this pull request as ready for review November 16, 2026 14:59
# Conflicts:
#	crates/pgls_analyser/tests/snapshots/postgres_regress__typecheck_15.snap
#	crates/pgls_analyser/tests/snapshots/postgres_regress__typecheck_16.snap
#	crates/pgls_analyser/tests/snapshots/postgres_regress__typecheck_17.snap
#	crates/pgls_analyser/tests/snapshots/postgres_regress__typecheck_18.snap
@psteinroe
psteinroe merged commit ab43386 into main Nov 16, 2026
12 checks passed
@psteinroe
psteinroe deleted the fix/splitter-cte-search-cycle branch November 16, 2026 20:42
psteinroe added a commit that referenced this pull request Nov 16, 2026
Stacked on #1191. **Blocked on:**
- #1186, #1188 and #1189: pretty printer round-trips of `IS JSON`,
`ALTER TABLE ... RENAME RULE` and `MERGE` actions;
- #1190: the analyser's `unknownRelation` on `matview.sql:261`, a table
created by a PL/pgSQL function that the file defines and calls.

The statements this PR now splits like the parser hit those bugs.

### Fixes
All in `pgls_statement_splitter`.

- **`WITH` that starts no CTE**: `WITH` starts a statement only when
CTEs follow, i.e. `WITH RECURSIVE`, `WITH name (` or `WITH name AS [NOT]
MATERIALIZED (`
([gram.y](https://github.com/postgres/postgres/blob/REL_18_6/src/backend/parser/gram.y#L13040)).
`COPY ... WITH (...)`, `COPY ... WITH DELIMITER AS '|'`, `DECLARE ...
WITH HOLD`, `FETCH ... WITH TIES`, `DROP DATABASE ... WITH (FORCE)`,
`REFRESH ... WITH NO DATA`, `IS JSON WITH UNIQUE KEYS` and `CREATE CAST
... WITH INOUT AS IMPLICIT` stay in their statement. This replaces the
list of exceptions (`WITH ORDINALITY`, `WITH CHECK`, `WITH TIME ZONE`,
...) and the `WITH` loop of `CREATE`. At the end of the text, a
half-written `WITH`, `WITH name` or `WITH name AS` still starts a
statement.
- **No `(` to match**: the skip of a parenthesized part (a CTE's body)
ran to the next `)` when the `(` was missing, which could be hundreds of
statements later. It now reports the missing `(` and stops.

**Cost:** a `WITH` looks at most three tokens ahead instead of one. No
extra pass and no recursion.

### Tests
- Splitter unit tests `with_that_starts_no_cte` and `half_written_with`
(a missing `;` before a CTE, `COPY ... WITH` at the end of a line, a CTE
without `(`).
- The mismatch list of #1183 shrinks to the one case below.

### Fixtures
The files whose statement starts change are re-recorded with `just
record-regress <major>`, one version at a time: 10–12 verdict files per
version (`copy`, `copy2`, `drop_if_exists`, `guc`, `limit`, `matview`,
`portals`, `rowsecurity`, `rules`, `xmlmap`, plus `sqljson` from 16 and
`copyencoding` in 18). The text that a `WITH` swallowed is now split and
run, so the corpus grows by about 2300–2500 statements per version, e.g.
`rowsecurity` from 111 to 1223 statements and `rules` from 71 to 621.
The `postgres_regress__typecheck_*`, spans and splitter vs parser
snapshots follow.

### Expected CI
Until the PRs above merge: the printer's `postgres_regress` round-trips,
and the analyser's `postgres_regress` on `matview.sql:261` (one finding
per version).

### Not covered
- `CREATE SCHEMA` with blank lines between its elements
(`namespace.sql:9`) is still cut at the blank lines; it stays in the
mismatch list.
- A half-written `COPY t FROM stdin WITH DELIMITER AS` at the end of the
text looks like a CTE and starts a statement until the string is typed.



**Update:** #1186, #1188, #1189 and #1190 are merged, so this is no
longer blocked. #1191 and main are merged in (9eb2c65). The
`postgres_regress__typecheck_*` snapshots are regenerated on top of
#1190: `matview.sql:261` has no finding any more, and 33–43 findings per
version are gone, all on statements Postgres rejected, after calls to
PL/pgSQL functions of the file (`incremental_sort.sql`, `memoize.sql`,
`explain.sql`, …), like the ones #1190 removed on main. CI is green,
including the printer's round-trips and `postgres_regress`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant