Repository navigation
fix(splitter): skip the SEARCH and CYCLE clauses of a CTE - #1191
Merged
Merged
Conversation
Inside a BEGIN ATOMIC body the END of a CASE expression closed the body, so the real END became a statement of its own, which the formatter printed as COMMIT. CASE now opens a level like BEGIN, as in psql, and nested CASE expressions are tracked. END, CASE and BEGIN used as column labels or names don't count.
Each statement is checked to parse back the same on its own, which relies on the statement boundaries. A wrong boundary cuts a valid statement into fragments, and one of them can format into something else, e.g. the END of a BEGIN ATOMIC body into COMMIT. When a statement doesn't parse, the text from the statement before it to the statement after the last one that doesn't parse is now parsed as a whole before and after formatting. If the result has a syntax error or different statements, the file is left as written with a diagnostic, for --write, --stdin-file-path and the editor alike. --write no longer rewrites a file whose content doesn't change.
functest_S_15 in create_function_sql now splits into one statement, which
Postgres accepts, and pg_get_functiondef('functest_S_15') finds it. With
the rest of the file no longer behind an unparsable statement, the
CREATE FUNCTION test1 that returns text for int is reported as well.
…ction_bodies off Postgres doesn't validate the body of a SQL function created while check_function_bodies is off, so its statements may use tables that an earlier statement of the body creates.
…ges it A statement that calls a SQL function of the file whose body may change the catalog, directly or through another such function, taints the catalog like CALL does.
# Conflicts: # crates/pgls_walk/src/walk.rs
# Conflicts: # crates/pgls_analyser/tests/snapshots/postgres_regress__typecheck_15.snap # crates/pgls_analyser/tests/snapshots/postgres_regress__typecheck_16.snap # crates/pgls_analyser/tests/snapshots/postgres_regress__typecheck_17.snap # crates/pgls_analyser/tests/snapshots/postgres_regress__typecheck_18.snap
The splitter now keeps the CASE ... END of a BEGIN ATOMIC body in one statement, so more of create_function_sql resolves.
# Conflicts: # crates/pgls_analyser/src/lib.rs # crates/pgls_walk/src/walk.rs
# Conflicts: # crates/pgls_workspace/src/workspace/server/formatting.rs
# Conflicts: # crates/pgls_analyser/tests/snapshots/postgres_regress__typecheck_15.snap # crates/pgls_analyser/tests/snapshots/postgres_regress__typecheck_16.snap # crates/pgls_analyser/tests/snapshots/postgres_regress__typecheck_17.snap # crates/pgls_analyser/tests/snapshots/postgres_regress__typecheck_18.snap
# Conflicts: # crates/pgls_analyser/tests/snapshots/postgres_regress__typecheck_15.snap # crates/pgls_analyser/tests/snapshots/postgres_regress__typecheck_16.snap # crates/pgls_analyser/tests/snapshots/postgres_regress__typecheck_17.snap # crates/pgls_analyser/tests/snapshots/postgres_regress__typecheck_18.snap
…the regression suite From each statement of the splitter, the check looks for the statement psql sends: the shortest text up to a `;` that libpg_query's parser reads as one statement. Where that isn't one statement of the splitter, the splitter cut it in pieces or ran past its end. The mismatches are snapshotted per version.
…d set operations like the parser - `CREATE SCHEMA s CREATE TABLE ...` keeps its schema elements. - `WITH` starts a statement only when CTEs follow: `WITH RECURSIVE`, `WITH name (` or `WITH name AS [NOT] MATERIALIZED (`. `COPY ... WITH (...)`, `WITH HOLD`, `WITH TIES`, `WITH NO DATA`, `WITH (FORCE)` and `IS JSON WITH UNIQUE KEYS` no longer start one. - A missing `(` no longer runs to the next `)`, which swallowed the rest of a file after such a cut. - The `SEARCH` and `CYCLE` clauses of a CTE are skipped. - `SELECT` after `INTERSECT` or `DISTINCT`, `ON COMMIT DELETE ROWS`, `SHARE UPDATE EXCLUSIVE` and `PREPARE p AS INSERT/UPDATE/DELETE` stay in the statement. - CTEs after `AS`, `FOR` and in rule actions, and the query of an `INSERT INTO` there, belong to the statement.
…plits like the parser The statements that were cut in pieces now run as one, and the text that a `WITH` swallowed up to a later `)` is split again: guc, copy2, limit, matview, portals, rules and rowsecurity were mostly hidden behind one statement. Files whose split didn't change keep their verdicts.
…of the parser's statement A cut at a `;` inside a statement, e.g. in a `BEGIN ATOMIC` body, leaves a fragment that ends with a `;` and doesn't parse. The `;` of the statements after it are tried too. The parser's statement has to start where the splitter's does, so a lone `;` doesn't pass for the start of the next statement.
…tches # Conflicts: # crates/pgls_postgres_regress/tests/snapshots/splitter_vs_parser__splitter_vs_parser_15.snap # crates/pgls_postgres_regress/tests/snapshots/splitter_vs_parser__splitter_vs_parser_16.snap # crates/pgls_postgres_regress/tests/snapshots/splitter_vs_parser__splitter_vs_parser_17.snap # crates/pgls_postgres_regress/tests/snapshots/splitter_vs_parser__splitter_vs_parser_18.snap
The WITH that starts no CTE, the missing `(` and the SEARCH and CYCLE clauses of a CTE move to their own PRs: they expose bugs of the pretty printer and a false positive of the analyser, which get fixed first. CTEs after AS, FOR and in rule actions stay, with the list of WITH clauses from before. They stop at SEARCH and CYCLE, like a top-level WITH.
Starting from the verdicts of main, only the files whose statement starts change are re-recorded. The typecheck, spans and splitter vs parser snapshots follow.
The SEARCH and CYCLE clauses after a CTE are skipped, up to the column after SET or USING. Without the end, e.g. while typing, the clause ends at the next `;`. CTEs in another statement, e.g. after CREATE VIEW v AS, no longer stop at them either.
psteinroe
added a commit
that referenced
this pull request
Nov 16, 2026
…d set operations like the parser (#1184) Stacked on #1183, which checks the splitter's boundaries against the parser on the regression suite. This fixes what it found: the mismatches go from 144–150 per version to 1. ### Fixes All in `pgls_statement_splitter`. Each follows what psql sends as one statement (up to the `;`) and Postgres' grammar. - **`CREATE SCHEMA s CREATE TABLE ...`**: `CREATE` no longer starts a statement inside `CREATE SCHEMA`, whose schema elements are `CREATE TABLE/VIEW/INDEX/SEQUENCE/TRIGGER` ([gram.y](https://github.com/postgres/postgres/blob/REL_18_6/src/backend/parser/gram.y#L1517)). - **`WITH` that starts no CTE**: `WITH` starts a statement only when CTEs follow, i.e. `WITH RECURSIVE`, `WITH name (` or `WITH name AS [NOT] MATERIALIZED (` ([gram.y](https://github.com/postgres/postgres/blob/REL_18_6/src/backend/parser/gram.y#L13040)). `COPY ... WITH (...)`, `COPY ... WITH DELIMITER AS '|'`, `DECLARE ... WITH HOLD`, `FETCH ... WITH TIES`, `DROP DATABASE ... WITH (FORCE)`, `REFRESH ... WITH NO DATA`, `IS JSON WITH UNIQUE KEYS` and `CREATE CAST ... WITH INOUT AS IMPLICIT` stay in their statement. This replaces the list of exceptions (`WITH ORDINALITY`, `WITH CHECK`, `WITH TIME ZONE`, ...). At the end of the text, a half-written `WITH`, `WITH name` or `WITH name AS` still starts a statement. - **No `(` to match**: the skip of a parenthesized part (a CTE's body) ran to the next `)` when the `(` was missing, which could be hundreds of statements later. It now reports the missing `(` and stops. - **CTE in another statement**: CTEs after `AS` (`CREATE VIEW v AS`, `PREPARE p AS`), `FOR` (`DECLARE c CURSOR FOR`) and `DO`/`INSTEAD`/`ALSO` (rule actions) belong to that statement, and so does the statement after them. - **`SEARCH` and `CYCLE`** after a CTE are skipped, up to the column after `SET` or `USING` ([gram.y](https://github.com/postgres/postgres/blob/REL_18_6/src/backend/parser/gram.y#L13061)). Without the end, e.g. while typing, the clause ends at the next `;`. - **`SELECT` after `INTERSECT` and `DISTINCT`** (`UNION DISTINCT SELECT`) stays in the statement ([gram.y](https://github.com/postgres/postgres/blob/REL_18_6/src/backend/parser/gram.y#L12935)), like after `UNION` and `EXCEPT`. - **`DELETE ROWS`** (`ON COMMIT DELETE ROWS`, [gram.y](https://github.com/postgres/postgres/blob/REL_18_6/src/backend/parser/gram.y#L4674)) starts no statement, like `delete(` before. - **`INSERT`/`UPDATE`/`DELETE` after `AS`** (`PREPARE p AS UPDATE ...`, [gram.y](https://github.com/postgres/postgres/blob/REL_18_6/src/backend/parser/gram.y#L12107)) and **`UPDATE` after `SHARE`** (`LOCK t IN SHARE UPDATE EXCLUSIVE MODE`, [gram.y](https://github.com/postgres/postgres/blob/REL_18_6/src/backend/parser/gram.y#L12469)) stay in the statement. - **`INSERT INTO ... SELECT` in another statement** (a rule action, `PREPARE`): after a nested `INSERT INTO`, `SELECT` and `WITH` no longer start a statement, like in a top-level `INSERT`. **Cost:** no extra pass and no recursion. A `WITH` looks at most three tokens ahead instead of one. The CTE and `INSERT INTO` cases are two booleans in the loop of a statement. `SEARCH`/`CYCLE` move one token per iteration and stop at the next `;`. ### Tests - Splitter unit tests: `create_schema_with_elements`, `with_that_starts_no_cte`, `cte_search_and_cycle`, `intersect_and_distinct_before_select`, `on_commit_delete_rows`, `lock_share_update_exclusive`, `queries_of_prepare_declare_and_rules`, and `half_written_with` for half-written SQL (a missing `;` before a CTE, `COPY ... WITH` at the end of a line, a CTE without `(`). - The mismatch list of #1183 shrinks to the one case below. ### Fixtures The verdicts were recorded with the splitter's boundaries, so the files it now splits differently are re-recorded with `just record-regress <major>`, one version at a time: 31–35 verdict files per version. Only files whose statement starts changed were kept; the rest, and the catalogs, are unchanged. The text that a `WITH` swallowed is now split and run, so the corpus grows by 2100–2300 statements per version, e.g. `rowsecurity` from 111 to 1223 statements and `rules` from 71 to 621. The `postgres_regress__typecheck_*` snapshots and the spans snapshot of the regression suite follow from the new verdicts; they're updated from CI in a follow-up commit. ### Not covered - `CREATE SCHEMA` with blank lines between its elements (`namespace.sql:9`) is still cut at the blank lines. Each element parses on its own, and that's what the blank-line recovery takes for a missing `;` (`SELECT a FROM t` + blank line + `BEGIN;`). It stays in the mismatch list. - A half-written `COPY t FROM stdin WITH DELIMITER AS` at the end of the text looks like a CTE and starts a statement until the string is typed. **Update:** #1184 now keeps only the fixes that are green on their own: `CREATE SCHEMA` elements, `SELECT` after `INTERSECT`/`DISTINCT`, `ON COMMIT DELETE ROWS`, `LOCK ... SHARE UPDATE EXCLUSIVE`, `INSERT`/`UPDATE`/`DELETE` after `AS`, `INSERT INTO ... SELECT` nested in a rule action or `PREPARE`, and CTEs nested after `AS`/`FOR`/`DO`/`INSTEAD`/`ALSO`. The others move to two draft PRs stacked on this one, because the statements they now split like the parser expose bugs that get fixed first, in their own PRs on main: - `SEARCH`/`CYCLE` → #1191, blocked on the printer's round-trip of those clauses (#1187). - `WITH` that starts no CTE, with the fix for a missing `(` → #1192, blocked on the printer's round-trips of `IS JSON`, `ALTER TABLE ... RENAME RULE` and `MERGE` actions (#1186, #1188, #1189) and on the analyser's false positive on `matview.sql:261` (#1190). The nested CTEs worked without the new `WITH` check, so they stay here: a `WITH` after `AS`, `FOR`, `DO`, `INSTEAD` or `ALSO` that isn't in the old list of `WITH` clauses starts the CTEs of the statement's query. They stop at `SEARCH`/`CYCLE` after a CTE, like a top-level `WITH` did, so `CREATE VIEW v AS WITH ... SEARCH ... SELECT` is still cut before the `SELECT` until #1191. `CREATE` keeps its loop over `WITH (...)` options. Fixtures and snapshots now: 22–24 verdict files per version are re-recorded from main's verdicts (`alter_table`, `collate*`, `create_index`, `create_schema`, `create_view`, `event_trigger`, `foreign_key`, `guc`, `insert`, `lock`, `maintain_every`, `namespace`, `plancache`, `publication`, `stats_ext`, `temp`, `transactions`, `union`, `updatable_views`, `with`, plus `prepare` from 16 and `partition_prune` in 18); the catalogs are unchanged. Cuts that are merged back shrink the corpus by about 150 statements per version (39966 → 39817 for 15, 47288 → 47134 for 18). The mismatch list goes from 144–150 to 48–49 per version; the entries left are the ones #1191 and #1192 fix, and `namespace.sql:9`. The `postgres_regress__typecheck_*` snapshots (1622–1914 findings on rejected statements, none on accepted ones) and the spans snapshot are in this PR now, so the sentence above about updating them from CI in a follow-up commit is outdated.
psteinroe
marked this pull request as ready for review
November 16, 2026 14:59
# Conflicts: # crates/pgls_analyser/tests/snapshots/postgres_regress__typecheck_15.snap # crates/pgls_analyser/tests/snapshots/postgres_regress__typecheck_16.snap # crates/pgls_analyser/tests/snapshots/postgres_regress__typecheck_17.snap # crates/pgls_analyser/tests/snapshots/postgres_regress__typecheck_18.snap
psteinroe
added a commit
that referenced
this pull request
Nov 16, 2026
Stacked on #1191. **Blocked on:** - #1186, #1188 and #1189: pretty printer round-trips of `IS JSON`, `ALTER TABLE ... RENAME RULE` and `MERGE` actions; - #1190: the analyser's `unknownRelation` on `matview.sql:261`, a table created by a PL/pgSQL function that the file defines and calls. The statements this PR now splits like the parser hit those bugs. ### Fixes All in `pgls_statement_splitter`. - **`WITH` that starts no CTE**: `WITH` starts a statement only when CTEs follow, i.e. `WITH RECURSIVE`, `WITH name (` or `WITH name AS [NOT] MATERIALIZED (` ([gram.y](https://github.com/postgres/postgres/blob/REL_18_6/src/backend/parser/gram.y#L13040)). `COPY ... WITH (...)`, `COPY ... WITH DELIMITER AS '|'`, `DECLARE ... WITH HOLD`, `FETCH ... WITH TIES`, `DROP DATABASE ... WITH (FORCE)`, `REFRESH ... WITH NO DATA`, `IS JSON WITH UNIQUE KEYS` and `CREATE CAST ... WITH INOUT AS IMPLICIT` stay in their statement. This replaces the list of exceptions (`WITH ORDINALITY`, `WITH CHECK`, `WITH TIME ZONE`, ...) and the `WITH` loop of `CREATE`. At the end of the text, a half-written `WITH`, `WITH name` or `WITH name AS` still starts a statement. - **No `(` to match**: the skip of a parenthesized part (a CTE's body) ran to the next `)` when the `(` was missing, which could be hundreds of statements later. It now reports the missing `(` and stops. **Cost:** a `WITH` looks at most three tokens ahead instead of one. No extra pass and no recursion. ### Tests - Splitter unit tests `with_that_starts_no_cte` and `half_written_with` (a missing `;` before a CTE, `COPY ... WITH` at the end of a line, a CTE without `(`). - The mismatch list of #1183 shrinks to the one case below. ### Fixtures The files whose statement starts change are re-recorded with `just record-regress <major>`, one version at a time: 10–12 verdict files per version (`copy`, `copy2`, `drop_if_exists`, `guc`, `limit`, `matview`, `portals`, `rowsecurity`, `rules`, `xmlmap`, plus `sqljson` from 16 and `copyencoding` in 18). The text that a `WITH` swallowed is now split and run, so the corpus grows by about 2300–2500 statements per version, e.g. `rowsecurity` from 111 to 1223 statements and `rules` from 71 to 621. The `postgres_regress__typecheck_*`, spans and splitter vs parser snapshots follow. ### Expected CI Until the PRs above merge: the printer's `postgres_regress` round-trips, and the analyser's `postgres_regress` on `matview.sql:261` (one finding per version). ### Not covered - `CREATE SCHEMA` with blank lines between its elements (`namespace.sql:9`) is still cut at the blank lines; it stays in the mismatch list. - A half-written `COPY t FROM stdin WITH DELIMITER AS` at the end of the text looks like a CTE and starts a statement until the string is typed. **Update:** #1186, #1188, #1189 and #1190 are merged, so this is no longer blocked. #1191 and main are merged in (9eb2c65). The `postgres_regress__typecheck_*` snapshots are regenerated on top of #1190: `matview.sql:261` has no finding any more, and 33–43 findings per version are gone, all on statements Postgres rejected, after calls to PL/pgSQL functions of the file (`incremental_sort.sql`, `memoize.sql`, `explain.sql`, …), like the ones #1190 removed on main. CI is green, including the printer's round-trips and `postgres_regress`.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1184. Blocked on #1187, which makes the pretty printer round-trip the
SEARCHandCYCLEclauses of a CTE. Until that merges, the printer'spostgres_regresstest fails on the statements this PR now splits like the parser.Fix
In
pgls_statement_splitter, theSEARCHandCYCLEclauses after a CTE are skipped, up to the column afterSETorUSING(gram.y). Without the end, e.g. while typing, the clause ends at the next;. Before, the query after them started a new statement:CTEs inside another statement (
CREATE VIEW v AS WITH ...) no longer stop at these clauses either; #1184 left that out on purpose.Cost: one token per iteration, up to the end of the clause or the next
;. No recursion.Tests
cte_search_and_cycle.with.sqlentries: 10–11 left per version, all fixed by fix(splitter): start a statement at WITH only when a CTE follows #1192 exceptCREATE SCHEMAwith blank lines.Fixtures
with.sqlis re-recorded withjust record-regress <major>for every version; nothing else changes. Thepostgres_regress__typecheck_*, spans and splitter vs parser snapshots follow, with no finding on a statement Postgres accepted.Expected CI
Only the printer's
postgres_regresstest fails, on 23 statements ofwith.sqlper version, until #1187 merges.Update: #1183, #1184 and #1187 are merged. Main is merged in (814909b); the spans snapshot change moved with the snapshot to
pgls_resolve. CI is green, including the printer'spostgres_regresstest, so this is no longer blocked.