Skip to content

feat: add block_on_with_check_interrupts - #670

Open
mkaruza wants to merge 1 commit into
mainfrom
mkaruza/block-on-with-timeout
Open

mkaruza wants to merge 1 commit into
mainfrom
mkaruza/block-on-with-timeout

Conversation

@mkaruza

@mkaruza mkaruza commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Add block_on_with_check_interrupts() that checks Postgres interrupts
after each timeout interval.

The new function polls the future in with 250ms timeout. After each timeout
it calls check_for_interrupts!(), outside the runtime, to check if an
interrupt is pending, Postgres raises its usual error and the future is dropped
during unwinding; otherwise it keeps waiting.

It has no deadline of its own and only interrupts futures that yield;
blocking code inside the future is not interrupted.

Switch iceberg_fdw to the new function for all remote calls.

@mkaruza
mkaruza requested a review from a team as a code owner October 22, 2026 13:34
@mkaruza
mkaruza force-pushed the mkaruza/block-on-with-timeout branch from 8f72bc4 to 47b0580 Compare October 22, 2026 13:58
@imor

imor commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

@mkaruza the problem of non-responsive backends running wrapper queries needs to be solved at an architectural level. For example by making the tokio runtime wait on the process latch (in addition to I/O) or to offload all I/O to a separate thread (which ultimately will still mean the main thread of the process waiting on the proc latch). A user should be able to cancel a query by calling pg_cancel_backend(<pid>) or terminate a backend by calling pg_terminate_backend(<pid>) at any time and it should response reasonably quickly. This PR introduces a a fixed timeout for each blocking operation, and although that does solve the blocking problem to some extent it introduces another: what if a user wants to run a query which takes longer than 5 minutes? We could make the timeout configurable but that has its own tradeoffs. First, it is different from statement_timeout which is the standard way a Postgres query timeout is applied. Second, there's no correct timeout value: too short a timeout means users can't run queries longer than that and too long a timeout means it'll stay stuck for that long timeout without responding to pg_terminate_backend or similar functions.

For the above reasons I don't think we should merge this PR and instead work on the architectural level issues plaguing the wrappers framework with regards to blocking I/O.

@mkaruza
mkaruza force-pushed the mkaruza/block-on-with-timeout branch from 47b0580 to c17f5d7 Compare October 26, 2026 10:15
@mkaruza

mkaruza commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor Author

@imor you are right, previous implementation was limited and gave each fixe timeout, which was not complete

I've replaced it with block_on_with_check_interrupts which polls the future in fixed 250ms intervals (we can lower that if needed). After each interval it checks for pending postgres interrupts, and if there are any, Postgres raises its usual error. If there are none, it keeps polling the same future.

I think this is a much simpler way to solve the problem.

We would need change wrapper wide block_on calls with this approach function but we can do it in another PR.

Add `block_on_with_check_interrupts()` that checks Postgres interrupts
after each timeout interval.

The new function polls the future in with 250ms timeout. After each timeout
it calls `check_for_interrupts!()`, outside the runtime, to check if an
interrupt is pending, Postgres raises its usual error and the future is dropped
during unwinding; otherwise it keeps waiting.

It has no deadline of its own and only interrupts futures that yield;
blocking code inside the future is not interrupted.

Switch iceberg_fdw to the new function for all remote calls.
@mkaruza
mkaruza force-pushed the mkaruza/block-on-with-timeout branch from c17f5d7 to 5a32f15 Compare October 26, 2026 10:27
@mkaruza mkaruza changed the title feat: add block_on_with_timeout feat: add block_on_with_check_interrupts Oct 27, 2026
@mkaruza
mkaruza requested a review from imor October 28, 2026 07:59
@imor

imor commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

Hey @mkaruza, what I meant by fixing this at the architectural level is that we should fix this in a fundamental way such that we drastically reduce the chances of backends getting stuck. This means not only in iceberg but all wrappers. This also meant to write tests which would reproduce the backends getting stuck and then fixing it should pass those tests. I've attempted one approach with AI's help here: https://github.com/supabase/wrappers/pull/673/changes to give an idea about the scope of work this fix needs. The PR is not ready to merge and it's not necessarily the final approach but it acts as a catalyst to think more deeply through the structural issues in the code. Please go through it and let me know what you think.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants