Skip to content

Thread: route a burst of messages in one call - #109

Merged
LPX-E5BD8 merged 1 commit into
mainfrom
router-batch
Nov 9, 2026
Merged

LPX-E5BD8 merged 1 commit into
mainfrom
router-batch

Conversation

@LPX-E5BD8

Copy link
Copy Markdown
Contributor

Routing batches the messages that wait on one routing lane, for example a burst in a Feishu group or several messages sent quickly.

  • One call per burst. A single fast-tier call (gateway::route_many) returns a JSON array with one route per message. Each message is decided on its own and handled in order, through the same pre-routed path as Thread: route messages while Steve is still answering #106.
  • What stays out of a batch. Plain questions, which need no router, and direct messages. With fewer than 2 messages left, or when the call or its reply is unusable, every message is routed alone as before.
  • While Steve is still answering. The batch is told what Steve is answering, and each message still goes through decide::now_while_answering.
  • Edits. A message edited after its route was made is routed again.
  • Single messages are unaffected. For one message the prompt is byte-identical to before (route_prompt_for with one text), so the prompt cache does not change.

Eval: batch_eval_real_model, with recorded outputs replayed in CI. 20 synthetic cases in batches of 5, on real Haiku:

  • 20/20 hits, and no question became an item;
  • 1.7 s per message instead of 4.7 s;
  • 4 calls instead of 20.

Tests:

  • steve-core: 650 passed, including new gateway and burst tests.
  • steve-hub --no-fail-fast: everything passed except the known /tmp → /private/tmp failure in outputs_api, which is unrelated and passes in CI.
  • clippy: clean.
  • lint and the strict boundary check: passed.

🤖 Generated with Claude Code

Messages waiting on the same routing lane are routed together: one
fast-tier call returns a JSON array with one route per message, each
decided on its own and handled in order as if routed alone. Plain
questions and messages with a fixed way stay out; with fewer than two
left, or if the call or its reply fails, each is routed alone as before.
While Steve is still answering, the burst is told what (pending), and
each message still goes ahead or waits by decide::now_while_answering.
A message edited after its route was made is routed again.

Batch eval (20 synthetic cases, bursts of 5, real Haiku, outputs
replayed in CI): 20/20 hits, no question made an item; 1.7 s per
message instead of 4.7 s, 4 calls instead of 20.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@LPX-E5BD8
LPX-E5BD8 merged commit 1dec8b8 into main Nov 9, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant