Repository navigation
Thread: route a burst of messages in one call - #109
Merged
Merged
Conversation
Messages waiting on the same routing lane are routed together: one fast-tier call returns a JSON array with one route per message, each decided on its own and handled in order as if routed alone. Plain questions and messages with a fixed way stay out; with fewer than two left, or if the call or its reply fails, each is routed alone as before. While Steve is still answering, the burst is told what (pending), and each message still goes ahead or waits by decide::now_while_answering. A message edited after its route was made is routed again. Batch eval (20 synthetic cases, bursts of 5, real Haiku, outputs replayed in CI): 20/20 hits, no question made an item; 1.7 s per message instead of 4.7 s, 4 calls instead of 20. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Routing batches the messages that wait on one routing lane, for example a burst in a Feishu group or several messages sent quickly.
gateway::route_many) returns a JSON array with one route per message. Each message is decided on its own and handled in order, through the same pre-routed path as Thread: route messages while Steve is still answering #106.decide::now_while_answering.route_prompt_forwith one text), so the prompt cache does not change.Eval:
batch_eval_real_model, with recorded outputs replayed in CI. 20 synthetic cases in batches of 5, on real Haiku:Tests:
steve-core: 650 passed, including new gateway and burst tests.steve-hub --no-fail-fast: everything passed except the known/tmp→/private/tmpfailure inoutputs_api, which is unrelated and passes in CI.🤖 Generated with Claude Code