Repository navigation
[Benchmark] RTX 3090 + i9-10900K, Flash-Next IQ3_S at 512K context (engines 0.1.44.1 / 0.1.48 / 0.1.50) + seven-configuration A/B - #4054
Open
smallmj wants to merge 2 commits into
Conversation
…engine 0.1.48), with a seven-configuration A/B on engine 0.1.44.1
|
Thanks for the pull request. To keep the tracker readable, every item needs an area prefix in the title and the parts of the pull request template filled in. This one is missing:
Edit the pull request (title and description) and this note goes away on its own. If it is still missing parts in 5 days it will be closed, which is not a rejection: fix it and comment Details: CONTRIBUTING.md |
|
Thanks, this follows the format now. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds eight results-only folders under
bench/results/(no engine changes):--kv-grow, spec 4 / spec-min-p 0.70, MTP rt. BENCH suite (tools/strata_bench.py, 3 runs per test): short chat 86.9 tok/s decode (TTFT 0.73 s), 4K prompt 1102 tok/s, 16K prompt 1675 tok/s, agent session cache hit 70.2% / follow-up decode 94.4 tok/s. Its README carries an appendix summarizing the A/B below.--pool-workers 4/6/4 + --pcie-frac 0.20/--pcie-frac 0.00 + STRATA_PF_FUSED=1/--pcie-frac 0.00 + STRATA_LOOKAHEAD=0/ combo. Main finding:STRATA_PF_FUSED=1is the only change beyond noise (+3.6% on 16K prompt, decode unchanged); pool-workers below the default of 9 cost chat/agent decode on this CPU;STRATA_LOOKAHEAD=0neutral.Validation
Passed:
python tools/strata_bench.py(viaBENCH.bat) on Windows 11, engine 0.1.48 / project 0.1.48.1 (commit 7ddc17a), RTX 3090 + i9-10900K: suite completed, 3 runs per test, all requests returned tokens; report atbench/results/2026-10-24-community-rtx-3090-bench-v-base-iq3-s/(README.md,results.json,engine-log-excerpt.txt).results.json.Failed: none observed (no failed or cancelled requests in any suite; all runs returned tokens).
Not run: the standard benchmark
strata bench run(draft protocol in BENCHMARK_PROTOCOL.md); correctness/recall checks (tools/needle_bench.py, HumanEval) — these suites are speed-only; the--clientsconcurrency and--long(32K) arms of the suite.Limits: results are unmodified tool output except an appendix added to the 2026-10-24 README; one suite per A/B arm (within-suite ranges from 3 runs); the two dates are not directly comparable (different engine, suite's short-chat prompts changed between tool versions); background load and power limits were not recorded. Happy to restructure or add a standard
strata bench runif useful.Update 2026-10-25: engine 0.1.50 re-run
The same base configuration and the same suite, re-run the day after the 0.2.5 release
(project v0.2.5, commit 41cf05a, engine 0.1.50) — report at
bench/results/2026-10-25-community-rtx-3090-bench-v-base-iq3-s/. Same machine, samemodel, same flags; the suite's own README has the full tables. Medians against the
2026-10-24 run (engine 0.1.48), same JSON, cross-day comparison so treat with the usual
noise caveats:
Decode medians moved up (chat +14%, agent +30%) but the run ranges overlap, and this
machine has shown ±15% cross-day drift before — the prefill numbers (same to 0.2%) are
the stable reference. No claim beyond "no regression and possibly faster decode".