Skip to content

[Benchmark] RTX 3090 + i9-10900K, Flash-Next IQ3_S at 512K context (engines 0.1.44.1 / 0.1.48 / 0.1.50) + seven-configuration A/B - #4054

Open
smallmj wants to merge 2 commits into
Niko1221:mainfrom
smallmj:community-rtx-3090-bench-v-2026-10-24
Open

smallmj wants to merge 2 commits into
Niko1221:mainfrom
smallmj:community-rtx-3090-bench-v-2026-10-24

Conversation

@smallmj

@smallmj smallmj commented Oct 25, 2026 •

Copy link
Copy Markdown

Summary

Adds eight results-only folders under bench/results/ (no engine changes):

  • 2026-10-24-community-rtx-3090-bench-v-base-iq3-s — single RTX 3090 (24 GiB), i9-10900K, 64 GB RAM, Windows 11; Strata project 0.1.48.1 (commit 7ddc17a), engine 0.1.48, original Flash-Next IQ3_S, 524,288-token context, KV k8v4 with --kv-grow, spec 4 / spec-min-p 0.70, MTP rt. BENCH suite (tools/strata_bench.py, 3 runs per test): short chat 86.9 tok/s decode (TTFT 0.73 s), 4K prompt 1102 tok/s, 16K prompt 1675 tok/s, agent session cache hit 70.2% / follow-up decode 94.4 tok/s. Its README carries an appendix summarizing the A/B below.
  • Seven A/B arms on the same machine from 2026-10-16 (engine 0.1.44.1, commit 296b59d, one suite per arm, sequential, fresh model load between arms): base / --pool-workers 4 / 6 / 4 + --pcie-frac 0.20 / --pcie-frac 0.00 + STRATA_PF_FUSED=1 / --pcie-frac 0.00 + STRATA_LOOKAHEAD=0 / combo. Main finding: STRATA_PF_FUSED=1 is the only change beyond noise (+3.6% on 16K prompt, decode unchanged); pool-workers below the default of 9 cost chat/agent decode on this CPU; STRATA_LOOKAHEAD=0 neutral.

Validation

Passed:

  • python tools/strata_bench.py (via BENCH.bat) on Windows 11, engine 0.1.48 / project 0.1.48.1 (commit 7ddc17a), RTX 3090 + i9-10900K: suite completed, 3 runs per test, all requests returned tokens; report at bench/results/2026-10-24-community-rtx-3090-bench-v-base-iq3-s/ (README.md, results.json, engine-log-excerpt.txt).
  • Same command on engine 0.1.44.1 (commit 296b59d) for the seven A/B arms (2026-10-16); all seven suites completed with per-run data in their results.json.

Failed: none observed (no failed or cancelled requests in any suite; all runs returned tokens).

Not run: the standard benchmark strata bench run (draft protocol in BENCHMARK_PROTOCOL.md); correctness/recall checks (tools/needle_bench.py, HumanEval) — these suites are speed-only; the --clients concurrency and --long (32K) arms of the suite.

Limits: results are unmodified tool output except an appendix added to the 2026-10-24 README; one suite per A/B arm (within-suite ranges from 3 runs); the two dates are not directly comparable (different engine, suite's short-chat prompts changed between tool versions); background load and power limits were not recorded. Happy to restructure or add a standard strata bench run if useful.

Update 2026-10-25: engine 0.1.50 re-run

The same base configuration and the same suite, re-run the day after the 0.2.5 release
(project v0.2.5, commit 41cf05a, engine 0.1.50) — report at
bench/results/2026-10-25-community-rtx-3090-bench-v-base-iq3-s/. Same machine, same
model, same flags; the suite's own README has the full tables. Medians against the
2026-10-24 run (engine 0.1.48), same JSON, cross-day comparison so treat with the usual
noise caveats:

Metric 0.1.48 (10-24) 0.1.50 (10-25)
chat decode tok/s 86.9 (70.9-104.7) 99.0 (67.9-114.3)
chat first token (s) 0.73 0.73
4K prompt tok/s 1102.0 1102.0
16K prompt tok/s 1675.1 (1672.8-1676.1) 1679.0 (1677.3-1681.3)
agent decode tok/s 94.4 (78.7-120.4) 123.0 (98.1-141.4)
agent cache hit % 70.2 69.8

Decode medians moved up (chat +14%, agent +30%) but the run ranges overlap, and this
machine has shown ±15% cross-day drift before — the prefill numbers (same to 0.2%) are
the stable reference. No claim beyond "no regression and possibly faster decode".

…engine 0.1.48), with a seven-configuration A/B on engine 0.1.44.1
@github-actions github-actions Bot added the needs-format Title or template does not follow CONTRIBUTING.md yet label Oct 25, 2026
@github-actions

Copy link
Copy Markdown

Thanks for the pull request. To keep the tracker readable, every item needs an area prefix in the title and the parts of the pull request template filled in. This one is missing:

  • Start the title with its area: [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other], then a short description. Example: "[Engine] fix decode stop on long prompts".
  • Fill in "Validation": the exact commands, the revision and environment tested, what you saw. Keep Passed, Failed and Not run apart.

Edit the pull request (title and description) and this note goes away on its own. If it is still missing parts in 5 days it will be closed, which is not a rejection: fix it and comment /reopen.

Details: CONTRIBUTING.md

@smallmj smallmj changed the title Community benchmark: RTX 3090 + i9-10900K, Flash-Next IQ3_S at 512K context (engine 0.1.48) + seven-configuration A/B on engine 0.1.44.1 [Benchmark] RTX 3090 + i9-10900K, Flash-Next IQ3_S at 512K context (engine 0.1.48) + seven-configuration A/B on engine 0.1.44.1 Oct 25, 2026
@github-actions github-actions Bot removed the needs-format Title or template does not follow CONTRIBUTING.md yet label Oct 25, 2026
@github-actions

Copy link
Copy Markdown

Thanks, this follows the format now.

@smallmj smallmj changed the title [Benchmark] RTX 3090 + i9-10900K, Flash-Next IQ3_S at 512K context (engine 0.1.48) + seven-configuration A/B on engine 0.1.44.1 [Benchmark] RTX 3090 + i9-10900K, Flash-Next IQ3_S at 512K context (engines 0.1.44.1 / 0.1.48 / 0.1.50) + seven-configuration A/B Oct 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant