[Licenser]

Problem and result

kq_cpu_bench had two problems with expert formats the engine uses:

• I-quant pairs crashed. make_blob quantized its random test rows with ggml-cpu's type traits' from_float,
  which the i-quants do not have, so an IQ3_S pair called a null pointer: an access violation on Windows, SIGSEGV
  (exit 139) on Linux, before the first timing.
• Q2_0 was timed on a path the engine does not take. A Q2_0 gate/up or down was timed with ggml-cpu's Q2_0 dot.
  The CPU pool runs Q2_0 experts on Strata's own rows instead (pool.cpp, drain modes 5 and 6), so the bench
  reported a Q2_0 cost about 2.7x higher than the pool's.

After this change all pairs run, and a Q2_0 matrix is timed the way the pool runs it. Threadripper 3960X, one
thread, --threads 1 --nt 1 --reps 3:

pair	upstream main (7ddc17a)	this branch
iq3_s/q2_0	SIGSEGV (exit 139)	0.573 ms per expert per thread
q2_0/q2_0	0.742 ms (ggml-cpu's Q2_0 dot)	0.272 ms (the pool's Q2_0 rows)

Related discussion

None. The change only touches the bench tool (src/kernels/cpu/kq_cpu_bench.cpp); the engine is unchanged.

Approach

• make_blob quantizes with ggml_quantize_chunk, with a flat importance matrix where the type requires one
  (ggml_quantize_requires_imatrix). The blobs keep valid scales and the real sizes, as before.
• A Q2_0 gate/up or down is timed as the pool runs it: the activations as an ActQ (act_quant_any), Strata's Q2_0
  rows (q2_rows_any), SwiGLU in between. The pair's header line says when that path is taken
  ("Q2_0 down on Strata's rows (the pool's path)"). Other formats keep ggml-cpu's dot, as before.

Validation

Passed:

• This branch on v0.2.3.1 (7ddc17a), CPU-only build (-DCMAKE_BUILD_TYPE=Release), Linux, GCC 15.2, Threadripper
  3960X: kq_cpu_bench --pairs iq3_s/q2_0,iq2_xs/q2_0,q2_0/q2_0 --threads 1 --nt 1 --reps 3 runs all three pairs,
  exit 0 (0.573, 0.349 and 0.294 ms per expert per thread). --pairs q2_0/q2_0 alone: 0.272 ms.
• The same build of upstream main 7ddc17a on the same machine: --pairs iq3_s/q2_0 ends with SIGSEGV (exit 139);
  --pairs q2_0/q2_0: 0.742 ms per expert per thread.
• An earlier base of the same change (477ac13, CUDA sm_75 build): --pairs iq3_s/iq4_nl,iq2_xs/q2_0 ran, 0.661 and
  0.391 ms per expert per thread.

Failed:

• None observed.

Not run:

• Windows / MSVC, where the crash was first seen as an access violation: not built here.
• HIP, SYCL and Metal builds (the tool is CPU-only code).

Limits

• The test experts are random weights quantized with a flat importance matrix: right for timing, not for accuracy.
• The Q2_0 timing follows pool.cpp's current drain modes 5 and 6; if the pool's Q2_0 path changes, the bench has to
  follow it.

Used Claude.