[Engine] V100 (sm_70) falls off bus — Xid 79 + PCIe CmpltTO during long-context prefill after 0.1.44.1 -> 0.1.47 #4085

Summary

After upgrading the engine from a self-built 0.1.44.1 to the official portable 0.1.47 release package, a Tesla V100-SXM2-16GB (sm_70) drops off the bus under long-context prefill (~81K tokens) with NVIDIA Xid 79 + 154 and a PCIe Completion Timeout ([14] CmpltTO) on the socket-1 root port. It happened twice on 0.1.47 (≈8.5 h and ≈33.9 h after the upgrade). The same hardware ran 0.1.44.1 stably for weeks; after rolling back to 0.1.44.1 the crash has not recurred (9+ h and counting). This looks like a regression specific to V100 / sm_70 introduced between 0.1.44.1 and 0.1.47.

Environment

• GPU: 4× Tesla V100-SXM2-16GB (sm_70). Failing card: GPU2 = 0000:82:00.0, on socket 1 / NUMA node 1, PCIe x8 (board-limited).
• CPU / board: 2× Xeon E5-2696 v4 (44C/88T), X99 mining board, 256 GiB RAM.
• OS / driver: Ubuntu 26.04, kernel 7.0.0-38, NVIDIA driver 580.178.04, CUDA 12.
• Model: Qwen3.8-Flash-Next, huihui v2 abliterated, UD-Q4_K_XL (103.7 GiB, 4-shard).
• Config: --layer-split 9,25,34 --spec 4 --pipeline-windows 2, KV int8, --conversation-cache-mib 32768 --conversation-cache-slots 4, --vram-reserve-mib 1400 --vram-reserve-later-mib 1600.
• Affected build: strata-engine 0.1.47 — official portable package, build_id 76f90e58436d, source branch f-022-rel, multi-arch sm_60/61/70/75/80/86/89 + PTX JIT, ed25519-signed.
• Known-good build: 0.1.44.1 — self-built from source, compiled for sm_70 only (no PTX), static cubin.

What happened (crash signature, identical on both occurrences)

Kernel logs at crash time:

[08:35:39] pcieport 0000:80:01.0: PCIe Bus Error: severity=Uncorrectable (Non-Fatal),
[08:35:39]      [14] CmpltTO                (First)          <- Completion Timeout
[08:35:39] nvidia 0000:82:00.0: AER: can't recover (no error_detected callback)
[08:35:39] pcieport 0000:80:01.0: AER: device recovery failed
[08:35:39] pcieport 0000:80:01.0: AER: Multiple Uncorrectable (Fatal) error message received
[08:35:39] NVRM: Xid (PCI:0000:82:00): 79, GPU has fallen off the bus.
[08:35:39] NVRM: Xid (PCI:0000:82:00): 154, GPU recovery action changed from 0x0 to 0x1
[08:35:39] pcieport 0000:80:01.0: AER: Root Port link has been reset (0)

• Root port: 0000:80:01.0 (device 8086:6f02).
• The GPU simply stops answering PCIe completions; the request times out, escalates to Fatal, the link resets, and the GPU drops off the bus.
• Engine-side symptom: the crash is hit during prefill, reading the prompt (batched). Engine log:

  prefill copy_i32: unknown error          ×3
    (reading the prompt (batched): layer 8 of the prompt chunk from token 45396)
    stage "reading the prompt (batched): layer 8 of the prompt chunk from token 45396" for 60 s
    VRAM free when this request began: 1065 MiB
  

• After the crash the engine process is stuck (unkillable, kill -9 does not free VRAM); only a full host reboot recovers the card.

Regression evidence

Both 0.1.47 crashes are on the same card and same root port as each other, and on the socket-1 / NUMA-node-1 side — which is also the side that, on this board, has a known ~55% ceiling on concurrent device-side DMA versus socket 0 (independent pre-existing bottleneck, not the crash cause but the likely stress point).

Ruled out (so it is not a red herring)

• Not VRAM exhaustion: free VRAM at crash was 1065 MiB (>1 GB, reserve already raised to 1400/1600). The earlier "low-VRAM → driver paging → AER" theory is disproven.
• Not host RAM / ECC: EDAC mc0–mc3 ce_count/ue_count all 0; host RAM 126 GiB free.
• Not auto-update: no cron job pulls/updates the engine; 0.1.47 was installed manually.

Likely technical direction (hypotheses for the maintainer)

1. PCIe transfer path change between 0.1.44.1 and 0.1.47. The repo's source tree carries a branch f-142-pciefrac ("PCIe fractional / sliced transfer") merged into the f-142 line. The failure mode here is exactly a PCIe Completion Timeout — strongly suggestive that a PCIe-transfer-related change regresses on V100/sm_70 (or on this older PCIe-gen board).
2. Multi-arch + PTX build on sm_70. 0.1.47 is a fat binary with PTX JIT, whereas 0.1.44.1 was a clean sm_70-only static build. A PTX-JIT-compiled kernel path on sm_70 (vs. a shipped sm_70 cubin) could behave differently under the long-prefill batched-copy (copy_i32) workload.

Steps to reproduce (on V100 / sm_70)

1. Run Strata 0.1.47 (official portable engine package) on a V100 with a large MoE model that does not fit entirely in VRAM (experts spill to host RAM / NVMe).
2. Send a long prompt (~81K tokens) to exercise the batched prefill copy path.
3. Observe: within hours-to-2-days, GPU on socket 1 drops off the bus with Xid 79 + [14] CmpltTO on its root port.
4. Roll back to a sm_70-only 0.1.44.1 build → crash stops.

Logs / files attached

• Full dmesg excerpt (AER + Xid 79/154) — see above and V100_GPU2二次掉卡_取证_2026-10-25.md.
• Engine log tail showing prefill copy_i32: unknown error at token 45396, 60 s stall, VRAM free ... 1065 MiB.
• nvidia-smi -q ECC report (all zero) and PCIe LnkSta (link recovers to x8 @ 8 GT/s after reboot, error bits clear).

Offer to maintainer

Happy to (a) retest on 0.1.48 / latest to confirm whether it is already fixed, (b) capture a nvml/engine debug trace around the prefill copy, or (c) bisect against the f-142-pciefrac merge if you point me at the right commit range. V100 is not on the officially listed supported GPUs, but it ran 0.1.44.1 rock-solid, so from our side this is a clear regression, not "unsupported hardware".

Submitted on behalf of a production V100 deployment (account: zealotq). Full dmesg / engine-log取证 is in a separate local report; the key crash signature is inline above. Related prior art found while searching: #4007 mentions a pcie_frac startup parameter (the repo's f-142-pciefrac branch) — same PCIe-transfer area I suspect.