[Gasoonjia]

Stack from ghstack (oldest at bottom):

• [executorch][cuda] Grow sequence-layout ring layers on demand #24076
• [executorch][muse-glimmer] Bound the CUDA prefill logits selector by the sampler rows #24107
• -> [executorch][muse] Serve Muse Glimmer CUDA batching on a batched-sequence cache #24075
• [executorch][cuda] Lower off-graph KV to the batched-sequence layout #24074
• [executorch][cuda] Add a batched-sequence CUDA KV cache #24073
• [executorch][cuda] Add a length-aware Triton SDPA over per-sequence KV columns #24072
• [executorch][muse] Accept images in CUDA DFlash batching #24066
• [executorch][muse] Serve images on CUDA batching #24065
• [executorch][muse] Export vision with the CUDA batching contract #24064
• [executorch][cuda] Run CudaExecutor steps from prepared embeddings #24063
• [executorch][muse] Share the Muse Glimmer embeddings engine across backends #24062

Adds --kv-layout {cell,sequence} and --max-kv-seqs to the Muse Glimmer CUDA batching exports (solo and DFlash). sequence writes a batched-sequence manifest for target and draft and publishes get_offgraph_kv_max_seqs (and get_draft_offgraph_kv_max_seqs / get_draft_offgraph_kv_seq_capacity for the draft). CudaExecutor and the CUDA DFlash executor pick kBatchedSequence when the artifact declares a sequence count, kBatchedCell otherwise, so the served feature set matches MLX's batched-sequence path. Cell artifacts are unchanged. The layout is chosen at export; switching it at runtime on one .pte is out of scope.

Differential Revision: D125987299

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/24075

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures, 1 Cancelled Job, 1 Pending, 4 Unclassified Failures

As of commit 01a1fde with merge base 2c3e231 ():

NEW FAILURES - The following jobs have failed:

• pull / unittest / macos / macos-job (gh)
  examples/models/muse-glimmer/tests/test_cuda_sequence_export_inputs.py::SequenceExportInputsTest::test_small_logits_example_preserves_token_and_embedding_capacity
• pull / unittest-editable / macos / macos-job (gh)
  examples/models/muse-glimmer/tests/test_cuda_sequence_export_inputs.py::SequenceExportInputsTest::test_small_logits_example_preserves_token_and_embedding_capacity

UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:

• Test CUDA Builds / test-muse-glimmer-cuda-e2e-kquant-17gb-solo-text-batching / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Test CUDA Builds / unittest-cuda-quantized-gemm (int4) / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  ##[error]received SIGINT
• Test ROCm AOTI / test-voxtral-realtime-rocm-gfx950-rocm7.2 / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command docker exec -t a8f0d11d4a87b3ad5439707e52a0dfe9e395981b0d16810d49da4b409baee049 /exec failed with exit code 137
• Test ROCm AOTI / unittest-rocm-gfx950-rocm7.2 / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command docker exec -t eb54346e8bc20d7bf9c83372804118cb9c7dec50bb7f8d264acba7c59f031898 /exec failed with exit code 137

CANCELLED JOB - The following job was cancelled. Please retry:

• pull / test-qnn-python-imports-linux / linux-job (gh)
  curl: (22) The requested URL returned error:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.