Repository navigation
Description
Describe the bug
GitHub Actions jobs that build inside the Intel container
(ghcr.io/deepmodeling/abacus-intel) intermittently abort at the very
first step, "Initialize containers", during docker pull. Docker
retries the pull 3 times and each attempt fails with:
failed to register layer: write /opt/intel/oneapi/mkl/2025.2/lib/libmkl_sycl_blas.so.5:
no space left on device
The job exits before compiling anything. Re-running the same commit on
a different hosted runner usually passes, so this is a runner
disk-capacity lottery: the image only fits on runners that happen to
have enough free disk. Affected checks (any job using the Intel image):
"Build with Intel toolchain", "Build with Makefile & Intel compilers",
"Build extra components with Intel toolchain".
Expected behavior
The container pull and the CI build should succeed reliably on any
standard hosted runner for an unchanged, otherwise-green commit — the
same code passes these checks on other runners and passes every other
check (GNU/CUDA builds, Test) in the same run.
To Reproduce
No source change is needed — the failure is environmental:
- Open or update any PR (or re-run a completed run) so that the
Intel-container checks are scheduled, e.g. any commit of refactor(dfpt): replace the shared fixture base class with a free-function test tool #8270 / refactor(dfpt): replace the stru fixture base class with a free-function test tool #8283. - With roughly 50% probability during 2026-11-04 (6 failures across 3
PRs in ~3 hours; first seen on refactor(dfpt): split the oversized DFPT serial tests along the source boundaries #8259 on 11-02), the job dies at
"Initialize containers" with the error above. - Re-running the failed jobs (write access required) or re-pushing the
same commit lands on a different runner and typically passes.
Example failing runs (all at "Initialize containers", identical error
line):
- https://github.com/deepmodeling/abacus-develop/actions/runs/41456220294/job/125313452810 (refactor(dfpt): split the oversized DFPT serial tests along the source boundaries #8259, 11-02 13:48 UTC)
- https://github.com/deepmodeling/abacus-develop/actions/runs/41770195925/job/126377837739 (refactor(dfpt): replace the shared fixture base class with a free-function test tool #8270, 11-04 03:57 UTC)
- https://github.com/deepmodeling/abacus-develop/actions/runs/41786476186/job/126429662988 (refactor(dfpt): replace the stru fixture base class with a free-function test tool #8283, 11-04 06:44 UTC)
Environment
- Host: GitHub Actions hosted runners (the workflow's Intel-container
jobs; not a local environment) - Container: ghcr.io/deepmodeling/abacus-intel (pulled fresh each run)
- Failure point: docker layer registration while writing
/opt/intel/oneapi/mkl/2025.2/lib/libmkl_sycl_blas.so.5 - Not related to any compiler flags or ABACUS code: zero compilation
has run when the job fails
Additional Context
Impact:
- Blocks otherwise-green PRs; the red X looks like a code failure but
nothing was compiled. - Fork contributors cannot use "Re-run failed jobs" (needs write
access), so the only workaround is pushing an empty commit and hoping
the next runner has enough free disk.
Cluster observed on 2026-11-04 (UTC):
| Time | PR | Check |
|---|---|---|
| 03:57 | #8270 | Build with Makefile & Intel compilers |
| 04:22 | #8270 | Build with Intel toolchain (and Makefile one) |
| 04:47 | #8283 | Build extra components with Intel toolchain |
| 06:44 | #8283 | Build extra components with Intel toolchain |
| (first occurrence: #8259 on 11-02 13:48) |
Task list for Issue attackers (only for developers)
- Verify the issue is not a duplicate.
- Describe the bug.
- Steps to reproduce.
- Expected behavior.
- Error message.
- Environment details.
- Additional context.
- Assign a priority level (low, medium, high, urgent).
- Assign the issue to a team member.
- Label the issue with relevant tags.
- Identify possible related issues.
- Create a unit test or automated test to reproduce the bug (if applicable).
- Fix the bug.
- Test the fix.
- Update documentation (if necessary).
- Close the issue and inform the reporter (if applicable).
Activity
ABACUS-PR-BOT commentedon Nov 4, 2026
The supplied error indicates storage exhaustion while Docker imports the Intel image, before ABACUS compilation. At
bcf2fe79, the Intel CMake checks and Makefile check use job-level containers. A cleanup step inside these jobs runs after container initialization, so it cannot prevent this pull failure.The practical directions are a smaller published Intel image, sufficient runner storage, or host-side job orchestration that prepares storage before explicitly pulling and starting the container. Deleting inherited oneAPI files in a later Docker layer only hides them; the original layer bytes remain. An image change must reduce the actual final layer footprint while preserving the compilers and dependencies needed by all three Intel checks. The separate
X64image builder does not establish the storage budget of theubuntu-latestconsumers.A bounded cold-pull/startup check can establish the cause without a full ABACUS build: record the resolved image digest, runner image/version, Docker storage backend, and free bytes and inodes on the relevant storage filesystems through extraction. Leave headroom for subsequent builds. The workflows use mutable
latest, so the same source commit does not guarantee the same image across attempts.Related #7851 concerns PEXSI/image-tag contents; it does not establish a fix for this startup failure.
This diagnosis uses the supplied report, pinned workflow/Dockerfile source and public runner/Docker documentation. The linked failed-job diagnostics, exact storage limit, image footprint and reported failure rate remain unverified; no image pull or build was run.