Skip to content

CI: Intel-container jobs intermittently fail at "Initialize containers" with "no space left on device" during the image pull #8286

Open

Description

@Zanthoxylum

Describe the bug

GitHub Actions jobs that build inside the Intel container
(ghcr.io/deepmodeling/abacus-intel) intermittently abort at the very
first step, "Initialize containers", during docker pull. Docker
retries the pull 3 times and each attempt fails with:

failed to register layer: write /opt/intel/oneapi/mkl/2025.2/lib/libmkl_sycl_blas.so.5:
no space left on device

The job exits before compiling anything. Re-running the same commit on
a different hosted runner usually passes, so this is a runner
disk-capacity lottery: the image only fits on runners that happen to
have enough free disk. Affected checks (any job using the Intel image):
"Build with Intel toolchain", "Build with Makefile & Intel compilers",
"Build extra components with Intel toolchain".

Expected behavior

The container pull and the CI build should succeed reliably on any
standard hosted runner for an unchanged, otherwise-green commit — the
same code passes these checks on other runners and passes every other
check (GNU/CUDA builds, Test) in the same run.

To Reproduce

No source change is needed — the failure is environmental:

  1. Open or update any PR (or re-run a completed run) so that the
    Intel-container checks are scheduled, e.g. any commit of refactor(dfpt): replace the shared fixture base class with a free-function test tool #8270 / refactor(dfpt): replace the stru fixture base class with a free-function test tool #8283.
  2. With roughly 50% probability during 2026-11-04 (6 failures across 3
    PRs in ~3 hours; first seen on refactor(dfpt): split the oversized DFPT serial tests along the source boundaries #8259 on 11-02), the job dies at
    "Initialize containers" with the error above.
  3. Re-running the failed jobs (write access required) or re-pushing the
    same commit lands on a different runner and typically passes.

Example failing runs (all at "Initialize containers", identical error
line):

Environment

  • Host: GitHub Actions hosted runners (the workflow's Intel-container
    jobs; not a local environment)
  • Container: ghcr.io/deepmodeling/abacus-intel (pulled fresh each run)
  • Failure point: docker layer registration while writing
    /opt/intel/oneapi/mkl/2025.2/lib/libmkl_sycl_blas.so.5
  • Not related to any compiler flags or ABACUS code: zero compilation
    has run when the job fails

Additional Context

Impact:

  • Blocks otherwise-green PRs; the red X looks like a code failure but
    nothing was compiled.
  • Fork contributors cannot use "Re-run failed jobs" (needs write
    access), so the only workaround is pushing an empty commit and hoping
    the next runner has enough free disk.

Cluster observed on 2026-11-04 (UTC):

Time PR Check
03:57 #8270 Build with Makefile & Intel compilers
04:22 #8270 Build with Intel toolchain (and Makefile one)
04:47 #8283 Build extra components with Intel toolchain
06:44 #8283 Build extra components with Intel toolchain
(first occurrence: #8259 on 11-02 13:48)

Task list for Issue attackers (only for developers)

  • Verify the issue is not a duplicate.
  • Describe the bug.
  • Steps to reproduce.
  • Expected behavior.
  • Error message.
  • Environment details.
  • Additional context.
  • Assign a priority level (low, medium, high, urgent).
  • Assign the issue to a team member.
  • Label the issue with relevant tags.
  • Identify possible related issues.
  • Create a unit test or automated test to reproduce the bug (if applicable).
  • Fix the bug.
  • Test the fix.
  • Update documentation (if necessary).
  • Close the issue and inform the reporter (if applicable).

Activity

  1. added
    BugsBugs that only solvable with sufficient knowledge of DFT
    Tests/ExamplesIssues/PR related to unit tests and integrate tests
    on Nov 4, 2026
  2. ABACUS-PR-BOT commented on Nov 4, 2026

    @ABACUS-PR-BOT
    Collaborator

    The supplied error indicates storage exhaustion while Docker imports the Intel image, before ABACUS compilation. At bcf2fe79, the Intel CMake checks and Makefile check use job-level containers. A cleanup step inside these jobs runs after container initialization, so it cannot prevent this pull failure.

    The practical directions are a smaller published Intel image, sufficient runner storage, or host-side job orchestration that prepares storage before explicitly pulling and starting the container. Deleting inherited oneAPI files in a later Docker layer only hides them; the original layer bytes remain. An image change must reduce the actual final layer footprint while preserving the compilers and dependencies needed by all three Intel checks. The separate X64 image builder does not establish the storage budget of the ubuntu-latest consumers.

    A bounded cold-pull/startup check can establish the cause without a full ABACUS build: record the resolved image digest, runner image/version, Docker storage backend, and free bytes and inodes on the relevant storage filesystems through extraction. Leave headroom for subsequent builds. The workflows use mutable latest, so the same source commit does not guarantee the same image across attempts.

    Related #7851 concerns PEXSI/image-tag contents; it does not establish a fix for this startup failure.

    This diagnosis uses the supplied report, pinned workflow/Dockerfile source and public runner/Docker documentation. The linked failed-job diagnostics, exact storage limit, image footprint and reported failure rate remain unverified; no image pull or build was run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    BugsBugs that only solvable with sufficient knowledge of DFTTests/ExamplesIssues/PR related to unit tests and integrate tests

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions