feat(ci): linux arm64 images for docker by LexLuthr · Pull Request #13784 · filecoin-project/lotus

LexLuthr · 2026-09-09T14:29:43Z

Related Issues

Fixes #13783

Proposed Changes

Additional Info

Checklist

Before you mark the PR ready for review, please make sure that:

Commits have a clear commit message.
PR title conforms with contribution conventions
Update CHANGELOG.md or signal that this change does not need it per contribution conventions
New features have usage guidelines and / or documentation updates in
- Lotus Documentation
- Discussion Tutorials
Tests exist for new functionality or change in behavior
CI is green

github-actions

Please update the PR title to match https://github.com/filecoin-project/lotus/blob/master/CONTRIBUTING.md#pr-title-conventions

github-actions

Please update the PR title to match https://github.com/filecoin-project/lotus/blob/master/CONTRIBUTING.md#pr-title-conventions

github-actions

Please update the PR title to match https://github.com/filecoin-project/lotus/blob/master/CONTRIBUTING.md#pr-title-conventions

github-actions

Please update the PR title to match https://github.com/filecoin-project/lotus/blob/master/CONTRIBUTING.md#pr-title-conventions

Copilot

🟡 Changes recommended

A critical runtime dependency issue and a moderate failed-run cleanup issue remain unresolved.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds native Linux ARM64 Docker builds and multi-platform manifest publishing, while updating container build dependencies.

Changes:

Adds AMD64 and ARM64 Docker matrix builds.
Publishes combined multi-platform manifests.
Updates Debian, Rust, and Docker compatibility documentation.

File summaries

File	Summary
`Dockerfile`	Updates builder/runtime images; critically, `libudev.so.1` is missing for `libhwloc15`.
`CHANGELOG.md`	Documents Docker base-image and ARM64 changes.
`.github/workflows/docker.yml`	Adds architecture-specific builds and manifest publishing; failed runs can leave temporary tags behind (moderate).

Review details

Suppressed comments (1)

.github/workflows/docker.yml:191

This cleanup step uses the default success() condition, so it is skipped whenever the manifest preflight or imagetools create fails (and the whole publish job is skipped if any docker matrix entry fails). The architecture-specific images have already been pushed at that point, so failed runs leave $BUILD_TAG-amd64/$BUILD_TAG-arm64 tags in Docker Hub indefinitely; add a failure-safe cleanup path (covering both the publish and build failures) for these run-scoped tags.

      - name: Remove temporary architecture tags
        # A cleanup failure must not invalidate a successfully published image.
        continue-on-error: true

Files reviewed: 3/3 changed files
Comments generated: 1
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

PR title now matches the required format.

BigLep · 2026-09-10T15:41:14Z

+          platforms: linux/${{ matrix.platform.arch }}
          push: ${{ env.PUBLISH == 'true' }}
-          tags: ${{ steps.meta.outputs.tags }}
+          tags: filecoin/${{ matrix.image }}:${{ env.BUILD_TAG }}-${{ matrix.platform.arch }}


Per 2026-09-10 verbal, this is the recommended change from agent.

Recommended change. Docker's documented pattern for multi-platform builds across separate runners pushes each arch image by digest instead of by a temporary tag, then assembles the index from the digests. That removes the build-<run_id>-* tags, the "Remove temporary architecture tags" step below, and the need for a Docker Hub token with delete rights. It also has no failure-path cleanup problem, because nothing tagged is ever created except the final tags.

Sketch:

- name: Build image id: build uses: docker/build-push-action@v5 with: context: . platforms: linux/${{ matrix.platform.arch }} labels: ${{ steps.meta.outputs.labels }} outputs: type=image,name=filecoin/${{ matrix.image }},push-by-digest=true,name-canonical=true,push=${{ env.PUBLISH == 'true' }} build-args: | ${{ matrix.network != 'mainnet' && format('GOFLAGS=-tags={0}', matrix.network) || ''}} - if: env.PUBLISH == 'true' run: | mkdir -p "$RUNNER_TEMP/digests" echo -n "${{ steps.build.outputs.digest }}" > "$RUNNER_TEMP/digests/${{ matrix.platform.arch }}" - if: env.PUBLISH == 'true' uses: actions/upload-artifact@v4 with: name: digests-${{ matrix.image }}-${{ matrix.network }}-${{ matrix.platform.arch }} path: ${{ runner.temp }}/digests/*

and in publish, actions/download-artifact with pattern: digests-${{ matrix.image }}-${{ matrix.network }}-* + merge-multiple: true, then

docker buildx imagetools create "${tag_args[@]}" \ "$IMAGE@$(cat digests/amd64)" "$IMAGE@$(cat digests/arm64)"

The existing --dry-run + jq -e preflight works unchanged on digest sources. Reference: https://docs.docker.com/build/ci/github-actions/multi-platform/#distribute-build-across-multiple-runners

Motivation for insisting: the tag-deletion step is currently failing on every publish (see my comment on that step), and the digest approach makes that whole class of problem go away rather than patching it.

BigLep · 2026-09-10T15:41:14Z

+            | sort | unique == ["amd64", "arm64"]
+          ' "$RUNNER_TEMP/manifest.json"
+          docker buildx imagetools create "${tag_args[@]}" "${sources[@]}"
+      - name: Remove temporary architecture tags


I think you're aware of this agent flag.

Let me know if we need to set a new PAT in lotus (although I would want to make sure you have perms to do this yourself).

Only relevant if you keep the temp-tag approach instead of push-by-digest (see my comment on the tags: line above; with digests this whole step is deleted).

This step does not work today. In the one real publish run on this branch (job 102569293673, Publish Docker (lotus / mainnet) in run 34381379467) the Docker Hub DELETE returned 403 and continue-on-error: true swallowed it:

curl: (22) The requested URL returned error: 403 ##[error]Process completed with exit code 22.

All twelve build-34381379467-{network}-{arch} tags are still on Docker Hub for filecoin/lotus and filecoin/lotus-all-in-one. As written, every master push, nightly and release adds another twelve permanent tags per run.

Likely cause: the DOCKERHUB_TOKEN PAT is read/write only; tag deletion needs the delete scope (or a bot user with delete rights on the org repos), so someone with access to the filecoin Hub org would have to rotate the secret. You would also need an if: always()-style cleanup for the failure path, since a failed imagetools create (or a failed build leg) currently leaves the arch tags behind too.

Either way, please do not keep continue-on-error: true silent: let the step fail, or at least echo "::warning::..." so a broken cleanup shows in the run summary.

BigLep · 2026-09-10T15:41:14Z

+
+  publish:
+    name: Publish Docker (${{ matrix.image }} / ${{ matrix.network }})
+    needs: docker


It looks like there's a behaviour change. IIUC, before this PR each network published independently. Now publish needs every one of the twelve docker matrix legs, so a single failure (say 2k/arm64) blocks filecoin/lotus:master too.

If that is intended, fine, but maybe say so in PR description?

AI solution if not the intended path:

if: ${{ !cancelled() && needs.docker.result != 'cancelled' && (github.event.inputs.publish == 'true' || github.event_name != 'pull_request') }}

lets the surviving networks publish, and the existing --dry-run + jq -e preflight already fails cleanly for a network whose arch image is missing.

BigLep · 2026-09-10T15:41:15Z

          username: ${{ vars.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}
-      - name: Build and push if channel is set (channel=${{ steps.channel.outputs.channel }})
+      - name: Build image


AI flag:

Nothing in CI ever runs the image, so runtime library gaps in lotus-base are only discovered by users (this is what the libudev1 question below was about). Cheap smoke test for the PR path:

- name: Build image uses: docker/build-push-action@v5 with: ... load: ${{ env.PUBLISH != 'true' }} - if: env.PUBLISH != 'true' name: Smoke test run: docker run --rm "filecoin/${{ matrix.image }}:${{ env.BUILD_TAG }}-${{ matrix.platform.arch }}" lotus --version

(load and push/digest outputs are mutually exclusive on this action, so on publish runs you would pull by digest first, or just rely on the PR run.) It also catches the case where the prebuilt libfilcrypto picks CPU features the runner has but the target does not; the arm64 prebuilt is selected from the runner's /proc/cpuinfo. Related: has anyone run filecoin/lotus:cf64f7ffd under Docker Desktop on Apple Silicon? That is the use case in #13783 and it is the one place a SIGILL would show up.

Yes. I tested it with Curio devnet using this image to build lotus and miner containers.

We can probably test in foc-devnet as well.

BigLep · 2026-09-10T15:41:15Z

 #####################################
 FROM debian:trixie AS lotus-base
-MAINTAINER Lotus Development Team
+RUN apt-get update && apt-get install -y --no-install-recommends libudev1 \


I believe this can be dropped.

In my agent discussions:
debian:trixie already ships libudev1 in the base image (dpkg -s libudev1 succeeds in a fresh debian:trixie container, and ldd libhwloc.so.15 resolves libudev.so.1 without installing anything).

BigLep · 2026-09-10T15:41:15Z

    CARGO_HOME=/usr/local/cargo \
    PATH=/usr/local/cargo/bin:$PATH \
-    RUST_VERSION=1.86.0
+    RUST_VERSION=1.94.0


Agent flag. I leave to you on whether this is relevant right now.

Is this bump needed? Both arches fetch the prebuilt libfilcrypto (the arm64 build log shows successfully installed prebuilt libfilcrypto from filecoin-ffi v1.36.1), so the Rust toolchain is only used when someone builds with FFI_BUILD_FROM_SOURCE=1. If you keep it, note that the rustup-init download a few lines down is still pinned to rustup 1.25.1 (2022) with hard-coded checksums; a 2026 toolchain installed by a 2022 rustup is untested here. I'd leave RUST_VERSION alone in this PR so the Dockerfile diff is limited to the arm64 change.

1.86.0 is pretty old, IIRC we're at least 1.94.0 across everything now so I'm fine with this as a housekeeping task

Even FFI_BUILD_FROM_SOURCE AFAIK will overwrite it by rust_toolchain file in the ffi repo.

BigLep · 2026-09-10T15:41:15Z

+      BUILD_TAG: &build-tag build-${{ github.run_id }}-${{ matrix.network }}
    steps:
-      - id: channel
+      - &channel


Nit: YAML anchors are supported by Actions now and the runs prove it parses, but this is the first use in the repo. A one-line comment ("shared with the publish job below via YAML anchors") would save the next reader a search.

LexLuthr requested review from Kubuxu and rvagg as code owners September 9, 2026 14:29

github-project-automation Bot added this to FilOz Sep 9, 2026

github-project-automation Bot moved this to 📌 Triage in FilOz Sep 9, 2026

github-actions Bot previously requested changes Sep 9, 2026

View reviewed changes

github-project-automation Bot moved this from 📌 Triage to ⌨️ In Progress in FilOz Sep 9, 2026

github-actions Bot previously requested changes Sep 9, 2026

View reviewed changes

LexLuthr changed the title ~~feat(ci): Linux arm64 images for docker~~ ci(release): Linux arm64 images for docker Sep 9, 2026

github-actions Bot previously requested changes Sep 9, 2026

View reviewed changes

LexLuthr changed the title ~~ci(release): Linux arm64 images for docker~~ ci: Linux arm64 images for docker Sep 9, 2026

github-actions Bot requested changes Sep 9, 2026

View reviewed changes

BigLep requested a lite review from Copilot September 10, 2026 00:52

Copilot started reviewing on behalf of BigLep September 10, 2026 00:53 View session

BigLep reviewed Sep 10, 2026

View reviewed changes

Comment thread Dockerfile

Copilot AI reviewed Sep 10, 2026

View reviewed changes

Comment thread Dockerfile

BigLep mentioned this pull request Sep 10, 2026

fix(docker): move builder and runtime images to Debian trixie #13785

Merged

4 tasks

rvagg changed the title ~~ci: Linux arm64 images for docker~~ chore(ci): linux arm64 images for docker Sep 10, 2026

rvagg changed the title ~~chore(ci): linux arm64 images for docker~~ feat: linux arm64 images for docker Sep 10, 2026

rvagg reviewed Sep 10, 2026

View reviewed changes

Comment thread .github/workflows/docker.yml

rvagg reviewed Sep 10, 2026

View reviewed changes

Comment thread CHANGELOG.md Outdated

LexLuthr requested a review from rvagg September 10, 2026 07:49

LexLuthr added this pull request to stack #13789 September 10, 2026 12:47

LexLuthr force-pushed the feat/arm-images branch from 2894aab to 411cd90 Compare September 10, 2026 13:51

Linux arm64 images for docker

03f3c1d

LexLuthr force-pushed the feat/arm-images branch from 63e6739 to 03f3c1d Compare September 10, 2026 14:15

LexLuthr changed the title ~~feat: linux arm64 images for docker~~ feat(ci): linux arm64 images for docker Sep 10, 2026

BigLep reviewed Sep 10, 2026

View reviewed changes

BigLep mentioned this pull request Sep 11, 2026

test(itests): stop double proving-period waits at null-round deadline boundaries #13793

Draft

use digest instead of temp tags

bab03fc

Uh oh!

Conversation

LexLuthr commented Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Related Issues

Proposed Changes

Additional Info

Checklist

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Suppressed comments (1)

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

LexLuthr commented Sep 9, 2026 •

edited

Loading