Repository navigation
[Hotfix][Build] Backport Docker pipeline fix and Hudi Hadoop exclusion to 0.9.0 (#4485) - #4495
Conversation
…modules Same change as #4491 on master: the repository Actions allowlist rejects the third-party docker/* actions at workflow startup, so the image publishing pipeline has been failing since the policy was tightened. Vendor the five docker actions as pinned submodules under .github/actions/ and reference them as local actions; adds the workflow_dispatch trigger with a dry-run switch and the Flink optimizer matrix cleanup (1.14.6 -> 1.18.1).
…#4494) hudi-java-client pulls hudi-common -> hbase-client 2.4.9, which drags in a full Hadoop 2.10 stack (hadoop-auth, hadoop-common, hadoop-hdfs, hadoop-hdfs-client, yarn client/api/common, mapreduce-client-core, hadoop-distcp, hadoop-annotations - ten artifacts). In distributions that bundle the Hudi format (e.g. the Docker image built with -Psupport-all-formats) these shadow the managed hadoop-client-api/runtime 3.4.3: KerberosName gets loaded from hadoop-auth 2.10.0 while HadoopKerberosName from hadoop-client-api 3.4.3 calls the missing setRuleMechanism, breaking the AMS Terminal with NoSuchMethodError (issue #4485). Exclude org.apache.hadoop:* from the Hudi dependency so the host supplies the single managed Hadoop client - the same boundary the AMS module already applies to its Spark dependencies. Verified: dependency:tree for amoro-format-hudi goes from 10 Hadoop-2.10.0 artifacts to 0 (only hadoop-client-api/runtime 3.4.3 remain, provided by amoro-common), and the module still compiles.
|
Why this PR also carries the workflow fix (and not only the Hudi exclusion) GitHub Actions reads the workflow file from the ref being built, not from master. The only way to rebuild the 0.9.0-incubating images is to dispatch the workflow on this
Neither fix alone is sufficient: without the workflow fix nothing can be built; without the Hudi exclusion the built image would still carry the #4485 Terminal bug. The two commits together are the minimal set for publishing a healthy 0.9.0 image. Dispatching from master is not an option either: master has already moved to 1.0.0-SNAPSHOT (#4493) and would build the wrong version. |
The v0.9.0-incubating tag is immutable, so the 0.9.0-incubating Docker images are rebuilt from this branch (per #4495). The semver/latest tag rules only fire on tag pushes, and a branch build would otherwise tag images as 0.9.0-snapshot. Add explicit raw rules that produce the release tags (0.9.0-incubating, -hadoop3, optimizer -flink1.18/-1.20/ -spark3.5 variants, latest) when building from refs/heads/0.9.0, and scope the -snapshot rules to master builds.
Backports the two fixes needed to rebuild and publish the official 0.9.0-incubating Docker images, addressing #4485.
What's included
Docker pipeline fix (from [Build] Fix Docker image publishing: vendor docker/* actions as submodules, drop legacy Flink 1.14.6 #4491, master): vendor the five docker/* GitHub Actions as pinned git submodules under
.github/actions/(the repository Actions allowlist rejects third-party actions at workflow startup), add theworkflow_dispatchtrigger with the dry-run switch, and update the Flink optimizer matrix (1.14.6 → 1.18.1). Also carries master's workflow shape for the image jobs (single default Hadoop 3 image — Hadoop 2 support ends with 0.9.0 per [AMORO-4451][Build] Discontinue Hadoop 2 support #4452, so no Hadoop 2 image variant is built for this release).Hudi Hadoop exclusion (from [Hudi] Exclude the transitive Hadoop 2.10 stack from hudi-java-client #4494, master): exclude
org.apache.hadoop:*fromhudi-java-clientso Hudi's transitivehbase-client:2.4.9no longer drags a full Hadoop 2.10 stack into-Psupport-all-formatsdistributions — the KerberosName shadowing that breaks the AMS Terminal ([Bug]: Terminal fails with NoSuchMethodError in the apache/amoro Docker image because Hudi brings in hadoop-auth 2.10.0 #4485).Why a release-branch PR
The
v0.9.0-incubatingtag cannot be moved; images for the release are rebuilt by dispatching the fixed workflow on the0.9.0branch (dry-run=false), which builds from the same source as the released tag plus exactly these two fixes. No jar artifacts are re-published; only Docker Hub images change.Verification plan
dry-run=falseafter merge → verifyapache/amoro:0.9.0-incubatingtags on Docker Hub