Skip to content

[AMORO-4381][Build] Port 0.9.x license completion to master's Hadoop 3-only distribution - #4504

Open
czy006 wants to merge 2 commits into
masterfrom
license-port-0.9.x
Open

czy006 wants to merge 2 commits into
masterfrom
license-port-0.9.x

Conversation

@czy006

@czy006 czy006 commented Nov 15, 2026

Copy link
Copy Markdown
Contributor

What

Ports the license/attribution work that shipped in 0.9.0-incubating (RC14) to master, so 1.0.0 does not cut its RC from pre-RC13 LICENSE files:

Adaptation to master (single commit = cherry-picks + adaptation)

Master builds only the Hadoop 3 convenience binary (the hadoop2 profile was removed), so beyond the clean cherry-picks:

  • Dropped the hadoop2 Maven profile (conflict resolution) and the orphan NOTICE-binary-hadoop2, LICENSE.xmlenc-0.52, LICENSE.mssql-jdbc
  • Removed Hadoop 2-only attributions from LICENSE-binary / README-LICENSES / licenses-binary/: xmlenc, javax.activation:activation, jaxb-api 2.2.2, stax-api, mssql-jdbc, snappy-java 1.1.8.4, httpclient 4.5.13, xbean-asm9-shaded 4.20, spark-core 3.3.4 web resources (JSON formatter, vis.js, DataTables 1.10.25), hadoop-yarn-common 2.10.2 web resources (DataTables 1.9.4)
  • Reconciled versions with master's dependency set: Iceberg 1.8.1 → 1.10.2, commons-compress 1.23.0 → 1.26.2, postgresql 42.7.2 → 42.7.14, amoro-web jar placeholder (<version>)
  • licenses-binary/LICENSE.snappy-java kept byte-identical (hash-pinned by tools/releasing; its dual recipe targets describe prebuilts retained inside snappy-java 1.1.10.5 itself)
  • Updated tools/releasing regression checks to single-distribution semantics (removed the xmlenc branch, simplified the JTA assertion)

License-motivated code changes (from #4473)

  • Replaces Lucene RamUsageEstimator usage with an adapted in-house ObjectSizeEstimator; Lucene moves to test-only scope and NOTICE retains the adapted-code attribution
  • Adds Apache headers + Presto provenance to the Spark SQL grammar files (.g4 + regenerated parsers)

Verification

  • tools/releasing python suite: 11/11 (mutation-tested — corrupted snappy hash and JTA window were both caught)
  • mvn validate (checkstyle + spotless): pass
  • amoro-common + amoro-format-iceberg test-compile; TestObjectSizeEstimator 3/3; zero remaining org.apache.lucene imports in main code
  • All 38 licenses/ references resolve (repo file / jar-internal path / URL — no dangling pointers)
  • Spark web-resource removals verified empirically against the spark-core_2.12-3.5.8.jar ui/static/ inventory
  • Zero Hadoop 2 residue outside intentionally kept spots (snappy recipe label — hash-pinned; Hudi pom comment describing Hudi's own transitive stack; site download archive links)

Note

licenses/maven/ per-artifact NOTICE inventories are ported snapshots with version reconciliation; they should be regenerated against master's dependency trees at the 1.0.0 RC audit (that is what tools/releasing is for).

…3-only distribution

Port the license/attribution work that shipped in 0.9.0-incubating
(RC14) so 1.0.0 does not cut its RC from pre-RC13 LICENSE files:

- #4456 Fill license attribution gaps in 0.9.x artifacts
- #4473 Complete licenses in source and Maven distributions

Adapted to master, which builds only the Hadoop 3 convenience binary
(the hadoop2 profile is gone):

- drop the hadoop2 Maven profile and the orphan NOTICE-binary-hadoop2
- remove Hadoop 2-only attributions: xmlenc, javax.activation:activation,
  jaxb-api 2.2.2, stax-api, mssql-jdbc, snappy-java 1.1.8.4,
  httpclient 4.5.13, xbean-asm9-shaded 4.20, spark-core 3.3.4 web
  resources (JSON formatter, vis.js, DataTables 1.10.25) and
  hadoop-yarn-common 2.10.2 web resources (DataTables 1.9.4)
- reconcile versions with master's dependency set: Iceberg 1.8.1 ->
  1.10.2, commons-compress 1.23.0 -> 1.26.2, postgresql 42.7.2 ->
  42.7.14, amoro-web jar placeholder
- keep licenses-binary/LICENSE.snappy-java byte-identical (hash-pinned
  by tools/releasing; its dual recipe targets describe prebuilts
  retained inside snappy-java 1.1.10.5 itself)
- update the releasing regression checks to single-distribution semantics

License-motivated code changes from #4473 come along:

- replace Lucene RamUsageEstimator usage with an adapted in-house
  ObjectSizeEstimator; Lucene moves to test-only scope and NOTICE
  retains the adapted-code attribution
- add Apache headers and Presto provenance to the Spark SQL grammar
  files (.g4 plus regenerated parsers)

Verified: tools/releasing suite 11/11 (mutation-tested), mvn validate,
amoro-common/amoro-format-iceberg test-compile, TestObjectSizeEstimator
3/3, all 38 licenses/ references resolve, zero Hadoop 2 residue.
@github-actions github-actions Bot added type:docs Improvements or additions to documentation module:mixed-flink Flink moduel for Mixed Format module:mixed-spark Spark module for Mixed Format module:ams-server Ams server module module:ams-optimizer AMS optimizer module type:infra type:build module:ams-dashboard Ams dashboard module module:common labels Nov 15, 2026
Record the resolved Hadoop 3 distribution dependencies after the license port pins jsr305 to 3.0.2 and moves Lucene to test scope.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

module:ams-dashboard Ams dashboard module module:ams-optimizer AMS optimizer module module:ams-server Ams server module module:common module:mixed-flink Flink moduel for Mixed Format module:mixed-spark Spark module for Mixed Format type:build type:docs Improvements or additions to documentation type:infra

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant