Skip to content

New Production Release Procedure - #1684

Merged
oblodgett merged 1883 commits into
productionfrom
stage
Nov 3, 2026
Merged

oblodgett merged 1883 commits into
productionfrom
stage

Conversation

@oblodgett

Copy link
Copy Markdown
Member

No description provided.

oblodgett added 30 commits May 15, 2026 07:40
Delete unused Neo4j-backed indexer classes (DatasetIndexer, DiseaseIndexer,
GeneIndexer, GoIndexer, ModelIndexer, VariantIndexer) and move curation
indexers, interfaces, and services up from indexers/curation/* into
top-level indexers/, interfaces/, service/ packages.
The class-level @JSONVIEW on GeneMolecularInteractionDocument and
GeneGeneticInteractionDocument was a no-op: the documents have a single
field with no field-level @JSONVIEW, and the indexer ObjectMapper uses
the default DEFAULT_VIEW_INCLUSION=true. Remove the view argument from
indexDocuments() calls and drop the now-unused @JsonView/PublicView
imports from both documents and indexers. agr_indexer now has zero
references to org.alliancegenome.neo4j.
Pull species from curation API once in SourceDocumentCreationManager and
pass the matching Species into each SourceDocumentCreation (mirrors the
geneCacheMap + severityRanking pattern). VariantSummaryConverter now
takes a curation Species instead of the deprecated SpeciesType enum and
reads fullName / taxon.curie / abbreviation / assembly_curie directly.

agr_variant_indexer is now Neo4j-namespace clean.
Delete three Groovy integration specs that exercised the now-removed
Neo4j-backed indexers (GeneDocumentIntegrationSpec, GOTermIntegrationSpec,
CategoriesAndFieldsIntegrationForLaterSpec), and remove the
org.neo4j.ogm logger from log4j2.xml in both indexer modules.
SCRUM-5973 remove Neo4j from indexers
Remove unused @Inject fields from AlleleController, DiseaseController,
and GeneController. Delete 56 unreferenced classes from agr_java_core
(filtering/sorting helpers, ES doclets, Neo4j repositories/entities/views,
TDF/doclet translators, parallel GZIP utilities, ColumnFieldMapping
classes, and the core DiseaseService). Relocate single-module-consumer
code out of agr_java_core into the consuming modules:
- StatsCollector to agr_indexer
- core.filedownload package and VariantConfigHelper to agr_variant_indexer
- data_extractor.extractors.fms package to agr_intermine_data_extractor
  (with OpenAPI annotations stripped from the rescu client interfaces
  since that module excludes Quarkus).
Also drop one obsolete VCF test script under agr_variant_indexer.
Remove unreachable classes orphaned by SCRUM-5973's indexer cleanup:
- Neo4j GoRepository and all five *IndexerRepository classes
- All six Neo4j-entity-to-ES document translators plus their EntityDocumentTranslator / Translator bases
- All six site-index *DocumentCache classes and IndexerCache base
- Indexer-side GeneToGeneOrthologyService and GeneExpressionAnnotationService
- Five unused Rescu client interfaces in agr_indexer
- Twelve dev test scripts under agr_indexer and agr_variant_indexer including S3ProgressListener
- agr_api DTOs (DiseaseEntitySlim, GeneList, JoinTypeValue x2) and agr_file_generator GeneDescriptionFileGenerator
- Unused Validate main in agr_schema_validation
- Cascade orphans SearchableItemDocument, ESDoclet, CrossReferenceDoclet, and the four api.json Jackson serializer classes (plus the commented-out reference in RestDefaultObjectMapper)

Also drop the unused 7-arg constructor on TranscriptLevelConsequence and the corresponding GeneDocumentCache import, and clean up two obsolete test classes and references.
Replace the in-memory filter pass in OrthologyESService with a term
filter on stringencyFilter.keyword inside the ES query so total counts
and pagination match the filtered result set. Remove the unused
filter.method query parameter from /gene/{id}/orthologs and
/gene/{id}/paralogs (the UI never sent it and does its own client-side
method filtering on the returned data).

With the only caller gone, delete the in-memory filter machinery:
FilterService, AnnotationFiltering, FilterFunction, and OrthologyFiltering.
Also remove the now-unused FieldFilter.ORTHOLOGY_METHOD enum value, the
obsolete checkOrthologyAPIWithMethods integration test, and the dead
Neo4j-backed getVariants method on VariantService.
DiseaseController was the only caller of the SortingField enum. Inline
the single use as a string literal and delete both the SortingField enum
and the unimplemented Sorting interface alongside it.
…tion

The 5-arg Pagination constructor stored a ColumnFieldMapping for filter
validation, but the two endpoints that used it (genetic-interactions and
molecular-interactions in GeneController) only called addFilterOption()
afterwards, never addFieldFilter() -- so the mapping was never consulted
at runtime. validateFilterValues() had its only caller commented out.

Delete the entire dead cluster -- Page, Table, Column, ColumnFieldMapping,
and InteractionColumnFieldMapping (the last one also dragged in a Neo4j
InteractionGeneJoin entity coupling). Drop the mapping field, 5-arg
constructor, checkIfMappingExists(), and validateFilterValues() from
Pagination. Update the two GeneController call sites to use the 4-arg
constructor.
Replace the three Neo4j repository lookups with a single SearchDAO
query against the *_summary categories in the site index, returning
only the name field via _source filtering. Drops the static
GeneRepository / DiseaseRepository / AlleleRepository instances
and removes the related Neo4j imports.
…eESDAO

Extract the gene_summary ES query into a new GeneESDAO under
agr_java_core (mirrors the existing VariantESDAO pattern). Refactor
GeneESService.getById to delegate to it, and replace the
geneRepository.getShallowGene(geneID) call in
DiseaseESService.getDiseaseRibbonSummary with geneESDAO.getById(geneID).

populateDiseaseRibbonSummary now takes the curation Gene from the
gene_summary doc instead of the Neo4j Gene entity, and reads
geneSymbol.displayText / taxon.curie / taxon.name from it.

Drops the static GeneRepository field from DiseaseESService. The only
remaining caller of GeneRepository is the agr_api_tester load-test
harness.
…ules

The api_tester module hit Neo4j for a gene ID list and pinged each REST
endpoint. With production data sourced from ES, this Neo4j-backed
smoke test is no longer needed. The intermine_data_extractor is being
replaced by direct curation DB calls.

Remove both modules from the parent pom and delete their directories.

GeneRepository and VariantRepository both cascade to zero external
refs after these deletions, so delete them too. DiseaseRepository
still has two callers (DiseaseRibbonService and DiseaseESService) and
remains for now.
DataExtractorRepository was only used by the deleted intermine
extractor module. ResultTSVTranslator and EntityTSVTranslator are
abstract bases with zero remaining subclasses.
…ugh ES

Add DiseaseESDAO mirroring the GeneESDAO/VariantESDAO pattern and use
it to replace two Neo4j call sites in the disease ribbon path:

- DiseaseRibbonService.getShallowDiseaseTerm(id) -> DiseaseESDAO.getById
  for the disease-summary section descriptions (5 hardcoded categories +
  one per top-level slim term).
- DiseaseESService.getDiseaseAnnotationHistogram now reads the
  pre-computed annotation.parentSlimIDs directly instead of asking
  DiseaseRibbonService to walk the DO ontology via Neo4j per annotation.

The dynamic per-request histogram path is fully off Neo4j. The startup
template build still uses DiseaseRepository.getAgrDoSlim and
getDOParentTermIDs (the AGR slim list and the section assignment for
each slim term); those will be migrated once we settle on a data source.

DiseaseRibbonService.getAllParentIDs becomes unused and is deleted
along with its helper. DiseaseRibbonService now takes a DiseaseESDAO
in its constructor; DiseaseESService wires it via @PostConstruct since
the DAOs are CDI-managed.
Replace the remaining Neo4j calls in the disease ribbon path with
disease_summary ES lookups via DiseaseESDAO:

- getAgrSlimDocs() returns disease_summary docs flagged with the
  doTerm.subsets keyword DO_AGR_slim (v0.49.2 of curation_api exposes
  subsets and a flat parentClosureIDs set on disease_summary).
- DiseaseRibbonService now iterates those slim docs and uses each
  doc's parentClosureIDs to assign the slim to one of the five
  section categories, replacing diseaseRepository.getDOParentTermIDs.
- DiseaseESService.populateDiseaseRibbonSummary now derives the slim
  ID list from getAgrSlimDocs instead of getAgrDoSlim.

With DiseaseRepository gone, the Neo4jRepository abstract base has
zero subclasses and is deleted too. agr_java_software no longer has
any Neo4j repository classes.

pom.xml curation.version bump to v0.49.2 is intentionally not
included here -- v0.49.2 hasn't been published to Maven Central yet.
…lator

Only 4 of 11 public methods on this translator had callers (all using
curation_api Document types). Removed the 7 Neo4j-typed dead methods,
their 4 private helpers, 2 commented-out helpers, and 1 dead overload.
All 7 imports from org.alliancegenome.neo4j.entity drop with them.

Also removed the unused diseaseTranslator field in GeneController.
Four entities had no references anywhere in the codebase
(not even from other Neo4j entities):
- DiseaseSummary
- EntitySummary
- HTPDataset
- ParaAlgorithm

The rest of the Neo4j entity graveyard is anchored by ~10 externally-
imported types (Gene, Allele, Variant, AffectedGenomicModel, etc.) that
hold the cluster together via same-package field references. Migrating
those consumers off the Neo4j types is needed before further deletions.
Five production references were holding the dead island alive:
- ESService.DOTerm.getAllOtherDiseaseTerms() -> moved constant to DiseaseRibbonSummary
- GeneRESTInterface.PhenotypeAnnotation -> fixed stale @Schema to point at GenePhenotypeAnnotationDocument
- VariantServiceHelper (only called by dead TranscriptLevelConsequence)
- SourceServiceHelper (only called by dead DiseaseEntityJoin)
- PhenotypeIT test references to GeneticEntity.CrossReferenceType enum -> inlined as string

With those gone, all 47 remaining files under neo4j/entity/ were dead-island
(no external imports). Deleted in one shot.

Surviving in the neo4j package (genuinely live, just historical naming):
- Neo4jEntity (still extended by AllianceReleaseInfo / ModFileMetadata / OntologyFileMetadata)
- SpeciesType enum (used by ES services)
- ReleaseSummary, AllianceReleaseInfo, ModFileMetadata, OntologyFileMetadata (release-info flow)
- view/PublicView, view/BaseFilter (Jackson + pagination helpers)
These classes have no actual Neo4j coupling -- they're a Jackson @JSONVIEW
marker and a pagination base filter that lived in org.alliancegenome.neo4j.view
for historical reasons.

Moved to org.alliancegenome.view. Updated 28 imports.
The ReleaseInfoService.getSummary() method was a no-op since the Neo4j
repos were deleted -- it returned an empty ReleaseSummary with all
setter lines commented out. UI doesn't call /releaseInfo/summary
anywhere. Removed:

- ReleaseInfoRESTInterface.getReleaseInfoSummary()
- ReleaseInfoController.getReleaseInfoSummary()
- ReleaseInfoService.getSummary()
- ReleaseSummary, AllianceReleaseInfo, ModFileMetadata, OntologyFileMetadata
- Neo4jEntity abstract base (no more subclasses)
- DateConverter (only used by the three deleted @NodeEntity classes)
All translator + download-row classes lived under
agr_java_core/.../core/translators/tdf/ but were only consumed by
agr_api controllers -- zero internal users in agr_java_core. Moved
all 11 files alongside the agr_api-resident BaseToTdfTranslator and
sibling translators in agr_api/.../api/translators/tdf/, updated
package declarations and imports.
Audit found 36 files in agr_java_core that were only consumed by agr_api
code. Moved them all with git mv to preserve history, preserving the
existing package paths (no source edits needed).

Clusters moved:
- JsonResultResponse + Request + Duration
- SearchApiResponse + SearchResult
- AggResult + AggDocCount
- Disease ribbon: DiseaseRibbonSummary/Entity/Section/SubgroupSlim, SectionSlim, DiseaseRibbonService
- SiteMap + SiteMapIndex + XMLURL + XMLURLSet + package-info
- RestError: RestErrorException + RestErrorMessage + RestErrorExceptionMapper
- ES DAOs: ESDAO (base) + AutoCompleteDAO + DiseaseESDAO + GeneESDAO + SearchDAO + VariantESDAO

Standalone moves:
- Pagination, FieldFilter, AutoCompleteResult, RelatedDataLink, AlleleVariantIndexService
- Category, SpeciesType (verified agr_api-only after earlier neo4j cleanup)
- BaseFilter (caught by compiler -- agr_api-only)

Also deleted PresentationEntity.java (dead -- zero implementers, zero consumers).

14 empty directories cleaned up in agr_java_core including neo4j/ which
fully dissolves task #8.
Renamed the legacy top-level packages in agr_api so everything now lives
under org.alliancegenome.api.* instead of being scattered across cache.*,
core.*, es.*, and view.*:

- cache.repository.helper -> api.response (JsonResultResponse, Request, Duration)
- core.api.service -> api.service (DiseaseRibbonService)
- core.variant.service -> api.service (AlleleVariantIndexService)
- core.exceptions -> api.exceptions (RestError*)
- es.index + es.index.site.dao -> api.es.dao (ESDAO + 5 DAOs, flattened)
- es.model.query -> api.es.query (Pagination, FieldFilter)
- es.model.search -> api.es.search (7 search/agg classes)
- view -> api.view (BaseFilter)

SpeciesType left in neo4j.entity intentionally -- it's slated for deletion.
…e.core

Renames:
- api.entity, es.index, es.model -> core.document
  (15 + 1 + 2 = 18 ES doc classes merged; ESDocument joins the doc family)
- es.index.site.schema -> core.es.schema
- es.index.site.schema.settings -> core.es.schema.settings
- es.rest.RestConfig -> core.config.RestConfig
  (RestConfig joins ConfigHelper in core.config)
- es.util -> core.es.util
- view -> core.view
- core.helpers -> core.helper (singularize)
- core.variant.converters -> core.variant.converter (singularize)

Left alone: core.config (now also holds RestConfig), core.util.

Also bumped curation.version to v0.49.2 -- now published on Maven Central.
oblodgett and others added 26 commits September 8, 2026 12:02
SCRUM-6031 Create literature_search_result Elasticsearch document
…ghlighting

- add literature_search_result to categoryFilters with authors and publicationYear facets
- add abstractText, authors, citation and publicationYear to searchFields with boosts so
  reference search matches those fields
- return abstractText, authors, citation, shortCitation and publicationYear in responseFields
- highlight abstractText as a whole field and blacklist abstractText.standardText so the
  abstract carries a single full-field highlight
- map publicationYear as text with a keyword subfield so the facet and filter work
SCRUM-6394 Set fields for reference search
Add an explicit mapping for subject.curie (text, with .keyword and .sort
subfields) alongside the existing subject.primaryExternalId entry.

agr_api already sends a bool.should clause on subject.curie.keyword for
gene/annotation lookups, but the field is not mapped on either the stage
or production cluster, so that clause matches zero documents and is dead
weight on every such query. Matching currently happens entirely via
subject.primaryExternalId.keyword.

Mapping the field now means it exists in the index ahead of the data, so
the clause becomes valid as soon as we start populating subject.curie
with our own minted IDs.

Follows the existing curie/primaryExternalId pairing used for
object.curie, allele.taxon.species.genomeAssembly.curie and
geneMolecularInteraction.geneAssociationSubject.curie. Generated mapping
is byte-identical in shape to subject.primaryExternalId.
…urie

feat: index subject.curie in site_index mapping
Populate referenceType on the literature search result document from the ABC category field, map it as text with a keyword subfield so it is aggregatable, and register it as the first facet for the literature search category.

The ABC category value (Research_Article, Review_Article, Conference_Publication, Thesis, etc.) previously survived only inside literatureSummary.category in _source, which is unindexed, so it could not be searched, filtered or faceted.

SCRUM-6467
Populate resourceTitle on the literature search result document from the ABC resource_title field, map it as text with a keyword subfield so it is aggregatable, and register it as a facet for the literature search category.

Journal was previously only reachable as a substring of the citation field, so it could be matched by free-text search but never filtered on. resource_title is present on 95.7% of ABC references.

SCRUM-6551
SCRUM-6555: add referenceType facet to literature search results
SCRUM-6551: add resourceTitle (journal) facet to literature search results
AGM phenotype annotations with no gene/allele reference (subject,
inferred, or asserted) were never surfaced anywhere: PhenotypeAnnotationCurationIndexer
populated agmMap but only ever used it to roll gene/allele associations
into geneMap/alleleMap, and index() never built or indexed an AGM
document at all -- so such annotations silently dropped out of the
Phenotype download files (both JSON and TSV).

Adds AGMPhenotypeAnnotationDocument (category "agm_phenotype_annotation"),
mirroring the already-shipped AGMDiseaseAnnotationDocument, and wires
PhenotypeAnnotationCurationIndexer.index() to build and index it via a
new createAGMPhenotypeAnnotationDocuments(), mirroring
DiseaseAnnotationCurationIndexer.createAGMDiseaseAnnotationDocuments().

No other changes needed: FileGeneratorConfig.Phenotype already lists
"agm_phenotype_annotation" as an expected ES category and
PhenotypeFileGenerator already branches on AGMPhenotypeAnnotation for
TSV columns -- both were written anticipating this document type, they
just never received any. AGMPhenotypeAnnotationService.getFiltered()
already pulls the full public AGM phenotype annotation population
(gene/allele-referenced or not) via findForPublic against the curation
API, and there's no phenotype-specific ES field-mapping section in
Mapping.java to update (unlike disease_annotation's), so no further
indexing config is needed either.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
SCRUM-6445 index AGM-only phenotype annotations for public downloads
…ism gene counts (#1676)

Co-authored-by: Joe Wang <dongzhuo.wang@gmail.com>
…rie (#1679)

Since curation v0.53 mints AGRKB curies on genes, Gene.getIdentifier()
returns the AGRKB curie instead of the MOD id. The disease indexer keys
its gene map by getIdentifier() and passed that key as
subjectGene.primaryExternalId to /orthologygenerated/findForPublic, so
every orthology lookup returned nothing and no via-orthology gene disease
annotations were indexed (0 since the 2026-10-02 stage run, ~60K before).

Use the focus gene's primaryExternalId for the orthology lookup.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
SCRUM-6682 reorder literature facets and list years newest first
SCRUM-6730 remove UniqueID column from Disease download TSV
SCRUM-6682 hide the literature author facet
im.createSnapShot(repo, snapShotName, index_name);
String snapshotType = args.remove(0);
List<String> indexes = im.getIndexList();
for(String index: indexes) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚫 [checkstyle] <com.puppycrawl.tools.checkstyle.checks.whitespace.WhitespaceAroundCheck> reported by reviewdog 🐶
'for' is not followed by whitespace.

String snapshotType = args.remove(0);
List<String> indexes = im.getIndexList();
for(String index: indexes) {
if(index.startsWith(snapshotType)) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚫 [checkstyle] <com.puppycrawl.tools.checkstyle.checks.whitespace.WhitespaceAroundCheck> reported by reviewdog 🐶
'if' is not followed by whitespace.

@claude

claude Bot commented Nov 3, 2026

Copy link
Copy Markdown

Review summary (stage → production release, 712 files / ~1.9k commits)

This is too large for a line-by-line review, and almost all of it is commits that were already reviewed when they merged into stage. I read the newest change (#1683) closely and spot-checked the new public endpoints.

  • SCRUM-6793 return MOD IDs from allele-viewer-ids so the viewer matches the VCF #1683 (AlleleESService.resolveAlleleIdentifier): correct. Checking primaryExternalId and modInternalId before curie matches the MOD IDs in the VCF, and the updated test covers every fallback case.
  • Breaking API removals, for the release notes: this release deletes the Orthology, Genes, Go, Entity, Cache and Phenotype REST interfaces, and changes the signatures of several others. External API consumers will see 404s on those routes, so it's worth stating that in the release announcement or on the API Changes mailing list.

Worth hardening in DownloadService / DownloadController (new in this release; not blocking):

  1. openDownloadStream calls new URL(url).openStream() with no connect or read timeout. Every /download/{filename} request holds a worker thread while the file streams from download.alliancegenome.org. If that host stalls, requests pile up until the worker pool runs out, and the whole API stops responding. The same applies to listDownloads. Use URLConnection with setConnectTimeout and setReadTimeout, or the HTTP client already used elsewhere.
  2. filename goes into the upstream URL and the Content-Disposition header without validation. Once decoded, a path parameter can contain /, .. or ?. Requests can only reach the fixed public download host, but they are not confined to {release}/downloads/. Accept only something like ^[A-Za-z0-9._-]+$ and reject everything else.
  3. Every exception becomes a 404, including the case where the release lookup fails (IllegalStateException). An Elasticsearch or upstream outage then looks like "file not found". Return 404 only for FileNotFoundException and 5xx for everything else.

I found no other correctness problems in the parts I checked.

@oblodgett
oblodgett merged commit fb9adbd into production Nov 3, 2026
5 of 6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

10 participants