Repository navigation
New Production Release Procedure - #1684
Conversation
Delete unused Neo4j-backed indexer classes (DatasetIndexer, DiseaseIndexer, GeneIndexer, GoIndexer, ModelIndexer, VariantIndexer) and move curation indexers, interfaces, and services up from indexers/curation/* into top-level indexers/, interfaces/, service/ packages.
The class-level @JSONVIEW on GeneMolecularInteractionDocument and GeneGeneticInteractionDocument was a no-op: the documents have a single field with no field-level @JSONVIEW, and the indexer ObjectMapper uses the default DEFAULT_VIEW_INCLUSION=true. Remove the view argument from indexDocuments() calls and drop the now-unused @JsonView/PublicView imports from both documents and indexers. agr_indexer now has zero references to org.alliancegenome.neo4j.
Pull species from curation API once in SourceDocumentCreationManager and pass the matching Species into each SourceDocumentCreation (mirrors the geneCacheMap + severityRanking pattern). VariantSummaryConverter now takes a curation Species instead of the deprecated SpeciesType enum and reads fullName / taxon.curie / abbreviation / assembly_curie directly. agr_variant_indexer is now Neo4j-namespace clean.
Delete three Groovy integration specs that exercised the now-removed Neo4j-backed indexers (GeneDocumentIntegrationSpec, GOTermIntegrationSpec, CategoriesAndFieldsIntegrationForLaterSpec), and remove the org.neo4j.ogm logger from log4j2.xml in both indexer modules.
SCRUM-5973 remove Neo4j from indexers
Remove unused @Inject fields from AlleleController, DiseaseController, and GeneController. Delete 56 unreferenced classes from agr_java_core (filtering/sorting helpers, ES doclets, Neo4j repositories/entities/views, TDF/doclet translators, parallel GZIP utilities, ColumnFieldMapping classes, and the core DiseaseService). Relocate single-module-consumer code out of agr_java_core into the consuming modules: - StatsCollector to agr_indexer - core.filedownload package and VariantConfigHelper to agr_variant_indexer - data_extractor.extractors.fms package to agr_intermine_data_extractor (with OpenAPI annotations stripped from the rescu client interfaces since that module excludes Quarkus). Also drop one obsolete VCF test script under agr_variant_indexer.
Remove unreachable classes orphaned by SCRUM-5973's indexer cleanup: - Neo4j GoRepository and all five *IndexerRepository classes - All six Neo4j-entity-to-ES document translators plus their EntityDocumentTranslator / Translator bases - All six site-index *DocumentCache classes and IndexerCache base - Indexer-side GeneToGeneOrthologyService and GeneExpressionAnnotationService - Five unused Rescu client interfaces in agr_indexer - Twelve dev test scripts under agr_indexer and agr_variant_indexer including S3ProgressListener - agr_api DTOs (DiseaseEntitySlim, GeneList, JoinTypeValue x2) and agr_file_generator GeneDescriptionFileGenerator - Unused Validate main in agr_schema_validation - Cascade orphans SearchableItemDocument, ESDoclet, CrossReferenceDoclet, and the four api.json Jackson serializer classes (plus the commented-out reference in RestDefaultObjectMapper) Also drop the unused 7-arg constructor on TranscriptLevelConsequence and the corresponding GeneDocumentCache import, and clean up two obsolete test classes and references.
Replace the in-memory filter pass in OrthologyESService with a term
filter on stringencyFilter.keyword inside the ES query so total counts
and pagination match the filtered result set. Remove the unused
filter.method query parameter from /gene/{id}/orthologs and
/gene/{id}/paralogs (the UI never sent it and does its own client-side
method filtering on the returned data).
With the only caller gone, delete the in-memory filter machinery:
FilterService, AnnotationFiltering, FilterFunction, and OrthologyFiltering.
Also remove the now-unused FieldFilter.ORTHOLOGY_METHOD enum value, the
obsolete checkOrthologyAPIWithMethods integration test, and the dead
Neo4j-backed getVariants method on VariantService.
DiseaseController was the only caller of the SortingField enum. Inline the single use as a string literal and delete both the SortingField enum and the unimplemented Sorting interface alongside it.
…tion The 5-arg Pagination constructor stored a ColumnFieldMapping for filter validation, but the two endpoints that used it (genetic-interactions and molecular-interactions in GeneController) only called addFilterOption() afterwards, never addFieldFilter() -- so the mapping was never consulted at runtime. validateFilterValues() had its only caller commented out. Delete the entire dead cluster -- Page, Table, Column, ColumnFieldMapping, and InteractionColumnFieldMapping (the last one also dragged in a Neo4j InteractionGeneJoin entity coupling). Drop the mapping field, 5-arg constructor, checkIfMappingExists(), and validateFilterValues() from Pagination. Update the two GeneController call sites to use the 4-arg constructor.
Replace the three Neo4j repository lookups with a single SearchDAO query against the *_summary categories in the site index, returning only the name field via _source filtering. Drops the static GeneRepository / DiseaseRepository / AlleleRepository instances and removes the related Neo4j imports.
…eESDAO Extract the gene_summary ES query into a new GeneESDAO under agr_java_core (mirrors the existing VariantESDAO pattern). Refactor GeneESService.getById to delegate to it, and replace the geneRepository.getShallowGene(geneID) call in DiseaseESService.getDiseaseRibbonSummary with geneESDAO.getById(geneID). populateDiseaseRibbonSummary now takes the curation Gene from the gene_summary doc instead of the Neo4j Gene entity, and reads geneSymbol.displayText / taxon.curie / taxon.name from it. Drops the static GeneRepository field from DiseaseESService. The only remaining caller of GeneRepository is the agr_api_tester load-test harness.
…ules The api_tester module hit Neo4j for a gene ID list and pinged each REST endpoint. With production data sourced from ES, this Neo4j-backed smoke test is no longer needed. The intermine_data_extractor is being replaced by direct curation DB calls. Remove both modules from the parent pom and delete their directories. GeneRepository and VariantRepository both cascade to zero external refs after these deletions, so delete them too. DiseaseRepository still has two callers (DiseaseRibbonService and DiseaseESService) and remains for now.
DataExtractorRepository was only used by the deleted intermine extractor module. ResultTSVTranslator and EntityTSVTranslator are abstract bases with zero remaining subclasses.
…ugh ES Add DiseaseESDAO mirroring the GeneESDAO/VariantESDAO pattern and use it to replace two Neo4j call sites in the disease ribbon path: - DiseaseRibbonService.getShallowDiseaseTerm(id) -> DiseaseESDAO.getById for the disease-summary section descriptions (5 hardcoded categories + one per top-level slim term). - DiseaseESService.getDiseaseAnnotationHistogram now reads the pre-computed annotation.parentSlimIDs directly instead of asking DiseaseRibbonService to walk the DO ontology via Neo4j per annotation. The dynamic per-request histogram path is fully off Neo4j. The startup template build still uses DiseaseRepository.getAgrDoSlim and getDOParentTermIDs (the AGR slim list and the section assignment for each slim term); those will be migrated once we settle on a data source. DiseaseRibbonService.getAllParentIDs becomes unused and is deleted along with its helper. DiseaseRibbonService now takes a DiseaseESDAO in its constructor; DiseaseESService wires it via @PostConstruct since the DAOs are CDI-managed.
Replace the remaining Neo4j calls in the disease ribbon path with disease_summary ES lookups via DiseaseESDAO: - getAgrSlimDocs() returns disease_summary docs flagged with the doTerm.subsets keyword DO_AGR_slim (v0.49.2 of curation_api exposes subsets and a flat parentClosureIDs set on disease_summary). - DiseaseRibbonService now iterates those slim docs and uses each doc's parentClosureIDs to assign the slim to one of the five section categories, replacing diseaseRepository.getDOParentTermIDs. - DiseaseESService.populateDiseaseRibbonSummary now derives the slim ID list from getAgrSlimDocs instead of getAgrDoSlim. With DiseaseRepository gone, the Neo4jRepository abstract base has zero subclasses and is deleted too. agr_java_software no longer has any Neo4j repository classes. pom.xml curation.version bump to v0.49.2 is intentionally not included here -- v0.49.2 hasn't been published to Maven Central yet.
…lator Only 4 of 11 public methods on this translator had callers (all using curation_api Document types). Removed the 7 Neo4j-typed dead methods, their 4 private helpers, 2 commented-out helpers, and 1 dead overload. All 7 imports from org.alliancegenome.neo4j.entity drop with them. Also removed the unused diseaseTranslator field in GeneController.
Four entities had no references anywhere in the codebase (not even from other Neo4j entities): - DiseaseSummary - EntitySummary - HTPDataset - ParaAlgorithm The rest of the Neo4j entity graveyard is anchored by ~10 externally- imported types (Gene, Allele, Variant, AffectedGenomicModel, etc.) that hold the cluster together via same-package field references. Migrating those consumers off the Neo4j types is needed before further deletions.
Five production references were holding the dead island alive: - ESService.DOTerm.getAllOtherDiseaseTerms() -> moved constant to DiseaseRibbonSummary - GeneRESTInterface.PhenotypeAnnotation -> fixed stale @Schema to point at GenePhenotypeAnnotationDocument - VariantServiceHelper (only called by dead TranscriptLevelConsequence) - SourceServiceHelper (only called by dead DiseaseEntityJoin) - PhenotypeIT test references to GeneticEntity.CrossReferenceType enum -> inlined as string With those gone, all 47 remaining files under neo4j/entity/ were dead-island (no external imports). Deleted in one shot. Surviving in the neo4j package (genuinely live, just historical naming): - Neo4jEntity (still extended by AllianceReleaseInfo / ModFileMetadata / OntologyFileMetadata) - SpeciesType enum (used by ES services) - ReleaseSummary, AllianceReleaseInfo, ModFileMetadata, OntologyFileMetadata (release-info flow) - view/PublicView, view/BaseFilter (Jackson + pagination helpers)
These classes have no actual Neo4j coupling -- they're a Jackson @JSONVIEW marker and a pagination base filter that lived in org.alliancegenome.neo4j.view for historical reasons. Moved to org.alliancegenome.view. Updated 28 imports.
The ReleaseInfoService.getSummary() method was a no-op since the Neo4j repos were deleted -- it returned an empty ReleaseSummary with all setter lines commented out. UI doesn't call /releaseInfo/summary anywhere. Removed: - ReleaseInfoRESTInterface.getReleaseInfoSummary() - ReleaseInfoController.getReleaseInfoSummary() - ReleaseInfoService.getSummary() - ReleaseSummary, AllianceReleaseInfo, ModFileMetadata, OntologyFileMetadata - Neo4jEntity abstract base (no more subclasses) - DateConverter (only used by the three deleted @NodeEntity classes)
All translator + download-row classes lived under agr_java_core/.../core/translators/tdf/ but were only consumed by agr_api controllers -- zero internal users in agr_java_core. Moved all 11 files alongside the agr_api-resident BaseToTdfTranslator and sibling translators in agr_api/.../api/translators/tdf/, updated package declarations and imports.
Audit found 36 files in agr_java_core that were only consumed by agr_api code. Moved them all with git mv to preserve history, preserving the existing package paths (no source edits needed). Clusters moved: - JsonResultResponse + Request + Duration - SearchApiResponse + SearchResult - AggResult + AggDocCount - Disease ribbon: DiseaseRibbonSummary/Entity/Section/SubgroupSlim, SectionSlim, DiseaseRibbonService - SiteMap + SiteMapIndex + XMLURL + XMLURLSet + package-info - RestError: RestErrorException + RestErrorMessage + RestErrorExceptionMapper - ES DAOs: ESDAO (base) + AutoCompleteDAO + DiseaseESDAO + GeneESDAO + SearchDAO + VariantESDAO Standalone moves: - Pagination, FieldFilter, AutoCompleteResult, RelatedDataLink, AlleleVariantIndexService - Category, SpeciesType (verified agr_api-only after earlier neo4j cleanup) - BaseFilter (caught by compiler -- agr_api-only) Also deleted PresentationEntity.java (dead -- zero implementers, zero consumers). 14 empty directories cleaned up in agr_java_core including neo4j/ which fully dissolves task #8.
Renamed the legacy top-level packages in agr_api so everything now lives under org.alliancegenome.api.* instead of being scattered across cache.*, core.*, es.*, and view.*: - cache.repository.helper -> api.response (JsonResultResponse, Request, Duration) - core.api.service -> api.service (DiseaseRibbonService) - core.variant.service -> api.service (AlleleVariantIndexService) - core.exceptions -> api.exceptions (RestError*) - es.index + es.index.site.dao -> api.es.dao (ESDAO + 5 DAOs, flattened) - es.model.query -> api.es.query (Pagination, FieldFilter) - es.model.search -> api.es.search (7 search/agg classes) - view -> api.view (BaseFilter) SpeciesType left in neo4j.entity intentionally -- it's slated for deletion.
…e.core Renames: - api.entity, es.index, es.model -> core.document (15 + 1 + 2 = 18 ES doc classes merged; ESDocument joins the doc family) - es.index.site.schema -> core.es.schema - es.index.site.schema.settings -> core.es.schema.settings - es.rest.RestConfig -> core.config.RestConfig (RestConfig joins ConfigHelper in core.config) - es.util -> core.es.util - view -> core.view - core.helpers -> core.helper (singularize) - core.variant.converters -> core.variant.converter (singularize) Left alone: core.config (now also holds RestConfig), core.util. Also bumped curation.version to v0.49.2 -- now published on Maven Central.
SCRUM-6031 Create literature_search_result Elasticsearch document
…ghlighting - add literature_search_result to categoryFilters with authors and publicationYear facets - add abstractText, authors, citation and publicationYear to searchFields with boosts so reference search matches those fields - return abstractText, authors, citation, shortCitation and publicationYear in responseFields - highlight abstractText as a whole field and blacklist abstractText.standardText so the abstract carries a single full-field highlight - map publicationYear as text with a keyword subfield so the facet and filter work
SCRUM-6394 Set fields for reference search
Add an explicit mapping for subject.curie (text, with .keyword and .sort subfields) alongside the existing subject.primaryExternalId entry. agr_api already sends a bool.should clause on subject.curie.keyword for gene/annotation lookups, but the field is not mapped on either the stage or production cluster, so that clause matches zero documents and is dead weight on every such query. Matching currently happens entirely via subject.primaryExternalId.keyword. Mapping the field now means it exists in the index ahead of the data, so the clause becomes valid as soon as we start populating subject.curie with our own minted IDs. Follows the existing curie/primaryExternalId pairing used for object.curie, allele.taxon.species.genomeAssembly.curie and geneMolecularInteraction.geneAssociationSubject.curie. Generated mapping is byte-identical in shape to subject.primaryExternalId.
…urie feat: index subject.curie in site_index mapping
Populate referenceType on the literature search result document from the ABC category field, map it as text with a keyword subfield so it is aggregatable, and register it as the first facet for the literature search category. The ABC category value (Research_Article, Review_Article, Conference_Publication, Thesis, etc.) previously survived only inside literatureSummary.category in _source, which is unindexed, so it could not be searched, filtered or faceted. SCRUM-6467
Populate resourceTitle on the literature search result document from the ABC resource_title field, map it as text with a keyword subfield so it is aggregatable, and register it as a facet for the literature search category. Journal was previously only reachable as a substring of the citation field, so it could be matched by free-text search but never filtered on. resource_title is present on 95.7% of ABC references. SCRUM-6551
SCRUM-6555: add referenceType facet to literature search results
SCRUM-6551: add resourceTitle (journal) facet to literature search results
AGM phenotype annotations with no gene/allele reference (subject, inferred, or asserted) were never surfaced anywhere: PhenotypeAnnotationCurationIndexer populated agmMap but only ever used it to roll gene/allele associations into geneMap/alleleMap, and index() never built or indexed an AGM document at all -- so such annotations silently dropped out of the Phenotype download files (both JSON and TSV). Adds AGMPhenotypeAnnotationDocument (category "agm_phenotype_annotation"), mirroring the already-shipped AGMDiseaseAnnotationDocument, and wires PhenotypeAnnotationCurationIndexer.index() to build and index it via a new createAGMPhenotypeAnnotationDocuments(), mirroring DiseaseAnnotationCurationIndexer.createAGMDiseaseAnnotationDocuments(). No other changes needed: FileGeneratorConfig.Phenotype already lists "agm_phenotype_annotation" as an expected ES category and PhenotypeFileGenerator already branches on AGMPhenotypeAnnotation for TSV columns -- both were written anticipating this document type, they just never received any. AGMPhenotypeAnnotationService.getFiltered() already pulls the full public AGM phenotype annotation population (gene/allele-referenced or not) via findForPublic against the curation API, and there's no phenotype-specific ES field-mapping section in Mapping.java to update (unlike disease_annotation's), so no further indexing config is needed either. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
SCRUM-6445 index AGM-only phenotype annotations for public downloads
…ism gene counts (#1676) Co-authored-by: Joe Wang <dongzhuo.wang@gmail.com>
curation library version bump
…rie (#1679) Since curation v0.53 mints AGRKB curies on genes, Gene.getIdentifier() returns the AGRKB curie instead of the MOD id. The disease indexer keys its gene map by getIdentifier() and passed that key as subjectGene.primaryExternalId to /orthologygenerated/findForPublic, so every orthology lookup returned nothing and no via-orthology gene disease annotations were indexed (0 since the 2026-10-02 stage run, ~60K before). Use the focus gene's primaryExternalId for the orthology lookup. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
SCRUM-6682 reorder literature facets and list years newest first
SCRUM-6730 remove UniqueID column from Disease download TSV
SCRUM-6682 hide the literature author facet
| im.createSnapShot(repo, snapShotName, index_name); | ||
| String snapshotType = args.remove(0); | ||
| List<String> indexes = im.getIndexList(); | ||
| for(String index: indexes) { |
There was a problem hiding this comment.
🚫 [checkstyle] <com.puppycrawl.tools.checkstyle.checks.whitespace.WhitespaceAroundCheck> reported by reviewdog 🐶
'for' is not followed by whitespace.
| String snapshotType = args.remove(0); | ||
| List<String> indexes = im.getIndexList(); | ||
| for(String index: indexes) { | ||
| if(index.startsWith(snapshotType)) { |
There was a problem hiding this comment.
🚫 [checkstyle] <com.puppycrawl.tools.checkstyle.checks.whitespace.WhitespaceAroundCheck> reported by reviewdog 🐶
'if' is not followed by whitespace.
|
Review summary (stage → production release, 712 files / ~1.9k commits) This is too large for a line-by-line review, and almost all of it is commits that were already reviewed when they merged into
Worth hardening in
I found no other correctness problems in the parts I checked. |
No description provided.