Repository navigation
Index content-element id, so a document's ingest seeks the nodes it writes - #78
Merged
Merged
Conversation
…rites
Every save, parent link and next-chunk link the GOM store writes is a
MERGE or MATCH of `(n:<labels> {id: ...})`, and no index served `id`
under ContentElement or Chunk. Each such statement scanned its label, so
one document's ingest cost the size of the whole store, once per
section, chunk and link (320 label scans for a 40-section document). On
an engine that holds its write lock for a statement, every reader waited
out each scan: an appliance ingesting ~450 documents saw saved views go
from 0.4 s to 112-124 s.
provision() now ensures RANGE indexes on ContentElement(id) and
Chunk(id). A range index rather than a uniqueness constraint, so it asks
nothing of the data already stored.
GraphProvisioner no longer recreates an existing range index under the
convention name: only vector and full-text indexes are read by name, and
the existing one may belong to a uniqueness constraint (Chunk(id)),
which cannot be dropped, so the recreate failed provisioning outright.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F6e7yYAKD1C2yMoRqM7Neb
jasperblues
approved these changes
Nov 16, 2026
jasperblues
left a comment
Collaborator
There was a problem hiding this comment.
The diagnosis and the fix are right: every content save and link MERGEs by id, and with no index under those labels each one scanned the label. Merging. A follow-up comment has two notes for later.
Collaborator
|
Merged. Thanks for tracking this down. Two follow-ups:
|
jasperblues
added a commit
that referenced
this pull request
Nov 16, 2026
The range indexes on ContentElement(id) and Chunk(id) that #78 added were listed by hand in GraphObjectManagerStore.provision(). They now sit beside the @nodeid they serve, as @nodeid @RangeIndex on ContentElementFragment and ChunkPlaceFragment, and provision() reads them off those fragments with SchemaCatalog.fromFragments. The indexes provisioned are the same. ContentIdIndexesFollowTheModelTest checks that the derived list equals the old hand list, and scans every @NodeFragment in the model package: any fragment labelled ContentElement or Chunk whose @nodeid no declared index serves fails the test, so a new fragment can't leave its saves unindexed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Symptom
An Embabel appliance (image sha-8241100, 15 Nov 2026) ingesting ~450 documents from a realm
documentssource: every saved-view invoke (e.g.POST /api/v1/views/DiseaseCard/invoke, normally 0.4 s) took 112–124 s, container CPU near idle. The ingest thread was inGraphObjectManagerStore.linkToParents→gom.saveAll(ParentLinkView…), inside the in-process engine'smergeClause→nodeFits, reading properties from page files node by node while holding the exclusive statement lock. Every reader queued behind it.Cause
Each per-link statement Drivine 0.5.0 generates for
ParentLinkView(abridged,__drivine.*bookkeeping elided):The pattern does carry a label. What was missing is an index:
provision()declared none onidunderContentElementorChunk. So the MERGE had nothing to seek on and scanned the label. Other statements have the same shape: the section, chunk and document saves (MERGE (n:LeafSection:ContentElement {id}),MERGE (n:Chunk:ContentElement {id}),MERGE (n:Document:ContentRoot:ContentElement {id})) and the next-chunk link (MERGE (n:Chunk {id: row.id})). Each scans too. One 40-section document into a store holding 10k content nodes made 320 label scans (ContentElement 80, LeafSection 80, Chunk 159, Document 1). On Neo4j the same statements are label scans as well.Fix
provision()now ensuresRANGEindexes onContentElement(id)andChunk(id)(the configured chunk label). Every save carriesContentElement, and the next-chunk link namesChunkalone. I chose a range index over a uniqueness constraint because it asks nothing of the data a store already holds, so it can't fail at startup.GraphProvisionerno longer recreates an existing range index under the convention name. Only vector and full-text indexes are looked up by name. An existingChunk(id)index may belong to a uniqueness constraint, which can't be dropped, and the recreate then failed provisioning outright. The new test reproduces that case.Tests
Neo4jContentElementIdIndexTest:__Entity__(id)is indexed.Chunk(id)is already uniquely constrained provisions, and the constraint survives.mvn clean install): 491 tests, 0 failures.🤖 Generated with Claude Code
https://claude.ai/code/session_01F6e7yYAKD1C2yMoRqM7Neb