Skip to content

Index content-element id, so a document's ingest seeks the nodes it writes - #78

Merged
jasperblues merged 1 commit into
mainfrom
content-element-id-index
Nov 16, 2026
Merged

jasperblues merged 1 commit into
mainfrom
content-element-id-index

Conversation

@johnsonr

Copy link
Copy Markdown
Contributor

Symptom

An Embabel appliance (image sha-8241100, 15 Nov 2026) ingesting ~450 documents from a realm documents source: every saved-view invoke (e.g. POST /api/v1/views/DiseaseCard/invoke, normally 0.4 s) took 112–124 s, container CPU near idle. The ingest thread was in GraphObjectManagerStore.linkToParents → gom.saveAll(ParentLinkView…), inside the in-process engine's mergeClause → nodeFits, reading properties from page files node by node while holding the exclusive statement lock. Every reader queued behind it.

Cause

Each per-link statement Drivine 0.5.0 generates for ParentLinkView (abridged, __drivine.* bookkeeping elided):

MERGE (n:ContentElement {id: $p0_id}) …
UNWIND $p1_rows AS row
MERGE (n:ContentElement {id: row.id}) SET n += row.props …
OPTIONAL MATCH (_r0)-[x:HAS_PARENT]->(n)
MERGE (_r0)-[r:HAS_PARENT]->(n) …

The pattern does carry a label. What was missing is an index: provision() declared none on id under ContentElement or Chunk. So the MERGE had nothing to seek on and scanned the label. Other statements have the same shape: the section, chunk and document saves (MERGE (n:LeafSection:ContentElement {id}), MERGE (n:Chunk:ContentElement {id}), MERGE (n:Document:ContentRoot:ContentElement {id})) and the next-chunk link (MERGE (n:Chunk {id: row.id})). Each scans too. One 40-section document into a store holding 10k content nodes made 320 label scans (ContentElement 80, LeafSection 80, Chunk 159, Document 1). On Neo4j the same statements are label scans as well.

Fix

  • provision() now ensures RANGE indexes on ContentElement(id) and Chunk(id) (the configured chunk label). Every save carries ContentElement, and the next-chunk link names Chunk alone. I chose a range index over a uniqueness constraint because it asks nothing of the data a store already holds, so it can't fail at startup.
  • GraphProvisioner no longer recreates an existing range index under the convention name. Only vector and full-text indexes are looked up by name. An existing Chunk(id) index may belong to a uniqueness constraint, which can't be dropped, and the recreate then failed provisioning outright. The new test reproduces that case.

Tests

  • Neo4jContentElementIdIndexTest:
    • provisioning creates both indexes. This fails before the fix: only __Entity__(id) is indexed.
    • a store whose Chunk(id) is already uniquely constrained provisions, and the constraint survives.
  • Full suite (mvn clean install): 491 tests, 0 failures.
  • In embabel/me, against this build (companion PR): one ingest into a 10k-node store makes 0 content label scans, down from 320. A read issued during an ingest, against a store whose label scans each cost 300 ms, never waits as long as one scan.

🤖 Generated with Claude Code

https://claude.ai/code/session_01F6e7yYAKD1C2yMoRqM7Neb

…rites

Every save, parent link and next-chunk link the GOM store writes is a
MERGE or MATCH of `(n:<labels> {id: ...})`, and no index served `id`
under ContentElement or Chunk. Each such statement scanned its label, so
one document's ingest cost the size of the whole store, once per
section, chunk and link (320 label scans for a 40-section document). On
an engine that holds its write lock for a statement, every reader waited
out each scan: an appliance ingesting ~450 documents saw saved views go
from 0.4 s to 112-124 s.

provision() now ensures RANGE indexes on ContentElement(id) and
Chunk(id). A range index rather than a uniqueness constraint, so it asks
nothing of the data already stored.

GraphProvisioner no longer recreates an existing range index under the
convention name: only vector and full-text indexes are read by name, and
the existing one may belong to a uniqueness constraint (Chunk(id)),
which cannot be dropped, so the recreate failed provisioning outright.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F6e7yYAKD1C2yMoRqM7Neb
@johnsonr
johnsonr requested a review from jasperblues November 15, 2026 21:52

@jasperblues jasperblues left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The diagnosis and the fix are right: every content save and link MERGEs by id, and with no index under those labels each one scanned the label. Merging. A follow-up comment has two notes for later.

@jasperblues
jasperblues merged commit de132c1 into main Nov 16, 2026
8 checks passed
@jasperblues

Copy link
Copy Markdown
Collaborator

Merged. Thanks for tracking this down. Two follow-ups:

jasperblues added a commit that referenced this pull request Nov 16, 2026
The range indexes on ContentElement(id) and Chunk(id) that #78 added were
listed by hand in GraphObjectManagerStore.provision(). They now sit beside
the @nodeid they serve, as @nodeid @RangeIndex on ContentElementFragment
and ChunkPlaceFragment, and provision() reads them off those fragments
with SchemaCatalog.fromFragments. The indexes provisioned are the same.

ContentIdIndexesFollowTheModelTest checks that the derived list equals the
old hand list, and scans every @NodeFragment in the model package: any
fragment labelled ContentElement or Chunk whose @nodeid no declared index
serves fails the test, so a new fragment can't leave its saves unindexed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants