Skip to content

About

Keep unaffected data products running when schemas break.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

Schema Dam

An agent that reads DataHub, contains only the real blast radius, and writes the verified incident back.

When an upstream schema breaks, Schema Dam turns live DataHub context into bounded runtime action. It protects only affected products, validates a repair, waits for exact-artifact approval, and leaves the verified recovery in DataHub for the next operator or agent.

CI Python DataHub License

Watch the demo (2:50) · Try the interactive simulator

See the agent loop · Run with DataHub · Fast local proof · Design

Schema Dam keeps affected features in safe continuity modes while unaffected features continue

The agent loop

A Schema Registry can check compatibility. It cannot tell an operator which downstream products actually use the changed field. Schema Dam uses DataHub to close that operational gap.

READ from DataHub ACT on the runtime WRITE BACK to DataHub
Schema, column lineage, asset identity, ownership Contain affected inputs, activate fallbacks, shadow-test, replay after approval One record with impact, owners, approval evidence, replay, and resolution — a native DATA_SCHEMA incident where privileges allow, otherwise resolved tags plus a summary document

3 affected contained · 5 unaffected kept running · 10 consumer-events replayed · 0 production checkpoint movement before approval

The agent reads context, explains the plan, and invokes bounded tools. A deterministic controller owns gates, checkpoints, and replay; a human owns the release decision. DataHub carries the durable handoff, never runtime authority.

Demo scenario

In the synthetic ecommerce demo, a producer changes category_code to category.code. DataHub lineage identifies three affected products while five others continue normally.

Product boundary Incident behavior
Category Analytics Pause input; serve last-good output
Inventory Forecast Pause Orders and companion Product input; serve last-good output
Recommendation Pause both inputs; activate the configured fallback
Five unaffected consumers Keep processing

Accepted events stay in the append-only log. After approval, Schema Dam replays the affected backlog in order, verifies recovery, resumes primary serving, and updates the same DataHub incident.

Run with DataHub

Requires Python 3.11+, uv, Docker, and a local DataHub quickstart.

git clone https://github.com/noooey/schema-dam.git
cd schema-dam
uv tool install acryl-datahub
datahub docker quickstart
export DATAHUB_GMS_URL=http://localhost:8080
uv sync --extra datahub
uv run python scripts/seed_datahub.py
uv run schema-dam-demo contain --reset
uv run schema-dam-agent prepare --trace generated/agent-trace.json

The final command reads live responses from the official DataHub MCP Server, reconciles them with the persisted containment plan, tests the candidate, and must stop here:

Incident      READY_TO_RELEASE
Candidate SHA <full SHA-256>
Active SHA    -
Approval      WAITING

There is no automatic release. After a separate exact-digest approval, the repository Skill coordinates promotion, ordered replay, product-health checks, resume, and verified DataHub lifecycle write-back. The resulting incident lets the next operator or agent inherit the affected assets, owners, decision, artifact digest, approval, replay evidence, and resolution.

Follow the agent workflow or judge quickstart for the complete path.

Run the local proof

For a fast deterministic proof without a live DataHub instance:

uv sync
uv run schema-dam-webhook-e2e
Lineage targets   3 affected
Unaffected        5 consumers kept processing
Fallback          recommendation_consumer:FALLBACK_ACTIVE
Replay            10 consumer-events · checkpoint 15/15
Result            RESOLVED · gate OPEN · order preserved TRUE

This synthetic HTTP-boundary run proves the controller contract; it is not presented as a live DataHub or agent recording.

Control contract

Boundary Guarantee
Uncertain impact Empty, stale, truncated, or unbound lineage fails closed
Selective containment Only affected input boundaries close; companion inputs are protected together
Shadow validation Candidate tests cannot advance production checkpoints
Release Approval must match the full canary-tested SHA-256
Replay Recovery is ordered and at-least-once; sinks must be transactional or idempotent
Authority DataHub mirrors verified state; the controller remains runtime authority

Scope and evidence

Schema Dam is a control-plane reference implementation, not a universal adapter generator. The reproducible demo uses synthetic ecommerce data, SQLite, and eight Python consumers; tested integrations, deployment assumptions, and remaining production work are tracked in Production readiness.

The dedicated performance soak and target-environment game day remain open and are not claimed as completed evidence.

Documentation

License

Apache License 2.0

About

Keep unaffected data products running when schemas break.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages