An agent that reads DataHub, contains only the real blast radius, and writes the verified incident back.
When an upstream schema breaks, Schema Dam turns live DataHub context into bounded runtime action. It protects only affected products, validates a repair, waits for exact-artifact approval, and leaves the verified recovery in DataHub for the next operator or agent.
Watch the demo (2:50) · Try the interactive simulator
See the agent loop · Run with DataHub · Fast local proof · Design
A Schema Registry can check compatibility. It cannot tell an operator which downstream products actually use the changed field. Schema Dam uses DataHub to close that operational gap.
| READ from DataHub | ACT on the runtime | WRITE BACK to DataHub |
|---|---|---|
| Schema, column lineage, asset identity, ownership | Contain affected inputs, activate fallbacks, shadow-test, replay after approval | One record with impact, owners, approval evidence, replay, and resolution — a native DATA_SCHEMA incident where privileges allow, otherwise resolved tags plus a summary document |
3 affected contained · 5 unaffected kept running · 10 consumer-events replayed · 0 production checkpoint movement before approval
The agent reads context, explains the plan, and invokes bounded tools. A deterministic controller owns gates, checkpoints, and replay; a human owns the release decision. DataHub carries the durable handoff, never runtime authority.
In the synthetic ecommerce demo, a producer changes category_code to
category.code. DataHub lineage identifies three affected products while five
others continue normally.
| Product boundary | Incident behavior |
|---|---|
| Category Analytics | Pause input; serve last-good output |
| Inventory Forecast | Pause Orders and companion Product input; serve last-good output |
| Recommendation | Pause both inputs; activate the configured fallback |
| Five unaffected consumers | Keep processing |
Accepted events stay in the append-only log. After approval, Schema Dam replays the affected backlog in order, verifies recovery, resumes primary serving, and updates the same DataHub incident.
Requires Python 3.11+, uv, Docker, and a local
DataHub quickstart.
git clone https://github.com/noooey/schema-dam.git
cd schema-dam
uv tool install acryl-datahub
datahub docker quickstart
export DATAHUB_GMS_URL=http://localhost:8080
uv sync --extra datahub
uv run python scripts/seed_datahub.py
uv run schema-dam-demo contain --reset
uv run schema-dam-agent prepare --trace generated/agent-trace.jsonThe final command reads live responses from the official DataHub MCP Server, reconciles them with the persisted containment plan, tests the candidate, and must stop here:
Incident READY_TO_RELEASE
Candidate SHA <full SHA-256>
Active SHA -
Approval WAITING
There is no automatic release. After a separate exact-digest approval, the repository Skill coordinates promotion, ordered replay, product-health checks, resume, and verified DataHub lifecycle write-back. The resulting incident lets the next operator or agent inherit the affected assets, owners, decision, artifact digest, approval, replay evidence, and resolution.
Follow the agent workflow or judge quickstart for the complete path.
For a fast deterministic proof without a live DataHub instance:
uv sync
uv run schema-dam-webhook-e2eLineage targets 3 affected
Unaffected 5 consumers kept processing
Fallback recommendation_consumer:FALLBACK_ACTIVE
Replay 10 consumer-events · checkpoint 15/15
Result RESOLVED · gate OPEN · order preserved TRUE
This synthetic HTTP-boundary run proves the controller contract; it is not presented as a live DataHub or agent recording.
| Boundary | Guarantee |
|---|---|
| Uncertain impact | Empty, stale, truncated, or unbound lineage fails closed |
| Selective containment | Only affected input boundaries close; companion inputs are protected together |
| Shadow validation | Candidate tests cannot advance production checkpoints |
| Release | Approval must match the full canary-tested SHA-256 |
| Replay | Recovery is ordered and at-least-once; sinks must be transactional or idempotent |
| Authority | DataHub mirrors verified state; the controller remains runtime authority |
Schema Dam is a control-plane reference implementation, not a universal adapter generator. The reproducible demo uses synthetic ecommerce data, SQLite, and eight Python consumers; tested integrations, deployment assumptions, and remaining production work are tracked in Production readiness.
- HTTP control-boundary transcript
- Generated compatibility adapter and tests
- Registry-v3 evidence validator
The dedicated performance soak and target-environment game day remain open and are not claimed as completed evidence.
- Start: Quickstart, CLI, agent runbook, evidence, Devpost
- Core: Design, frontier, workflows, artifacts, transformation, lineage trust
- Operations: Adoption, intake, mitigation, operator workflow, readiness, DR
- Runtimes: Kafka/Redpanda, Airflow, Dagster, dbt Cloud, Kafka Connect/CDC, and multi-region control
- Product: DataHub incidents, semantic contracts, governance, usefulness, roadmap
