Skip to content

feat(memory): make keyed remember_memory writes idempotent - #2583

Open
Doar999 wants to merge 10 commits into
oceanbase:masterfrom
Doar999:feat/2521-idempotent-remember-memory
Open

Doar999 wants to merge 10 commits into
oceanbase:masterfrom
Doar999:feat/2521-idempotent-remember-memory

Conversation

@Doar999

@Doar999 Doar999 commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Which issue or RFC does this PR close?

Closes #2521.

Rationale for this change

A client that times out on remember_memory cannot tell whether the write was committed, and a blind retry creates a duplicate memory. An optional idempotency key lets the caller retry the same creation safely.

What changes are included in this PR?

  • remember_memory accepts an optional idempotency_key (^[!-~]{1,256}$, compared byte for byte). The same rule applies over HTTP and the in-process SDK.
  • A retry with the same key and equivalent stored content replays the original creation: revision 1 with its creation-time state and changed: true. Replays are re-authorized and do not extend the key.
  • The same key with different content returns 409 idempotency_conflict with details.kind = "memory_idempotency_key" and writes nothing.
  • Keys are scoped per Scope and per principal (one shared namespace per Scope when access control is off) and live for a fixed 24 hours on the database clock. An expired key behaves as unused.
  • The receipt is written in the same transaction as the creation. Only SHA-256 digests of the namespace and of the RFC 8785 content are stored. Expired receipts are removed in bounded batches of 32 after commit.
  • Migration v1_5_2_001 adds pc_atomic_memory_idempotency for SQLite and the MySQL dialect; manifests and the release upgrade gate are updated.
  • OpenAPI, generated HTTP models, the Python client, MCP tool description, Desktop generated bindings, the shared Skill guidance, and English and Chinese docs describe the field. The Dify plugin keeps it hidden.
  • A failed cleanup is logged as atomic_memory.idempotency.cleanup_failed and never changes the write result.

Out of scope: deduplication across different keys, changes to expected_revision, automatic client retries, and a general idempotency framework.

Are there any user-facing changes?

Yes, additive. remember_memory gains an optional idempotency_key, and a new 409 idempotency_conflict response applies only to keyed requests. Requests without a key behave as before. Upgrading runs migration v1_5_2_001; no existing data is rewritten.

How was this change tested?

  • tests/e2e/test_remember_memory_idempotency.py (new, 15 cases): replay, creation-state replay after later changes, conflict, key validation over HTTP and the SDK, expected_revision rejection, expiry and bounded cleanup, concurrent requests in one process and across processes, and per-principal namespaces with re-authorization.
  • MCP, API contract, Desktop contract, migration revision naming, full Server migration, release version, and observability suites pass.
  • Real OceanBase CE 4.3.5.6 (the CI-pinned image): tests/builtin/persistence/test_mysql_migration_runner.py -k oceanbase 38 passed, 1 failed; tests/e2e/test_database_bootstrap.py -k oceanbase 5 passed; tests/e2e/test_atomic_memory_read_snapshot.py -k oceanbase 5 passed. A keyed-write scenario through the Server HTTP path (replay, conflict, 3 and 8 concurrent requests on one key, concurrent requests on an expired key) created exactly one memory in every round.
  • The one OceanBase migration failure, test_full_server_upgrade_with_renamed_check[oceanbase-True] (stale_ack on the no-op apply after a successful upgrade), fails identically on master at 63f021d6 and is not part of this change.
  • Desktop on Linux with Rust 1.95: cargo fmt --check, cargo clippy --all-targets -- -D warnings, and pnpm ipc:check pass; cargo test passed 130 cases before the profile existed, and the session suite (9 cases, including the two that needed the profile) passed after it was added. Desktop UI typecheck and tests pass.
  • Desktop compatibility profile sqlite-memory-idempotency-df79da6b-v1: desktop/tests/real_server.py --product-qualification passed anonymous loopback, static Bearer, HTTPS with a base path, and Provider modes against a wheel built from df79da6b; see desktop/VALIDATION.md.
  • Dify plugin contract tests, DSH diagnostics tests, generate_api.py --check, generate_js_operations.py --check, release_version.py --check, upgrade_release_gate.py, integration guidance and lifecycle checks, sync_skill.py --check, and ruff pass.

CI: dsh-package fails in setup validates native include trees observed by a real DSH startup. The same case fails on master (push run 42951538729 and scheduled run 42919036989) and on unrelated pull requests, so it is not addressed here.

Not done: the Skill pins are updated to the new guidance, but no new model evaluation run has been made for it.

AI usage statement

Built with Claude Code (Claude Fable 5.2): design analysis, implementation, tests, documentation, and verification runs. The author reviewed the decisions and results.

Add an optional idempotency_key to remember_memory. A retry with the same
key and content replays the original creation; different content returns
409 idempotency_conflict. Keys live 24 hours on the database clock in a
per-Scope, per-principal namespace and are stored as digests only.

Adds migration v1_5_2_001 for SQLite and MySQL-dialect backends, exposes
the field over HTTP, the SDK, and MCP, keeps it hidden in Dify, and
documents the behavior in both locales.

Closes oceanbase#2521

@hidb4ai hidb4ai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The keyed in-process SDK path can replay a creation after scope contribution has been revoked. Please apply the required current authorization before receipt lookup; the inline comment includes a reproduced case.

Comment on lines +467 to +469
stored = await find_receipt(self._database, key)
if stored is not None and stored.live:
return _replay_receipt(stored, key, scope_id)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Authorize scope contribution before replaying SDK creates

This early return skips _prepare_atomic_creations, where in-process calls normally reach authorize_creates. With real SQLite and built-in access control, I created a keyed memory as Alice, revoked her scope.contributor binding, then called runtime.atomic_memory.for_scope(scope).create(..., idempotency_key=key, context=alice_context) again: it succeeded and returned the original result. An unkeyed call and a new-key call both raised AccessDeniedError, and the equivalent HTTP retry returned 403. Alice retains ownership of the artifact, so the later exact-revision read checks only artifact.read and cannot enforce the required scope.contribute check. A different payload with the old key also returns BaseValueConflictError before authorization. Enforce current scope contribution before receipt lookup/conflict/replay for the in-process entry point too, as required by the RFC's authorization contract.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rechecked 122dd8679efdd9ca9a20396e51dd9267e3112134: the revoked-contributor reproduction is fixed, and a trusted local context now shares the anonymous key namespace correctly.

One P2 authorization case remains in the new preflight guard. It skips authorization whenever execution_context.access is None, whereas the domain policy rejects an explicit context unless trusted_local is true. Reproduced with real SQLite: create a memory locally with key known, then call memory.create(..., context=ArtifactSearchExecutionContext()). No key, an unused key, and known with the original content all raise AccessDeniedError; known with different content instead raises BaseValueConflictError. Thus an explicitly untrusted invocation can distinguish an occupied key before authorization. No memory body was returned and no additional memory was written.

Please preserve the domain's distinction between absent context, trusted local context, and untrusted context before any receipt lookup/conflict response. Add this case alongside the revoked-contributor regression.

An in-process keyed create returned a replay or a conflict before the
caller's creation authority was checked. Require it before the receipt
lookup so a revoked contributor learns nothing from an old key.
raise InvalidBaseAccessRequestError(
"idempotency_key", "must be 1 to 256 printable ASCII characters, no spaces"
)
principal = None if context is None or context.principal is None else context.principal

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Share the SDK key namespace when access control is disabled

This branch includes the principal from a valid ArtifactSearchExecutionContext(principal=alice, trusted_local=True) despite its absent access service, so submitting the same key/content without context and retrying with that context selects different receipts and creates duplicate memories, contrary to RFC 1809. Use an empty namespace principal whenever access control is disabled.

@AlexStocks AlexStocks added the 1.6.0 v1.6.0 release tracking label Nov 12, 2026
A context that named a principal without an access service put that
principal into the key namespace, so the same key and content created
a second memory. Use the empty principal unless access is enforced.

@hireview001 hireview001 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • CI failed: CI result (failure).
  • CI failed: Upgrade release gate (failure).
  • CI failed: CI preflight (failure).
  • CI failed: Main (failure). Failed steps: CI preflight / Check OpenAPI contract; Upgrade release gate / Require the selected parallel acceptance workflow; CI result / Require successful selected checks.

@@ -1,5 +1,5 @@
{
"contractSha256": "e1ff2e3ad1d6a09a9288c4c66e111a54c4979026e29806ff70543988b8e29220",
"contractSha256": "36eb4ebc989a34be1872cd220c5289df81d690ea57d9e5565d50d06046ac9a81",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Register qualification for the new Desktop contract

The new bundled digest matches neither a profile nor an authenticated-client mapping in compatibility.json, so check_expected disables business operations on saved connections and connect_local rejects local setup with CompatibilityUnverified. Qualify this exact contract and register its evidence so Desktop connection workflows remain usable.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

1.6.0 v1.6.0 release tracking

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(memory): define and implement idempotent remember_memory writes

4 participants