Cache-root admission compares path strings: moving the cache dir + symlink while the daemon runs leaves every new client permanently refused (silent MCP "Request timed out") #2788

Summary

While the resident daemon was running, I moved the contents of ~/.cache/codebase-memory-mcp to another disk (/mnt/ssassd/cache/codebase-memory-mcp) and left a symlink at the old location. From that moment on, every new CBM client is refused by the daemon's cache-root admission check and surfaces this to the user only as a silent MCP Request timed out. The daemon itself stays alive and idle forever, so the outage never self-heals.

This report documents the root cause (with hard evidence), and four findings: the three defects — Defect 1 (identity primitive is path-string based; two candidate replacements are laid out below, choice left to the maintainer), Defect 2 (failure surfaces as a silent timeout instead of the actionable error), Defect 3 (no self-healing or retirement path) — plus a fourth, separate item: an internal canonicalization inconsistency that is bug-level regardless of the identity primitive chosen (reported separately below, distinct from the three defects).

Environment

• codebase-memory-mcp 0.12.1 (AUR codebase-memory-mcp-bin 0.12.1-2), Linux x86_64 (CachyOS)
• Architecture in play: stdio frontend + per-user resident daemon (/tmp/cbm-daemon-<uid>/) with version-cohort admission (version and cache root must match)
• Clients affected: OpenCode MCP sessions, session hooks (hook-augment), index workers

What actually happens

The admission handshake compares a fingerprint of the cache root. That fingerprint is the sha256 of the path string:

sha256("/home/vincenty/.cache/codebase-memory-mcp") = dd2b3d9737d1ef6d…   ← daemon's active_cache
sha256("/mnt/ssassd/cache/codebase-memory-mcp")      = f18a2ac03a5c0ebc…   ← requested_cache of every new client

Timeline (all timestamps same day):

1. 00:13 — daemon self-starts. Its env carries CBM_CACHE_DIR=/home/vincenty/.cache/codebase-memory-mcp (verified via /proc/<pid>/environ); at this time the path is still a real directory (no symlink), so its canonical form is itself. Daemon log: daemon.start version=0.12.1 pid=1984 cache_fingerprint=dd2b3d97….
2. 04:56 — cache dir moved to /mnt/ssassd/cache/codebase-memory-mcp, symlink left at ~/.cache/codebase-memory-mcp.
3. 05:16 onward — every new process (hooks, index workers, fresh MCP frontends) resolves the default cache path through the symlink to its realpath and requests cohort admission with fingerprint f18a2ac0…. The daemon refuses: {"event":"daemon.version_conflict","reason":"cache_root", …} — 89 entries in daemon-conflicts.ndjson over ~5.5 hours, all with identical active_version/requested_version (0.12.1) and identical build hashes; the cache hash is the only difference.
4. Client-visible symptom: OpenCode shows MCP server: codebase-memory-mcp — Request timed out. Session hooks log hook-augment: deadline_exceeded ms=2000 repeatedly (hook-augment-timeouts.log).

Data was never at risk — file access through the symlink is transparent and the daemon keeps reading/writing the same physical directory. The failure is purely in the identity comparison: the same physical directory gets two different identities because its canonical path string changed when the symlink was introduced.

An index worker even prints the exact, actionable diagnosis — but only into a log file deep inside the cache dir:

CBM index worker could not start: CBM could not start because the active account daemon uses a different cache directory (active cache dd2b3d97…; requested cache f18a2ac0…). Close all CBM sessions and commands, then retry with one consistent CBM_CACHE_DIR.

Reproduction

1. Start any CBM client so the resident daemon self-starts with the default cache dir (a real directory, no symlink).
2. mv ~/.cache/codebase-memory-mcp /mnt/elsewhere/ && ln -s /mnt/elsewhere/codebase-memory-mcp ~/.cache/codebase-memory-mcp
3. Start any new CBM client (or trigger a session hook).

Expected: clients and daemon recognize the same physical cache directory through the symlink and continue — or at minimum fail loudly with the actionable message above.
Actual: daemon.version_conflict reason=cache_root per request, client retries until the host's MCP deadline, user sees only "Request timed out".

Findings

Defect 1. Path-string-as-identity is the wrong identity primitive (design choice — maintainer to decide)

The handshake treats "which directory is this" as "what is the canonical path string". That identity is not stable across legitimate filesystem changes (move + symlink, bind mount, parent rename). Two stronger primitives, either of which makes the move+symlink case invisible:

• (a) inode/device identity: stat() the cache root; st_dev+st_ino identifies the physical directory regardless of path. Both sides stat through whatever path they hold and compare.
• (b) ID file in the cache: write a UUID file into the cache dir on first use; the handshake reads the ID through each side's own path and compares IDs.

Which one to adopt (or another approach with the same property) is left to the maintainer; the point is that the identity should belong to the directory, not to the spelling of the path used to reach it.

Defect 2. The failure surfaces as a silent timeout instead of the actionable error that already exists

The message "active daemon uses a different cache directory … retry with one consistent CBM_CACHE_DIR" is exactly what a user needs, but it never reaches the MCP client — frontends retry quietly until the host's request deadline fires. Please propagate the refusal reason to the MCP client as the tool/initialize error (or at least to stderr) so a misconfigured pair is diagnosable in seconds instead of requiring log spelunking under the cache dir.

Defect 3. No self-healing or retirement path

After the mismatch, the daemon (with zero admitted clients left, and every arriving request conflicting) stays alive indefinitely, and frontends retry indefinitely — a permanent deadlock until a human kills the daemon. A graceful daemon-restart/retire command, or automatic renegotiation when the daemon sees sustained cache-root conflicts and no live admitted clients, would bound the outage.

4. Additional bug, separate from the three defects above: the two sides of the same comparison canonicalize differently

• The daemon pins the string given at spawn time (the spawning frontend writes CBM_CACHE_DIR into the daemon env; before the move, realpath == the literal string, so both agree at spawn).
• New clients resolve realpath per connection and present that.

Same binary, two different canonicalization algorithms on the two sides of one equality check — so the same directory legitimately gets two identities at two points in time. Even keeping path strings as the identity, both sides should canonicalize identically (live) at handshake time; with Defect 1 adopted (stable directory identity), this class of split cannot occur at all.

Workaround (user side)

Kill the stale --cbm-daemon-internal process and every stale MCP frontend still holding flock on cbm-version-cohort-lifetime-v1.lock (check with fuser; they keep the old cohort alive and block any new daemon), then restart every host application (editor/agent) that had connected to the MCP server — apps that observed the timeouts do not recover or respawn their MCP child processes on their own. In practice the only reliable one-shot recovery was rebooting the machine. Optionally pin one stable CBM_CACHE_DIR everywhere to avoid the split in the first place. (See the addendum comment for the measured recovery sequence.)

Acknowledgment

The conservative admission check itself is sound — it protects cache coherence (and has caught real version-skew incidents, e.g. 0.11 daemon vs 0.12 frontends). These findings are about the identity primitive it compares, how its refusal is surfaced, and its lack of a recovery path — not about refusing being wrong.