[YoavLax]

What does this PR do?

Closes #1462 (step 1: hybrid retrieval via RRF). Refs #1417.

semantic_query ranked nodes by per-keyword min-cosine over the RI node vectors alone. On this repo's own index those vectors are strongly collapsed (mean cosine to the centroid is 0.981). As a result, ranking was close to noise, test code filled the top results, and natural-language phrases found nothing.

cbm_store_vector_search now fuses two channels with reciprocal rank fusion (k=60, lexical weight 5, vector weight 1), keyed on node id:

• Lexical: FTS5 bm25() over nodes_fts, using the same exact-name / case-insensitive / callable / type tiers as the BM25 query path.
• Vector: one composed, IDF-weighted query vector from token_vectors, scored with cbm_cosine_i8 over node_vectors. Query elements can be keywords or phrases. They are lowercased, stopwords are dropped, and words are split on camelCase/acronym boundaries.
• Test demotion: test symbols sort below production code unless the query mentions tests. A symbol counts as a test if it is in a test directory, has a test file name, is named test_* / Test*, or is a *_for_test hook.
• Deterministic ordering, independent of limit. The certification loop is removed.

There is no index format change. xxhash is no longer included in store.c. Docs are updated: the store.h contract, the README semantic search line, and the semantic_query type-error hint in mcp.c.

Behavior change: if no query word is known to token_vectors and nothing matches lexically, the result is now empty instead of the whole graph ranked against a random fallback vector.

Measurements

• Setup: 81 hand-labelled queries over this repository's own index, in two phrasings per intent (keywords and sentences). Linux x86_64 (Docker), -O2 build, baseline main @ 10ab613.
• The full table and caveats are in Feature: hybrid BM25+vector via RRF, and option to embed function bodies in semantic_query #1462 (comment).

semantic_query input	R@10 main → PR	MRR main → PR	tests in top 10 main → PR
keywords	0.099 → 0.901	0.047 → 0.675	39–55% → 0.7%
sentences	0.000 → 0.642	0.000 → 0.408	39–55% → 1.6%

The eval set is small, comes from a single repo, and was also used to tune the weights, so the numbers are optimistic.

Tests

• New: tool_search_graph_semantic_hybrid_ranking. It covers:
  • a sentence resolved through its known words;
  • a test twin demoted to a negative score, with scores monotone;
  • a test-oriented query that keeps the test first;
  • a vector-less type reachable lexically;
  • an all-unknown query returning no rows.
• Updated: tool_search_graph_semantic_store_error_fails_closed now creates token_vectors with the query token. The vector scan only runs when at least one query word is known, so this keeps the test exercising the forced scan failure.

Size

This is about 790 added / 230 removed lines, mostly the rewritten search block in store.c. That is over the 500-line guideline. It is submitted as one PR so the design can be reviewed as a whole; I'm happy to split it (test demotion and deterministic order / RRF fusion / phrase and camelCase handling) on request.

AI disclosure

This PR was written with an AI coding agent (GitHub Copilot) acting on my behalf. I reviewed it and am accountable for it; the commit carries my DCO sign-off.

Checklist

• Every commit is signed off (git commit -s) — required, CI rejects
  unsigned commits (DCO, see CONTRIBUTING.md)
• Tests pass locally (make -f Makefile.cbm test)
  I ran build/c/test-runner mcp only, not the full ASan/UBSan suite. All semantic tests pass.
  search_code_scan_live_child_deadline_is_bounded_and_fails_closed failed in 2 of 3 runs (a 100 ms subprocess timeout in search_code). This change does not touch that path, but I did not run it on unchanged main.
• Lint passes (make -f Makefile.cbm lint-ci)
  Only clang-format-18 was run, and it is clean on the changed ranges. clang-tidy and cppcheck were not run locally.
• New behavior is covered by a test (reproduce-first for bug fixes)

[github-actions]

Thanks for opening this — it has been seen, and it is queued.

This note is automated, but it is not a brush-off: it exists so you know where your PR stands instead of having to guess from silence.

Current review status: working through a backlog. 0.9.1-rc.1 is out, so the release freeze that held reviews is over — but it left a large queue of open pull requests behind it, and we are reading through them oldest-first. The background is in discussion #1144.

What that means for this PR, concretely:

• It will not be closed for inactivity. No stale bot touches pull requests here.
• It may still sit a while before a human reads it. That is on us, not on you.
• Older PRs are read first, so a recent one is not being skipped — it is behind a queue.

Things that will genuinely speed it up whenever review does happen:

• Keep it rebased on main — the tree is moving quickly right now, and a conflicting branch cannot be reviewed as the diff you intended.
• Get CI green, or say which failures you believe are pre-existing.
• Keep the change to one claim. Bundled features and refactors get split before they get merged, which costs you a round trip.
• Every commit needs a sign-off (git commit -s) — CI enforces DCO.

If this fixes a bug, a reproduction we can run is worth more than a description of the symptom.

Thanks for contributing, and sorry in advance for the wait.