Added 12 new fixtures across all three active projects:
- p04: 1 short/ambiguous case ('current status')
- p05: 1 CGH calibration case with cross-project bleed guard
- p06: 7 new fixtures targeting triage-promoted memories
(firmware interface, z-axis, cam encoder, telemetry rate,
offline design, USB SSD, Tailscale)
- Adversarial: cross-project-no-bleed (p04 query must not surface
p06 telemetry rate), no-project-hint (project memories must not
appear without a hint)
First run: 14/18 passing.
4 failures (p06-firmware-interface, p06-z-axis, p06-offline-design,
p06-tailscale) share the same root cause: long pre-existing p06
memories (530+ chars, confidence 0.9+) fill the 25% project-memory
budget before the query-relevant newly-promoted memories (shorter,
confidence 0.5) get a slot. Budget contention at equal overlap
score tiebroken by confidence. Day 7 ranking tweak target.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The fixture asserted 'GigaBIT M1' must not appear in a p05 pack,
but GigaBIT M1 is the mirror the interferometer measures, so its
name legitimately shows up in p05 source docs (CGH test setup
diagrams, AOM design input, etc.). Flagging it as bleed was false
positive.
Replace the assertion with genuinely p04-only material: the
'Option B' / 'conical back' architecture decision and a p06 tag,
neither of which has any reason to appear in a p05 configuration
answer.
Harness now passes 6/6 against live Dalidou at 38f6e52 — the
first clean baseline. Subsequent retrieval/ranking/ingestion
changes can be measured against this run.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
scripts/retrieval_eval.py walks a fixture file of project-hinted
questions, runs each against POST /context/build, and scores the
returned formatted_context against per-fixture expect_present and
expect_absent substring checklists. Exit 0 on all-pass, 1 on any
miss. Human-readable by default, --json for automation.
First live run against Dalidou at SHA 1161645: 4/6 pass. The two
failures are real findings, not harness bugs:
- p05-configuration FAIL: "GigaBIT M1" appears in the p05 pack.
Cross-project bleed from a shared p05 doc that legitimately
mentions the p04 mirror under test. Fixture kept strict so
future ranker tuning can close the gap.
- p05-vendor-signal FAIL: "Zygo" missing. The vendor memory exists
with confidence 0.9 but get_memories_for_context walks memories
in fixed order (effectively by updated_at / confidence), so lower-
ranked memories get pushed out of the per-project budget slice by
higher-confidence ones even when the query is specifically about
the lower-ranked content. Query-relevance ordering of memories is
the natural next fix.
Docs sync:
- master-plan-status.md: Phase 9 reflection entry now notes that
capture→reinforce runs automatically and project memories reach
the context pack, while extract remains batch/manual. First batch-
extract pass surfaced 1 candidate from 42 interactions — extractor
rule tuning is a known follow-up.
- next-steps.md: the 2026-04-11 retrieval quality review entry now
shows the project-memory-band work as DONE, and a new
"Reflection Loop Live Check" subsection records the extractor-
coverage finding from the first batch run.
- Both files now agree with the code; follow-up reviewers
(Codex, future Claude) should no longer see narrative drift.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
