docs: record fully green retrieval harness

DEV-LEDGER.md

@@ -6,11 +6,11 @@
 
 ## Orientation
 
-- **live_sha** (Dalidou `/health` build_sha): `a87d984` (verified 2026-04-25T00:37Z post memory-ranking stabilization deploy; status=ok)
-- **last_updated**: 2026-04-25 by Codex (project_id metadata backfill complete; memory ranking stabilized)
-- **main_tip**: `a87d984`
-- **test_count**: 571 on `main`
-- **harness**: `19/20 PASS` on live Dalidou, 0 blocking failures, 1 known content gap (`p04-constraints`)
+- **live_sha** (Dalidou `/health` build_sha): `d3de9f6` (verified 2026-04-25T01:01Z post trusted-state ranking deploy; status=ok)
+- **last_updated**: 2026-04-25 by Codex (retrieval harness fully green)
+- **main_tip**: `d3de9f6`
+- **test_count**: 572 on `main`
+- **harness**: `20/20 PASS` on live Dalidou, 0 blocking failures, 0 known issues
 - **vectors**: 33,253
 - **active_memories**: 290 (`/admin/dashboard` 2026-04-24; note integrity panel reports a separate active_memory_count=951 and needs reconciliation)
 - **candidate_memories**: 0 (triage queue drained)
@@ -23,9 +23,9 @@
 - **capture_clients**: claude-code (Stop hook + cwd project inference), openclaw (before_agent_start + llm_output plugin, verified live)
 - **wiki**: http://dalidou:8100/wiki (browse), /wiki/projects/{id}, /wiki/entities/{id}, /wiki/search
 - **dashboard**: http://dalidou:8100/admin/dashboard (now shows pipeline health, interaction totals by client, all registered projects)
-- **active_track**: Engineering V1 Completion (started 2026-04-22). V1-0 landed (`2712c5d`). V1-A density gate CLEARED (784 active ≫ 100 target as of 2026-04-23). V1-A soak gate at day 5/~7 (F4 first run 2026-04-19; nightly clean 2026-04-19 through 2026-04-23; failures confined to the known p04-constraints content gap). Plan: `docs/plans/engineering-v1-completion-plan.md`. Resume map: `docs/plans/v1-resume-state.md`.
+- **active_track**: Engineering V1 Completion (started 2026-04-22). V1-0 landed (`2712c5d`). V1-A density gate CLEARED (784 active ≫ 100 target as of 2026-04-23). V1-A soak gate at day 5/~7 (F4 first run 2026-04-19; nightly clean 2026-04-19 through 2026-04-23; live harness is now fully green as of 2026-04-25). Plan: `docs/plans/engineering-v1-completion-plan.md`. Resume map: `docs/plans/v1-resume-state.md`.
 - **last_nightly_pipeline**: `2026-04-23T03:00:20Z` — harness 17/18, triage promoted=3 rejected=7 human=0, dedup 7 clusters (1 tier1 + 6 tier2 auto-merged), graduation 30-skipped 0-graduated 0-errors, auto-triage drained the queue (0 new candidates 2026-04-22T00:52Z run)
-- **open_branches**: `codex/retrieval-memory-ranking-stabilization` pushed and fast-forwarded into `main` as `a87d984`; no active unmerged code branch for this tranche.
+- **open_branches**: `codex/p04-constraints-state-gap` pushed and fast-forwarded into `main` as `d3de9f6`; no active unmerged code branch for this tranche.
 
 ## Active Plan
 
@@ -170,6 +170,8 @@ One branch `codex/extractor-eval-loop` for Day 1-5, a second `codex/retrieval-ha
 
 ## Session Log
 
+- **2026-04-25 Codex (p04 constraint gap closed; harness fully green)** Root-caused the remaining `p04-constraints` fixture: the `Zerodur` / `1.2` fact already existed in Trusted Project State (`requirement/key_constraints`), but project-state formatting was category/key ordered and then truncated to the 20% state budget, so contacts/decisions consumed the budget before the relevant requirement. Added query-relevance ranking for Trusted Project State entries before formatting/truncation, with regression coverage in `test_project_state_query_relevance_before_truncation`. Removed the fixture's `known_issue` lane so future p04 constraint regressions are blocking. Cleaned up a duplicate live requirement entry created during diagnosis by invalidating `requirement/mirror-blank-core-constraints`; canonical `requirement/key_constraints` remains active. Verified focused suite: 35 passed. Verified full local suite: 572 passed. Deployed `d3de9f67eaa08dfc5b2d86e8221b8c70fef266d3`; live exact p04 probe now surfaces `[REQUIREMENT] key_constraints` with `1.2` and `Zerodur`. Live retrieval harness: 20/20, 0 known issues, 0 blocking failures.
+
 - **2026-04-25 Codex (project_id backfill + retrieval stabilization closed)** Merged `codex/project-id-metadata-retrieval` into `main` (`867a1ab`) and deployed to Dalidou. Took Chroma-inclusive backup `/srv/storage/atocore/backups/snapshots/20260424T154358Z`, then ran `scripts/backfill_chunk_project_ids.py` per project; populated projects `p04-gigabit`, `p05-interferometer`, `p06-polisher`, `atomizer-v2`, and `atocore` applied cleanly for 33,253 vectors total, with 0 missing/malformed and an immediate final dry-run showing 33,253 already tagged / 0 updates. Post-backfill harness exposed p06 memory-ranking misses (`Tailscale`, `encoder`), so Codex shipped `4744c69` then `a87d984 fix(memory): widen query-time context candidates`. Full local suite: 571 passed. Live `/health` reports `a87d9845a8c34395a02890f0cf22aa7a46afaf62`, vectors=33,253, sources_ready=true. Live retrieval harness: 19/20, 0 blocking failures, 1 known issue (`p04-constraints` missing `Zerodur` / `1.2`). A repeat backfill dry-run after the code-only stabilization deploy was aborted after the one-off container ran too long; the live service stayed healthy and the earlier post-apply idempotency result remains the migration acceptance record. Dalidou HTTP push credentials are still not configured; this session pushed through the Windows credential path.
 
 - **2026-04-24 Codex (retrieval boundary deployed + project_id metadata tranche)** Merged `codex/audit-improvements-foundation` to `main` as `f44a211` and pushed to Dalidou Gitea. Took pre-deploy runtime backup `/srv/storage/atocore/backups/snapshots/20260424T144810Z` (DB + registry, no Chroma). Deployed via `papa@dalidou` canonical `deploy/dalidou/deploy.sh`; live `/health` reports build_sha `f44a2114970008a7eec4e7fc2860c8f072914e38`, build_time `2026-04-24T14:48:44Z`, status ok. Post-deploy retrieval harness: 20 fixtures, 19 pass, 0 blocking failures, 1 known issue (`p04-constraints`). The former blocker `p05-broad-status-no-atomizer` now passes. Manual p05 `context-build "current status"` spot check shows no p04/Atomizer source bleed in retrieved chunks. Started follow-up branch `codex/project-id-metadata-retrieval`: registered-project ingestion now writes explicit `project_id` into DB chunk metadata and Chroma vector metadata; retrieval prefers exact `project_id` when present and keeps path/tag matching as legacy fallback; added dry-run-by-default `scripts/backfill_chunk_project_ids.py` to backfill SQLite + Chroma metadata; added tests for project-id ingestion, registered refresh propagation, exact project-id retrieval, and collision fallback. Verified targeted suite (`test_ingestion.py`, `test_project_registry.py`, `test_retrieval.py`): 36 passed. Verified full suite: 556 passed in 72.44s. Branch not merged or deployed yet.

docs/current-state.md

@@ -1,10 +1,10 @@
 # AtoCore - Current State (2026-04-25)
 
-Update 2026-04-25: project-id chunk/vector metadata is deployed and backfilled.
-Live Dalidou is on `a87d984`; `/health` is ok with 33,253 vectors and sources
-ready. Live retrieval harness is 19/20 with 0 blocking failures and 1 known
-content gap (`p04-constraints` missing `Zerodur` / `1.2`). Full local suite:
-571 passed.
+Update 2026-04-25: project-id chunk/vector metadata is deployed and backfilled,
+and the final p04 Trusted Project State budget/ranking gap is closed. Live
+Dalidou is on `d3de9f6`; `/health` is ok with 33,253 vectors and sources
+ready. Live retrieval harness is 20/20 with 0 blocking failures and 0 known
+issues. Full local suite: 572 passed.
 
 The project-id backfill was applied per populated project after a
 Chroma-inclusive backup at
@@ -76,10 +76,10 @@ Last nightly run (2026-04-19 03:00 UTC): **31 promoted · 39 rejected · 0 needs
 | 7G | Re-extraction on prompt version bump | pending |
 | 7H | Chroma vector hygiene (delete vectors for superseded memories) | pending |
 
-## Known gaps (honest, refreshed 2026-04-24)
+## Known gaps (honest, refreshed 2026-04-25)
 
 1. **Capture surface is Claude-Code-and-OpenClaw only.** Conversations in Claude Desktop, Claude.ai web, phone, or any other LLM UI are NOT captured. Example: the rotovap/mushroom chat yesterday never reached AtoCore because no hook fired. See Q4 below.
 2. **Project-scoped retrieval guard is deployed and passing.** Explicit `project_id` chunk/vector metadata is now present in SQLite and Chroma for the 33,253-vector corpus. Retrieval prefers exact metadata ownership and keeps path/tag matching as a legacy fallback.
 3. **Human interface is useful but not yet the V1 Human Mirror.** Wiki/dashboard pages exist, but the spec routes, deterministic mirror files, disputed markers, and curated annotations remain V1-D work.
-4. **Harness known issue:** `p04-constraints` wants "Zerodur" and "1.2"; live retrieval surfaces related constraints but not those exact strings. Treat as content/state gap until fixed.
+4. **Harness is currently green.** The former `p04-constraints` known issue is closed; query-relevant Trusted Project State entries now rank before state-budget truncation.
 5. **Formal docs lag the ledger during fast work.** Use `DEV-LEDGER.md` and `python scripts/live_status.py` for live truth, then copy verified claims into these docs.

docs/master-plan-status.md

@@ -133,7 +133,7 @@ deferred from the shared client until their workflows are exercised.
 
 ## What Is Real Today (updated 2026-04-25)
 
-- canonical AtoCore runtime on Dalidou (`a87d984`, deploy.sh verified)
+- canonical AtoCore runtime on Dalidou (`d3de9f6`, deploy.sh verified)
 - 33,253 vectors across 6 registered projects, with explicit `project_id`
   metadata backfilled into SQLite and Chroma after snapshot
   `/srv/storage/atocore/backups/snapshots/20260424T154358Z`
@@ -154,10 +154,10 @@ deferred from the shared client until their workflows are exercised.
 - query-relevance memory ranking with overlap-density scoring and widened
   query-time candidate pools so older exact-intent project memories can rank
   ahead of generic high-confidence notes
-- retrieval eval harness: 20 fixtures; current live has 19 pass, 1 known
-  content gap, and 0 blocking failures after the project-id backfill and
-  memory-ranking stabilization deploy
-- 571 tests passing on `main`
+- query-relevance Trusted Project State ranking before state-budget truncation
+- retrieval eval harness: 20 fixtures; current live has 20 pass, 0 known
+  issues, and 0 blocking failures
+- 572 tests passing on `main`
 - nightly pipeline: backup → cleanup → rsync → OpenClaw import → vault refresh → extract → triage → **auto-promote/expire** → weekly synth/lint → **retrieval harness** → **pipeline summary to project state**
 - Phase 10 operational: reinforcement-based auto-promotion (ref_count ≥ 3, confidence ≥ 0.7) + stale candidate expiry (14 days unreinforced)
 - pipeline health visible in dashboard: interaction totals by client, pipeline last_run, harness results, triage stats
@@ -178,10 +178,7 @@ These are the current practical priorities.
    Target: 100+ active memories.
 3. **Multi-model triage** (Phase 11 entry) — switch auto-triage to a
    different model than the extractor for independent validation
-4. **Fix p04-constraints harness failure** — retrieval doesn't surface
-   "Zerodur" for p04 constraint queries. Investigate if it's a missing
-   memory or retrieval ranking issue.
-5. **Fix Dalidou Git credentials** — the host checkout can fetch but cannot
+4. **Fix Dalidou Git credentials** — the host checkout can fetch but cannot
    push to Gitea over HTTP in non-interactive SSH sessions. Prefer switching
    the deploy checkout to a Gitea SSH key; PAT-backed `credential.helper store`
    is the fallback.