diff --git a/DEV-LEDGER.md b/DEV-LEDGER.md index 6ab21f9..9c1bb4f 100644 --- a/DEV-LEDGER.md +++ b/DEV-LEDGER.md @@ -7,16 +7,16 @@ ## Orientation - **live_sha** (Dalidou `/health` build_sha): `2b86543` (verified 2026-04-23T15:20:53Z post-R14 deploy; status=ok) -- **last_updated**: 2026-04-23 by Claude (R14 squash-merged + deployed; Orientation refreshed) +- **last_updated**: 2026-04-24 by Codex (audit-improvements foundation branch; live status refreshed) - **main_tip**: `2b86543` -- **test_count**: 548 (547 + 1 R14 regression test) -- **harness**: `17/18 PASS` on live Dalidou (p04-constraints expects "Zerodur" — known content gap, not regression; consistent since 2026-04-19) +- **test_count**: 553 +- **harness**: `18/20 PASS` on live Dalidou plus 1 known content gap and 1 blocking project-bleed guard pending deploy of this branch - **vectors**: 33,253 -- **active_memories**: 784 (up from 84 pre-density-batch — density gate CRUSHED vs V1-A's 100-target) -- **candidate_memories**: 2 (triage queue drained) -- **interactions**: 500+ (limit=2000 query returned 500 — density batch has been running; actual may be higher, confirm via /stats next update) +- **active_memories**: 290 (`/admin/dashboard` 2026-04-24; note integrity panel reports a separate active_memory_count=951 and needs reconciliation) +- **candidate_memories**: 0 (triage queue drained) +- **interactions**: 950 (`/admin/dashboard` 2026-04-24) - **registered_projects**: atocore, p04-gigabit, p05-interferometer, p06-polisher, atomizer-v2, abb-space (aliased p08) -- **project_state_entries**: 63 (atocore alone; full cross-project count not re-sampled this update) +- **project_state_entries**: 128 across registered projects (`/admin/dashboard` 2026-04-24) - **entities**: 66 (up from 35 — V1-0 backfill + ongoing work; 0 open conflicts) - **off_host_backup**: `papa@192.168.86.39:/home/papa/atocore-backups/` via cron, verified - **nightly_pipeline**: backup → cleanup → rsync → OpenClaw import → vault refresh → extract → auto-triage → **auto-promote/expire (NEW)** → weekly synth/lint Sundays → **retrieval harness (NEW)** → **pipeline summary (NEW)** @@ -170,6 +170,10 @@ One branch `codex/extractor-eval-loop` for Day 1-5, a second `codex/retrieval-ha ## Session Log +- **2026-04-24 Codex (audit improvements foundation)** Started implementation of the audit recommendations on branch `codex/audit-improvements-foundation` from `origin/main@c53e61e`. First tranche: registry-aware project-scoped retrieval filtering (`ATOCORE_RANK_PROJECT_SCOPE_FILTER`, widened candidate pull before filtering), eval harness known-issue lane, two p05 project-bleed fixtures, `scripts/live_status.py`, README/current-state/master-plan status refresh. Verified `pytest -q`: 550 passed in 67.11s. Live retrieval harness against undeployed production: 20 fixtures, 18 pass, 1 known issue (`p04-constraints` Zerodur/1.2 content gap), 1 blocking guard (`p05-broad-status-no-atomizer`) still failing because production has not yet deployed the retrieval filter and currently pulls `P04-GigaBIT-M1-KB-design` into broad p05 status context. Live dashboard refresh: health ok, build `2b86543`, docs 1748, chunks/vectors 33253, interactions 948, active memories 289, candidates 0, project_state total 128. Noted count discrepancy: dashboard memories.active=289 while integrity active_memory_count=951; schedule reconciliation in a follow-up. + +- **2026-04-24 Codex (independent-audit hardening)** Applied the Opus independent audit's fast follow-ups before merge/deploy. Closed the two P1s by making project-scope ownership path/tag-based only, adding path-segment/tag-exact matching to avoid short-alias substring collisions, and keeping title/heading text out of provenance decisions. Added regression tests for title poisoning, substring collision, and unknown-project fallback. Added retrieval log fields `raw_results_count`, `post_filter_count`, `post_filter_dropped`, and `underfilled`. Added retrieval-eval run metadata (`generated_at`, `base_url`, `/health`) and `live_status.py` auth-token/status support. README now documents the ranking knobs and clarifies that the hard scope filter and soft project match boost are separate controls. Verified `pytest -q`: 553 passed in 66.07s. Live production remains expected-predeploy: 20 fixtures, 18 pass, 1 known content gap, 1 blocking p05 bleed guard. Latest live dashboard: build `2b86543`, docs 1748, chunks/vectors 33253, interactions 950, active memories 290, candidates 0, project_state total 128. + - **2026-04-23 Codex + Claude (R14 closed)** Codex reviewed `claude/r14-promote-400` at `3888db9`, no findings: "The route change is narrowly scoped: `promote_entity()` still returns False for not-found/not-candidate cases, so the existing 404 behavior remains intact, while caller-fixable validation failures now surface as 400." Ran `pytest tests/test_v1_0_write_invariants.py -q` from an isolated worktree: 15 passed in 1.91s. Claude squash-merged to main as `0989fed`, followed by ledger close-out `2b86543`, then deployed via canonical script. Dalidou `/health` reports build_sha=`2b86543e6ad26011b39a44509cc8df3809725171`, build_time `2026-04-23T15:20:53Z`, status=ok. R14 closed. Orientation refreshed earlier this session also reflected the V1-A gate status: **density gate CLEARED** (784 active memories vs 100 target — density batch-extract ran between 2026-04-22 and 2026-04-23 and more than crushed the gate), **soak gate at day 5 of ~7** (F4 first run 2026-04-19; nightly clean 2026-04-19 through 2026-04-23; only chronic failure is the known p04-constraints "Zerodur" content gap). V1-A branches from a clean V1-0 baseline as soon as the soak is called done. - **2026-04-22 Codex + Antoine (V1-0 closed)** Codex approved `f16cd52` after re-running both original probes (legacy-candidate promote + supersede hook — both correct) and the three targeted regression suites (`test_v1_0_write_invariants.py`, `test_engineering_v1_phase5.py`, `test_inbox_crossproject.py` — all pass). Squash-merged to main as `2712c5d` ("feat(engineering): enforce V1-0 write invariants"). Deployed to Dalidou via the canonical deploy script; `/health` build_sha=`2712c5d2d03cb2a6af38b559664afd1c4cd0e050` status=ok. Validated backup snapshot at `/srv/storage/atocore/backups/snapshots/20260422T190624Z` taken BEFORE prod backfill. Prod backfill of `scripts/v1_0_backfill_provenance.py` against live DB: dry-run found 31 active/superseded entities with no provenance, list reviewed and looked sane; live run with default `hand_authored=1` flag path updated 31 rows; follow-up dry-run returned 0 rows remaining → no lingering F-8 violations in prod. Codex logged one residual P2 (R14): HTTP `POST /entities/{id}/promote` route doesn't translate the new service-layer `ValueError` into 400 — legacy bad candidate promoted through the API surfaces as 500. Not blocking. V1-0 closed. **Gates for V1-A**: soak window ends ~2026-04-26; 100-active-memory density target (currently 84 active + the ~31 newly flagged ones — need to check how those count in density math). V1-A holds until both gates clear. diff --git a/README.md b/README.md index d46df7a..70d77d4 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ Personal context engine that enriches LLM interactions with durable memory, stru ```bash pip install -e . -uvicorn src.atocore.main:app --port 8100 +uvicorn atocore.main:app --port 8100 ``` ## Usage @@ -37,6 +37,10 @@ python scripts/atocore_client.py audit-query "gigabit" 5 | POST | /ingest | Ingest markdown file or folder | | POST | /query | Retrieve relevant chunks | | POST | /context/build | Build full context pack | +| POST | /interactions | Capture prompt/response interactions | +| GET/POST | /memory | List/create durable memories | +| GET/POST | /entities | Engineering entity graph surface | +| GET | /admin/dashboard | Operator dashboard | | GET | /health | Health check | | GET | /debug/context | Inspect last context pack | @@ -66,8 +70,10 @@ unversioned forms. FastAPI (port 8100) |- Ingestion: markdown -> parse -> chunk -> embed -> store |- Retrieval: query -> embed -> vector search -> rank - |- Context Builder: retrieve -> boost -> budget -> format - |- SQLite (documents, chunks, memories, projects, interactions) + |- Context Builder: project state -> memories -> entities -> retrieval -> budget + |- Reflection: capture -> reinforce -> extract -> triage -> promote/expire + |- Engineering: typed entities, relationships, conflicts, wiki/mirror + |- SQLite (documents, chunks, memories, projects, interactions, entities) '- ChromaDB (vector embeddings) ``` @@ -82,6 +88,16 @@ Set via environment variables (prefix `ATOCORE_`): | ATOCORE_CHUNK_MAX_SIZE | 800 | Max chunk size (chars) | | ATOCORE_CONTEXT_BUDGET | 3000 | Context pack budget (chars) | | ATOCORE_EMBEDDING_MODEL | paraphrase-multilingual-MiniLM-L12-v2 | Embedding model | +| ATOCORE_RANK_PROJECT_MATCH_BOOST | 2.0 | Soft boost for chunks whose metadata matches the project hint | +| ATOCORE_RANK_PROJECT_SCOPE_FILTER | true | Filter project-hinted retrieval away from other registered project corpora | +| ATOCORE_RANK_PROJECT_SCOPE_CANDIDATE_MULTIPLIER | 4 | Widen candidate pull before project-scope filtering | +| ATOCORE_RANK_QUERY_TOKEN_STEP | 0.08 | Per-token boost when query terms appear in high-signal metadata | +| ATOCORE_RANK_QUERY_TOKEN_CAP | 1.32 | Maximum query-token boost multiplier | +| ATOCORE_RANK_PATH_HIGH_SIGNAL_BOOST | 1.18 | Boost current decision/status/requirements-like paths | +| ATOCORE_RANK_PATH_LOW_SIGNAL_PENALTY | 0.72 | Down-rank archive/history-like paths | + +`ATOCORE_RANK_PROJECT_SCOPE_FILTER` gates the hard cross-project filter only. +`ATOCORE_RANK_PROJECT_MATCH_BOOST` remains the separate soft-ranking knob. ## Testing @@ -93,7 +109,10 @@ pytest ## Operations - `scripts/atocore_client.py` provides a live API client for project refresh, project-state inspection, and retrieval-quality audits. +- `scripts/retrieval_eval.py` runs the live retrieval/context harness, separates blocking failures from known content gaps, and stamps JSON output with target/build metadata. +- `scripts/live_status.py` renders a compact read-only status report from `/health`, `/stats`, `/projects`, and `/admin/dashboard`; set `ATOCORE_AUTH_TOKEN` or `--auth-token` when those endpoints are gated. - `docs/operations.md` captures the current operational priority order: retrieval quality, Wave 2 trusted-operational ingestion, AtoDrive scoping, and restore validation. +- `DEV-LEDGER.md` is the fast-moving source of operational truth during active development; copy claims into docs only after checking the live service. ## Architecture Notes diff --git a/docs/current-state.md b/docs/current-state.md index 1758ef6..32a46e2 100644 --- a/docs/current-state.md +++ b/docs/current-state.md @@ -1,6 +1,7 @@ -# AtoCore — Current State (2026-04-22) +# AtoCore - Current State (2026-04-24) -Live deploy: `2712c5d` · Dalidou health: ok · Harness: 17/18 · Tests: 547 passing. +Live deploy: `2b86543` · Dalidou health: ok · Harness: 18/20 with 1 known +content gap and 1 current blocking project-bleed guard · Tests: 553 passing. ## V1-0 landed 2026-04-22 @@ -13,9 +14,8 @@ supersede) with Q-3 fail-open. Prod backfill ran cleanly — 31 legacy active/superseded entities flagged `hand_authored=1`, follow-up dry-run returned 0 remaining rows. Test count 533 → 547 (+14). -R14 (P2, non-blocking): `POST /entities/{id}/promote` route fix translates -the new `ValueError` into 400. Branch `claude/r14-promote-400` pending -Codex review + squash-merge. +R14 is closed: `POST /entities/{id}/promote` now translates the new +caller-fixable V1-0 `ValueError` into HTTP 400. **Next in the V1 track:** V1-A (minimal query slice + Q-6 killer-correctness integration). Gated on pipeline soak (~2026-04-26) + 100+ active memory @@ -65,10 +65,10 @@ Last nightly run (2026-04-19 03:00 UTC): **31 promoted · 39 rejected · 0 needs | 7G | Re-extraction on prompt version bump | pending | | 7H | Chroma vector hygiene (delete vectors for superseded memories) | pending | -## Known gaps (honest) +## Known gaps (honest, refreshed 2026-04-24) 1. **Capture surface is Claude-Code-and-OpenClaw only.** Conversations in Claude Desktop, Claude.ai web, phone, or any other LLM UI are NOT captured. Example: the rotovap/mushroom chat yesterday never reached AtoCore because no hook fired. See Q4 below. -2. **OpenClaw is capture-only, not context-grounded.** The plugin POSTs `/interactions` on `llm_output` but does NOT call `/context/build` on `before_agent_start`. OpenClaw's underlying agent runs blind. See Q2 below. -3. **Human interface (wiki) is thin and static.** 5 project cards + a "System" line. No dashboard for the autonomous activity. No per-memory detail page. See Q3/Q5. -4. **Harness 17/18** — the `p04-constraints` fixture wants "Zerodur" but retrieval surfaces related-not-exact terms. Content gap, not a retrieval regression. -5. **Two projects under-populated**: p05-interferometer (4 memories, 18 state) and atomizer-v2 (1 memory, 6 state). Batch re-extract with the new llm-0.6.0 prompt would help. +2. **Project-scoped retrieval still needs deployment verification.** The April 24 audit reproduced cross-project competition on broad p05 prompts. The current branch adds registry-aware project filtering and a harness guard; verify after deploy. +3. **Human interface is useful but not yet the V1 Human Mirror.** Wiki/dashboard pages exist, but the spec routes, deterministic mirror files, disputed markers, and curated annotations remain V1-D work. +4. **Harness known issue:** `p04-constraints` wants "Zerodur" and "1.2"; live retrieval surfaces related constraints but not those exact strings. Treat as content/state gap until fixed. +5. **Formal docs lag the ledger during fast work.** Use `DEV-LEDGER.md` and `python scripts/live_status.py` for live truth, then copy verified claims into these docs. diff --git a/docs/master-plan-status.md b/docs/master-plan-status.md index 29d492c..d7e17f7 100644 --- a/docs/master-plan-status.md +++ b/docs/master-plan-status.md @@ -70,9 +70,14 @@ read-only additive mode. - Phase 6 - AtoDrive - Phase 10 - Write-back - Phase 11 - Multi-model -- Phase 12 - Evaluation - Phase 13 - Hardening +### Partial / Operational Baseline + +- Phase 12 - Evaluation. The retrieval/context harness exists and runs + against live Dalidou, but coverage is still intentionally small and + should grow before this is complete in the intended sense. + ### Engineering Layer Planning Sprint **Status: complete.** All 8 architecture docs are drafted. The @@ -126,11 +131,13 @@ This sits implicitly between Phase 8 (OpenClaw) and Phase 11 (multi-model). Memory-review and engineering-entity commands are deferred from the shared client until their workflows are exercised. -## What Is Real Today (updated 2026-04-16) +## What Is Real Today (updated 2026-04-24) -- canonical AtoCore runtime on Dalidou (`775960c`, deploy.sh verified) +- canonical AtoCore runtime on Dalidou (`2b86543`, deploy.sh verified) - 33,253 vectors across 6 registered projects -- 234 captured interactions (192 claude-code, 38 openclaw, 4 test) +- 950 captured interactions as of the 2026-04-24 live dashboard; refresh + exact live counts with + `python scripts/live_status.py` - 6 registered projects: - `p04-gigabit` (483 docs, 15 state entries) - `p05-interferometer` (109 docs, 18 state entries) @@ -138,12 +145,15 @@ deferred from the shared client until their workflows are exercised. - `atomizer-v2` (568 docs, 5 state entries) - `abb-space` (6 state entries) - `atocore` (drive source, 47 state entries) -- 110 Trusted Project State entries across all projects (decisions, requirements, facts, contacts, milestones) -- 84 active memories (31 project, 23 knowledge, 10 episodic, 8 adaptation, 7 preference, 5 identity) +- 128 Trusted Project State entries across all projects (decisions, requirements, facts, contacts, milestones) +- 290 active memories and 0 candidate memories as of the 2026-04-24 live + dashboard - context pack assembly with 4 tiers: Trusted Project State > identity/preference > project memories > retrieved chunks - query-relevance memory ranking with overlap-density scoring -- retrieval eval harness: 18 fixtures, 17/18 passing on live -- 303 tests passing +- retrieval eval harness: 20 fixtures; current live has 18 pass, 1 known + content gap, and 1 blocking cross-project bleed guard targeted by the + current retrieval-scoping branch +- 553 tests passing on the audit-improvements branch - nightly pipeline: backup → cleanup → rsync → OpenClaw import → vault refresh → extract → triage → **auto-promote/expire** → weekly synth/lint → **retrieval harness** → **pipeline summary to project state** - Phase 10 operational: reinforcement-based auto-promotion (ref_count ≥ 3, confidence ≥ 0.7) + stale candidate expiry (14 days unreinforced) - pipeline health visible in dashboard: interaction totals by client, pipeline last_run, harness results, triage stats @@ -190,9 +200,9 @@ where surfaces are disjoint, pauses when they collide. | V1-E | Memory→entity graduation end-to-end + remaining Q-4 trust tests | pending V1-D (note: collides with memory extractor; pauses for multi-model triage work) | | V1-F | F-5 detector generalization + route alias + O-1/O-2/O-3 operational + D-1/D-3/D-4 docs | finish line | -R14 (P2, non-blocking): `POST /entities/{id}/promote` route returns 500 -on the new V1-0 `ValueError` instead of 400. Fix on branch -`claude/r14-promote-400`, pending Codex review. +R14 is closed: `POST /entities/{id}/promote` now translates +caller-fixable V1-0 provenance validation failures into HTTP 400 instead +of leaking as HTTP 500. ## Next diff --git a/scripts/live_status.py b/scripts/live_status.py new file mode 100644 index 0000000..c8a3ec2 --- /dev/null +++ b/scripts/live_status.py @@ -0,0 +1,131 @@ +"""Render a compact live-status report from a running AtoCore instance. + +This is intentionally read-only and stdlib-only so it can be used from a +fresh checkout, a cron job, or a Codex/Claude session without installing the +full app package. The output is meant to reduce docs drift: copy the report +into status docs only after it was generated from the live service. +""" + +from __future__ import annotations + +import argparse +import errno +import json +import os +import sys +import urllib.error +import urllib.request +from typing import Any + + +DEFAULT_BASE_URL = os.environ.get("ATOCORE_BASE_URL", "http://dalidou:8100").rstrip("/") +DEFAULT_TIMEOUT = int(os.environ.get("ATOCORE_TIMEOUT_SECONDS", "30")) +DEFAULT_AUTH_TOKEN = os.environ.get("ATOCORE_AUTH_TOKEN", "").strip() + + +def request_json(base_url: str, path: str, timeout: int, auth_token: str = "") -> dict[str, Any]: + headers = {"Authorization": f"Bearer {auth_token}"} if auth_token else {} + req = urllib.request.Request(f"{base_url}{path}", method="GET", headers=headers) + with urllib.request.urlopen(req, timeout=timeout) as response: + body = response.read().decode("utf-8") + status = getattr(response, "status", None) + payload = json.loads(body) if body.strip() else {} + if not isinstance(payload, dict): + payload = {"value": payload} + if status is not None: + payload["_http_status"] = status + return payload + + +def collect_status(base_url: str, timeout: int, auth_token: str = "") -> dict[str, Any]: + payload: dict[str, Any] = {"base_url": base_url} + for name, path in { + "health": "/health", + "stats": "/stats", + "projects": "/projects", + "dashboard": "/admin/dashboard", + }.items(): + try: + payload[name] = request_json(base_url, path, timeout, auth_token) + except (urllib.error.URLError, TimeoutError, OSError, json.JSONDecodeError) as exc: + payload[name] = {"error": str(exc)} + return payload + + +def render_markdown(status: dict[str, Any]) -> str: + health = status.get("health", {}) + stats = status.get("stats", {}) + projects = status.get("projects", {}).get("projects", []) + dashboard = status.get("dashboard", {}) + memories = dashboard.get("memories", {}) if isinstance(dashboard.get("memories"), dict) else {} + project_state = dashboard.get("project_state", {}) if isinstance(dashboard.get("project_state"), dict) else {} + interactions = dashboard.get("interactions", {}) if isinstance(dashboard.get("interactions"), dict) else {} + pipeline = dashboard.get("pipeline", {}) if isinstance(dashboard.get("pipeline"), dict) else {} + + lines = [ + "# AtoCore Live Status", + "", + f"- base_url: `{status.get('base_url', '')}`", + "- endpoint_http_statuses: " + f"`health={health.get('_http_status', 'error')}, " + f"stats={stats.get('_http_status', 'error')}, " + f"projects={status.get('projects', {}).get('_http_status', 'error')}, " + f"dashboard={dashboard.get('_http_status', 'error')}`", + f"- service_status: `{health.get('status', 'unknown')}`", + f"- code_version: `{health.get('code_version', health.get('version', 'unknown'))}`", + f"- build_sha: `{health.get('build_sha', 'unknown')}`", + f"- build_branch: `{health.get('build_branch', 'unknown')}`", + f"- build_time: `{health.get('build_time', 'unknown')}`", + f"- env: `{health.get('env', 'unknown')}`", + f"- documents: `{stats.get('total_documents', 'unknown')}`", + f"- chunks: `{stats.get('total_chunks', 'unknown')}`", + f"- vectors: `{stats.get('total_vectors', health.get('vectors_count', 'unknown'))}`", + f"- registered_projects: `{len(projects)}`", + f"- active_memories: `{memories.get('active', 'unknown')}`", + f"- candidate_memories: `{memories.get('candidates', 'unknown')}`", + f"- interactions: `{interactions.get('total', 'unknown')}`", + f"- project_state_entries: `{project_state.get('total', 'unknown')}`", + f"- pipeline_last_run: `{pipeline.get('last_run', 'unknown')}`", + ] + + if projects: + lines.extend(["", "## Projects"]) + for project in projects: + aliases = ", ".join(project.get("aliases", [])) + suffix = f" ({aliases})" if aliases else "" + lines.append(f"- `{project.get('id', '')}`{suffix}") + + return "\n".join(lines) + "\n" + + +def main() -> int: + parser = argparse.ArgumentParser(description="Render live AtoCore status") + parser.add_argument("--base-url", default=DEFAULT_BASE_URL) + parser.add_argument("--timeout", type=int, default=DEFAULT_TIMEOUT) + parser.add_argument( + "--auth-token", + default=DEFAULT_AUTH_TOKEN, + help="Bearer token; defaults to ATOCORE_AUTH_TOKEN when set", + ) + parser.add_argument("--json", action="store_true", help="emit raw JSON") + args = parser.parse_args() + + status = collect_status(args.base_url.rstrip("/"), args.timeout, args.auth_token) + if args.json: + output = json.dumps(status, indent=2, ensure_ascii=True) + "\n" + else: + output = render_markdown(status) + + try: + sys.stdout.write(output) + except BrokenPipeError: + return 0 + except OSError as exc: + if exc.errno in {errno.EINVAL, errno.EPIPE}: + return 0 + raise + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/retrieval_eval.py b/scripts/retrieval_eval.py index b067da8..6f20117 100644 --- a/scripts/retrieval_eval.py +++ b/scripts/retrieval_eval.py @@ -44,6 +44,7 @@ import urllib.error import urllib.parse import urllib.request from dataclasses import dataclass, field +from datetime import datetime, timezone from pathlib import Path DEFAULT_BASE_URL = os.environ.get("ATOCORE_BASE_URL", "http://dalidou:8100") @@ -52,6 +53,13 @@ DEFAULT_BUDGET = 3000 DEFAULT_FIXTURES = Path(__file__).parent / "retrieval_eval_fixtures.json" +def request_json(base_url: str, path: str, timeout: int) -> dict: + req = urllib.request.Request(f"{base_url}{path}", method="GET") + with urllib.request.urlopen(req, timeout=timeout) as resp: + body = resp.read().decode("utf-8") + return json.loads(body) if body.strip() else {} + + @dataclass class Fixture: name: str @@ -60,6 +68,7 @@ class Fixture: budget: int = DEFAULT_BUDGET expect_present: list[str] = field(default_factory=list) expect_absent: list[str] = field(default_factory=list) + known_issue: bool = False notes: str = "" @@ -70,8 +79,13 @@ class FixtureResult: missing_present: list[str] unexpected_absent: list[str] total_chars: int + known_issue: bool = False error: str = "" + @property + def blocking_failure(self) -> bool: + return not self.ok and not self.known_issue + def load_fixtures(path: Path) -> list[Fixture]: data = json.loads(path.read_text(encoding="utf-8")) @@ -89,6 +103,7 @@ def load_fixtures(path: Path) -> list[Fixture]: budget=int(raw.get("budget", DEFAULT_BUDGET)), expect_present=list(raw.get("expect_present", [])), expect_absent=list(raw.get("expect_absent", [])), + known_issue=bool(raw.get("known_issue", False)), notes=raw.get("notes", ""), ) ) @@ -117,6 +132,7 @@ def run_fixture(fixture: Fixture, base_url: str, timeout: int) -> FixtureResult: missing_present=list(fixture.expect_present), unexpected_absent=[], total_chars=0, + known_issue=fixture.known_issue, error=f"http_error: {exc}", ) @@ -129,16 +145,26 @@ def run_fixture(fixture: Fixture, base_url: str, timeout: int) -> FixtureResult: missing_present=missing, unexpected_absent=unexpected, total_chars=len(formatted), + known_issue=fixture.known_issue, ) -def print_human_report(results: list[FixtureResult]) -> None: +def print_human_report(results: list[FixtureResult], metadata: dict) -> None: total = len(results) passed = sum(1 for r in results if r.ok) + known = sum(1 for r in results if not r.ok and r.known_issue) + blocking = sum(1 for r in results if r.blocking_failure) print(f"Retrieval eval: {passed}/{total} fixtures passed") + print( + "Target: " + f"{metadata.get('base_url', 'unknown')} " + f"build={metadata.get('health', {}).get('build_sha', 'unknown')}" + ) + if known or blocking: + print(f"Blocking failures: {blocking} Known issues: {known}") print() for r in results: - marker = "PASS" if r.ok else "FAIL" + marker = "PASS" if r.ok else ("KNOWN" if r.known_issue else "FAIL") print(f"[{marker}] {r.fixture.name} project={r.fixture.project} chars={r.total_chars}") if r.error: print(f" error: {r.error}") @@ -150,15 +176,21 @@ def print_human_report(results: list[FixtureResult]) -> None: print(f" notes: {r.fixture.notes}") -def print_json_report(results: list[FixtureResult]) -> None: +def print_json_report(results: list[FixtureResult], metadata: dict) -> None: payload = { + "generated_at": metadata.get("generated_at"), + "base_url": metadata.get("base_url"), + "health": metadata.get("health", {}), "total": len(results), "passed": sum(1 for r in results if r.ok), + "known_issues": sum(1 for r in results if not r.ok and r.known_issue), + "blocking_failures": sum(1 for r in results if r.blocking_failure), "fixtures": [ { "name": r.fixture.name, "project": r.fixture.project, "ok": r.ok, + "known_issue": r.known_issue, "total_chars": r.total_chars, "missing_present": r.missing_present, "unexpected_absent": r.unexpected_absent, @@ -179,15 +211,26 @@ def main() -> int: parser.add_argument("--json", action="store_true", help="emit machine-readable JSON") args = parser.parse_args() + base_url = args.base_url.rstrip("/") + try: + health = request_json(base_url, "/health", args.timeout) + except (urllib.error.URLError, TimeoutError, OSError, json.JSONDecodeError) as exc: + health = {"error": str(exc)} + metadata = { + "generated_at": datetime.now(timezone.utc).isoformat(), + "base_url": base_url, + "health": health, + } + fixtures = load_fixtures(args.fixtures) - results = [run_fixture(f, args.base_url, args.timeout) for f in fixtures] + results = [run_fixture(f, base_url, args.timeout) for f in fixtures] if args.json: - print_json_report(results) + print_json_report(results, metadata) else: - print_human_report(results) + print_human_report(results, metadata) - return 0 if all(r.ok for r in results) else 1 + return 0 if not any(r.blocking_failure for r in results) else 1 if __name__ == "__main__": diff --git a/scripts/retrieval_eval_fixtures.json b/scripts/retrieval_eval_fixtures.json index c3e29ff..2379ccb 100644 --- a/scripts/retrieval_eval_fixtures.json +++ b/scripts/retrieval_eval_fixtures.json @@ -27,7 +27,8 @@ "expect_absent": [ "polisher suite" ], - "notes": "Key constraints are in Trusted Project State and in the mission-framing memory" + "known_issue": true, + "notes": "Known content gap as of 2026-04-24: live retrieval surfaces related constraints but not the exact Zerodur / 1.2 strings. Keep visible, but do not make nightly harness red until the source/state gap is fixed." }, { "name": "p04-short-ambiguous", @@ -80,6 +81,36 @@ ], "notes": "CGH is a core p05 concept. Should surface via chunks and possibly the architecture memory. Must not bleed p06 polisher-suite terms." }, + { + "name": "p05-broad-status-no-atomizer", + "project": "p05-interferometer", + "prompt": "current status", + "expect_present": [ + "--- Trusted Project State ---", + "--- Project Memories ---", + "Zygo" + ], + "expect_absent": [ + "atomizer-v2", + "ATOMIZER_PODCAST_BRIEFING", + "[Source: atomizer-v2/", + "P04-GigaBIT-M1-KB-design" + ], + "notes": "Regression guard for the April 24 audit finding: broad p05 status queries must not pull Atomizer/archive context into project-scoped packs." + }, + { + "name": "p05-vendor-decision-no-archive-first", + "project": "p05-interferometer", + "prompt": "vendor selection decision", + "expect_present": [ + "Selection-Decision" + ], + "expect_absent": [ + "[Source: atomizer-v2/", + "ATOMIZER_PODCAST_BRIEFING" + ], + "notes": "Project-scoped decision query should stay inside p05 and prefer current decision/vendor material over unrelated project archives." + }, { "name": "p06-suite-split", "project": "p06-polisher", diff --git a/src/atocore/config.py b/src/atocore/config.py index 1ac5308..882ca65 100644 --- a/src/atocore/config.py +++ b/src/atocore/config.py @@ -46,6 +46,8 @@ class Settings(BaseSettings): # All multipliers default to the values used since Wave 1; tighten or # loosen them via ATOCORE_* env vars without touching code. rank_project_match_boost: float = 2.0 + rank_project_scope_filter: bool = True + rank_project_scope_candidate_multiplier: int = 4 rank_query_token_step: float = 0.08 rank_query_token_cap: float = 1.32 rank_path_high_signal_boost: float = 1.18 diff --git a/src/atocore/retrieval/retriever.py b/src/atocore/retrieval/retriever.py index 24f2dd2..26e0182 100644 --- a/src/atocore/retrieval/retriever.py +++ b/src/atocore/retrieval/retriever.py @@ -1,5 +1,6 @@ """Retrieval: query to ranked chunks.""" +import json import re import time from dataclasses import dataclass @@ -7,7 +8,7 @@ from dataclasses import dataclass import atocore.config as _config from atocore.models.database import get_connection from atocore.observability.logger import get_logger -from atocore.projects.registry import get_registered_project +from atocore.projects.registry import RegisteredProject, get_registered_project, load_project_registry from atocore.retrieval.embeddings import embed_query from atocore.retrieval.vector_store import get_vector_store @@ -83,6 +84,19 @@ def retrieve( """Retrieve the most relevant chunks for a query.""" top_k = top_k or _config.settings.context_top_k start = time.time() + scoped_project = get_registered_project(project_hint) if project_hint else None + scope_filter_enabled = bool(scoped_project and _config.settings.rank_project_scope_filter) + registered_projects = None + query_top_k = top_k + if scope_filter_enabled: + query_top_k = max( + top_k, + top_k * max(1, _config.settings.rank_project_scope_candidate_multiplier), + ) + try: + registered_projects = load_project_registry() + except Exception: + registered_projects = None query_embedding = embed_query(query) store = get_vector_store() @@ -101,11 +115,12 @@ def retrieve( results = store.query( query_embedding=query_embedding, - top_k=top_k, + top_k=query_top_k, where=where, ) chunks = [] + raw_result_count = len(results["ids"][0]) if results and results["ids"] and results["ids"][0] else 0 if results and results["ids"] and results["ids"][0]: existing_ids = _existing_chunk_ids(results["ids"][0]) for i, chunk_id in enumerate(results["ids"][0]): @@ -117,6 +132,13 @@ def retrieve( meta = results["metadatas"][0][i] if results["metadatas"] else {} content = results["documents"][0][i] if results["documents"] else "" + if scope_filter_enabled and not _is_allowed_for_project_scope( + scoped_project, + meta, + registered_projects, + ): + continue + score *= _query_match_boost(query, meta) score *= _path_signal_boost(meta) if project_hint: @@ -137,42 +159,139 @@ def retrieve( duration_ms = int((time.time() - start) * 1000) chunks.sort(key=lambda chunk: chunk.score, reverse=True) + post_filter_count = len(chunks) + chunks = chunks[:top_k] log.info( "retrieval_done", query=query[:100], top_k=top_k, + query_top_k=query_top_k, + raw_results_count=raw_result_count, + post_filter_count=post_filter_count, results_count=len(chunks), + post_filter_dropped=max(0, raw_result_count - post_filter_count), + underfilled=bool(raw_result_count >= query_top_k and len(chunks) < top_k), duration_ms=duration_ms, ) return chunks +def _is_allowed_for_project_scope( + project: RegisteredProject, + metadata: dict, + registered_projects: list[RegisteredProject] | None = None, +) -> bool: + """Return True when a chunk is target-project or not project-owned. + + Project-hinted retrieval should not let one registered project's corpus + compete with another's. At the same time, unowned/global sources should + remain eligible because shared docs and cross-project references can be + genuinely useful. The registry gives us the boundary: if metadata matches + a registered project and it is not the requested project, filter it out. + """ + if _metadata_matches_project(project, metadata): + return True + + if registered_projects is None: + try: + registered_projects = load_project_registry() + except Exception: + return True + + for other in registered_projects: + if other.project_id == project.project_id: + continue + if _metadata_matches_project(other, metadata): + return False + return True + + +def _metadata_matches_project(project: RegisteredProject, metadata: dict) -> bool: + path = _metadata_source_path(metadata) + tags = _metadata_tags(metadata) + for term in _project_scope_terms(project): + if _path_matches_term(path, term) or term in tags: + return True + return False + + +def _project_scope_terms(project: RegisteredProject) -> set[str]: + terms = {project.project_id.lower()} + terms.update(alias.lower() for alias in project.aliases) + for source_ref in project.ingest_roots: + normalized = source_ref.subpath.replace("\\", "/").strip("/").lower() + if normalized: + terms.add(normalized) + terms.add(normalized.split("/")[-1]) + return {term for term in terms if term} + + +def _metadata_searchable(metadata: dict) -> str: + return " ".join( + [ + str(metadata.get("source_file", "")).replace("\\", "/").lower(), + str(metadata.get("title", "")).lower(), + str(metadata.get("heading_path", "")).lower(), + str(metadata.get("tags", "")).lower(), + ] + ) + + +def _metadata_source_path(metadata: dict) -> str: + return str(metadata.get("source_file", "")).replace("\\", "/").strip("/").lower() + + +def _metadata_tags(metadata: dict) -> set[str]: + raw_tags = metadata.get("tags", []) + if isinstance(raw_tags, (list, tuple, set)): + return {str(tag).strip().lower() for tag in raw_tags if str(tag).strip()} + if isinstance(raw_tags, str): + try: + parsed = json.loads(raw_tags) + except json.JSONDecodeError: + parsed = [raw_tags] + if isinstance(parsed, (list, tuple, set)): + return {str(tag).strip().lower() for tag in parsed if str(tag).strip()} + if isinstance(parsed, str) and parsed.strip(): + return {parsed.strip().lower()} + return set() + + +def _path_matches_term(path: str, term: str) -> bool: + normalized = term.replace("\\", "/").strip("/").lower() + if not path or not normalized: + return False + if "/" in normalized: + return path == normalized or path.startswith(f"{normalized}/") + return normalized in set(path.split("/")) + + +def _metadata_has_term(metadata: dict, term: str) -> bool: + normalized = term.replace("\\", "/").strip("/").lower() + if not normalized: + return False + if _path_matches_term(_metadata_source_path(metadata), normalized): + return True + if normalized in _metadata_tags(metadata): + return True + return re.search( + rf"(? float: """Return a project-aware relevance multiplier for raw retrieval.""" hint_lower = project_hint.strip().lower() if not hint_lower: return 1.0 - source_file = str(metadata.get("source_file", "")).lower() - title = str(metadata.get("title", "")).lower() - tags = str(metadata.get("tags", "")).lower() - searchable = " ".join([source_file, title, tags]) - project = get_registered_project(project_hint) - candidate_names = {hint_lower} - if project is not None: - candidate_names.add(project.project_id.lower()) - candidate_names.update(alias.lower() for alias in project.aliases) - candidate_names.update( - source_ref.subpath.replace("\\", "/").strip("/").split("/")[-1].lower() - for source_ref in project.ingest_roots - if source_ref.subpath.strip("/\\") - ) - + candidate_names = _project_scope_terms(project) if project is not None else {hint_lower} for candidate in candidate_names: - if candidate and candidate in searchable: + if _metadata_has_term(metadata, candidate): return _config.settings.rank_project_match_boost return 1.0 diff --git a/tests/test_config.py b/tests/test_config.py index c6d5428..3c40b2e 100644 --- a/tests/test_config.py +++ b/tests/test_config.py @@ -46,6 +46,8 @@ def test_settings_keep_legacy_db_path_when_present(tmp_path, monkeypatch): def test_ranking_weights_are_tunable_via_env(monkeypatch): monkeypatch.setenv("ATOCORE_RANK_PROJECT_MATCH_BOOST", "3.5") + monkeypatch.setenv("ATOCORE_RANK_PROJECT_SCOPE_FILTER", "false") + monkeypatch.setenv("ATOCORE_RANK_PROJECT_SCOPE_CANDIDATE_MULTIPLIER", "6") monkeypatch.setenv("ATOCORE_RANK_QUERY_TOKEN_STEP", "0.12") monkeypatch.setenv("ATOCORE_RANK_QUERY_TOKEN_CAP", "1.5") monkeypatch.setenv("ATOCORE_RANK_PATH_HIGH_SIGNAL_BOOST", "1.25") @@ -54,6 +56,8 @@ def test_ranking_weights_are_tunable_via_env(monkeypatch): settings = config.Settings() assert settings.rank_project_match_boost == 3.5 + assert settings.rank_project_scope_filter is False + assert settings.rank_project_scope_candidate_multiplier == 6 assert settings.rank_query_token_step == 0.12 assert settings.rank_query_token_cap == 1.5 assert settings.rank_path_high_signal_boost == 1.25 diff --git a/tests/test_retrieval.py b/tests/test_retrieval.py index 121575d..9049fae 100644 --- a/tests/test_retrieval.py +++ b/tests/test_retrieval.py @@ -70,8 +70,28 @@ def test_retrieve_skips_stale_vector_entries(tmp_data_dir, sample_markdown, monk def test_retrieve_project_hint_boosts_matching_chunks(monkeypatch): + target_project = type( + "Project", + (), + { + "project_id": "p04-gigabit", + "aliases": ("p04", "gigabit"), + "ingest_roots": (), + }, + )() + other_project = type( + "Project", + (), + { + "project_id": "p05-interferometer", + "aliases": ("p05", "interferometer"), + "ingest_roots": (), + }, + )() + class FakeStore: def query(self, query_embedding, top_k=10, where=None): + assert top_k == 8 return { "ids": [["chunk-a", "chunk-b"]], "documents": [["project doc", "other doc"]], @@ -102,22 +122,308 @@ def test_retrieve_project_hint_boosts_matching_chunks(monkeypatch): ) monkeypatch.setattr( "atocore.retrieval.retriever.get_registered_project", - lambda project_name: type( - "Project", - (), - { - "project_id": "p04-gigabit", - "aliases": ("p04", "gigabit"), - "ingest_roots": (), - }, - )(), + lambda project_name: target_project, + ) + monkeypatch.setattr( + "atocore.retrieval.retriever.load_project_registry", + lambda: [target_project, other_project], ) results = retrieve("mirror architecture", top_k=2, project_hint="p04") - assert len(results) == 2 + assert len(results) == 1 assert results[0].chunk_id == "chunk-a" - assert results[0].score > results[1].score + + +def test_retrieve_project_scope_allows_unowned_global_chunks(monkeypatch): + target_project = type( + "Project", + (), + { + "project_id": "p04-gigabit", + "aliases": ("p04", "gigabit"), + "ingest_roots": (), + }, + )() + + class FakeStore: + def query(self, query_embedding, top_k=10, where=None): + return { + "ids": [["chunk-a", "chunk-global"]], + "documents": [["project doc", "global doc"]], + "metadatas": [[ + { + "heading_path": "Overview", + "source_file": "p04-gigabit/pkm/_index.md", + "tags": '["p04-gigabit"]', + "title": "P04", + "document_id": "doc-a", + }, + { + "heading_path": "Overview", + "source_file": "shared/engineering-rules.md", + "tags": "[]", + "title": "Shared engineering rules", + "document_id": "doc-global", + }, + ]], + "distances": [[0.2, 0.21]], + } + + monkeypatch.setattr("atocore.retrieval.retriever.get_vector_store", lambda: FakeStore()) + monkeypatch.setattr("atocore.retrieval.retriever.embed_query", lambda query: [0.0, 0.1]) + monkeypatch.setattr( + "atocore.retrieval.retriever._existing_chunk_ids", + lambda chunk_ids: set(chunk_ids), + ) + monkeypatch.setattr( + "atocore.retrieval.retriever.get_registered_project", + lambda project_name: target_project, + ) + monkeypatch.setattr( + "atocore.retrieval.retriever.load_project_registry", + lambda: [target_project], + ) + + results = retrieve("mirror architecture", top_k=2, project_hint="p04") + + assert [r.chunk_id for r in results] == ["chunk-a", "chunk-global"] + + +def test_retrieve_project_scope_filter_can_be_disabled(monkeypatch): + target_project = type( + "Project", + (), + { + "project_id": "p04-gigabit", + "aliases": ("p04", "gigabit"), + "ingest_roots": (), + }, + )() + other_project = type( + "Project", + (), + { + "project_id": "p05-interferometer", + "aliases": ("p05", "interferometer"), + "ingest_roots": (), + }, + )() + + class FakeStore: + def query(self, query_embedding, top_k=10, where=None): + assert top_k == 2 + return { + "ids": [["chunk-a", "chunk-b"]], + "documents": [["project doc", "other project doc"]], + "metadatas": [[ + { + "heading_path": "Overview", + "source_file": "p04-gigabit/pkm/_index.md", + "tags": '["p04-gigabit"]', + "title": "P04", + "document_id": "doc-a", + }, + { + "heading_path": "Overview", + "source_file": "p05-interferometer/pkm/_index.md", + "tags": '["p05-interferometer"]', + "title": "P05", + "document_id": "doc-b", + }, + ]], + "distances": [[0.2, 0.2]], + } + + monkeypatch.setattr("atocore.config.settings.rank_project_scope_filter", False) + monkeypatch.setattr("atocore.retrieval.retriever.get_vector_store", lambda: FakeStore()) + monkeypatch.setattr("atocore.retrieval.retriever.embed_query", lambda query: [0.0, 0.1]) + monkeypatch.setattr( + "atocore.retrieval.retriever._existing_chunk_ids", + lambda chunk_ids: set(chunk_ids), + ) + monkeypatch.setattr( + "atocore.retrieval.retriever.get_registered_project", + lambda project_name: target_project, + ) + monkeypatch.setattr( + "atocore.retrieval.retriever.load_project_registry", + lambda: [target_project, other_project], + ) + + results = retrieve("mirror architecture", top_k=2, project_hint="p04") + + assert {r.chunk_id for r in results} == {"chunk-a", "chunk-b"} + + +def test_retrieve_project_scope_ignores_title_for_ownership(monkeypatch): + target_project = type( + "Project", + (), + { + "project_id": "p04-gigabit", + "aliases": ("p04", "gigabit"), + "ingest_roots": (), + }, + )() + other_project = type( + "Project", + (), + { + "project_id": "p06-polisher", + "aliases": ("p06", "polisher", "p11"), + "ingest_roots": (), + }, + )() + + class FakeStore: + def query(self, query_embedding, top_k=10, where=None): + return { + "ids": [["chunk-target", "chunk-poisoned-title"]], + "documents": [["p04 doc", "p06 doc"]], + "metadatas": [[ + { + "heading_path": "Overview", + "source_file": "p04-gigabit/pkm/_index.md", + "tags": '["p04-gigabit"]', + "title": "P04", + "document_id": "doc-a", + }, + { + "heading_path": "Overview", + "source_file": "p06-polisher/pkm/architecture.md", + "tags": '["p06-polisher"]', + "title": "GigaBIT M1 mirror lessons", + "document_id": "doc-b", + }, + ]], + "distances": [[0.2, 0.19]], + } + + monkeypatch.setattr("atocore.retrieval.retriever.get_vector_store", lambda: FakeStore()) + monkeypatch.setattr("atocore.retrieval.retriever.embed_query", lambda query: [0.0, 0.1]) + monkeypatch.setattr( + "atocore.retrieval.retriever._existing_chunk_ids", + lambda chunk_ids: set(chunk_ids), + ) + monkeypatch.setattr( + "atocore.retrieval.retriever.get_registered_project", + lambda project_name: target_project, + ) + monkeypatch.setattr( + "atocore.retrieval.retriever.load_project_registry", + lambda: [target_project, other_project], + ) + + results = retrieve("mirror architecture", top_k=2, project_hint="p04") + + assert [r.chunk_id for r in results] == ["chunk-target"] + + +def test_retrieve_project_scope_uses_path_segments_not_substrings(monkeypatch): + target_project = type( + "Project", + (), + { + "project_id": "p05-interferometer", + "aliases": ("p05", "interferometer"), + "ingest_roots": (), + }, + )() + abb_project = type( + "Project", + (), + { + "project_id": "abb-space", + "aliases": ("abb",), + "ingest_roots": (), + }, + )() + + class FakeStore: + def query(self, query_embedding, top_k=10, where=None): + return { + "ids": [["chunk-target", "chunk-global"]], + "documents": [["p05 doc", "global doc"]], + "metadatas": [[ + { + "heading_path": "Overview", + "source_file": "p05-interferometer/pkm/_index.md", + "tags": '["p05-interferometer"]', + "title": "P05", + "document_id": "doc-a", + }, + { + "heading_path": "Abbreviation notes", + "source_file": "shared/cabbage-abbreviations.md", + "tags": "[]", + "title": "ABB-style abbreviations", + "document_id": "doc-global", + }, + ]], + "distances": [[0.2, 0.21]], + } + + monkeypatch.setattr("atocore.retrieval.retriever.get_vector_store", lambda: FakeStore()) + monkeypatch.setattr("atocore.retrieval.retriever.embed_query", lambda query: [0.0, 0.1]) + monkeypatch.setattr( + "atocore.retrieval.retriever._existing_chunk_ids", + lambda chunk_ids: set(chunk_ids), + ) + monkeypatch.setattr( + "atocore.retrieval.retriever.get_registered_project", + lambda project_name: target_project, + ) + monkeypatch.setattr( + "atocore.retrieval.retriever.load_project_registry", + lambda: [target_project, abb_project], + ) + + results = retrieve("abbreviations", top_k=2, project_hint="p05") + + assert [r.chunk_id for r in results] == ["chunk-target", "chunk-global"] + + +def test_retrieve_unknown_project_hint_does_not_widen_or_filter(monkeypatch): + class FakeStore: + def query(self, query_embedding, top_k=10, where=None): + assert top_k == 2 + return { + "ids": [["chunk-a", "chunk-b"]], + "documents": [["doc a", "doc b"]], + "metadatas": [[ + { + "heading_path": "Overview", + "source_file": "project-a/file.md", + "tags": "[]", + "title": "A", + "document_id": "doc-a", + }, + { + "heading_path": "Overview", + "source_file": "project-b/file.md", + "tags": "[]", + "title": "B", + "document_id": "doc-b", + }, + ]], + "distances": [[0.2, 0.21]], + } + + monkeypatch.setattr("atocore.retrieval.retriever.get_vector_store", lambda: FakeStore()) + monkeypatch.setattr("atocore.retrieval.retriever.embed_query", lambda query: [0.0, 0.1]) + monkeypatch.setattr( + "atocore.retrieval.retriever._existing_chunk_ids", + lambda chunk_ids: set(chunk_ids), + ) + monkeypatch.setattr( + "atocore.retrieval.retriever.get_registered_project", + lambda project_name: None, + ) + + results = retrieve("overview", top_k=2, project_hint="unknown-project") + + assert [r.chunk_id for r in results] == ["chunk-a", "chunk-b"] def test_retrieve_downranks_archive_noise_and_prefers_high_signal_paths(monkeypatch):