merge: project-scoped retrieval audit improvements
This commit is contained in:
@@ -7,16 +7,16 @@
|
||||
## Orientation
|
||||
|
||||
- **live_sha** (Dalidou `/health` build_sha): `2b86543` (verified 2026-04-23T15:20:53Z post-R14 deploy; status=ok)
|
||||
- **last_updated**: 2026-04-23 by Claude (R14 squash-merged + deployed; Orientation refreshed)
|
||||
- **last_updated**: 2026-04-24 by Codex (audit-improvements foundation branch; live status refreshed)
|
||||
- **main_tip**: `2b86543`
|
||||
- **test_count**: 548 (547 + 1 R14 regression test)
|
||||
- **harness**: `17/18 PASS` on live Dalidou (p04-constraints expects "Zerodur" — known content gap, not regression; consistent since 2026-04-19)
|
||||
- **test_count**: 553
|
||||
- **harness**: `18/20 PASS` on live Dalidou plus 1 known content gap and 1 blocking project-bleed guard pending deploy of this branch
|
||||
- **vectors**: 33,253
|
||||
- **active_memories**: 784 (up from 84 pre-density-batch — density gate CRUSHED vs V1-A's 100-target)
|
||||
- **candidate_memories**: 2 (triage queue drained)
|
||||
- **interactions**: 500+ (limit=2000 query returned 500 — density batch has been running; actual may be higher, confirm via /stats next update)
|
||||
- **active_memories**: 290 (`/admin/dashboard` 2026-04-24; note integrity panel reports a separate active_memory_count=951 and needs reconciliation)
|
||||
- **candidate_memories**: 0 (triage queue drained)
|
||||
- **interactions**: 950 (`/admin/dashboard` 2026-04-24)
|
||||
- **registered_projects**: atocore, p04-gigabit, p05-interferometer, p06-polisher, atomizer-v2, abb-space (aliased p08)
|
||||
- **project_state_entries**: 63 (atocore alone; full cross-project count not re-sampled this update)
|
||||
- **project_state_entries**: 128 across registered projects (`/admin/dashboard` 2026-04-24)
|
||||
- **entities**: 66 (up from 35 — V1-0 backfill + ongoing work; 0 open conflicts)
|
||||
- **off_host_backup**: `papa@192.168.86.39:/home/papa/atocore-backups/` via cron, verified
|
||||
- **nightly_pipeline**: backup → cleanup → rsync → OpenClaw import → vault refresh → extract → auto-triage → **auto-promote/expire (NEW)** → weekly synth/lint Sundays → **retrieval harness (NEW)** → **pipeline summary (NEW)**
|
||||
@@ -170,6 +170,10 @@ One branch `codex/extractor-eval-loop` for Day 1-5, a second `codex/retrieval-ha
|
||||
|
||||
## Session Log
|
||||
|
||||
- **2026-04-24 Codex (audit improvements foundation)** Started implementation of the audit recommendations on branch `codex/audit-improvements-foundation` from `origin/main@c53e61e`. First tranche: registry-aware project-scoped retrieval filtering (`ATOCORE_RANK_PROJECT_SCOPE_FILTER`, widened candidate pull before filtering), eval harness known-issue lane, two p05 project-bleed fixtures, `scripts/live_status.py`, README/current-state/master-plan status refresh. Verified `pytest -q`: 550 passed in 67.11s. Live retrieval harness against undeployed production: 20 fixtures, 18 pass, 1 known issue (`p04-constraints` Zerodur/1.2 content gap), 1 blocking guard (`p05-broad-status-no-atomizer`) still failing because production has not yet deployed the retrieval filter and currently pulls `P04-GigaBIT-M1-KB-design` into broad p05 status context. Live dashboard refresh: health ok, build `2b86543`, docs 1748, chunks/vectors 33253, interactions 948, active memories 289, candidates 0, project_state total 128. Noted count discrepancy: dashboard memories.active=289 while integrity active_memory_count=951; schedule reconciliation in a follow-up.
|
||||
|
||||
- **2026-04-24 Codex (independent-audit hardening)** Applied the Opus independent audit's fast follow-ups before merge/deploy. Closed the two P1s by making project-scope ownership path/tag-based only, adding path-segment/tag-exact matching to avoid short-alias substring collisions, and keeping title/heading text out of provenance decisions. Added regression tests for title poisoning, substring collision, and unknown-project fallback. Added retrieval log fields `raw_results_count`, `post_filter_count`, `post_filter_dropped`, and `underfilled`. Added retrieval-eval run metadata (`generated_at`, `base_url`, `/health`) and `live_status.py` auth-token/status support. README now documents the ranking knobs and clarifies that the hard scope filter and soft project match boost are separate controls. Verified `pytest -q`: 553 passed in 66.07s. Live production remains expected-predeploy: 20 fixtures, 18 pass, 1 known content gap, 1 blocking p05 bleed guard. Latest live dashboard: build `2b86543`, docs 1748, chunks/vectors 33253, interactions 950, active memories 290, candidates 0, project_state total 128.
|
||||
|
||||
- **2026-04-23 Codex + Claude (R14 closed)** Codex reviewed `claude/r14-promote-400` at `3888db9`, no findings: "The route change is narrowly scoped: `promote_entity()` still returns False for not-found/not-candidate cases, so the existing 404 behavior remains intact, while caller-fixable validation failures now surface as 400." Ran `pytest tests/test_v1_0_write_invariants.py -q` from an isolated worktree: 15 passed in 1.91s. Claude squash-merged to main as `0989fed`, followed by ledger close-out `2b86543`, then deployed via canonical script. Dalidou `/health` reports build_sha=`2b86543e6ad26011b39a44509cc8df3809725171`, build_time `2026-04-23T15:20:53Z`, status=ok. R14 closed. Orientation refreshed earlier this session also reflected the V1-A gate status: **density gate CLEARED** (784 active memories vs 100 target — density batch-extract ran between 2026-04-22 and 2026-04-23 and more than crushed the gate), **soak gate at day 5 of ~7** (F4 first run 2026-04-19; nightly clean 2026-04-19 through 2026-04-23; only chronic failure is the known p04-constraints "Zerodur" content gap). V1-A branches from a clean V1-0 baseline as soon as the soak is called done.
|
||||
|
||||
- **2026-04-22 Codex + Antoine (V1-0 closed)** Codex approved `f16cd52` after re-running both original probes (legacy-candidate promote + supersede hook — both correct) and the three targeted regression suites (`test_v1_0_write_invariants.py`, `test_engineering_v1_phase5.py`, `test_inbox_crossproject.py` — all pass). Squash-merged to main as `2712c5d` ("feat(engineering): enforce V1-0 write invariants"). Deployed to Dalidou via the canonical deploy script; `/health` build_sha=`2712c5d2d03cb2a6af38b559664afd1c4cd0e050` status=ok. Validated backup snapshot at `/srv/storage/atocore/backups/snapshots/20260422T190624Z` taken BEFORE prod backfill. Prod backfill of `scripts/v1_0_backfill_provenance.py` against live DB: dry-run found 31 active/superseded entities with no provenance, list reviewed and looked sane; live run with default `hand_authored=1` flag path updated 31 rows; follow-up dry-run returned 0 rows remaining → no lingering F-8 violations in prod. Codex logged one residual P2 (R14): HTTP `POST /entities/{id}/promote` route doesn't translate the new service-layer `ValueError` into 400 — legacy bad candidate promoted through the API surfaces as 500. Not blocking. V1-0 closed. **Gates for V1-A**: soak window ends ~2026-04-26; 100-active-memory density target (currently 84 active + the ~31 newly flagged ones — need to check how those count in density math). V1-A holds until both gates clear.
|
||||
|
||||
25
README.md
25
README.md
@@ -6,7 +6,7 @@ Personal context engine that enriches LLM interactions with durable memory, stru
|
||||
|
||||
```bash
|
||||
pip install -e .
|
||||
uvicorn src.atocore.main:app --port 8100
|
||||
uvicorn atocore.main:app --port 8100
|
||||
```
|
||||
|
||||
## Usage
|
||||
@@ -37,6 +37,10 @@ python scripts/atocore_client.py audit-query "gigabit" 5
|
||||
| POST | /ingest | Ingest markdown file or folder |
|
||||
| POST | /query | Retrieve relevant chunks |
|
||||
| POST | /context/build | Build full context pack |
|
||||
| POST | /interactions | Capture prompt/response interactions |
|
||||
| GET/POST | /memory | List/create durable memories |
|
||||
| GET/POST | /entities | Engineering entity graph surface |
|
||||
| GET | /admin/dashboard | Operator dashboard |
|
||||
| GET | /health | Health check |
|
||||
| GET | /debug/context | Inspect last context pack |
|
||||
|
||||
@@ -66,8 +70,10 @@ unversioned forms.
|
||||
FastAPI (port 8100)
|
||||
|- Ingestion: markdown -> parse -> chunk -> embed -> store
|
||||
|- Retrieval: query -> embed -> vector search -> rank
|
||||
|- Context Builder: retrieve -> boost -> budget -> format
|
||||
|- SQLite (documents, chunks, memories, projects, interactions)
|
||||
|- Context Builder: project state -> memories -> entities -> retrieval -> budget
|
||||
|- Reflection: capture -> reinforce -> extract -> triage -> promote/expire
|
||||
|- Engineering: typed entities, relationships, conflicts, wiki/mirror
|
||||
|- SQLite (documents, chunks, memories, projects, interactions, entities)
|
||||
'- ChromaDB (vector embeddings)
|
||||
```
|
||||
|
||||
@@ -82,6 +88,16 @@ Set via environment variables (prefix `ATOCORE_`):
|
||||
| ATOCORE_CHUNK_MAX_SIZE | 800 | Max chunk size (chars) |
|
||||
| ATOCORE_CONTEXT_BUDGET | 3000 | Context pack budget (chars) |
|
||||
| ATOCORE_EMBEDDING_MODEL | paraphrase-multilingual-MiniLM-L12-v2 | Embedding model |
|
||||
| ATOCORE_RANK_PROJECT_MATCH_BOOST | 2.0 | Soft boost for chunks whose metadata matches the project hint |
|
||||
| ATOCORE_RANK_PROJECT_SCOPE_FILTER | true | Filter project-hinted retrieval away from other registered project corpora |
|
||||
| ATOCORE_RANK_PROJECT_SCOPE_CANDIDATE_MULTIPLIER | 4 | Widen candidate pull before project-scope filtering |
|
||||
| ATOCORE_RANK_QUERY_TOKEN_STEP | 0.08 | Per-token boost when query terms appear in high-signal metadata |
|
||||
| ATOCORE_RANK_QUERY_TOKEN_CAP | 1.32 | Maximum query-token boost multiplier |
|
||||
| ATOCORE_RANK_PATH_HIGH_SIGNAL_BOOST | 1.18 | Boost current decision/status/requirements-like paths |
|
||||
| ATOCORE_RANK_PATH_LOW_SIGNAL_PENALTY | 0.72 | Down-rank archive/history-like paths |
|
||||
|
||||
`ATOCORE_RANK_PROJECT_SCOPE_FILTER` gates the hard cross-project filter only.
|
||||
`ATOCORE_RANK_PROJECT_MATCH_BOOST` remains the separate soft-ranking knob.
|
||||
|
||||
## Testing
|
||||
|
||||
@@ -93,7 +109,10 @@ pytest
|
||||
## Operations
|
||||
|
||||
- `scripts/atocore_client.py` provides a live API client for project refresh, project-state inspection, and retrieval-quality audits.
|
||||
- `scripts/retrieval_eval.py` runs the live retrieval/context harness, separates blocking failures from known content gaps, and stamps JSON output with target/build metadata.
|
||||
- `scripts/live_status.py` renders a compact read-only status report from `/health`, `/stats`, `/projects`, and `/admin/dashboard`; set `ATOCORE_AUTH_TOKEN` or `--auth-token` when those endpoints are gated.
|
||||
- `docs/operations.md` captures the current operational priority order: retrieval quality, Wave 2 trusted-operational ingestion, AtoDrive scoping, and restore validation.
|
||||
- `DEV-LEDGER.md` is the fast-moving source of operational truth during active development; copy claims into docs only after checking the live service.
|
||||
|
||||
## Architecture Notes
|
||||
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
# AtoCore — Current State (2026-04-22)
|
||||
# AtoCore - Current State (2026-04-24)
|
||||
|
||||
Live deploy: `2712c5d` · Dalidou health: ok · Harness: 17/18 · Tests: 547 passing.
|
||||
Live deploy: `2b86543` · Dalidou health: ok · Harness: 18/20 with 1 known
|
||||
content gap and 1 current blocking project-bleed guard · Tests: 553 passing.
|
||||
|
||||
## V1-0 landed 2026-04-22
|
||||
|
||||
@@ -13,9 +14,8 @@ supersede) with Q-3 fail-open. Prod backfill ran cleanly — 31 legacy
|
||||
active/superseded entities flagged `hand_authored=1`, follow-up dry-run
|
||||
returned 0 remaining rows. Test count 533 → 547 (+14).
|
||||
|
||||
R14 (P2, non-blocking): `POST /entities/{id}/promote` route fix translates
|
||||
the new `ValueError` into 400. Branch `claude/r14-promote-400` pending
|
||||
Codex review + squash-merge.
|
||||
R14 is closed: `POST /entities/{id}/promote` now translates the new
|
||||
caller-fixable V1-0 `ValueError` into HTTP 400.
|
||||
|
||||
**Next in the V1 track:** V1-A (minimal query slice + Q-6 killer-correctness
|
||||
integration). Gated on pipeline soak (~2026-04-26) + 100+ active memory
|
||||
@@ -65,10 +65,10 @@ Last nightly run (2026-04-19 03:00 UTC): **31 promoted · 39 rejected · 0 needs
|
||||
| 7G | Re-extraction on prompt version bump | pending |
|
||||
| 7H | Chroma vector hygiene (delete vectors for superseded memories) | pending |
|
||||
|
||||
## Known gaps (honest)
|
||||
## Known gaps (honest, refreshed 2026-04-24)
|
||||
|
||||
1. **Capture surface is Claude-Code-and-OpenClaw only.** Conversations in Claude Desktop, Claude.ai web, phone, or any other LLM UI are NOT captured. Example: the rotovap/mushroom chat yesterday never reached AtoCore because no hook fired. See Q4 below.
|
||||
2. **OpenClaw is capture-only, not context-grounded.** The plugin POSTs `/interactions` on `llm_output` but does NOT call `/context/build` on `before_agent_start`. OpenClaw's underlying agent runs blind. See Q2 below.
|
||||
3. **Human interface (wiki) is thin and static.** 5 project cards + a "System" line. No dashboard for the autonomous activity. No per-memory detail page. See Q3/Q5.
|
||||
4. **Harness 17/18** — the `p04-constraints` fixture wants "Zerodur" but retrieval surfaces related-not-exact terms. Content gap, not a retrieval regression.
|
||||
5. **Two projects under-populated**: p05-interferometer (4 memories, 18 state) and atomizer-v2 (1 memory, 6 state). Batch re-extract with the new llm-0.6.0 prompt would help.
|
||||
2. **Project-scoped retrieval still needs deployment verification.** The April 24 audit reproduced cross-project competition on broad p05 prompts. The current branch adds registry-aware project filtering and a harness guard; verify after deploy.
|
||||
3. **Human interface is useful but not yet the V1 Human Mirror.** Wiki/dashboard pages exist, but the spec routes, deterministic mirror files, disputed markers, and curated annotations remain V1-D work.
|
||||
4. **Harness known issue:** `p04-constraints` wants "Zerodur" and "1.2"; live retrieval surfaces related constraints but not those exact strings. Treat as content/state gap until fixed.
|
||||
5. **Formal docs lag the ledger during fast work.** Use `DEV-LEDGER.md` and `python scripts/live_status.py` for live truth, then copy verified claims into these docs.
|
||||
|
||||
@@ -70,9 +70,14 @@ read-only additive mode.
|
||||
- Phase 6 - AtoDrive
|
||||
- Phase 10 - Write-back
|
||||
- Phase 11 - Multi-model
|
||||
- Phase 12 - Evaluation
|
||||
- Phase 13 - Hardening
|
||||
|
||||
### Partial / Operational Baseline
|
||||
|
||||
- Phase 12 - Evaluation. The retrieval/context harness exists and runs
|
||||
against live Dalidou, but coverage is still intentionally small and
|
||||
should grow before this is complete in the intended sense.
|
||||
|
||||
### Engineering Layer Planning Sprint
|
||||
|
||||
**Status: complete.** All 8 architecture docs are drafted. The
|
||||
@@ -126,11 +131,13 @@ This sits implicitly between Phase 8 (OpenClaw) and Phase 11
|
||||
(multi-model). Memory-review and engineering-entity commands are
|
||||
deferred from the shared client until their workflows are exercised.
|
||||
|
||||
## What Is Real Today (updated 2026-04-16)
|
||||
## What Is Real Today (updated 2026-04-24)
|
||||
|
||||
- canonical AtoCore runtime on Dalidou (`775960c`, deploy.sh verified)
|
||||
- canonical AtoCore runtime on Dalidou (`2b86543`, deploy.sh verified)
|
||||
- 33,253 vectors across 6 registered projects
|
||||
- 234 captured interactions (192 claude-code, 38 openclaw, 4 test)
|
||||
- 950 captured interactions as of the 2026-04-24 live dashboard; refresh
|
||||
exact live counts with
|
||||
`python scripts/live_status.py`
|
||||
- 6 registered projects:
|
||||
- `p04-gigabit` (483 docs, 15 state entries)
|
||||
- `p05-interferometer` (109 docs, 18 state entries)
|
||||
@@ -138,12 +145,15 @@ deferred from the shared client until their workflows are exercised.
|
||||
- `atomizer-v2` (568 docs, 5 state entries)
|
||||
- `abb-space` (6 state entries)
|
||||
- `atocore` (drive source, 47 state entries)
|
||||
- 110 Trusted Project State entries across all projects (decisions, requirements, facts, contacts, milestones)
|
||||
- 84 active memories (31 project, 23 knowledge, 10 episodic, 8 adaptation, 7 preference, 5 identity)
|
||||
- 128 Trusted Project State entries across all projects (decisions, requirements, facts, contacts, milestones)
|
||||
- 290 active memories and 0 candidate memories as of the 2026-04-24 live
|
||||
dashboard
|
||||
- context pack assembly with 4 tiers: Trusted Project State > identity/preference > project memories > retrieved chunks
|
||||
- query-relevance memory ranking with overlap-density scoring
|
||||
- retrieval eval harness: 18 fixtures, 17/18 passing on live
|
||||
- 303 tests passing
|
||||
- retrieval eval harness: 20 fixtures; current live has 18 pass, 1 known
|
||||
content gap, and 1 blocking cross-project bleed guard targeted by the
|
||||
current retrieval-scoping branch
|
||||
- 553 tests passing on the audit-improvements branch
|
||||
- nightly pipeline: backup → cleanup → rsync → OpenClaw import → vault refresh → extract → triage → **auto-promote/expire** → weekly synth/lint → **retrieval harness** → **pipeline summary to project state**
|
||||
- Phase 10 operational: reinforcement-based auto-promotion (ref_count ≥ 3, confidence ≥ 0.7) + stale candidate expiry (14 days unreinforced)
|
||||
- pipeline health visible in dashboard: interaction totals by client, pipeline last_run, harness results, triage stats
|
||||
@@ -190,9 +200,9 @@ where surfaces are disjoint, pauses when they collide.
|
||||
| V1-E | Memory→entity graduation end-to-end + remaining Q-4 trust tests | pending V1-D (note: collides with memory extractor; pauses for multi-model triage work) |
|
||||
| V1-F | F-5 detector generalization + route alias + O-1/O-2/O-3 operational + D-1/D-3/D-4 docs | finish line |
|
||||
|
||||
R14 (P2, non-blocking): `POST /entities/{id}/promote` route returns 500
|
||||
on the new V1-0 `ValueError` instead of 400. Fix on branch
|
||||
`claude/r14-promote-400`, pending Codex review.
|
||||
R14 is closed: `POST /entities/{id}/promote` now translates
|
||||
caller-fixable V1-0 provenance validation failures into HTTP 400 instead
|
||||
of leaking as HTTP 500.
|
||||
|
||||
## Next
|
||||
|
||||
|
||||
131
scripts/live_status.py
Normal file
131
scripts/live_status.py
Normal file
@@ -0,0 +1,131 @@
|
||||
"""Render a compact live-status report from a running AtoCore instance.
|
||||
|
||||
This is intentionally read-only and stdlib-only so it can be used from a
|
||||
fresh checkout, a cron job, or a Codex/Claude session without installing the
|
||||
full app package. The output is meant to reduce docs drift: copy the report
|
||||
into status docs only after it was generated from the live service.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import errno
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from typing import Any
|
||||
|
||||
|
||||
DEFAULT_BASE_URL = os.environ.get("ATOCORE_BASE_URL", "http://dalidou:8100").rstrip("/")
|
||||
DEFAULT_TIMEOUT = int(os.environ.get("ATOCORE_TIMEOUT_SECONDS", "30"))
|
||||
DEFAULT_AUTH_TOKEN = os.environ.get("ATOCORE_AUTH_TOKEN", "").strip()
|
||||
|
||||
|
||||
def request_json(base_url: str, path: str, timeout: int, auth_token: str = "") -> dict[str, Any]:
|
||||
headers = {"Authorization": f"Bearer {auth_token}"} if auth_token else {}
|
||||
req = urllib.request.Request(f"{base_url}{path}", method="GET", headers=headers)
|
||||
with urllib.request.urlopen(req, timeout=timeout) as response:
|
||||
body = response.read().decode("utf-8")
|
||||
status = getattr(response, "status", None)
|
||||
payload = json.loads(body) if body.strip() else {}
|
||||
if not isinstance(payload, dict):
|
||||
payload = {"value": payload}
|
||||
if status is not None:
|
||||
payload["_http_status"] = status
|
||||
return payload
|
||||
|
||||
|
||||
def collect_status(base_url: str, timeout: int, auth_token: str = "") -> dict[str, Any]:
|
||||
payload: dict[str, Any] = {"base_url": base_url}
|
||||
for name, path in {
|
||||
"health": "/health",
|
||||
"stats": "/stats",
|
||||
"projects": "/projects",
|
||||
"dashboard": "/admin/dashboard",
|
||||
}.items():
|
||||
try:
|
||||
payload[name] = request_json(base_url, path, timeout, auth_token)
|
||||
except (urllib.error.URLError, TimeoutError, OSError, json.JSONDecodeError) as exc:
|
||||
payload[name] = {"error": str(exc)}
|
||||
return payload
|
||||
|
||||
|
||||
def render_markdown(status: dict[str, Any]) -> str:
|
||||
health = status.get("health", {})
|
||||
stats = status.get("stats", {})
|
||||
projects = status.get("projects", {}).get("projects", [])
|
||||
dashboard = status.get("dashboard", {})
|
||||
memories = dashboard.get("memories", {}) if isinstance(dashboard.get("memories"), dict) else {}
|
||||
project_state = dashboard.get("project_state", {}) if isinstance(dashboard.get("project_state"), dict) else {}
|
||||
interactions = dashboard.get("interactions", {}) if isinstance(dashboard.get("interactions"), dict) else {}
|
||||
pipeline = dashboard.get("pipeline", {}) if isinstance(dashboard.get("pipeline"), dict) else {}
|
||||
|
||||
lines = [
|
||||
"# AtoCore Live Status",
|
||||
"",
|
||||
f"- base_url: `{status.get('base_url', '')}`",
|
||||
"- endpoint_http_statuses: "
|
||||
f"`health={health.get('_http_status', 'error')}, "
|
||||
f"stats={stats.get('_http_status', 'error')}, "
|
||||
f"projects={status.get('projects', {}).get('_http_status', 'error')}, "
|
||||
f"dashboard={dashboard.get('_http_status', 'error')}`",
|
||||
f"- service_status: `{health.get('status', 'unknown')}`",
|
||||
f"- code_version: `{health.get('code_version', health.get('version', 'unknown'))}`",
|
||||
f"- build_sha: `{health.get('build_sha', 'unknown')}`",
|
||||
f"- build_branch: `{health.get('build_branch', 'unknown')}`",
|
||||
f"- build_time: `{health.get('build_time', 'unknown')}`",
|
||||
f"- env: `{health.get('env', 'unknown')}`",
|
||||
f"- documents: `{stats.get('total_documents', 'unknown')}`",
|
||||
f"- chunks: `{stats.get('total_chunks', 'unknown')}`",
|
||||
f"- vectors: `{stats.get('total_vectors', health.get('vectors_count', 'unknown'))}`",
|
||||
f"- registered_projects: `{len(projects)}`",
|
||||
f"- active_memories: `{memories.get('active', 'unknown')}`",
|
||||
f"- candidate_memories: `{memories.get('candidates', 'unknown')}`",
|
||||
f"- interactions: `{interactions.get('total', 'unknown')}`",
|
||||
f"- project_state_entries: `{project_state.get('total', 'unknown')}`",
|
||||
f"- pipeline_last_run: `{pipeline.get('last_run', 'unknown')}`",
|
||||
]
|
||||
|
||||
if projects:
|
||||
lines.extend(["", "## Projects"])
|
||||
for project in projects:
|
||||
aliases = ", ".join(project.get("aliases", []))
|
||||
suffix = f" ({aliases})" if aliases else ""
|
||||
lines.append(f"- `{project.get('id', '')}`{suffix}")
|
||||
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description="Render live AtoCore status")
|
||||
parser.add_argument("--base-url", default=DEFAULT_BASE_URL)
|
||||
parser.add_argument("--timeout", type=int, default=DEFAULT_TIMEOUT)
|
||||
parser.add_argument(
|
||||
"--auth-token",
|
||||
default=DEFAULT_AUTH_TOKEN,
|
||||
help="Bearer token; defaults to ATOCORE_AUTH_TOKEN when set",
|
||||
)
|
||||
parser.add_argument("--json", action="store_true", help="emit raw JSON")
|
||||
args = parser.parse_args()
|
||||
|
||||
status = collect_status(args.base_url.rstrip("/"), args.timeout, args.auth_token)
|
||||
if args.json:
|
||||
output = json.dumps(status, indent=2, ensure_ascii=True) + "\n"
|
||||
else:
|
||||
output = render_markdown(status)
|
||||
|
||||
try:
|
||||
sys.stdout.write(output)
|
||||
except BrokenPipeError:
|
||||
return 0
|
||||
except OSError as exc:
|
||||
if exc.errno in {errno.EINVAL, errno.EPIPE}:
|
||||
return 0
|
||||
raise
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -44,6 +44,7 @@ import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
DEFAULT_BASE_URL = os.environ.get("ATOCORE_BASE_URL", "http://dalidou:8100")
|
||||
@@ -52,6 +53,13 @@ DEFAULT_BUDGET = 3000
|
||||
DEFAULT_FIXTURES = Path(__file__).parent / "retrieval_eval_fixtures.json"
|
||||
|
||||
|
||||
def request_json(base_url: str, path: str, timeout: int) -> dict:
|
||||
req = urllib.request.Request(f"{base_url}{path}", method="GET")
|
||||
with urllib.request.urlopen(req, timeout=timeout) as resp:
|
||||
body = resp.read().decode("utf-8")
|
||||
return json.loads(body) if body.strip() else {}
|
||||
|
||||
|
||||
@dataclass
|
||||
class Fixture:
|
||||
name: str
|
||||
@@ -60,6 +68,7 @@ class Fixture:
|
||||
budget: int = DEFAULT_BUDGET
|
||||
expect_present: list[str] = field(default_factory=list)
|
||||
expect_absent: list[str] = field(default_factory=list)
|
||||
known_issue: bool = False
|
||||
notes: str = ""
|
||||
|
||||
|
||||
@@ -70,8 +79,13 @@ class FixtureResult:
|
||||
missing_present: list[str]
|
||||
unexpected_absent: list[str]
|
||||
total_chars: int
|
||||
known_issue: bool = False
|
||||
error: str = ""
|
||||
|
||||
@property
|
||||
def blocking_failure(self) -> bool:
|
||||
return not self.ok and not self.known_issue
|
||||
|
||||
|
||||
def load_fixtures(path: Path) -> list[Fixture]:
|
||||
data = json.loads(path.read_text(encoding="utf-8"))
|
||||
@@ -89,6 +103,7 @@ def load_fixtures(path: Path) -> list[Fixture]:
|
||||
budget=int(raw.get("budget", DEFAULT_BUDGET)),
|
||||
expect_present=list(raw.get("expect_present", [])),
|
||||
expect_absent=list(raw.get("expect_absent", [])),
|
||||
known_issue=bool(raw.get("known_issue", False)),
|
||||
notes=raw.get("notes", ""),
|
||||
)
|
||||
)
|
||||
@@ -117,6 +132,7 @@ def run_fixture(fixture: Fixture, base_url: str, timeout: int) -> FixtureResult:
|
||||
missing_present=list(fixture.expect_present),
|
||||
unexpected_absent=[],
|
||||
total_chars=0,
|
||||
known_issue=fixture.known_issue,
|
||||
error=f"http_error: {exc}",
|
||||
)
|
||||
|
||||
@@ -129,16 +145,26 @@ def run_fixture(fixture: Fixture, base_url: str, timeout: int) -> FixtureResult:
|
||||
missing_present=missing,
|
||||
unexpected_absent=unexpected,
|
||||
total_chars=len(formatted),
|
||||
known_issue=fixture.known_issue,
|
||||
)
|
||||
|
||||
|
||||
def print_human_report(results: list[FixtureResult]) -> None:
|
||||
def print_human_report(results: list[FixtureResult], metadata: dict) -> None:
|
||||
total = len(results)
|
||||
passed = sum(1 for r in results if r.ok)
|
||||
known = sum(1 for r in results if not r.ok and r.known_issue)
|
||||
blocking = sum(1 for r in results if r.blocking_failure)
|
||||
print(f"Retrieval eval: {passed}/{total} fixtures passed")
|
||||
print(
|
||||
"Target: "
|
||||
f"{metadata.get('base_url', 'unknown')} "
|
||||
f"build={metadata.get('health', {}).get('build_sha', 'unknown')}"
|
||||
)
|
||||
if known or blocking:
|
||||
print(f"Blocking failures: {blocking} Known issues: {known}")
|
||||
print()
|
||||
for r in results:
|
||||
marker = "PASS" if r.ok else "FAIL"
|
||||
marker = "PASS" if r.ok else ("KNOWN" if r.known_issue else "FAIL")
|
||||
print(f"[{marker}] {r.fixture.name} project={r.fixture.project} chars={r.total_chars}")
|
||||
if r.error:
|
||||
print(f" error: {r.error}")
|
||||
@@ -150,15 +176,21 @@ def print_human_report(results: list[FixtureResult]) -> None:
|
||||
print(f" notes: {r.fixture.notes}")
|
||||
|
||||
|
||||
def print_json_report(results: list[FixtureResult]) -> None:
|
||||
def print_json_report(results: list[FixtureResult], metadata: dict) -> None:
|
||||
payload = {
|
||||
"generated_at": metadata.get("generated_at"),
|
||||
"base_url": metadata.get("base_url"),
|
||||
"health": metadata.get("health", {}),
|
||||
"total": len(results),
|
||||
"passed": sum(1 for r in results if r.ok),
|
||||
"known_issues": sum(1 for r in results if not r.ok and r.known_issue),
|
||||
"blocking_failures": sum(1 for r in results if r.blocking_failure),
|
||||
"fixtures": [
|
||||
{
|
||||
"name": r.fixture.name,
|
||||
"project": r.fixture.project,
|
||||
"ok": r.ok,
|
||||
"known_issue": r.known_issue,
|
||||
"total_chars": r.total_chars,
|
||||
"missing_present": r.missing_present,
|
||||
"unexpected_absent": r.unexpected_absent,
|
||||
@@ -179,15 +211,26 @@ def main() -> int:
|
||||
parser.add_argument("--json", action="store_true", help="emit machine-readable JSON")
|
||||
args = parser.parse_args()
|
||||
|
||||
base_url = args.base_url.rstrip("/")
|
||||
try:
|
||||
health = request_json(base_url, "/health", args.timeout)
|
||||
except (urllib.error.URLError, TimeoutError, OSError, json.JSONDecodeError) as exc:
|
||||
health = {"error": str(exc)}
|
||||
metadata = {
|
||||
"generated_at": datetime.now(timezone.utc).isoformat(),
|
||||
"base_url": base_url,
|
||||
"health": health,
|
||||
}
|
||||
|
||||
fixtures = load_fixtures(args.fixtures)
|
||||
results = [run_fixture(f, args.base_url, args.timeout) for f in fixtures]
|
||||
results = [run_fixture(f, base_url, args.timeout) for f in fixtures]
|
||||
|
||||
if args.json:
|
||||
print_json_report(results)
|
||||
print_json_report(results, metadata)
|
||||
else:
|
||||
print_human_report(results)
|
||||
print_human_report(results, metadata)
|
||||
|
||||
return 0 if all(r.ok for r in results) else 1
|
||||
return 0 if not any(r.blocking_failure for r in results) else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
@@ -27,7 +27,8 @@
|
||||
"expect_absent": [
|
||||
"polisher suite"
|
||||
],
|
||||
"notes": "Key constraints are in Trusted Project State and in the mission-framing memory"
|
||||
"known_issue": true,
|
||||
"notes": "Known content gap as of 2026-04-24: live retrieval surfaces related constraints but not the exact Zerodur / 1.2 strings. Keep visible, but do not make nightly harness red until the source/state gap is fixed."
|
||||
},
|
||||
{
|
||||
"name": "p04-short-ambiguous",
|
||||
@@ -80,6 +81,36 @@
|
||||
],
|
||||
"notes": "CGH is a core p05 concept. Should surface via chunks and possibly the architecture memory. Must not bleed p06 polisher-suite terms."
|
||||
},
|
||||
{
|
||||
"name": "p05-broad-status-no-atomizer",
|
||||
"project": "p05-interferometer",
|
||||
"prompt": "current status",
|
||||
"expect_present": [
|
||||
"--- Trusted Project State ---",
|
||||
"--- Project Memories ---",
|
||||
"Zygo"
|
||||
],
|
||||
"expect_absent": [
|
||||
"atomizer-v2",
|
||||
"ATOMIZER_PODCAST_BRIEFING",
|
||||
"[Source: atomizer-v2/",
|
||||
"P04-GigaBIT-M1-KB-design"
|
||||
],
|
||||
"notes": "Regression guard for the April 24 audit finding: broad p05 status queries must not pull Atomizer/archive context into project-scoped packs."
|
||||
},
|
||||
{
|
||||
"name": "p05-vendor-decision-no-archive-first",
|
||||
"project": "p05-interferometer",
|
||||
"prompt": "vendor selection decision",
|
||||
"expect_present": [
|
||||
"Selection-Decision"
|
||||
],
|
||||
"expect_absent": [
|
||||
"[Source: atomizer-v2/",
|
||||
"ATOMIZER_PODCAST_BRIEFING"
|
||||
],
|
||||
"notes": "Project-scoped decision query should stay inside p05 and prefer current decision/vendor material over unrelated project archives."
|
||||
},
|
||||
{
|
||||
"name": "p06-suite-split",
|
||||
"project": "p06-polisher",
|
||||
|
||||
@@ -46,6 +46,8 @@ class Settings(BaseSettings):
|
||||
# All multipliers default to the values used since Wave 1; tighten or
|
||||
# loosen them via ATOCORE_* env vars without touching code.
|
||||
rank_project_match_boost: float = 2.0
|
||||
rank_project_scope_filter: bool = True
|
||||
rank_project_scope_candidate_multiplier: int = 4
|
||||
rank_query_token_step: float = 0.08
|
||||
rank_query_token_cap: float = 1.32
|
||||
rank_path_high_signal_boost: float = 1.18
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
"""Retrieval: query to ranked chunks."""
|
||||
|
||||
import json
|
||||
import re
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
@@ -7,7 +8,7 @@ from dataclasses import dataclass
|
||||
import atocore.config as _config
|
||||
from atocore.models.database import get_connection
|
||||
from atocore.observability.logger import get_logger
|
||||
from atocore.projects.registry import get_registered_project
|
||||
from atocore.projects.registry import RegisteredProject, get_registered_project, load_project_registry
|
||||
from atocore.retrieval.embeddings import embed_query
|
||||
from atocore.retrieval.vector_store import get_vector_store
|
||||
|
||||
@@ -83,6 +84,19 @@ def retrieve(
|
||||
"""Retrieve the most relevant chunks for a query."""
|
||||
top_k = top_k or _config.settings.context_top_k
|
||||
start = time.time()
|
||||
scoped_project = get_registered_project(project_hint) if project_hint else None
|
||||
scope_filter_enabled = bool(scoped_project and _config.settings.rank_project_scope_filter)
|
||||
registered_projects = None
|
||||
query_top_k = top_k
|
||||
if scope_filter_enabled:
|
||||
query_top_k = max(
|
||||
top_k,
|
||||
top_k * max(1, _config.settings.rank_project_scope_candidate_multiplier),
|
||||
)
|
||||
try:
|
||||
registered_projects = load_project_registry()
|
||||
except Exception:
|
||||
registered_projects = None
|
||||
|
||||
query_embedding = embed_query(query)
|
||||
store = get_vector_store()
|
||||
@@ -101,11 +115,12 @@ def retrieve(
|
||||
|
||||
results = store.query(
|
||||
query_embedding=query_embedding,
|
||||
top_k=top_k,
|
||||
top_k=query_top_k,
|
||||
where=where,
|
||||
)
|
||||
|
||||
chunks = []
|
||||
raw_result_count = len(results["ids"][0]) if results and results["ids"] and results["ids"][0] else 0
|
||||
if results and results["ids"] and results["ids"][0]:
|
||||
existing_ids = _existing_chunk_ids(results["ids"][0])
|
||||
for i, chunk_id in enumerate(results["ids"][0]):
|
||||
@@ -117,6 +132,13 @@ def retrieve(
|
||||
meta = results["metadatas"][0][i] if results["metadatas"] else {}
|
||||
content = results["documents"][0][i] if results["documents"] else ""
|
||||
|
||||
if scope_filter_enabled and not _is_allowed_for_project_scope(
|
||||
scoped_project,
|
||||
meta,
|
||||
registered_projects,
|
||||
):
|
||||
continue
|
||||
|
||||
score *= _query_match_boost(query, meta)
|
||||
score *= _path_signal_boost(meta)
|
||||
if project_hint:
|
||||
@@ -137,42 +159,139 @@ def retrieve(
|
||||
|
||||
duration_ms = int((time.time() - start) * 1000)
|
||||
chunks.sort(key=lambda chunk: chunk.score, reverse=True)
|
||||
post_filter_count = len(chunks)
|
||||
chunks = chunks[:top_k]
|
||||
|
||||
log.info(
|
||||
"retrieval_done",
|
||||
query=query[:100],
|
||||
top_k=top_k,
|
||||
query_top_k=query_top_k,
|
||||
raw_results_count=raw_result_count,
|
||||
post_filter_count=post_filter_count,
|
||||
results_count=len(chunks),
|
||||
post_filter_dropped=max(0, raw_result_count - post_filter_count),
|
||||
underfilled=bool(raw_result_count >= query_top_k and len(chunks) < top_k),
|
||||
duration_ms=duration_ms,
|
||||
)
|
||||
|
||||
return chunks
|
||||
|
||||
|
||||
def _is_allowed_for_project_scope(
|
||||
project: RegisteredProject,
|
||||
metadata: dict,
|
||||
registered_projects: list[RegisteredProject] | None = None,
|
||||
) -> bool:
|
||||
"""Return True when a chunk is target-project or not project-owned.
|
||||
|
||||
Project-hinted retrieval should not let one registered project's corpus
|
||||
compete with another's. At the same time, unowned/global sources should
|
||||
remain eligible because shared docs and cross-project references can be
|
||||
genuinely useful. The registry gives us the boundary: if metadata matches
|
||||
a registered project and it is not the requested project, filter it out.
|
||||
"""
|
||||
if _metadata_matches_project(project, metadata):
|
||||
return True
|
||||
|
||||
if registered_projects is None:
|
||||
try:
|
||||
registered_projects = load_project_registry()
|
||||
except Exception:
|
||||
return True
|
||||
|
||||
for other in registered_projects:
|
||||
if other.project_id == project.project_id:
|
||||
continue
|
||||
if _metadata_matches_project(other, metadata):
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def _metadata_matches_project(project: RegisteredProject, metadata: dict) -> bool:
|
||||
path = _metadata_source_path(metadata)
|
||||
tags = _metadata_tags(metadata)
|
||||
for term in _project_scope_terms(project):
|
||||
if _path_matches_term(path, term) or term in tags:
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _project_scope_terms(project: RegisteredProject) -> set[str]:
|
||||
terms = {project.project_id.lower()}
|
||||
terms.update(alias.lower() for alias in project.aliases)
|
||||
for source_ref in project.ingest_roots:
|
||||
normalized = source_ref.subpath.replace("\\", "/").strip("/").lower()
|
||||
if normalized:
|
||||
terms.add(normalized)
|
||||
terms.add(normalized.split("/")[-1])
|
||||
return {term for term in terms if term}
|
||||
|
||||
|
||||
def _metadata_searchable(metadata: dict) -> str:
|
||||
return " ".join(
|
||||
[
|
||||
str(metadata.get("source_file", "")).replace("\\", "/").lower(),
|
||||
str(metadata.get("title", "")).lower(),
|
||||
str(metadata.get("heading_path", "")).lower(),
|
||||
str(metadata.get("tags", "")).lower(),
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
def _metadata_source_path(metadata: dict) -> str:
|
||||
return str(metadata.get("source_file", "")).replace("\\", "/").strip("/").lower()
|
||||
|
||||
|
||||
def _metadata_tags(metadata: dict) -> set[str]:
|
||||
raw_tags = metadata.get("tags", [])
|
||||
if isinstance(raw_tags, (list, tuple, set)):
|
||||
return {str(tag).strip().lower() for tag in raw_tags if str(tag).strip()}
|
||||
if isinstance(raw_tags, str):
|
||||
try:
|
||||
parsed = json.loads(raw_tags)
|
||||
except json.JSONDecodeError:
|
||||
parsed = [raw_tags]
|
||||
if isinstance(parsed, (list, tuple, set)):
|
||||
return {str(tag).strip().lower() for tag in parsed if str(tag).strip()}
|
||||
if isinstance(parsed, str) and parsed.strip():
|
||||
return {parsed.strip().lower()}
|
||||
return set()
|
||||
|
||||
|
||||
def _path_matches_term(path: str, term: str) -> bool:
|
||||
normalized = term.replace("\\", "/").strip("/").lower()
|
||||
if not path or not normalized:
|
||||
return False
|
||||
if "/" in normalized:
|
||||
return path == normalized or path.startswith(f"{normalized}/")
|
||||
return normalized in set(path.split("/"))
|
||||
|
||||
|
||||
def _metadata_has_term(metadata: dict, term: str) -> bool:
|
||||
normalized = term.replace("\\", "/").strip("/").lower()
|
||||
if not normalized:
|
||||
return False
|
||||
if _path_matches_term(_metadata_source_path(metadata), normalized):
|
||||
return True
|
||||
if normalized in _metadata_tags(metadata):
|
||||
return True
|
||||
return re.search(
|
||||
rf"(?<![a-z0-9]){re.escape(normalized)}(?![a-z0-9])",
|
||||
_metadata_searchable(metadata),
|
||||
) is not None
|
||||
|
||||
|
||||
def _project_match_boost(project_hint: str, metadata: dict) -> float:
|
||||
"""Return a project-aware relevance multiplier for raw retrieval."""
|
||||
hint_lower = project_hint.strip().lower()
|
||||
if not hint_lower:
|
||||
return 1.0
|
||||
|
||||
source_file = str(metadata.get("source_file", "")).lower()
|
||||
title = str(metadata.get("title", "")).lower()
|
||||
tags = str(metadata.get("tags", "")).lower()
|
||||
searchable = " ".join([source_file, title, tags])
|
||||
|
||||
project = get_registered_project(project_hint)
|
||||
candidate_names = {hint_lower}
|
||||
if project is not None:
|
||||
candidate_names.add(project.project_id.lower())
|
||||
candidate_names.update(alias.lower() for alias in project.aliases)
|
||||
candidate_names.update(
|
||||
source_ref.subpath.replace("\\", "/").strip("/").split("/")[-1].lower()
|
||||
for source_ref in project.ingest_roots
|
||||
if source_ref.subpath.strip("/\\")
|
||||
)
|
||||
|
||||
candidate_names = _project_scope_terms(project) if project is not None else {hint_lower}
|
||||
for candidate in candidate_names:
|
||||
if candidate and candidate in searchable:
|
||||
if _metadata_has_term(metadata, candidate):
|
||||
return _config.settings.rank_project_match_boost
|
||||
|
||||
return 1.0
|
||||
|
||||
@@ -46,6 +46,8 @@ def test_settings_keep_legacy_db_path_when_present(tmp_path, monkeypatch):
|
||||
|
||||
def test_ranking_weights_are_tunable_via_env(monkeypatch):
|
||||
monkeypatch.setenv("ATOCORE_RANK_PROJECT_MATCH_BOOST", "3.5")
|
||||
monkeypatch.setenv("ATOCORE_RANK_PROJECT_SCOPE_FILTER", "false")
|
||||
monkeypatch.setenv("ATOCORE_RANK_PROJECT_SCOPE_CANDIDATE_MULTIPLIER", "6")
|
||||
monkeypatch.setenv("ATOCORE_RANK_QUERY_TOKEN_STEP", "0.12")
|
||||
monkeypatch.setenv("ATOCORE_RANK_QUERY_TOKEN_CAP", "1.5")
|
||||
monkeypatch.setenv("ATOCORE_RANK_PATH_HIGH_SIGNAL_BOOST", "1.25")
|
||||
@@ -54,6 +56,8 @@ def test_ranking_weights_are_tunable_via_env(monkeypatch):
|
||||
settings = config.Settings()
|
||||
|
||||
assert settings.rank_project_match_boost == 3.5
|
||||
assert settings.rank_project_scope_filter is False
|
||||
assert settings.rank_project_scope_candidate_multiplier == 6
|
||||
assert settings.rank_query_token_step == 0.12
|
||||
assert settings.rank_query_token_cap == 1.5
|
||||
assert settings.rank_path_high_signal_boost == 1.25
|
||||
|
||||
@@ -70,8 +70,28 @@ def test_retrieve_skips_stale_vector_entries(tmp_data_dir, sample_markdown, monk
|
||||
|
||||
|
||||
def test_retrieve_project_hint_boosts_matching_chunks(monkeypatch):
|
||||
target_project = type(
|
||||
"Project",
|
||||
(),
|
||||
{
|
||||
"project_id": "p04-gigabit",
|
||||
"aliases": ("p04", "gigabit"),
|
||||
"ingest_roots": (),
|
||||
},
|
||||
)()
|
||||
other_project = type(
|
||||
"Project",
|
||||
(),
|
||||
{
|
||||
"project_id": "p05-interferometer",
|
||||
"aliases": ("p05", "interferometer"),
|
||||
"ingest_roots": (),
|
||||
},
|
||||
)()
|
||||
|
||||
class FakeStore:
|
||||
def query(self, query_embedding, top_k=10, where=None):
|
||||
assert top_k == 8
|
||||
return {
|
||||
"ids": [["chunk-a", "chunk-b"]],
|
||||
"documents": [["project doc", "other doc"]],
|
||||
@@ -102,7 +122,21 @@ def test_retrieve_project_hint_boosts_matching_chunks(monkeypatch):
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"atocore.retrieval.retriever.get_registered_project",
|
||||
lambda project_name: type(
|
||||
lambda project_name: target_project,
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"atocore.retrieval.retriever.load_project_registry",
|
||||
lambda: [target_project, other_project],
|
||||
)
|
||||
|
||||
results = retrieve("mirror architecture", top_k=2, project_hint="p04")
|
||||
|
||||
assert len(results) == 1
|
||||
assert results[0].chunk_id == "chunk-a"
|
||||
|
||||
|
||||
def test_retrieve_project_scope_allows_unowned_global_chunks(monkeypatch):
|
||||
target_project = type(
|
||||
"Project",
|
||||
(),
|
||||
{
|
||||
@@ -110,14 +144,286 @@ def test_retrieve_project_hint_boosts_matching_chunks(monkeypatch):
|
||||
"aliases": ("p04", "gigabit"),
|
||||
"ingest_roots": (),
|
||||
},
|
||||
)(),
|
||||
)()
|
||||
|
||||
class FakeStore:
|
||||
def query(self, query_embedding, top_k=10, where=None):
|
||||
return {
|
||||
"ids": [["chunk-a", "chunk-global"]],
|
||||
"documents": [["project doc", "global doc"]],
|
||||
"metadatas": [[
|
||||
{
|
||||
"heading_path": "Overview",
|
||||
"source_file": "p04-gigabit/pkm/_index.md",
|
||||
"tags": '["p04-gigabit"]',
|
||||
"title": "P04",
|
||||
"document_id": "doc-a",
|
||||
},
|
||||
{
|
||||
"heading_path": "Overview",
|
||||
"source_file": "shared/engineering-rules.md",
|
||||
"tags": "[]",
|
||||
"title": "Shared engineering rules",
|
||||
"document_id": "doc-global",
|
||||
},
|
||||
]],
|
||||
"distances": [[0.2, 0.21]],
|
||||
}
|
||||
|
||||
monkeypatch.setattr("atocore.retrieval.retriever.get_vector_store", lambda: FakeStore())
|
||||
monkeypatch.setattr("atocore.retrieval.retriever.embed_query", lambda query: [0.0, 0.1])
|
||||
monkeypatch.setattr(
|
||||
"atocore.retrieval.retriever._existing_chunk_ids",
|
||||
lambda chunk_ids: set(chunk_ids),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"atocore.retrieval.retriever.get_registered_project",
|
||||
lambda project_name: target_project,
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"atocore.retrieval.retriever.load_project_registry",
|
||||
lambda: [target_project],
|
||||
)
|
||||
|
||||
results = retrieve("mirror architecture", top_k=2, project_hint="p04")
|
||||
|
||||
assert len(results) == 2
|
||||
assert results[0].chunk_id == "chunk-a"
|
||||
assert results[0].score > results[1].score
|
||||
assert [r.chunk_id for r in results] == ["chunk-a", "chunk-global"]
|
||||
|
||||
|
||||
def test_retrieve_project_scope_filter_can_be_disabled(monkeypatch):
|
||||
target_project = type(
|
||||
"Project",
|
||||
(),
|
||||
{
|
||||
"project_id": "p04-gigabit",
|
||||
"aliases": ("p04", "gigabit"),
|
||||
"ingest_roots": (),
|
||||
},
|
||||
)()
|
||||
other_project = type(
|
||||
"Project",
|
||||
(),
|
||||
{
|
||||
"project_id": "p05-interferometer",
|
||||
"aliases": ("p05", "interferometer"),
|
||||
"ingest_roots": (),
|
||||
},
|
||||
)()
|
||||
|
||||
class FakeStore:
|
||||
def query(self, query_embedding, top_k=10, where=None):
|
||||
assert top_k == 2
|
||||
return {
|
||||
"ids": [["chunk-a", "chunk-b"]],
|
||||
"documents": [["project doc", "other project doc"]],
|
||||
"metadatas": [[
|
||||
{
|
||||
"heading_path": "Overview",
|
||||
"source_file": "p04-gigabit/pkm/_index.md",
|
||||
"tags": '["p04-gigabit"]',
|
||||
"title": "P04",
|
||||
"document_id": "doc-a",
|
||||
},
|
||||
{
|
||||
"heading_path": "Overview",
|
||||
"source_file": "p05-interferometer/pkm/_index.md",
|
||||
"tags": '["p05-interferometer"]',
|
||||
"title": "P05",
|
||||
"document_id": "doc-b",
|
||||
},
|
||||
]],
|
||||
"distances": [[0.2, 0.2]],
|
||||
}
|
||||
|
||||
monkeypatch.setattr("atocore.config.settings.rank_project_scope_filter", False)
|
||||
monkeypatch.setattr("atocore.retrieval.retriever.get_vector_store", lambda: FakeStore())
|
||||
monkeypatch.setattr("atocore.retrieval.retriever.embed_query", lambda query: [0.0, 0.1])
|
||||
monkeypatch.setattr(
|
||||
"atocore.retrieval.retriever._existing_chunk_ids",
|
||||
lambda chunk_ids: set(chunk_ids),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"atocore.retrieval.retriever.get_registered_project",
|
||||
lambda project_name: target_project,
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"atocore.retrieval.retriever.load_project_registry",
|
||||
lambda: [target_project, other_project],
|
||||
)
|
||||
|
||||
results = retrieve("mirror architecture", top_k=2, project_hint="p04")
|
||||
|
||||
assert {r.chunk_id for r in results} == {"chunk-a", "chunk-b"}
|
||||
|
||||
|
||||
def test_retrieve_project_scope_ignores_title_for_ownership(monkeypatch):
|
||||
target_project = type(
|
||||
"Project",
|
||||
(),
|
||||
{
|
||||
"project_id": "p04-gigabit",
|
||||
"aliases": ("p04", "gigabit"),
|
||||
"ingest_roots": (),
|
||||
},
|
||||
)()
|
||||
other_project = type(
|
||||
"Project",
|
||||
(),
|
||||
{
|
||||
"project_id": "p06-polisher",
|
||||
"aliases": ("p06", "polisher", "p11"),
|
||||
"ingest_roots": (),
|
||||
},
|
||||
)()
|
||||
|
||||
class FakeStore:
|
||||
def query(self, query_embedding, top_k=10, where=None):
|
||||
return {
|
||||
"ids": [["chunk-target", "chunk-poisoned-title"]],
|
||||
"documents": [["p04 doc", "p06 doc"]],
|
||||
"metadatas": [[
|
||||
{
|
||||
"heading_path": "Overview",
|
||||
"source_file": "p04-gigabit/pkm/_index.md",
|
||||
"tags": '["p04-gigabit"]',
|
||||
"title": "P04",
|
||||
"document_id": "doc-a",
|
||||
},
|
||||
{
|
||||
"heading_path": "Overview",
|
||||
"source_file": "p06-polisher/pkm/architecture.md",
|
||||
"tags": '["p06-polisher"]',
|
||||
"title": "GigaBIT M1 mirror lessons",
|
||||
"document_id": "doc-b",
|
||||
},
|
||||
]],
|
||||
"distances": [[0.2, 0.19]],
|
||||
}
|
||||
|
||||
monkeypatch.setattr("atocore.retrieval.retriever.get_vector_store", lambda: FakeStore())
|
||||
monkeypatch.setattr("atocore.retrieval.retriever.embed_query", lambda query: [0.0, 0.1])
|
||||
monkeypatch.setattr(
|
||||
"atocore.retrieval.retriever._existing_chunk_ids",
|
||||
lambda chunk_ids: set(chunk_ids),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"atocore.retrieval.retriever.get_registered_project",
|
||||
lambda project_name: target_project,
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"atocore.retrieval.retriever.load_project_registry",
|
||||
lambda: [target_project, other_project],
|
||||
)
|
||||
|
||||
results = retrieve("mirror architecture", top_k=2, project_hint="p04")
|
||||
|
||||
assert [r.chunk_id for r in results] == ["chunk-target"]
|
||||
|
||||
|
||||
def test_retrieve_project_scope_uses_path_segments_not_substrings(monkeypatch):
|
||||
target_project = type(
|
||||
"Project",
|
||||
(),
|
||||
{
|
||||
"project_id": "p05-interferometer",
|
||||
"aliases": ("p05", "interferometer"),
|
||||
"ingest_roots": (),
|
||||
},
|
||||
)()
|
||||
abb_project = type(
|
||||
"Project",
|
||||
(),
|
||||
{
|
||||
"project_id": "abb-space",
|
||||
"aliases": ("abb",),
|
||||
"ingest_roots": (),
|
||||
},
|
||||
)()
|
||||
|
||||
class FakeStore:
|
||||
def query(self, query_embedding, top_k=10, where=None):
|
||||
return {
|
||||
"ids": [["chunk-target", "chunk-global"]],
|
||||
"documents": [["p05 doc", "global doc"]],
|
||||
"metadatas": [[
|
||||
{
|
||||
"heading_path": "Overview",
|
||||
"source_file": "p05-interferometer/pkm/_index.md",
|
||||
"tags": '["p05-interferometer"]',
|
||||
"title": "P05",
|
||||
"document_id": "doc-a",
|
||||
},
|
||||
{
|
||||
"heading_path": "Abbreviation notes",
|
||||
"source_file": "shared/cabbage-abbreviations.md",
|
||||
"tags": "[]",
|
||||
"title": "ABB-style abbreviations",
|
||||
"document_id": "doc-global",
|
||||
},
|
||||
]],
|
||||
"distances": [[0.2, 0.21]],
|
||||
}
|
||||
|
||||
monkeypatch.setattr("atocore.retrieval.retriever.get_vector_store", lambda: FakeStore())
|
||||
monkeypatch.setattr("atocore.retrieval.retriever.embed_query", lambda query: [0.0, 0.1])
|
||||
monkeypatch.setattr(
|
||||
"atocore.retrieval.retriever._existing_chunk_ids",
|
||||
lambda chunk_ids: set(chunk_ids),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"atocore.retrieval.retriever.get_registered_project",
|
||||
lambda project_name: target_project,
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"atocore.retrieval.retriever.load_project_registry",
|
||||
lambda: [target_project, abb_project],
|
||||
)
|
||||
|
||||
results = retrieve("abbreviations", top_k=2, project_hint="p05")
|
||||
|
||||
assert [r.chunk_id for r in results] == ["chunk-target", "chunk-global"]
|
||||
|
||||
|
||||
def test_retrieve_unknown_project_hint_does_not_widen_or_filter(monkeypatch):
|
||||
class FakeStore:
|
||||
def query(self, query_embedding, top_k=10, where=None):
|
||||
assert top_k == 2
|
||||
return {
|
||||
"ids": [["chunk-a", "chunk-b"]],
|
||||
"documents": [["doc a", "doc b"]],
|
||||
"metadatas": [[
|
||||
{
|
||||
"heading_path": "Overview",
|
||||
"source_file": "project-a/file.md",
|
||||
"tags": "[]",
|
||||
"title": "A",
|
||||
"document_id": "doc-a",
|
||||
},
|
||||
{
|
||||
"heading_path": "Overview",
|
||||
"source_file": "project-b/file.md",
|
||||
"tags": "[]",
|
||||
"title": "B",
|
||||
"document_id": "doc-b",
|
||||
},
|
||||
]],
|
||||
"distances": [[0.2, 0.21]],
|
||||
}
|
||||
|
||||
monkeypatch.setattr("atocore.retrieval.retriever.get_vector_store", lambda: FakeStore())
|
||||
monkeypatch.setattr("atocore.retrieval.retriever.embed_query", lambda query: [0.0, 0.1])
|
||||
monkeypatch.setattr(
|
||||
"atocore.retrieval.retriever._existing_chunk_ids",
|
||||
lambda chunk_ids: set(chunk_ids),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"atocore.retrieval.retriever.get_registered_project",
|
||||
lambda project_name: None,
|
||||
)
|
||||
|
||||
results = retrieve("overview", top_k=2, project_hint="unknown-project")
|
||||
|
||||
assert [r.chunk_id for r in results] == ["chunk-a", "chunk-b"]
|
||||
|
||||
|
||||
def test_retrieve_downranks_archive_noise_and_prefers_high_signal_paths(monkeypatch):
|
||||
|
||||
Reference in New Issue
Block a user