Compare commits
27 Commits
d456d3c86a
...
claude/r14
| Author | SHA1 | Date | |
|---|---|---|---|
| 3888db926f | |||
| 22a37a7241 | |||
| 2712c5d2d0 | |||
| 9ab5b3c9d8 | |||
| 44724c81ab | |||
| ce3a87857e | |||
| e147ab2abd | |||
| b94f9dff56 | |||
| 081c058f77 | |||
| 069d155585 | |||
| b1a3dd071e | |||
| 5fbd7e6094 | |||
| 90001c1956 | |||
| 6a2471d509 | |||
| 83b4d78cb7 | |||
| 9c91d778d9 | |||
| 6e43cc7383 | |||
| 877b97ec78 | |||
| e840ef4be3 | |||
| 56d5df0ab4 | |||
| 028d4c3594 | |||
| 9f262a21b0 | |||
| 7863ab3825 | |||
| 3ba49e92a9 | |||
| 02055e8db3 | |||
| cc68839306 | |||
| 45196f352f |
5
.gitignore
vendored
5
.gitignore
vendored
@@ -14,3 +14,8 @@ venv/
|
|||||||
.claude/*
|
.claude/*
|
||||||
!.claude/commands/
|
!.claude/commands/
|
||||||
!.claude/commands/**
|
!.claude/commands/**
|
||||||
|
|
||||||
|
# Editor / IDE state — user-specific, not project config
|
||||||
|
.obsidian/
|
||||||
|
.vscode/
|
||||||
|
.idea/
|
||||||
|
|||||||
@@ -6,10 +6,10 @@
|
|||||||
|
|
||||||
## Orientation
|
## Orientation
|
||||||
|
|
||||||
- **live_sha** (Dalidou `/health` build_sha): `775960c` (verified 2026-04-16 via /health, build_time 2026-04-16T17:59:30Z)
|
- **live_sha** (Dalidou `/health` build_sha): `2712c5d` (verified 2026-04-22 via Codex after V1-0 deploy; status=ok)
|
||||||
- **last_updated**: 2026-04-16 by Claude ("Make It Actually Useful" sprint — observability + Phase 10)
|
- **last_updated**: 2026-04-22 by Claude (V1-0 write-time invariants landed + deployed + prod backfill complete)
|
||||||
- **main_tip**: `999788b`
|
- **main_tip**: `2712c5d`
|
||||||
- **test_count**: 303 (4 new Phase 10 tests)
|
- **test_count**: 547 (prior 533 + 10 V1-0 invariant tests + 4 Codex-review regression tests)
|
||||||
- **harness**: `17/18 PASS` on live Dalidou (p04-constraints expects "Zerodur" — retrieval content gap, not regression)
|
- **harness**: `17/18 PASS` on live Dalidou (p04-constraints expects "Zerodur" — retrieval content gap, not regression)
|
||||||
- **vectors**: 33,253
|
- **vectors**: 33,253
|
||||||
- **active_memories**: 84 (31 project, 23 knowledge, 10 episodic, 8 adaptation, 7 preference, 5 identity)
|
- **active_memories**: 84 (31 project, 23 knowledge, 10 episodic, 8 adaptation, 7 preference, 5 identity)
|
||||||
@@ -143,9 +143,15 @@ One branch `codex/extractor-eval-loop` for Day 1-5, a second `codex/retrieval-ha
|
|||||||
| R11 | Codex | P2 | src/atocore/api/routes.py:773-845 | `POST /admin/extract-batch` still accepts `mode="llm"` inside the container and returns a successful 0-candidate result instead of surfacing that host-only LLM extraction is unavailable from this runtime. That is a misleading API contract for operators. | fixed | Claude | 2026-04-12 | (pending) |
|
| R11 | Codex | P2 | src/atocore/api/routes.py:773-845 | `POST /admin/extract-batch` still accepts `mode="llm"` inside the container and returns a successful 0-candidate result instead of surfacing that host-only LLM extraction is unavailable from this runtime. That is a misleading API contract for operators. | fixed | Claude | 2026-04-12 | (pending) |
|
||||||
| R12 | Codex | P2 | scripts/batch_llm_extract_live.py:39-190 | The host-side extractor duplicates the LLM system prompt and JSON parsing logic from `src/atocore/memory/extractor_llm.py`. It works today, but this is now a prompt/parser drift risk across the container and host implementations. | fixed | Claude | 2026-04-12 | (pending) |
|
| R12 | Codex | P2 | scripts/batch_llm_extract_live.py:39-190 | The host-side extractor duplicates the LLM system prompt and JSON parsing logic from `src/atocore/memory/extractor_llm.py`. It works today, but this is now a prompt/parser drift risk across the container and host implementations. | fixed | Claude | 2026-04-12 | (pending) |
|
||||||
| R13 | Codex | P2 | DEV-LEDGER.md:12 | The new `286 passing` test-count claim is not reproducibly auditable from the current audit environments: neither Dalidou nor the clean worktree has `pytest` available. The claim may be true in Claude's dev shell, but it remains unverified in this audit. | fixed | Claude | 2026-04-12 | (pending) |
|
| R13 | Codex | P2 | DEV-LEDGER.md:12 | The new `286 passing` test-count claim is not reproducibly auditable from the current audit environments: neither Dalidou nor the clean worktree has `pytest` available. The claim may be true in Claude's dev shell, but it remains unverified in this audit. | fixed | Claude | 2026-04-12 | (pending) |
|
||||||
|
| R14 | Codex | P2 | src/atocore/api/routes.py (POST /entities/{id}/promote) | The HTTP `POST /entities/{id}/promote` route does not translate the new service-layer `ValueError("source_refs required: cannot promote a candidate with no provenance...")` into a 400. A legacy no-provenance candidate promoted through the API currently surfaces as a 500. Does not block V1-0 acceptance; tidy in a follow-up. | fixed | Claude | 2026-04-22 | (pending) |
|
||||||
|
|
||||||
## Recent Decisions
|
## Recent Decisions
|
||||||
|
|
||||||
|
- **2026-04-22** **V1-0 done: approved, merged, deployed, prod backfilled.** Codex pulled `f16cd52`, re-ran the two original probes (both pass), re-ran the three targeted regression suites (all pass). Squash-merged to main as `2712c5d`. Dalidou deployed via canonical deploy script; `/health` reports build_sha=`2712c5d2d03cb2a6af38b559664afd1c4cd0e050`, status=ok. Validated backup snapshot taken at `/srv/storage/atocore/backups/snapshots/20260422T190624Z` before backfill. Prod backfill: `--dry-run` found 31 active/superseded entities with no provenance; list reviewed and sane; live run updated 31 rows via the default `hand_authored=1` flag path; follow-up dry-run returned 0 rows remaining. Residual logged as R14 (P2): `POST /entities/{id}/promote` HTTP route doesn't translate the new service-layer `ValueError` into a 400 — legacy bad candidate promotes via the API return 500 instead. Does not block V1-0 acceptance. V1-0 closed. Next: V1-A (Q-001 subsystem-scoped variant + Q-6 integration). V1-A holds until the soak window ends ~2026-04-26 and the 100-memory density target is hit. *Approved + landed by:* Codex. *Ratified by:* Antoine.
|
||||||
|
- **2026-04-22** **Engineering V1 Completion Plan — Codex sign-off (third round)**. Codex's third-round audit closed the remaining five open questions with concrete resolutions, patched inline in `docs/plans/engineering-v1-completion-plan.md`: (1) F-7 row rewritten with ground truth — schema + preserve-original + test coverage already exist (`graduated_to_entity_id` at `database.py:143-146`, `graduated` status in memory service, promote hook at `service.py:354-356,389-451`, tests at `test_engineering_v1_phase5.py:67-90`); **real gaps** are missing direct `POST /memory/{id}/graduate` route and spec's `knowledge→Fact` mismatch (no `fact` entity type exists; reconcile to `parameter` or similar); V1-E 2 → **3–4 days**; (2) Q-5 determinism reframed — don't stabilize the call to `datetime.now()`, inject regenerated timestamp + checksum as renderer inputs, remove DB iteration ordering dependencies; V1-D scope updated; (3) `project` vs `project_id` — doc note only, no rename, resolved; (4) total estimate 16.5–17.5 → **17.5–19.5 focused days** with calendar buffer on top; (5) "Minions" must not be canonized in D-3 release notes — neutral wording ("queued background processing / async workers") only. **Agreement reached**: Claude + Codex + Antoine aligned. V1-0 is ready to start once the current pipeline soak window ends (~2026-04-26) and the 100-memory density target is hit. *Patched by:* Codex. *Signed off by:* Codex ("with those edits, I'd sign off on the five questions"). *Accepted by:* Antoine. *Executor (V1-0 onwards):* Claude.
|
||||||
|
- **2026-04-22** **Engineering V1 Completion Plan revised per Codex second-round file-level audit** — three findings folded in, all with exact file:line refs from Codex: (1) F-1 downgraded from ✅ to 🟡 — `extractor_version` and `canonical_home` missing from `Entity` dataclass and `entities` table per `engineering-v1-acceptance.md:45`; V1-0 scope now adds both fields via additive migration + doc note that `project` IS `project_id` per "fields equivalent to" spec wording; (2) F-2 replaced with ground-truth per-query status: 9 of 20 v1-required queries done (Q-004/Q-005/Q-006/Q-008/Q-009/Q-011/Q-013/Q-016/Q-017), 1 partial (Q-001 needs subsystem-scoped variant), 10 missing (Q-002/003/007/010/012/014/018/019/020); V1-A scope shrank to Q-001 shape fix + Q-6 integration (pillar queries already implemented); V1-C closes the 8 remaining new queries + Q-020 deferred to V1-D; (3) F-5 reframed — generic `conflicts` + `conflict_members` schema already present at `database.py:190`, no migration needed; divergence is detector body (per-type dispatch needs generalization) + routes (`/admin/conflicts/*` needs `/conflicts/*` alias). Total revised to 16.5–17.5 days, ~60 tests. Plan: `docs/plans/engineering-v1-completion-plan.md` at commit `ce3a878` (Codex pulled clean). Three of Codex's eight open questions now answered; remaining: F-7 graduation depth, mirror determinism, `project` rename question, velocity calibration, minions naming. *Proposed by:* Claude. *Reviewed by:* Codex (two rounds).
|
||||||
|
- **2026-04-22** **Engineering V1 Completion Plan revised per Codex first-round review** — original six-phase order (queries → ingest → mirror → graduation → provenance → ops) rejected by Codex as backward: provenance-at-write (F-8) and conflict-detection hooks (F-5 minimal) must precede any phase that writes active entities. Revised to seven phases: V1-0 write-time invariants (F-8 + F-5 hooks + F-1 audit) as hard prerequisite, V1-A minimum query slice proving the model, V1-B ingest, V1-C full query catalog, V1-D mirror, V1-E graduation, V1-F full F-5 spec + ops + docs. Also softened "parallel with Now list" — real collision points listed explicitly; schedule shifted ~4 weeks to reflect that V1-0 cannot start during pipeline soak. Withdrew the "50–70% built" global framing in favor of the per-criterion gap table. Workspace sync note added: Codex's Playground workspace can't see the plan file; canonical dev tree is Windows `C:\Users\antoi\ATOCore`. Plan: `docs/plans/engineering-v1-completion-plan.md`. Awaiting Codex file-level audit once workspace syncs. *Proposed by:* Claude. *First-round review by:* Codex.
|
||||||
|
- **2026-04-22** gbrain-inspired "Phase 8 Minions + typed edges" plan **rejected as packaged** — wrong sequencing (leapfrogged `master-plan-status.md` Now list), wrong predicate set (6 vs V1's 17), wrong canonical boundary (edges-on-wikilinks instead of typed entities+relationships per `memory-vs-entities.md`). Mechanic (durable jobs + typed graph) deferred to V1 home. Record: `docs/decisions/2026-04-22-gbrain-plan-rejection.md`. *Proposed by:* Claude. *Reviewed/rejected by:* Codex. *Ratified by:* Antoine.
|
||||||
- **2026-04-12** Day 4 gate cleared: LLM-assisted extraction via `claude -p` (OAuth, no API key) is the path forward. Rule extractor stays as default for structural cues. *Proposed by:* Claude. *Ratified by:* Antoine.
|
- **2026-04-12** Day 4 gate cleared: LLM-assisted extraction via `claude -p` (OAuth, no API key) is the path forward. Rule extractor stays as default for structural cues. *Proposed by:* Claude. *Ratified by:* Antoine.
|
||||||
- **2026-04-12** First live triage: 16 promoted, 35 rejected from 51 LLM-extracted candidates. 31% accept rate. Active memory count 20->36. *Executed by:* Claude. *Ratified by:* Antoine.
|
- **2026-04-12** First live triage: 16 promoted, 35 rejected from 51 LLM-extracted candidates. 31% accept rate. Active memory count 20->36. *Executed by:* Claude. *Ratified by:* Antoine.
|
||||||
- **2026-04-12** No API keys allowed in AtoCore — LLM-assisted features use OAuth via `claude -p` or equivalent CLI-authenticated paths. *Proposed by:* Antoine.
|
- **2026-04-12** No API keys allowed in AtoCore — LLM-assisted features use OAuth via `claude -p` or equivalent CLI-authenticated paths. *Proposed by:* Antoine.
|
||||||
@@ -160,6 +166,40 @@ One branch `codex/extractor-eval-loop` for Day 1-5, a second `codex/retrieval-ha
|
|||||||
|
|
||||||
## Session Log
|
## Session Log
|
||||||
|
|
||||||
|
- **2026-04-22 Codex + Antoine (V1-0 closed)** Codex approved `f16cd52` after re-running both original probes (legacy-candidate promote + supersede hook — both correct) and the three targeted regression suites (`test_v1_0_write_invariants.py`, `test_engineering_v1_phase5.py`, `test_inbox_crossproject.py` — all pass). Squash-merged to main as `2712c5d` ("feat(engineering): enforce V1-0 write invariants"). Deployed to Dalidou via the canonical deploy script; `/health` build_sha=`2712c5d2d03cb2a6af38b559664afd1c4cd0e050` status=ok. Validated backup snapshot at `/srv/storage/atocore/backups/snapshots/20260422T190624Z` taken BEFORE prod backfill. Prod backfill of `scripts/v1_0_backfill_provenance.py` against live DB: dry-run found 31 active/superseded entities with no provenance, list reviewed and looked sane; live run with default `hand_authored=1` flag path updated 31 rows; follow-up dry-run returned 0 rows remaining → no lingering F-8 violations in prod. Codex logged one residual P2 (R14): HTTP `POST /entities/{id}/promote` route doesn't translate the new service-layer `ValueError` into 400 — legacy bad candidate promoted through the API surfaces as 500. Not blocking. V1-0 closed. **Gates for V1-A**: soak window ends ~2026-04-26; 100-active-memory density target (currently 84 active + the ~31 newly flagged ones — need to check how those count in density math). V1-A holds until both gates clear.
|
||||||
|
|
||||||
|
- **2026-04-22 Claude (V1-0 patches per Codex review)** Codex audit of commit `cbf9e03` surfaced two P1 gaps + one P2 scope concern, all verified with code-level probes. **P1 #1**: `promote_entity` didn't re-check the F-8 invariant — a legacy candidate with empty `source_refs` and `hand_authored=0` could still promote to active, violating the plan's "invariant at both `create_entity` and `promote_entity`". Fixed: `promote_entity` at `service.py:365-379` now raises `ValueError("source_refs required: cannot promote a candidate with no provenance...")` before flipping status. Stays symmetric with the create-side error. **P1 #2**: `supersede_entity` was missing the F-5 hook the plan requires on every active-entity write path. The `supersedes` relationship rooted at the `superseded_by` entity can create a conflict the detector should catch. Fixed at `service.py:581-591`: calls `detect_conflicts_for_entity(superseded_by)` with fail-open per Q-3. **P2**: backfill script's `--invalidate-instead` flag queried both active AND superseded rows; invalidating already-superseded rows would collapse history. Fixed at `scripts/v1_0_backfill_provenance.py:52-63`: `--invalidate-instead` now scopes to `status='active'` only (default flag-hand_authored mode stays broad as it's additive/non-destructive). Help text tightened to make the destructive posture explicit. **Four new regression tests** in `test_v1_0_write_invariants.py`: (1) `test_promote_rejects_legacy_candidate_without_provenance` — directly inserts a legacy candidate and confirms promote raises + row stays candidate; (2) `test_promote_accepts_candidate_flagged_hand_authored` — symmetry check; (3) `test_supersede_runs_conflict_detection_on_new_active` — monkeypatches detector, confirms hook fires on `superseded_by`; (4) `test_supersede_hook_fails_open` — Q-3 check for supersede path. **Test count**: 543 → 547 (+4 regression). Full suite `547 passed in 81.07s`. Next: commit patches on branch, push, Codex re-review.
|
||||||
|
|
||||||
|
- **2026-04-22 Claude (V1-0 landed on branch)** First V1 completion phase done on branch `claude/v1-0-write-invariants`. **F-1 schema remediation**: added `extractor_version`, `canonical_home`, `hand_authored` columns to `entities` via idempotent ALTERs in both `_apply_migrations` (`database.py:148-170`) and `init_engineering_schema` (`service.py:95-139`). CREATE TABLE also updated so fresh DBs get the columns natively. New `_table_exists` helper at `database.py:378`. `Entity` dataclass gains the three fields with sensible defaults. `EXTRACTOR_VERSION = "v1.0.0"` module constant at top of `service.py`. `_row_to_entity` tolerates rows without the new columns so tests predating V1-0 still pass. **F-8 provenance enforcement**: `create_entity` raises `ValueError("source_refs required: ...")` when called without non-empty source_refs AND without `hand_authored=True`. New kwargs `hand_authored: bool = False` and `extractor_version: str | None = None` threaded through `service.create_entity`, the `EntityCreateRequest` Pydantic model, the API route, and the wiki `/wiki/new` form body (form writes `hand_authored: true` since human entries are hand-authored by definition). **F-5 hook on active create**: `create_entity(status="active")` now calls `detect_conflicts_for_entity` with fail-open per `conflict-model.md:256` (errors log warning, write still succeeds). The promote path's existing hook at `service.py:400-404` was kept as-is. **Doc note** added to `engineering-ontology-v1.md` recording that `project` IS the `project_id` per "fields equivalent to" wording. **Backfill script** at `scripts/v1_0_backfill_provenance.py` — idempotent, defaults to flagging no-provenance active entities as `hand_authored=1`, supports `--dry-run` and `--invalidate-instead`. **Tests**: 10 new in `tests/test_v1_0_write_invariants.py` covering F-1 fields, F-8 raise path, F-8 hand_authored bypass, F-5 active-create hook, F-5 candidate-no-hook, Q-3 fail-open on detector error, Q-4 partial (scope_only=active excludes candidates). **Test fixes**: three pre-existing tests adapted — `test_requirement_name_conflict_detected` + `test_conflict_resolution_dismiss_leaves_entities_alone` now read from `list_open_conflicts` because the V1-0 hook records the conflict at create-time (detector dedup returns [] on re-run); `test_api_post_entity_with_null_project_stores_global` sends `hand_authored: true` since the fixture has no source_refs. **conftest.py monkeypatch**: wraps `create_entity` so tests missing both source_refs and hand_authored default to `hand_authored=True` (reasonable since tests author their own fixture data). Production paths (API route, wiki form, graduation scripts) all pass explicit values and are unaffected by the monkeypatch. **Test count**: 533 → 543 (+10), full suite `543 passed in 77.86s`. **Not yet**: commit + push + Codex review + deploy. **Branch**: `claude/v1-0-write-invariants`.
|
||||||
|
|
||||||
|
- **2026-04-22 Codex (late night)** Third-round audit closed the remaining five open questions. Patched `docs/plans/engineering-v1-completion-plan.md` inline (no commit by Codex). **F-7 finding (P1):** graduation stack is partially built — `_graduation_prompt.py`, `scripts/graduate_memories.py`, `database.py:143-146` (`graduated_to_entity_id`), memory `graduated` status, promote-preserves-original at `service.py:354-356,389-451`, tests at `test_engineering_v1_phase5.py:67-90` all exist. Real gaps: no direct `POST /memory/{id}/graduate` route at `routes.py:756`; spec's `knowledge→Fact` doesn't match ontology (`service.py:16` has no `fact` type — reconcile to `parameter` or similar). V1-E estimate 2 → 3–4 days. **Q-5 finding (P2):** "stabilize timestamp" insufficient — renderer reads wall-clock in `_footer()` at `mirror.py:320`; fix is inject regenerated timestamp + checksum as renderer inputs + sort DB iteration + remove dict ordering deps. V1-D scope patched. **Remaining three (P3):** `project` stays as doc-note equivalence (no rename); total estimate 17.5–19.5 focused days; release notes must NOT canonize "Minions" — neutral "queued background processing / async workers" only. **Sign-off:** "with those edits, I'd sign off on the five questions. The only non-architectural uncertainty left in the plan is scheduling discipline against the current Now list; that does not block V1-0 once the soak window and memory-density gate clear." **Status:** Claude + Codex agreed. Plan frozen pending Antoine final accept and gate clearance. Claude to commit Codex's patches + push.
|
||||||
|
|
||||||
|
- **2026-04-22 Claude (late night)** Codex second-round review did the full file-level audit and came back with three P1/P2 findings, all with exact file:line refs. Verified each against current code before revising. (1) **F-1 not clean**: `Entity` dataclass at `service.py:67` and `entities` table schema are missing the `extractor_version` and `canonical_home` shared-header fields required by `engineering-v1-acceptance.md:45`; `project` field is the project identifier but not named `project_id` as spec writes (spec wording "fields equivalent to" allows the naming, but needs explicit doc note). V1-0 scope now includes adding both missing fields via additive `_apply_migrations` pattern. (2) **F-2 needed exact statuses, not guesses**: per-function audit gave ground truth — 9 of 20 v1-required queries done, 1 partial (Q-001 returns project-wide tree not subsystem-scoped expand=contains per `engineering-query-catalog.md:71`), 10 missing. V1-A scope shrank to Q-001 shape fix + Q-6 integration (most pillar queries already implemented); V1-C closes the 8 net-new queries + Q-020 to V1-D. (3) **F-5 misframed**: the generic `conflicts` + `conflict_members` schema is ALREADY spec-compliant at `database.py:190`; divergence is detector body at `conflicts.py:36` (per-type dispatch needs generalization) + route path (`/admin/conflicts/*` needs `/conflicts/*` alias). V1-F no longer includes a schema migration; detector generalization + route alignment only. Totals revised to 16.5–17.5 days, ~60 tests (down from 12–17 / 65 because V1-A and V1-F scopes both shrank after audit). Three of the eight open questions resolved. Remaining open: F-7 graduation depth, mirror determinism, `project` naming, velocity calibration, minions-as-V2 naming. No code changes this session — plan + ledger only. Next: commit + push revised plan, then await Antoine+Codex joint sign-off before V1-0 starts.
|
||||||
|
|
||||||
|
- **2026-04-22 Claude (night)** Codex first-round review of the V1 Completion Plan summary came back with four findings. Three substantive, one workspace-sync: (1) "50–70% built" too loose — replaced with per-criterion table, global framing withdrawn; (2) phase order backward — provenance-at-write (F-8) and conflict hooks (F-5 minimal) depend-upon by every later phase but were in V1-E; new V1-0 prerequisite phase inserted to establish write-time invariants, and V1-A shrunk to a minimum query slice (four pillars Q-001/Q-005/Q-006/Q-017 + Q-6 integration) rather than full catalog closure; (3) "parallel with Now list / disjoint surfaces" too strong — real collisions listed explicitly (V1-0 provenance + memory extractor write path, V1-E graduation + memory module, V1-F conflicts migration + memory promote); schedule shifted ~4 weeks, V1-0 cannot start during pipeline soak; (4) Codex's Playground workspace can't see the plan file or the `src/atocore/engineering/` code — added a Workspace note to the plan directing per-file audit at the Windows canonical dev tree (`C:\Users\antoi\ATOCore`) and noting the three visible file paths (`docs/plans/engineering-v1-completion-plan.md`, `docs/decisions/2026-04-22-gbrain-plan-rejection.md`, `DEV-LEDGER.md`). Revised plan estimate: 12–17 days across 7 phases (up from 11–14 / 6), ~65 tests added (up from ~50). V1-0 is a hard prerequisite; no later phase starts until it lands. Pending Antoine decision on workspace sync (commit+push vs paste-to-Codex) so Codex can do the file-level audit. No code changes this session.
|
||||||
|
|
||||||
|
- **2026-04-22 Claude (late eve)** After the rejection, read the four core V1 architecture docs end-to-end (`engineering-ontology-v1.md`, `engineering-query-catalog.md`, `memory-vs-entities.md`, `engineering-v1-acceptance.md`) plus the four supporting docs (`promotion-rules.md`, `conflict-model.md`, `human-mirror-rules.md`, `tool-handoff-boundaries.md`). Cross-referenced against current code in `src/atocore/engineering/`. **Key finding:** V1 is already 50–70% built — entity types (16, superset of V1's 12), all 18 V1 relationship types, 4-state lifecycle, CRUD + supersede + invalidate + PATCH, queries module with most killer-correctness queries (orphan_requirements, risky_decisions, unsupported_claims, impact_analysis, evidence_chain), conflicts module scaffolded, mirror scaffolded, graduation endpoint scaffolded. Recent commits e147ab2/b94f9df/081c058/069d155/b1a3dd0 are all V1 entity-layer work. Drafted `docs/plans/engineering-v1-completion-plan.md` reframing the work as **V1 completion, not V1 start**. Six sequential phases V1-A through V1-F, estimated 11–14 days, ~50 new tests (533 → ~580). Phases run in parallel with the Now list (pipeline soak + density + multi-model triage + p04-constraints) because surfaces are disjoint. Plan explicitly defers the minions/queue mechanic per acceptance-doc negative list. Pending Codex audit of the plan itself — especially the F-2 query gap list (Claude didn't read each query function end-to-end), F-5 conflicts schema divergence (per-type detectors vs spec's generic slot-keyed shape), and F-7 graduation depth. No code changes this session.
|
||||||
|
|
||||||
|
- **2026-04-22 Claude (eve)** gbrain review session. Antoine surfaced https://github.com/garrytan/gbrain for compare/contrast. Claude drafted a "Phase 8 Minions + typed edges" plan pairing a durable job queue with a 6-predicate edge upgrade over wikilinks. Codex reviewed and rejected as packaged: (1) sequencing leapfrogged the `master-plan-status.md` Now list (pipeline soak → 100+ memories → multi-model triage → p04-constraints fix); (2) 6 predicates vs V1's 17 across Structural/Intent/Validation/Provenance families — would have been schema debt on day one per `engineering-ontology-v1.md:112-137`; (3) "edges over wikilinks" bypassed the V1 canonical entity + promotion contract in `memory-vs-entities.md`. Claude verified each high finding against the cited files and concurred. The underlying mechanic (durable background jobs + typed relationship graph) is still a valid future direction, but its correct home is the Engineering V1 sprint under **Next** in `master-plan-status.md:179`, not a leapfrog phase. Decision record: `docs/decisions/2026-04-22-gbrain-plan-rejection.md`. No code changes this session. Next (pending Antoine ack): read the four V1 architecture docs end-to-end, then draft an Engineering V1 foundation plan that follows the existing contract, not a new external reference. Phase 8 (OpenClaw) name remains untouched — Claude's misuse of "Phase 8" in the rejected plan was a naming collision, not a renaming.
|
||||||
|
|
||||||
|
- **2026-04-22 Claude (pm)** Issue B (wiki redlinks) landed — last remaining P2 from Antoine's sprint plan. `_wikilink_transform(text, current_project)` in `src/atocore/engineering/wiki.py` replaces `[[Name]]` / `[[Name|Display]]` tokens (pre-markdown) with HTML anchors. Resolution order: same-project exact-name match → live `wikilink`; other-project match → live link with `(in project X)` scope indicator (`wikilink-cross`); no match → `redlink` pointing at `/wiki/new?name=<quoted>&project=<current>`. New route `GET /wiki/new` renders a pre-filled "create this entity" form that POSTs to `/v1/entities` via a minimal inline fetch() and redirects to the new entity's wiki page on success. Transform applied in `render_project` (over the mirror markdown) and `render_entity` (over the description body). CSS: dashed-underline accent for live wikilinks, red italic + dashed for redlinks. 12 new tests including the regression from the spec (entity A references `[[EntityB]]` → initial render has `class="redlink"`; after EntityB is created, re-render no longer has redlink and includes `/wiki/entities/{b.id}`). Tests 521 → 533. All 6 acceptance criteria from the sprint plan ("daily-usable") now green: retract/supersede, edit without cloning, cross-project has a home, visual evidence, wiki readable, AKC can capture reliably.
|
||||||
|
|
||||||
|
- **2026-04-22 Claude** PATCH `/entities/{id}` + Issue D (/v1/engineering/* aliases) landed. New `update_entity()` in `src/atocore/engineering/service.py` supports partial updates to description (replace), properties (shallow merge — `null` value deletes a key), confidence (0..1, 400 on bounds violation), source_refs (append + dedup). Writes an `updated` audit row with full before/after snapshots. Forbidden via this path: entity_type / project / name / status — those require supersede+create or the dedicated status endpoints, by design. New route `PATCH /entities/{id}` aliased under `/v1`. Issue D: all 10 `/engineering/*` query paths (decisions, systems, components/{id}/requirements, changes, gaps + sub-paths, impact, evidence) added to the `/v1` allowlist. 12 new PATCH tests (merge, null-delete, confidence bounds, source_refs dedup, 404, audit row, v1 alias). Tests 509 → 521. Next: commit + deploy, then Issue B (wiki redlinks) as the last remaining P2 per Antoine's sprint order.
|
||||||
|
|
||||||
|
- **2026-04-21 Claude (night)** Issue E (retraction path for active entities + memories) landed. Two new entity endpoints and two new memory endpoints, all aliased under `/v1`: `POST /entities/{id}/invalidate` (active→invalid, 200 idempotent on already-invalid, 409 if candidate/superseded, 404 if missing), `POST /entities/{id}/supersede` (active→superseded + auto-creates `supersedes` relationship from the new entity to the old one; rejects self-supersede and unknown superseded_by with 400), `POST /memory/{id}/invalidate`, `POST /memory/{id}/supersede`. `invalidate_memory`/`supersede_memory` in service.py now take a `reason` string that lands in the audit `note`. New service helper `invalidate_active_entity(id, reason)` returns `(ok, code)` where code is one of `invalidated | already_invalid | not_active | not_found` for a clean HTTP-status mapping. 15 new tests. Tests 494 → 509. Unblocks correction workflows — no more SQL required to retract mistakes.
|
||||||
|
|
||||||
|
- **2026-04-21 Claude (cleanup)** One-time SQL cleanup on live Dalidou: flipped 8 `status='active' → 'invalid'` rows in `entities` (CGH, tower, "interferometer mirror tower", steel, "steel (likely)" in p05-interferometer + 3 remaining `AKC-E2E-Test-*` rows that were still active). Each update paired with a `memory_audit` row (action=`invalidated`, actor=`sql-cleanup`, note references Issue E pending). Executed inside the `atocore` container via `docker exec` since `/srv/storage/atocore/data/db/atocore.db` is root-owned and the service holds write perms. Verification: `GET /entities?project=p05-interferometer&scope_only=true` now 21 active, zero pollution. Issue E (public `POST /v1/entities/{id}/invalidate` for active→invalid) remains open — this cleanup should not be needed again once E ships.
|
||||||
|
|
||||||
|
- **2026-04-21 Claude (evening)** Issue F (visual evidence) landed. New `src/atocore/assets/` module provides hash-dedup binary storage (`<assets_dir>/<hash[:2]>/<hash>.<ext>`) with on-demand JPEG thumbnails cached under `.thumbnails/<size>/`. New `assets` table (hash_sha256 unique, mime_type, size, width/height, source_refs, status). `artifact` added to `ENTITY_TYPES`; no schema change needed on entities (`properties` stays free-form JSON carrying `kind`/`asset_id`/`caption`/`capture_context`). `EVIDENCED_BY` already in the relationship enum — no change. New API: `POST /assets` (multipart, 20 MB cap, MIME allowlist: png/jpeg/webp/gif/pdf/step/iges), `GET /assets/{id}` (streams original), `GET /assets/{id}/thumbnail?size=N` (Pillow, 16-2048 px clamp), `GET /assets/{id}/meta`, `GET /admin/assets/orphans`, `DELETE /assets/{id}` (409 if referenced), `GET /entities/{id}/evidence` (returns EVIDENCED_BY artifacts with asset metadata resolved). All aliased under `/v1`. Wiki: artifact entity pages render full image + caption + capture_context; other entity pages render an "Visual evidence" strip of EVIDENCED_BY thumbnails linking to full-res + artifact detail page. PDFs render as a link; other artifact kinds render as labeled chips. Added `python-multipart` + `Pillow>=10.0.0` to deps; docker-compose gets an `${ATOCORE_ASSETS_DIR}` bind mount; Dalidou `.env` updated with `ATOCORE_ASSETS_DIR=/srv/storage/atocore/data/assets`. 16 new tests (hash dedup, size cap, mime allowlist, thumbnail cache, orphan detection, invalidate gating, multipart upload, evidence API, v1 aliases, wiki rendering). Tests 478 → 494.
|
||||||
|
|
||||||
|
- **2026-04-21 Claude (pm)** Issue C (inbox + cross-project entities) landed. `inbox` is a reserved pseudo-project: auto-exists, cannot be registered/updated/aliased (enforced in `src/atocore/projects/registry.py` via `is_reserved_project` + `register_project`/`update_project` guards). `project=""` remains the cross-project/global bucket for facts that apply to every project. `resolve_project_name("inbox")` is stable and does not hit the registry. `get_entities` now scopes: `project=""` → only globals; `project="inbox"` → only inbox; `project="<real>"` default → that project plus globals; `scope_only=true` → strict. `POST /entities` accepts `project=null` as equivalent to `""`. `POST /entities/{id}/promote` accepts `{target_project}` to retarget an inbox/global lead into a real project on promote (new "retargeted" audit action). Wiki homepage shows a new "📥 Inbox & Global" section with live counts, linking to scoped `/entities` lists. 15 new tests in `test_inbox_crossproject.py` cover reserved-name enforcement, scoping rules, API shape, and promote retargeting. Tests 463 → 478. Pending: commit, push, deploy. Issue B (wiki redlinks) deferred per AKC thread — P1 cosmetic, not a blocker.
|
||||||
|
|
||||||
|
- **2026-04-21 Claude** Issue A (API versioning) landed on `main` working tree (not yet committed/deployed). `src/atocore/main.py` now mounts a second `/v1` router that re-registers an explicit allowlist of public handlers (`_V1_PUBLIC_PATHS`) against the same endpoint functions — entities, relationships, ingest, context/build, query, projects, memory, interactions, project/state, health, sources, stats, and their sub-paths. Unversioned paths are untouched; OpenClaw and hooks keep working. Added `tests/test_v1_aliases.py` (5 tests: health parity, projects parity, entities reachable, v1 paths present in OpenAPI, unversioned paths still present in OpenAPI) and a "API versioning" section in the README documenting the rule (new endpoints at latest prefix, breaking changes bump prefix, unversioned retained for internal callers). Tests 459 → 463. Next: commit + deploy, then relay to the AKC thread so Phase 2 can code against `/v1`. Issues B (wiki redlinks) and C (inbox/cross-project) remain open, unstarted.
|
||||||
|
|
||||||
|
- **2026-04-19 Claude** Shipped Phases 7A.1 (tiered auto-merge), 7C (tag canonicalization), 7D (confidence decay), 7I (OpenClaw context injection), UI refresh (memory/domain/activity pages + topnav), and closed the Claude Code retrieval asymmetry. Builds deployed: `028d4c3` → `56d5df0` → `e840ef4` → `877b97e` → `6e43cc7` → `9c91d77`. New capture-surface scope: Claude Code (Stop + UserPromptSubmit hooks, both installed and verified live) + OpenClaw (v0.2.0 plugin with capture + context injection, verified loaded on T420 gateway). `/wiki/capture` paste form removed from topnav; kept as labeled fallback. Anthropic API polling explicitly out of scope per user. Tests 414 → 459. `docs/capture-surfaces.md` documents the sanctioned scope.
|
||||||
|
|
||||||
|
- **2026-04-18 Claude** **Phase 7A — Memory Consolidation V1 ("sleep cycle") landed on branch.** New `docs/PHASE-7-MEMORY-CONSOLIDATION.md` covers all 8 subphases (7A dedup, 7B contradictions, 7C tag canon, 7D confidence decay, 7E memory detail, 7F domain view, 7F re-extract, 7H vector hygiene). 7A implementation: schema migration `memory_merge_candidates`, `atocore.memory.similarity` (cosine + transitive cluster), stdlib-only `atocore.memory._dedup_prompt` (llm drafts unified content preserving all specifics), `merge_memories()` + `create_merge_candidate()` + `get_merge_candidates()` + `reject_merge_candidate()` in service.py, host-side `scripts/memory_dedup.py` (HTTP + claude -p, idempotent via sorted-id set), 5 new endpoints under `/admin/memory/merge-candidates*` + `/admin/memory/dedup-scan` + `/admin/memory/dedup-status`, purple-themed "🔗 Merge Candidates" section in /admin/triage with editable draft + approve/reject buttons, "🔗 Scan for duplicates" control bar with threshold slider, nightly Step B3 in batch-extract.sh (0.90 daily, 0.85 Sundays deep), `deploy/dalidou/dedup-watcher.sh` host watcher for UI-triggered scans (mirrors graduation-watcher pattern). 21 new tests (similarity, prompt parse, idempotency, merge happy path, override content/tags, audit rows, abort-if-source-tampered, reject leaves sources alone, schema). Tests 374 → 395. Not yet deployed; harness not re-run. Next: push + deploy, install `dedup-watcher.sh` in host cron, trigger first scan, review proposals in UI.
|
||||||
|
|
||||||
- **2026-04-16 Claude** `b687e7f..999788b` **"Make It Actually Useful" sprint.** Two-part session: ops fixes then consolidation sprint.
|
- **2026-04-16 Claude** `b687e7f..999788b` **"Make It Actually Useful" sprint.** Two-part session: ops fixes then consolidation sprint.
|
||||||
|
|
||||||
**Part 1 — Ops fixes:** Deployed `b687e7f` (project inference from cwd). Fixed cron logging (was `/dev/null` — redirected to `~/atocore-logs/`). Fixed OpenClaw gateway crash-loop (`discord.replyToMode: "any"` invalid → `"all"`). Deployed `atocore-capture` plugin on T420 OpenClaw using `before_agent_start` + `llm_output` hooks — verified end-to-end: 38 `client=openclaw` interactions captured. Backfilled project tags on 179/181 unscoped interactions (165 atocore, 8 p06, 6 p04).
|
**Part 1 — Ops fixes:** Deployed `b687e7f` (project inference from cwd). Fixed cron logging (was `/dev/null` — redirected to `~/atocore-logs/`). Fixed OpenClaw gateway crash-loop (`discord.replyToMode: "any"` invalid → `"all"`). Deployed `atocore-capture` plugin on T420 OpenClaw using `before_agent_start` + `llm_output` hooks — verified end-to-end: 38 `client=openclaw` interactions captured. Backfilled project tags on 179/181 unscoped interactions (165 atocore, 8 p06, 6 p04).
|
||||||
|
|||||||
20
README.md
20
README.md
@@ -40,6 +40,26 @@ python scripts/atocore_client.py audit-query "gigabit" 5
|
|||||||
| GET | /health | Health check |
|
| GET | /health | Health check |
|
||||||
| GET | /debug/context | Inspect last context pack |
|
| GET | /debug/context | Inspect last context pack |
|
||||||
|
|
||||||
|
## API versioning
|
||||||
|
|
||||||
|
The public contract for external clients (AKC, OpenClaw, future tools) is
|
||||||
|
served under a `/v1` prefix. Every public endpoint is available at both its
|
||||||
|
unversioned path and under `/v1` — e.g. `POST /entities` and `POST /v1/entities`
|
||||||
|
route to the same handler.
|
||||||
|
|
||||||
|
Rules:
|
||||||
|
|
||||||
|
- New public endpoints land at the latest version prefix.
|
||||||
|
- Backwards-compatible additions stay on the current version.
|
||||||
|
- Breaking schema changes to an existing endpoint bump the prefix (`/v2/...`)
|
||||||
|
and leave the prior version in place until clients migrate.
|
||||||
|
- Unversioned paths are retained for internal callers (hooks, scripts, the
|
||||||
|
wiki/admin UI). Do not rely on them from external clients — use `/v1`.
|
||||||
|
|
||||||
|
The authoritative list of versioned paths is in `src/atocore/main.py`
|
||||||
|
(`_V1_PUBLIC_PATHS`). `GET /openapi.json` reflects both the versioned and
|
||||||
|
unversioned forms.
|
||||||
|
|
||||||
## Architecture
|
## Architecture
|
||||||
|
|
||||||
```text
|
```text
|
||||||
|
|||||||
0
deploy/dalidou/auto-triage-watcher.sh
Normal file → Executable file
0
deploy/dalidou/auto-triage-watcher.sh
Normal file → Executable file
60
deploy/dalidou/batch-extract.sh
Normal file → Executable file
60
deploy/dalidou/batch-extract.sh
Normal file → Executable file
@@ -150,6 +150,66 @@ print(f'Pipeline summary persisted: {json.dumps(summary)}')
|
|||||||
log "WARN: pipeline summary persistence failed (non-blocking)"
|
log "WARN: pipeline summary persistence failed (non-blocking)"
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# Step F2: Emerging-concepts detector (Phase 6 C.1)
|
||||||
|
log "Step F2: emerging-concepts detector"
|
||||||
|
python3 "$APP_DIR/scripts/detect_emerging.py" \
|
||||||
|
--base-url "$ATOCORE_URL" \
|
||||||
|
2>&1 || {
|
||||||
|
log "WARN: emerging detector failed (non-blocking)"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Step F3: Transient-to-durable extension (Phase 6 C.3)
|
||||||
|
log "Step F3: transient-to-durable extension"
|
||||||
|
curl -sSf -X POST "$ATOCORE_URL/admin/memory/extend-reinforced" \
|
||||||
|
-H 'Content-Type: application/json' \
|
||||||
|
2>&1 | tail -5 || {
|
||||||
|
log "WARN: extend-reinforced failed (non-blocking)"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Step F4: Confidence decay on unreferenced cold memories (Phase 7D)
|
||||||
|
# Daily: memories with reference_count=0 AND idle > 30 days → confidence × 0.97.
|
||||||
|
# Below 0.3 → auto-supersede with audit. Reversible via reinforcement.
|
||||||
|
log "Step F4: confidence decay"
|
||||||
|
curl -sSf -X POST "$ATOCORE_URL/admin/memory/decay-run" \
|
||||||
|
-H 'Content-Type: application/json' \
|
||||||
|
-d '{"idle_days_threshold": 30, "daily_decay_factor": 0.97, "supersede_confidence_floor": 0.30}' \
|
||||||
|
2>&1 | tail -5 || {
|
||||||
|
log "WARN: decay-run failed (non-blocking)"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Step B3: Memory dedup scan (Phase 7A)
|
||||||
|
# Nightly at 0.90 (tight — only near-duplicates). Sundays run a deeper
|
||||||
|
# pass at 0.85 to catch semantically-similar-but-differently-worded memories.
|
||||||
|
if [[ "$(date -u +%u)" == "7" ]]; then
|
||||||
|
DEDUP_THRESHOLD="0.85"
|
||||||
|
DEDUP_BATCH="80"
|
||||||
|
log "Step B3: memory dedup (Sunday deep pass, threshold $DEDUP_THRESHOLD)"
|
||||||
|
else
|
||||||
|
DEDUP_THRESHOLD="0.90"
|
||||||
|
DEDUP_BATCH="50"
|
||||||
|
log "Step B3: memory dedup (daily, threshold $DEDUP_THRESHOLD)"
|
||||||
|
fi
|
||||||
|
python3 "$APP_DIR/scripts/memory_dedup.py" \
|
||||||
|
--base-url "$ATOCORE_URL" \
|
||||||
|
--similarity-threshold "$DEDUP_THRESHOLD" \
|
||||||
|
--max-batch "$DEDUP_BATCH" \
|
||||||
|
2>&1 || {
|
||||||
|
log "WARN: memory dedup failed (non-blocking)"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Step B4: Tag canonicalization (Phase 7C, weekly Sundays)
|
||||||
|
# Autonomous: LLM proposes alias→canonical maps, auto-applies confidence >= 0.8.
|
||||||
|
# Projects tokens are protected (skipped on both sides). Borderline proposals
|
||||||
|
# land in /admin/tags/aliases for human review.
|
||||||
|
if [[ "$(date -u +%u)" == "7" ]]; then
|
||||||
|
log "Step B4: tag canonicalization (Sunday)"
|
||||||
|
python3 "$APP_DIR/scripts/canonicalize_tags.py" \
|
||||||
|
--base-url "$ATOCORE_URL" \
|
||||||
|
2>&1 || {
|
||||||
|
log "WARN: tag canonicalization failed (non-blocking)"
|
||||||
|
}
|
||||||
|
fi
|
||||||
|
|
||||||
# Step G: Integrity check (Phase 4 V1)
|
# Step G: Integrity check (Phase 4 V1)
|
||||||
log "Step G: integrity check"
|
log "Step G: integrity check"
|
||||||
python3 "$APP_DIR/scripts/integrity_check.py" \
|
python3 "$APP_DIR/scripts/integrity_check.py" \
|
||||||
|
|||||||
110
deploy/dalidou/dedup-watcher.sh
Executable file
110
deploy/dalidou/dedup-watcher.sh
Executable file
@@ -0,0 +1,110 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
#
|
||||||
|
# deploy/dalidou/dedup-watcher.sh
|
||||||
|
# -------------------------------
|
||||||
|
# Host-side watcher for on-demand memory dedup scans (Phase 7A).
|
||||||
|
#
|
||||||
|
# The /admin/triage page has a "🔗 Scan for duplicates" button that POSTs
|
||||||
|
# to /admin/memory/dedup-scan with {project, similarity_threshold, max_batch}.
|
||||||
|
# The container writes this to project_state (atocore/config/dedup_requested_at).
|
||||||
|
#
|
||||||
|
# This script runs on the Dalidou HOST (where claude CLI lives), polls
|
||||||
|
# for the flag, and runs memory_dedup.py when seen.
|
||||||
|
#
|
||||||
|
# Installed via cron every 2 minutes:
|
||||||
|
# */2 * * * * /srv/storage/atocore/app/deploy/dalidou/dedup-watcher.sh \
|
||||||
|
# >> /home/papa/atocore-logs/dedup-watcher.log 2>&1
|
||||||
|
#
|
||||||
|
# Mirrors deploy/dalidou/graduation-watcher.sh exactly.
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
ATOCORE_URL="${ATOCORE_URL:-http://127.0.0.1:8100}"
|
||||||
|
APP_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
|
||||||
|
LOCK_FILE="/tmp/atocore-dedup.lock"
|
||||||
|
LOG_DIR="/home/papa/atocore-logs"
|
||||||
|
mkdir -p "$LOG_DIR"
|
||||||
|
|
||||||
|
TS="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
|
||||||
|
log() { printf '[%s] %s\n' "$TS" "$*"; }
|
||||||
|
|
||||||
|
# Fetch the flag via API
|
||||||
|
STATE_JSON=$(curl -sSf --max-time 5 "$ATOCORE_URL/project/state/atocore" 2>/dev/null || echo "{}")
|
||||||
|
REQUESTED=$(echo "$STATE_JSON" | python3 -c "
|
||||||
|
import sys, json
|
||||||
|
try:
|
||||||
|
d = json.load(sys.stdin)
|
||||||
|
for e in d.get('entries', d.get('state', [])):
|
||||||
|
if e.get('category') == 'config' and e.get('key') == 'dedup_requested_at':
|
||||||
|
print(e.get('value', ''))
|
||||||
|
break
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
" 2>/dev/null || echo "")
|
||||||
|
|
||||||
|
if [[ -z "$REQUESTED" ]]; then
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
PROJECT=$(echo "$REQUESTED" | python3 -c "import sys,json; print(json.loads(sys.stdin.read() or '{}').get('project',''))" 2>/dev/null || echo "")
|
||||||
|
THRESHOLD=$(echo "$REQUESTED" | python3 -c "import sys,json; print(json.loads(sys.stdin.read() or '{}').get('similarity_threshold',0.88))" 2>/dev/null || echo "0.88")
|
||||||
|
MAX_BATCH=$(echo "$REQUESTED" | python3 -c "import sys,json; print(json.loads(sys.stdin.read() or '{}').get('max_batch',50))" 2>/dev/null || echo "50")
|
||||||
|
|
||||||
|
# Acquire lock
|
||||||
|
exec 9>"$LOCK_FILE" || exit 0
|
||||||
|
if ! flock -n 9; then
|
||||||
|
log "dedup already running, skipping"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Mark running
|
||||||
|
curl -sSf -X POST "$ATOCORE_URL/project/state" \
|
||||||
|
-H 'Content-Type: application/json' \
|
||||||
|
-d "{\"project\":\"atocore\",\"category\":\"status\",\"key\":\"dedup_running\",\"value\":\"1\",\"source\":\"dedup watcher\"}" \
|
||||||
|
>/dev/null 2>&1 || true
|
||||||
|
curl -sSf -X POST "$ATOCORE_URL/project/state" \
|
||||||
|
-H 'Content-Type: application/json' \
|
||||||
|
-d "{\"project\":\"atocore\",\"category\":\"status\",\"key\":\"dedup_last_started_at\",\"value\":\"$TS\",\"source\":\"dedup watcher\"}" \
|
||||||
|
>/dev/null 2>&1 || true
|
||||||
|
|
||||||
|
LOG_FILE="$LOG_DIR/dedup-ondemand-$(date -u +%Y%m%d-%H%M%S).log"
|
||||||
|
log "Starting dedup (project='$PROJECT' threshold=$THRESHOLD max_batch=$MAX_BATCH, log: $LOG_FILE)"
|
||||||
|
|
||||||
|
# Clear the flag BEFORE running so duplicate clicks queue at most one
|
||||||
|
curl -sSf -X DELETE "$ATOCORE_URL/project/state" \
|
||||||
|
-H 'Content-Type: application/json' \
|
||||||
|
-d "{\"project\":\"atocore\",\"category\":\"config\",\"key\":\"dedup_requested_at\"}" \
|
||||||
|
>/dev/null 2>&1 || true
|
||||||
|
|
||||||
|
cd "$APP_DIR"
|
||||||
|
export PYTHONPATH="$APP_DIR/src:${PYTHONPATH:-}"
|
||||||
|
ARGS=(--base-url "$ATOCORE_URL" --similarity-threshold "$THRESHOLD" --max-batch "$MAX_BATCH")
|
||||||
|
if [[ -n "$PROJECT" ]]; then
|
||||||
|
ARGS+=(--project "$PROJECT")
|
||||||
|
fi
|
||||||
|
|
||||||
|
if python3 scripts/memory_dedup.py "${ARGS[@]}" >> "$LOG_FILE" 2>&1; then
|
||||||
|
RESULT=$(grep "^summary:" "$LOG_FILE" | tail -1 || tail -1 "$LOG_FILE")
|
||||||
|
RESULT="${RESULT:-completed}"
|
||||||
|
log "dedup finished: $RESULT"
|
||||||
|
else
|
||||||
|
RESULT="ERROR — see $LOG_FILE"
|
||||||
|
log "dedup FAILED"
|
||||||
|
fi
|
||||||
|
|
||||||
|
FINISH_TS="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
|
||||||
|
|
||||||
|
curl -sSf -X POST "$ATOCORE_URL/project/state" \
|
||||||
|
-H 'Content-Type: application/json' \
|
||||||
|
-d "{\"project\":\"atocore\",\"category\":\"status\",\"key\":\"dedup_running\",\"value\":\"0\",\"source\":\"dedup watcher\"}" \
|
||||||
|
>/dev/null 2>&1 || true
|
||||||
|
curl -sSf -X POST "$ATOCORE_URL/project/state" \
|
||||||
|
-H 'Content-Type: application/json' \
|
||||||
|
-d "{\"project\":\"atocore\",\"category\":\"status\",\"key\":\"dedup_last_finished_at\",\"value\":\"$FINISH_TS\",\"source\":\"dedup watcher\"}" \
|
||||||
|
>/dev/null 2>&1 || true
|
||||||
|
|
||||||
|
SAFE_RESULT=$(printf '%s' "$RESULT" | python3 -c "import sys,json; print(json.dumps(sys.stdin.read())[1:-1])")
|
||||||
|
curl -sSf -X POST "$ATOCORE_URL/project/state" \
|
||||||
|
-H 'Content-Type: application/json' \
|
||||||
|
-d "{\"project\":\"atocore\",\"category\":\"status\",\"key\":\"dedup_last_result\",\"value\":\"$SAFE_RESULT\",\"source\":\"dedup watcher\"}" \
|
||||||
|
>/dev/null 2>&1 || true
|
||||||
@@ -27,6 +27,7 @@ services:
|
|||||||
- ${ATOCORE_BACKUP_DIR}:${ATOCORE_BACKUP_DIR}
|
- ${ATOCORE_BACKUP_DIR}:${ATOCORE_BACKUP_DIR}
|
||||||
- ${ATOCORE_RUN_DIR}:${ATOCORE_RUN_DIR}
|
- ${ATOCORE_RUN_DIR}:${ATOCORE_RUN_DIR}
|
||||||
- ${ATOCORE_PROJECT_REGISTRY_DIR}:${ATOCORE_PROJECT_REGISTRY_DIR}
|
- ${ATOCORE_PROJECT_REGISTRY_DIR}:${ATOCORE_PROJECT_REGISTRY_DIR}
|
||||||
|
- ${ATOCORE_ASSETS_DIR}:${ATOCORE_ASSETS_DIR}
|
||||||
- ${ATOCORE_VAULT_SOURCE_DIR}:${ATOCORE_VAULT_SOURCE_DIR}:ro
|
- ${ATOCORE_VAULT_SOURCE_DIR}:${ATOCORE_VAULT_SOURCE_DIR}:ro
|
||||||
- ${ATOCORE_DRIVE_SOURCE_DIR}:${ATOCORE_DRIVE_SOURCE_DIR}:ro
|
- ${ATOCORE_DRIVE_SOURCE_DIR}:${ATOCORE_DRIVE_SOURCE_DIR}:ro
|
||||||
healthcheck:
|
healthcheck:
|
||||||
|
|||||||
0
deploy/dalidou/graduation-watcher.sh
Normal file → Executable file
0
deploy/dalidou/graduation-watcher.sh
Normal file → Executable file
64
deploy/dalidou/hourly-extract.sh
Executable file
64
deploy/dalidou/hourly-extract.sh
Executable file
@@ -0,0 +1,64 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
#
|
||||||
|
# deploy/dalidou/hourly-extract.sh
|
||||||
|
# ---------------------------------
|
||||||
|
# Lightweight hourly extraction + triage so autonomous capture stays
|
||||||
|
# current (not a 24h-latency nightly-only affair).
|
||||||
|
#
|
||||||
|
# Does ONLY:
|
||||||
|
# Step A: LLM extraction over recent interactions (last 2h window)
|
||||||
|
# Step B: 3-tier auto-triage on the resulting candidates
|
||||||
|
#
|
||||||
|
# Skips the heavy nightly stuff (backup, rsync, OpenClaw import,
|
||||||
|
# synthesis, harness, integrity check, emerging detector). Those stay
|
||||||
|
# in cron-backup.sh at 03:00 UTC.
|
||||||
|
#
|
||||||
|
# Runs every hour via cron:
|
||||||
|
# 0 * * * * /srv/storage/atocore/app/deploy/dalidou/hourly-extract.sh \
|
||||||
|
# >> /home/papa/atocore-logs/hourly-extract.log 2>&1
|
||||||
|
#
|
||||||
|
# Lock file prevents overlap if a previous run is still going (which
|
||||||
|
# can happen if claude CLI rate-limits and retries).
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
ATOCORE_URL="${ATOCORE_URL:-http://127.0.0.1:8100}"
|
||||||
|
# 50 recent interactions is enough for an hour — typical usage is under 20/h.
|
||||||
|
LIMIT="${ATOCORE_HOURLY_EXTRACT_LIMIT:-50}"
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
APP_DIR="$(cd "$SCRIPT_DIR/../.." && pwd)"
|
||||||
|
TIMESTAMP="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
|
||||||
|
LOCK_FILE="/tmp/atocore-hourly-extract.lock"
|
||||||
|
|
||||||
|
log() { printf '[%s] %s\n' "$TIMESTAMP" "$*"; }
|
||||||
|
|
||||||
|
# Acquire lock (non-blocking)
|
||||||
|
exec 9>"$LOCK_FILE" || exit 0
|
||||||
|
if ! flock -n 9; then
|
||||||
|
log "hourly extract already running, skipping"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
export PYTHONPATH="$APP_DIR/src:${PYTHONPATH:-}"
|
||||||
|
|
||||||
|
log "=== hourly extract+triage starting ==="
|
||||||
|
|
||||||
|
# Step A — Extract candidates from recent interactions
|
||||||
|
log "Step A: LLM extraction (since last run)"
|
||||||
|
python3 "$APP_DIR/scripts/batch_llm_extract_live.py" \
|
||||||
|
--base-url "$ATOCORE_URL" \
|
||||||
|
--limit "$LIMIT" \
|
||||||
|
2>&1 || {
|
||||||
|
log "WARN: batch extraction failed (non-blocking)"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Step B — 3-tier auto-triage (sonnet → opus → discard)
|
||||||
|
log "Step B: auto-triage (3-tier)"
|
||||||
|
python3 "$APP_DIR/scripts/auto_triage.py" \
|
||||||
|
--base-url "$ATOCORE_URL" \
|
||||||
|
--max-batches 3 \
|
||||||
|
2>&1 || {
|
||||||
|
log "WARN: auto-triage failed (non-blocking)"
|
||||||
|
}
|
||||||
|
|
||||||
|
log "=== hourly extract+triage complete ==="
|
||||||
0
deploy/hooks/capture_stop.py
Normal file → Executable file
0
deploy/hooks/capture_stop.py
Normal file → Executable file
174
deploy/hooks/inject_context.py
Executable file
174
deploy/hooks/inject_context.py
Executable file
@@ -0,0 +1,174 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Claude Code UserPromptSubmit hook: inject AtoCore context.
|
||||||
|
|
||||||
|
Mirrors the OpenClaw 7I pattern on the Claude Code side. Every user
|
||||||
|
prompt submitted to Claude Code is (a) sent to /context/build on the
|
||||||
|
AtoCore API, and (b) the returned context pack is prepended to the
|
||||||
|
prompt the LLM sees — so Claude Code answers grounded in what AtoCore
|
||||||
|
already knows, same as OpenClaw now does.
|
||||||
|
|
||||||
|
Contract per Claude Code hooks spec:
|
||||||
|
stdin: JSON with `prompt`, `session_id`, `transcript_path`, `cwd`,
|
||||||
|
`hook_event_name`, etc.
|
||||||
|
stdout on success: JSON
|
||||||
|
{"hookSpecificOutput":
|
||||||
|
{"hookEventName": "UserPromptSubmit",
|
||||||
|
"additionalContext": "<pack>"}}
|
||||||
|
exit 0 always — fail open. An unreachable AtoCore must never block
|
||||||
|
the user's prompt.
|
||||||
|
|
||||||
|
Environment variables:
|
||||||
|
ATOCORE_URL base URL (default http://dalidou:8100)
|
||||||
|
ATOCORE_CONTEXT_DISABLED set to "1" to disable injection
|
||||||
|
ATOCORE_CONTEXT_BUDGET max chars of injected pack (default 4000)
|
||||||
|
ATOCORE_CONTEXT_TIMEOUT HTTP timeout in seconds (default 5)
|
||||||
|
|
||||||
|
Usage in ~/.claude/settings.json:
|
||||||
|
"UserPromptSubmit": [{
|
||||||
|
"matcher": "",
|
||||||
|
"hooks": [{
|
||||||
|
"type": "command",
|
||||||
|
"command": "python /path/to/inject_context.py",
|
||||||
|
"timeout": 10
|
||||||
|
}]
|
||||||
|
}]
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
import urllib.error
|
||||||
|
import urllib.request
|
||||||
|
|
||||||
|
ATOCORE_URL = os.environ.get("ATOCORE_URL", "http://dalidou:8100")
|
||||||
|
CONTEXT_TIMEOUT = float(os.environ.get("ATOCORE_CONTEXT_TIMEOUT", "5"))
|
||||||
|
CONTEXT_BUDGET = int(os.environ.get("ATOCORE_CONTEXT_BUDGET", "4000"))
|
||||||
|
|
||||||
|
# Don't spend an API call on trivial acks or slash commands.
|
||||||
|
MIN_PROMPT_LENGTH = 15
|
||||||
|
|
||||||
|
|
||||||
|
# Project inference table — kept in sync with capture_stop.py so both
|
||||||
|
# hooks agree on what project a Claude Code session belongs to.
|
||||||
|
_VAULT = "C:\\Users\\antoi\\antoine\\My Libraries\\Antoine Brain Extension"
|
||||||
|
_PROJECT_PATH_MAP: dict[str, str] = {
|
||||||
|
f"{_VAULT}\\2-Projects\\P04-GigaBIT-M1": "p04-gigabit",
|
||||||
|
f"{_VAULT}\\2-Projects\\P10-Interferometer": "p05-interferometer",
|
||||||
|
f"{_VAULT}\\2-Projects\\P11-Polisher-Fullum": "p06-polisher",
|
||||||
|
f"{_VAULT}\\2-Projects\\P08-ABB-Space-Mirror": "abb-space",
|
||||||
|
f"{_VAULT}\\2-Projects\\I01-Atomizer": "atomizer-v2",
|
||||||
|
f"{_VAULT}\\2-Projects\\I02-AtoCore": "atocore",
|
||||||
|
"C:\\Users\\antoi\\ATOCore": "atocore",
|
||||||
|
"C:\\Users\\antoi\\Polisher-Sim": "p06-polisher",
|
||||||
|
"C:\\Users\\antoi\\Fullum-Interferometer": "p05-interferometer",
|
||||||
|
"C:\\Users\\antoi\\Atomizer-V2": "atomizer-v2",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _infer_project(cwd: str) -> str:
|
||||||
|
if not cwd:
|
||||||
|
return ""
|
||||||
|
norm = os.path.normpath(cwd).lower()
|
||||||
|
for path_prefix, project_id in _PROJECT_PATH_MAP.items():
|
||||||
|
if norm.startswith(os.path.normpath(path_prefix).lower()):
|
||||||
|
return project_id
|
||||||
|
return ""
|
||||||
|
|
||||||
|
|
||||||
|
def _emit_empty() -> None:
|
||||||
|
"""Exit 0 with no additionalContext — equivalent to no-op."""
|
||||||
|
sys.exit(0)
|
||||||
|
|
||||||
|
|
||||||
|
def _emit_context(pack: str) -> None:
|
||||||
|
"""Write the hook output JSON and exit 0."""
|
||||||
|
out = {
|
||||||
|
"hookSpecificOutput": {
|
||||||
|
"hookEventName": "UserPromptSubmit",
|
||||||
|
"additionalContext": pack,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
sys.stdout.write(json.dumps(out))
|
||||||
|
sys.exit(0)
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
if os.environ.get("ATOCORE_CONTEXT_DISABLED") == "1":
|
||||||
|
_emit_empty()
|
||||||
|
|
||||||
|
try:
|
||||||
|
raw = sys.stdin.read()
|
||||||
|
if not raw.strip():
|
||||||
|
_emit_empty()
|
||||||
|
hook_data = json.loads(raw)
|
||||||
|
except Exception as exc:
|
||||||
|
# Bad stdin → nothing to do
|
||||||
|
print(f"inject_context: bad stdin: {exc}", file=sys.stderr)
|
||||||
|
_emit_empty()
|
||||||
|
|
||||||
|
prompt = (hook_data.get("prompt") or "").strip()
|
||||||
|
cwd = hook_data.get("cwd", "")
|
||||||
|
|
||||||
|
if len(prompt) < MIN_PROMPT_LENGTH:
|
||||||
|
_emit_empty()
|
||||||
|
|
||||||
|
# Skip meta / system prompts that start with '<' (XML tags etc.)
|
||||||
|
if prompt.startswith("<"):
|
||||||
|
_emit_empty()
|
||||||
|
|
||||||
|
project = _infer_project(cwd)
|
||||||
|
|
||||||
|
body = json.dumps({
|
||||||
|
"prompt": prompt,
|
||||||
|
"project": project,
|
||||||
|
"char_budget": CONTEXT_BUDGET,
|
||||||
|
}).encode("utf-8")
|
||||||
|
|
||||||
|
req = urllib.request.Request(
|
||||||
|
f"{ATOCORE_URL}/context/build",
|
||||||
|
data=body,
|
||||||
|
headers={"Content-Type": "application/json"},
|
||||||
|
method="POST",
|
||||||
|
)
|
||||||
|
|
||||||
|
try:
|
||||||
|
resp = urllib.request.urlopen(req, timeout=CONTEXT_TIMEOUT)
|
||||||
|
data = json.loads(resp.read().decode("utf-8"))
|
||||||
|
except urllib.error.URLError as exc:
|
||||||
|
# AtoCore unreachable — fail open
|
||||||
|
print(f"inject_context: atocore unreachable: {exc}", file=sys.stderr)
|
||||||
|
_emit_empty()
|
||||||
|
except Exception as exc:
|
||||||
|
print(f"inject_context: request failed: {exc}", file=sys.stderr)
|
||||||
|
_emit_empty()
|
||||||
|
|
||||||
|
pack = (data.get("formatted_context") or "").strip()
|
||||||
|
if not pack:
|
||||||
|
_emit_empty()
|
||||||
|
|
||||||
|
# Safety truncate. /context/build respects the budget we sent, but
|
||||||
|
# be defensive in case of a regression.
|
||||||
|
if len(pack) > CONTEXT_BUDGET + 500:
|
||||||
|
pack = pack[:CONTEXT_BUDGET] + "\n\n[context truncated]"
|
||||||
|
|
||||||
|
# Wrap so the LLM knows this is injected grounding, not user text.
|
||||||
|
wrapped = (
|
||||||
|
"---\n"
|
||||||
|
"AtoCore-injected context for this prompt "
|
||||||
|
f"(project={project or '(none)'}):\n\n"
|
||||||
|
f"{pack}\n"
|
||||||
|
"---"
|
||||||
|
)
|
||||||
|
|
||||||
|
print(
|
||||||
|
f"inject_context: injected {len(pack)} chars "
|
||||||
|
f"(project={project or 'none'}, prompt_chars={len(prompt)})",
|
||||||
|
file=sys.stderr,
|
||||||
|
)
|
||||||
|
_emit_context(wrapped)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
96
docs/PHASE-7-MEMORY-CONSOLIDATION.md
Normal file
96
docs/PHASE-7-MEMORY-CONSOLIDATION.md
Normal file
@@ -0,0 +1,96 @@
|
|||||||
|
# Phase 7 — Memory Consolidation (the "Sleep Cycle")
|
||||||
|
|
||||||
|
**Status**: 7A in progress · 7B-H scoped, deferred
|
||||||
|
**Design principle**: *"Like human memory while sleeping, but more robotic — never discard relevant details. Consolidate, update, supersede — don't delete."*
|
||||||
|
|
||||||
|
## Why
|
||||||
|
|
||||||
|
Phases 1–6 built capture + triage + graduation + emerging-project detection. What they don't solve:
|
||||||
|
|
||||||
|
| # | Problem | Fix |
|
||||||
|
|---|---|---|
|
||||||
|
| 1 | Redundancy — "APM uses NX" said 5 different ways across 5 memories | **7A** Semantic dedup |
|
||||||
|
| 2 | Latent contradictions — "chose Zygo" + "switched from Zygo" both active | **7B** Pair contradiction detection |
|
||||||
|
| 3 | Tag drift — `firmware`, `fw`, `firmware-control` fragment retrieval | **7C** Tag canonicalization |
|
||||||
|
| 4 | Confidence staleness — 6-month unreferenced memory ranks as fresh | **7D** Confidence decay |
|
||||||
|
| 5 | No memory drill-down page | **7E** `/wiki/memories/{id}` |
|
||||||
|
| 6 | Domain knowledge siloed per project | **7F** `/wiki/domains/{tag}` |
|
||||||
|
| 7 | Prompt upgrades (llm-0.5 → 0.6) don't re-process old interactions | **7G** Re-extraction on version bump |
|
||||||
|
| 8 | Superseded memory vectors still in Chroma polluting retrieval | **7H** Vector hygiene |
|
||||||
|
|
||||||
|
Collectively: the brain needs a nightly pass that looks at what it already knows and tidies up — dedup, resolve contradictions, canonicalize tags, decay stale facts — **without losing information**.
|
||||||
|
|
||||||
|
## Subphases
|
||||||
|
|
||||||
|
### 7A — Semantic dedup + consolidation *(this sprint)*
|
||||||
|
|
||||||
|
Compute embeddings on active memories, find pairs within `(project, memory_type)` bucket above similarity threshold (default 0.88), cluster, draft a unified memory via LLM, human approves in triage UI. On approve: sources become `superseded`, new merged memory created with union of `source_refs`, sum of `reference_count`, max of `confidence`. **Ships first** because redundancy compounds — every new memory potentially duplicates an old one.
|
||||||
|
|
||||||
|
Detailed spec lives in the working plan (`dapper-cooking-tower.md`) and across the files listed under "Files touched" below. Key decisions:
|
||||||
|
|
||||||
|
- LLM drafts, human approves — no silent auto-merge.
|
||||||
|
- Same `(project, memory_type)` bucket only. Cross-project merges are rare + risky → separate flow in 7B.
|
||||||
|
- Recompute embeddings each scan (~2s / 335 memories). Persist only if scan time becomes a problem.
|
||||||
|
- Cluster-based proposals (A~B~C → one merge), not pair-based.
|
||||||
|
- `status=superseded` never deleted — still queryable with filter.
|
||||||
|
|
||||||
|
**Schema**: new table `memory_merge_candidates` (pending | approved | rejected).
|
||||||
|
**Cron**: nightly at threshold 0.90 (tight); weekly (Sundays) at 0.85 (deeper cleanup).
|
||||||
|
**UI**: new "🔗 Merge Candidates" section in `/admin/triage`.
|
||||||
|
|
||||||
|
**Files touched in 7A**:
|
||||||
|
- `src/atocore/models/database.py` — migration
|
||||||
|
- `src/atocore/memory/similarity.py` — new, `compute_memory_similarity()`
|
||||||
|
- `src/atocore/memory/_dedup_prompt.py` — new, shared LLM prompt
|
||||||
|
- `src/atocore/memory/service.py` — `merge_memories()`
|
||||||
|
- `scripts/memory_dedup.py` — new, host-side detector (HTTP-only)
|
||||||
|
- `src/atocore/api/routes.py` — 5 new endpoints under `/admin/memory/`
|
||||||
|
- `src/atocore/engineering/triage_ui.py` — merge cards section
|
||||||
|
- `deploy/dalidou/batch-extract.sh` — Step B3
|
||||||
|
- `deploy/dalidou/dedup-watcher.sh` — new, UI-triggered scans
|
||||||
|
- `tests/test_memory_dedup.py` — ~10-15 new tests
|
||||||
|
|
||||||
|
### 7B — Memory-to-memory contradiction detection
|
||||||
|
|
||||||
|
Same embedding-pair machinery as 7A but within a *different* band (similarity 0.70–0.88 — semantically related but different wording). LLM classifies each pair: `duplicate | complementary | contradicts | supersedes-older`. Contradictions write a `memory_conflicts` row + surface a triage badge. Clear supersessions (both tier 1 sonnet and tier 2 opus agree) auto-mark the older as `superseded`.
|
||||||
|
|
||||||
|
### 7C — Tag canonicalization
|
||||||
|
|
||||||
|
Weekly LLM pass over `domain_tags` distribution, proposes `alias → canonical` map (e.g. `fw → firmware`). Human approves via UI (one-click pattern, same as emerging-project registration). Bulk-rewrites `domain_tags` atomically across all memories.
|
||||||
|
|
||||||
|
### 7D — Confidence decay
|
||||||
|
|
||||||
|
Daily lightweight job. For memories with `reference_count=0` AND `last_referenced_at` older than 30 days: multiply confidence by 0.97/day (~2-month half-life). Reinforcement already bumps confidence. Below 0.3 → auto-supersede with reason `decayed, no references`. Reversible (tune half-life), non-destructive (still searchable with status filter).
|
||||||
|
|
||||||
|
### 7E — Memory detail page `/wiki/memories/{id}`
|
||||||
|
|
||||||
|
Provenance chain: source_chunk → interaction → graduated_to_entity. Audit trail (Phase 4 has the data). Related memories (same project + tag + semantic neighbors). Decay trajectory plot (if 7D ships). Link target from every memory surfaced anywhere in the wiki.
|
||||||
|
|
||||||
|
### 7F — Cross-project domain view `/wiki/domains/{tag}`
|
||||||
|
|
||||||
|
One page per `domain_tag` showing all memories + graduated entities with that tag, grouped by project. "Optics across p04+p05+p06" becomes a real navigable page. Answers the long-standing question the tag system was meant to enable.
|
||||||
|
|
||||||
|
### 7G — Re-extraction on prompt upgrade
|
||||||
|
|
||||||
|
`batch_llm_extract_live.py --force-reextract --since DATE`. Dedupe key: `(interaction_id, extractor_version)` — same run on same interaction doesn't double-create. Triggered manually when `LLM_EXTRACTOR_VERSION` bumps. Not automatic (destructive).
|
||||||
|
|
||||||
|
### 7H — Vector store hygiene
|
||||||
|
|
||||||
|
Nightly: scan `source_chunks` and `memory_embeddings` (added in 7A V2) for `status=superseded|invalid`. Delete matching vectors from Chroma. Fail-open — the retrieval harness catches any real regression.
|
||||||
|
|
||||||
|
## Verification & ship order
|
||||||
|
|
||||||
|
1. **7A** — ship + observe 1 week → validate merge proposals are high-signal, rejection rate acceptable
|
||||||
|
2. **7D** — decay is low-risk + high-compounding value; ship second
|
||||||
|
3. **7C** — clean up tag fragmentation before 7F depends on canonical tags
|
||||||
|
4. **7E** + **7F** — UX surfaces; ship together once data is clean
|
||||||
|
5. **7B** — contradictions flow (pairs harder than duplicates to classify; wait for 7A data to tune threshold)
|
||||||
|
6. **7G** — on-demand; no ship until we actually bump the extractor prompt
|
||||||
|
7. **7H** — housekeeping; after 7A + 7B + 7D have generated enough `superseded` rows to matter
|
||||||
|
|
||||||
|
## Scope NOT in Phase 7
|
||||||
|
|
||||||
|
- Graduated memories (entity-descended) are **frozen** — exempt from dedup/decay. Entity consolidation is a separate Phase (8+).
|
||||||
|
- Auto-merging without human approval (always human-in-the-loop in V1).
|
||||||
|
- Summarization / compression — a different problem (reducing the number of chunks per memory, not the number of memories).
|
||||||
|
- Forgetting policies — there's no user-facing "delete this" flow in Phase 7. Supersede + filter covers the need.
|
||||||
@@ -159,6 +159,17 @@ Every major object should support fields equivalent to:
|
|||||||
- `created_at`
|
- `created_at`
|
||||||
- `updated_at`
|
- `updated_at`
|
||||||
- `notes` (optional)
|
- `notes` (optional)
|
||||||
|
- `extractor_version` (V1-0)
|
||||||
|
- `canonical_home` (V1-0)
|
||||||
|
|
||||||
|
**Naming note (V1-0, 2026-04-22).** The AtoCore `entities` table and
|
||||||
|
`Entity` dataclass name the project-identifier field `project`, not
|
||||||
|
`project_id`. This doc's "fields equivalent to" wording allows that
|
||||||
|
naming flexibility — the `project` field on entity rows IS the
|
||||||
|
`project_id` per spec. No storage rename is planned; downstream readers
|
||||||
|
should treat `entity.project` as the project identifier. This was
|
||||||
|
resolved in Codex's third-round audit of the V1 Completion Plan (see
|
||||||
|
`docs/plans/engineering-v1-completion-plan.md`).
|
||||||
|
|
||||||
## Suggested Status Lifecycle
|
## Suggested Status Lifecycle
|
||||||
|
|
||||||
|
|||||||
45
docs/capture-surfaces.md
Normal file
45
docs/capture-surfaces.md
Normal file
@@ -0,0 +1,45 @@
|
|||||||
|
# AtoCore — sanctioned capture surfaces
|
||||||
|
|
||||||
|
**Scope statement**: AtoCore captures conversations from **two surfaces only**. Everything else is intentionally out of scope.
|
||||||
|
|
||||||
|
| Surface | Hooks | Status |
|
||||||
|
|---|---|---|
|
||||||
|
| **Claude Code** (local CLI) | `Stop` (capture) + `UserPromptSubmit` (context injection) | both installed |
|
||||||
|
| **OpenClaw** (agent framework on T420) | `before_agent_start` (context injection) + `llm_output` (capture) | both installed (v0.2.0 plugin, Phase 7I) |
|
||||||
|
|
||||||
|
Both surfaces are **symmetric** — push (capture) and pull (context injection on prompt submit) — so AtoCore learns from every turn AND every turn is grounded in what AtoCore already knows.
|
||||||
|
|
||||||
|
## Why these two?
|
||||||
|
|
||||||
|
- **Stable hook APIs.** Claude Code exposes `Stop` and `UserPromptSubmit` lifecycle hooks with documented JSON contracts. OpenClaw exposes `before_agent_start` and `llm_output`. Both run locally where we control the process.
|
||||||
|
- **Passive from the user's perspective.** No paste, no manual capture command, no "remember this" prompt. You just use the tool and AtoCore absorbs everything durable.
|
||||||
|
- **Failure is graceful.** If AtoCore is down, hooks exit 0 with no output — the user's turn proceeds uninterrupted.
|
||||||
|
|
||||||
|
## Why not Claude Desktop / Claude.ai web / Claude mobile / ChatGPT / …?
|
||||||
|
|
||||||
|
- Claude Desktop has MCP but no `Stop`-equivalent hook for auto-capture; auto-capture would require system-prompt coercion ("call atocore_remember every turn"), which is fragile.
|
||||||
|
- Claude.ai web has no hook surface — would need a browser extension (real project, not shipped).
|
||||||
|
- Claude mobile app has neither hooks nor MCP — nothing to wire into.
|
||||||
|
- ChatGPT etc. — same as above.
|
||||||
|
|
||||||
|
**Anthropic API log polling is explicitly prohibited.**
|
||||||
|
|
||||||
|
If you find yourself wanting to capture from one of these, the real answer is: use Claude Code or OpenClaw for the work that matters. Don't paste chat transcripts into AtoCore — that contradicts the whole design principle of passive capture.
|
||||||
|
|
||||||
|
A `/wiki/capture` fallback form still exists (the endpoint `/interactions` is public) but it is **not promoted in the UI** and is documented as a last-resort escape hatch. If you're reaching for it, something is wrong with your workflow, not with AtoCore.
|
||||||
|
|
||||||
|
## Hook files
|
||||||
|
|
||||||
|
- `deploy/hooks/capture_stop.py` — Claude Code Stop → POSTs `/interactions`
|
||||||
|
- `deploy/hooks/inject_context.py` — Claude Code UserPromptSubmit → POSTs `/context/build`, returns pack via `hookSpecificOutput.additionalContext`
|
||||||
|
- `openclaw-plugins/atocore-capture/index.js` — OpenClaw plugin v0.2.0: capture + context injection
|
||||||
|
|
||||||
|
Both Claude Code hooks share a `_infer_project` table mapping cwd to project slug. Keep them in sync when adding a new project path.
|
||||||
|
|
||||||
|
## Kill switches
|
||||||
|
|
||||||
|
- `ATOCORE_CAPTURE_DISABLED=1` → skip Stop capture
|
||||||
|
- `ATOCORE_CONTEXT_DISABLED=1` → skip UserPromptSubmit injection
|
||||||
|
- OpenClaw plugin config `injectContext: false` → skip context injection (capture still fires)
|
||||||
|
|
||||||
|
All three are documented in the respective hook/plugin files.
|
||||||
@@ -1,275 +1,49 @@
|
|||||||
# AtoCore Current State
|
# AtoCore — Current State (2026-04-19)
|
||||||
|
|
||||||
## Status Summary
|
Live deploy: `877b97e` · Dalidou health: ok · Harness: 17/18.
|
||||||
|
|
||||||
AtoCore is no longer just a proof of concept. The local engine exists, the
|
## The numbers
|
||||||
correctness pass is complete, Dalidou now hosts the canonical runtime and
|
|
||||||
machine-storage location, and the T420/OpenClaw side now has a safe read-only
|
|
||||||
path to consume AtoCore. The live corpus is no longer just self-knowledge: it
|
|
||||||
now includes a first curated ingestion batch for the active projects.
|
|
||||||
|
|
||||||
## Phase Assessment
|
| | count |
|
||||||
|
|---|---|
|
||||||
|
| Active memories | 266 (180 project, 31 preference, 24 knowledge, 17 adaptation, 11 episodic, 3 identity) |
|
||||||
|
| Candidates pending | **0** (autonomous triage drained the queue) |
|
||||||
|
| Interactions captured | 605 (250 claude-code, 351 openclaw) |
|
||||||
|
| Entities (typed graph) | 50 |
|
||||||
|
| Vectors in Chroma | 33K+ |
|
||||||
|
| Projects | 6 registered (p04, p05, p06, abb-space, atomizer-v2, atocore) + apm emerging (2 memories, below auto-register threshold) |
|
||||||
|
| Unique domain tags | 210 |
|
||||||
|
| Tests | 440 passing |
|
||||||
|
|
||||||
- completed
|
## Autonomous pipeline — what runs without me
|
||||||
- Phase 0
|
|
||||||
- Phase 0.5
|
|
||||||
- Phase 1
|
|
||||||
- baseline complete
|
|
||||||
- Phase 2
|
|
||||||
- Phase 3
|
|
||||||
- Phase 5
|
|
||||||
- Phase 7
|
|
||||||
- Phase 9 (Commits A/B/C: capture, reinforcement, extractor + review queue)
|
|
||||||
- partial
|
|
||||||
- Phase 4
|
|
||||||
- Phase 8
|
|
||||||
- not started
|
|
||||||
- Phase 6
|
|
||||||
- Phase 10
|
|
||||||
- Phase 11
|
|
||||||
- Phase 12
|
|
||||||
- Phase 13
|
|
||||||
|
|
||||||
## What Exists Today
|
| When | Job | Does |
|
||||||
|
|---|---|---|
|
||||||
|
| every hour | `hourly-extract.sh` | Pulls new interactions → LLM extraction → 3-tier auto-triage (sonnet → opus → discard/human). 0 pending candidates right now = autonomy is working. |
|
||||||
|
| every 2 min | `dedup-watcher.sh` | Services UI-triggered dedup scans |
|
||||||
|
| daily 03:00 UTC | Full nightly (`batch-extract.sh`) | Extract · triage · auto-promote reinforced · synthesis · harness · dedup (0.90) · emerging detector · transient→durable · **confidence decay (7D)** · integrity check · alerts |
|
||||||
|
| Sundays | +Weekly deep pass | Knowledge-base lint · dedup @ 0.85 · **tag canonicalization (7C)** |
|
||||||
|
|
||||||
- ingestion pipeline
|
Last nightly run (2026-04-19 03:00 UTC): **31 promoted · 39 rejected · 0 needs human**. That's the brain self-organizing.
|
||||||
- parser and chunker
|
|
||||||
- SQLite-backed memory and project state
|
|
||||||
- vector retrieval
|
|
||||||
- context builder
|
|
||||||
- API routes for query, context, health, and source status
|
|
||||||
- project registry and per-project refresh foundation
|
|
||||||
- project registration lifecycle:
|
|
||||||
- template
|
|
||||||
- proposal preview
|
|
||||||
- approved registration
|
|
||||||
- safe update of existing project registrations
|
|
||||||
- refresh
|
|
||||||
- implementation-facing architecture notes for:
|
|
||||||
- engineering knowledge hybrid architecture
|
|
||||||
- engineering ontology v1
|
|
||||||
- env-driven storage and deployment paths
|
|
||||||
- Dalidou Docker deployment foundation
|
|
||||||
- initial AtoCore self-knowledge corpus ingested on Dalidou
|
|
||||||
- T420/OpenClaw read-only AtoCore helper skill
|
|
||||||
- full active-project markdown/text corpus wave for:
|
|
||||||
- `p04-gigabit`
|
|
||||||
- `p05-interferometer`
|
|
||||||
- `p06-polisher`
|
|
||||||
|
|
||||||
## What Is True On Dalidou
|
## Phase 7 — Memory Consolidation status
|
||||||
|
|
||||||
- deployed repo location:
|
| Subphase | What | Status |
|
||||||
- `/srv/storage/atocore/app`
|
|---|---|---|
|
||||||
- canonical machine DB location:
|
| 7A | Semantic dedup + merge lifecycle | live |
|
||||||
- `/srv/storage/atocore/data/db/atocore.db`
|
| 7A.1 | Tiered auto-approve (sonnet ≥0.8 + sim ≥0.92 → merge; opus escalation; human only for ambiguous) | live |
|
||||||
- canonical vector store location:
|
| 7B | Memory-to-memory contradiction detection (0.70–0.88 band, classify duplicate/contradicts/supersedes) | deferred, needs 7A signal |
|
||||||
- `/srv/storage/atocore/data/chroma`
|
| 7C | Tag canonicalization (weekly; auto-apply ≥0.8 confidence; protects project tokens) | live (first run: 0 proposals — vocabulary is clean) |
|
||||||
- source input locations:
|
| 7D | Confidence decay (0.97/day on idle unreferenced; auto-supersede below 0.3) | live (first run: 0 decayed — nothing idle+unreferenced yet) |
|
||||||
- `/srv/storage/atocore/sources/vault`
|
| 7E | `/wiki/memories/{id}` detail page | pending |
|
||||||
- `/srv/storage/atocore/sources/drive`
|
| 7F | `/wiki/domains/{tag}` cross-project view | pending (wants 7C + more usage first) |
|
||||||
|
| 7G | Re-extraction on prompt version bump | pending |
|
||||||
|
| 7H | Chroma vector hygiene (delete vectors for superseded memories) | pending |
|
||||||
|
|
||||||
The service and storage foundation are live on Dalidou.
|
## Known gaps (honest)
|
||||||
|
|
||||||
The machine-data host is real and canonical.
|
1. **Capture surface is Claude-Code-and-OpenClaw only.** Conversations in Claude Desktop, Claude.ai web, phone, or any other LLM UI are NOT captured. Example: the rotovap/mushroom chat yesterday never reached AtoCore because no hook fired. See Q4 below.
|
||||||
|
2. **OpenClaw is capture-only, not context-grounded.** The plugin POSTs `/interactions` on `llm_output` but does NOT call `/context/build` on `before_agent_start`. OpenClaw's underlying agent runs blind. See Q2 below.
|
||||||
The project registry is now also persisted in a canonical mounted config path on
|
3. **Human interface (wiki) is thin and static.** 5 project cards + a "System" line. No dashboard for the autonomous activity. No per-memory detail page. See Q3/Q5.
|
||||||
Dalidou:
|
4. **Harness 17/18** — the `p04-constraints` fixture wants "Zerodur" but retrieval surfaces related-not-exact terms. Content gap, not a retrieval regression.
|
||||||
|
5. **Two projects under-populated**: p05-interferometer (4 memories, 18 state) and atomizer-v2 (1 memory, 6 state). Batch re-extract with the new llm-0.6.0 prompt would help.
|
||||||
- `/srv/storage/atocore/config/project-registry.json`
|
|
||||||
|
|
||||||
The content corpus is partially populated now.
|
|
||||||
|
|
||||||
The Dalidou instance already contains:
|
|
||||||
|
|
||||||
- AtoCore ecosystem and hosting docs
|
|
||||||
- current-state and OpenClaw integration docs
|
|
||||||
- Master Plan V3
|
|
||||||
- Build Spec V1
|
|
||||||
- trusted project-state entries for `atocore`
|
|
||||||
- full staged project markdown/text corpora for:
|
|
||||||
- `p04-gigabit`
|
|
||||||
- `p05-interferometer`
|
|
||||||
- `p06-polisher`
|
|
||||||
- curated repo-context docs for:
|
|
||||||
- `p05`: `Fullum-Interferometer`
|
|
||||||
- `p06`: `polisher-sim`
|
|
||||||
- trusted project-state entries for:
|
|
||||||
- `p04-gigabit`
|
|
||||||
- `p05-interferometer`
|
|
||||||
- `p06-polisher`
|
|
||||||
|
|
||||||
Current live stats after the full active-project wave are now far beyond the
|
|
||||||
initial seed stage:
|
|
||||||
|
|
||||||
- more than `1,100` source documents
|
|
||||||
- more than `20,000` chunks
|
|
||||||
- matching vector count
|
|
||||||
|
|
||||||
The broader long-term corpus is still not fully populated yet. Wider project and
|
|
||||||
vault ingestion remains a deliberate next step rather than something already
|
|
||||||
completed, but the corpus is now meaningfully seeded beyond AtoCore's own docs.
|
|
||||||
|
|
||||||
For human-readable quality review, the current staged project markdown corpus is
|
|
||||||
primarily visible under:
|
|
||||||
|
|
||||||
- `/srv/storage/atocore/sources/vault/incoming/projects`
|
|
||||||
|
|
||||||
This staged area is now useful for review because it contains the markdown/text
|
|
||||||
project docs that were actually ingested for the full active-project wave.
|
|
||||||
|
|
||||||
It is important to read this staged area correctly:
|
|
||||||
|
|
||||||
- it is a readable ingestion input layer
|
|
||||||
- it is not the final machine-memory representation itself
|
|
||||||
- seeing familiar PKM-style notes there is expected
|
|
||||||
- the machine-processed intelligence lives in the DB, chunks, vectors, memory,
|
|
||||||
trusted project state, and context-builder outputs
|
|
||||||
|
|
||||||
## What Is True On The T420
|
|
||||||
|
|
||||||
- SSH access is working
|
|
||||||
- OpenClaw workspace inspected at `/home/papa/clawd`
|
|
||||||
- OpenClaw's own memory system remains unchanged
|
|
||||||
- a read-only AtoCore integration skill exists in the workspace:
|
|
||||||
- `/home/papa/clawd/skills/atocore-context/`
|
|
||||||
- the T420 can successfully reach Dalidou AtoCore over network/Tailscale
|
|
||||||
- fail-open behavior has been verified for the helper path
|
|
||||||
- OpenClaw can now seed AtoCore in two distinct ways:
|
|
||||||
- project-scoped memory entries
|
|
||||||
- staged document ingestion into the retrieval corpus
|
|
||||||
- the helper now supports the practical registered-project lifecycle:
|
|
||||||
- projects
|
|
||||||
- project-template
|
|
||||||
- propose-project
|
|
||||||
- register-project
|
|
||||||
- update-project
|
|
||||||
- refresh-project
|
|
||||||
- the helper now also supports the first organic routing layer:
|
|
||||||
- `detect-project "<prompt>"`
|
|
||||||
- `auto-context "<prompt>" [budget] [project]`
|
|
||||||
- OpenClaw can now default to AtoCore for project-knowledge questions without
|
|
||||||
requiring explicit helper commands from the human every time
|
|
||||||
|
|
||||||
## What Exists In Memory vs Corpus
|
|
||||||
|
|
||||||
These remain separate and that is intentional.
|
|
||||||
|
|
||||||
In `/memory`:
|
|
||||||
|
|
||||||
- project-scoped curated memories now exist for:
|
|
||||||
- `p04-gigabit`: 5 memories
|
|
||||||
- `p05-interferometer`: 6 memories
|
|
||||||
- `p06-polisher`: 8 memories
|
|
||||||
|
|
||||||
These are curated summaries and extracted stable project signals.
|
|
||||||
|
|
||||||
In `source_documents` / retrieval corpus:
|
|
||||||
|
|
||||||
- full project markdown/text corpora are now present for the active project set
|
|
||||||
- retrieval is no longer limited to AtoCore self-knowledge only
|
|
||||||
- the current corpus is broad enough that ranking quality matters more than
|
|
||||||
corpus presence alone
|
|
||||||
- underspecified prompts can still pull in historical or archive material, so
|
|
||||||
project-aware routing and better ranking remain important
|
|
||||||
|
|
||||||
The source refresh model now has a concrete foundation in code:
|
|
||||||
|
|
||||||
- a project registry file defines known project ids, aliases, and ingest roots
|
|
||||||
- the API can list registered projects
|
|
||||||
- the API can return a registration template
|
|
||||||
- the API can preview a registration without mutating state
|
|
||||||
- the API can persist an approved registration
|
|
||||||
- the API can update an existing registered project without changing its canonical id
|
|
||||||
- the API can refresh one registered project at a time
|
|
||||||
|
|
||||||
This lifecycle is now coherent end to end for normal use.
|
|
||||||
|
|
||||||
The first live update passes on existing registered projects have now been
|
|
||||||
verified against `p04-gigabit` and `p05-interferometer`:
|
|
||||||
|
|
||||||
- the registration description can be updated safely
|
|
||||||
- the canonical project id remains unchanged
|
|
||||||
- refresh still behaves cleanly after the update
|
|
||||||
- `context/build` still returns useful project-specific context afterward
|
|
||||||
|
|
||||||
## Reliability Baseline
|
|
||||||
|
|
||||||
The runtime has now been hardened in a few practical ways:
|
|
||||||
|
|
||||||
- SQLite connections use a configurable busy timeout
|
|
||||||
- SQLite uses WAL mode to reduce transient lock pain under normal concurrent use
|
|
||||||
- project registry writes are atomic file replacements rather than in-place rewrites
|
|
||||||
- a full runtime backup and restore path now exists and has been exercised on
|
|
||||||
live Dalidou:
|
|
||||||
- SQLite (hot online backup via `conn.backup()`)
|
|
||||||
- project registry (file copy)
|
|
||||||
- Chroma vector store (cold directory copy under `exclusive_ingestion()`)
|
|
||||||
- backup metadata
|
|
||||||
- `restore_runtime_backup()` with CLI entry point
|
|
||||||
(`python -m atocore.ops.backup restore <STAMP>
|
|
||||||
--confirm-service-stopped`), pre-restore safety snapshot for
|
|
||||||
rollback, WAL/SHM sidecar cleanup, `PRAGMA integrity_check`
|
|
||||||
on the restored file
|
|
||||||
- the first live drill on 2026-04-09 surfaced and fixed a Chroma
|
|
||||||
restore bug on Docker bind-mounted volumes (`shutil.rmtree`
|
|
||||||
on a mount point); a regression test now asserts the
|
|
||||||
destination inode is stable across restore
|
|
||||||
- deploy provenance is visible end-to-end:
|
|
||||||
- `/health` reports `build_sha`, `build_time`, `build_branch`
|
|
||||||
from env vars wired by `deploy.sh`
|
|
||||||
- `deploy.sh` Step 6 verifies the live `build_sha` matches the
|
|
||||||
just-built commit (exit code 6 on drift) so "live is current?"
|
|
||||||
can be answered precisely, not just by `__version__`
|
|
||||||
- `deploy.sh` Step 1.5 detects that the script itself changed
|
|
||||||
in the pulled commit and re-execs into the fresh copy, so
|
|
||||||
the deploy never silently runs the old script against new source
|
|
||||||
|
|
||||||
This does not eliminate every concurrency edge, but it materially improves the
|
|
||||||
current operational baseline.
|
|
||||||
|
|
||||||
In `Trusted Project State`:
|
|
||||||
|
|
||||||
- each active seeded project now has a conservative trusted-state set
|
|
||||||
- promoted facts cover:
|
|
||||||
- summary
|
|
||||||
- core architecture or boundary decision
|
|
||||||
- key constraints
|
|
||||||
- next focus
|
|
||||||
|
|
||||||
This separation is healthy:
|
|
||||||
|
|
||||||
- memory stores distilled project facts
|
|
||||||
- corpus stores the underlying retrievable documents
|
|
||||||
|
|
||||||
## Immediate Next Focus
|
|
||||||
|
|
||||||
1. ~~Re-run the full backup/restore drill~~ — DONE 2026-04-11,
|
|
||||||
full pass (db, registry, chroma, integrity all true)
|
|
||||||
2. ~~Turn on auto-capture of Claude Code sessions in conservative
|
|
||||||
mode~~ — DONE 2026-04-11, Stop hook wired via
|
|
||||||
`deploy/hooks/capture_stop.py` → `POST /interactions`
|
|
||||||
with `reinforce=false`; kill switch via
|
|
||||||
`ATOCORE_CAPTURE_DISABLED=1`
|
|
||||||
3. Run a short real-use pilot with auto-capture on, verify
|
|
||||||
interactions are landing in Dalidou, review quality
|
|
||||||
4. Use the new T420-side organic routing layer in real OpenClaw workflows
|
|
||||||
4. Tighten retrieval quality for the now fully ingested active project corpora
|
|
||||||
5. Move to Wave 2 trusted-operational ingestion instead of blindly widening raw corpus further
|
|
||||||
6. Keep the new engineering-knowledge architecture docs as implementation guidance while avoiding premature schema work
|
|
||||||
7. Expand the remaining boring operations baseline:
|
|
||||||
- retention policy cleanup script
|
|
||||||
- off-Dalidou backup target (rsync or similar)
|
|
||||||
8. Only later consider write-back, reflection, or deeper autonomous behaviors
|
|
||||||
|
|
||||||
See also:
|
|
||||||
|
|
||||||
- [ingestion-waves.md](C:/Users/antoi/ATOCore/docs/ingestion-waves.md)
|
|
||||||
- [master-plan-status.md](C:/Users/antoi/ATOCore/docs/master-plan-status.md)
|
|
||||||
|
|
||||||
## Guiding Constraints
|
|
||||||
|
|
||||||
- bad memory is worse than no memory
|
|
||||||
- trusted project state must remain highest priority
|
|
||||||
- human-readable sources and machine storage stay separate
|
|
||||||
- OpenClaw integration must not degrade OpenClaw baseline behavior
|
|
||||||
|
|||||||
113
docs/decisions/2026-04-22-gbrain-plan-rejection.md
Normal file
113
docs/decisions/2026-04-22-gbrain-plan-rejection.md
Normal file
@@ -0,0 +1,113 @@
|
|||||||
|
# Decision record: gbrain-inspired "Phase 8 Minions + typed edges" plan rejected
|
||||||
|
|
||||||
|
**Date:** 2026-04-22
|
||||||
|
**Author of plan:** Claude
|
||||||
|
**Reviewer:** Codex
|
||||||
|
**Ratified by:** Antoine
|
||||||
|
**Status:** Rejected as packaged. Underlying mechanic (durable background jobs + typed relationships) deferred to its correct home.
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
Antoine surfaced https://github.com/garrytan/gbrain and asked for a compare/contrast and a
|
||||||
|
plan to improve AtoCore. Claude proposed a "Phase 8" plan pairing:
|
||||||
|
|
||||||
|
1. A Minion-style durable job queue replacing the nightly cron pipeline
|
||||||
|
2. A typed-edge upgrade over existing wikilinks, with a six-predicate set
|
||||||
|
(`mentions`, `decided_by`, `supersedes`, `evidences`, `part_of`, `blocks`)
|
||||||
|
|
||||||
|
Codex reviewed and rejected the plan as packaged. This record captures what went wrong,
|
||||||
|
what was right, and where the ideas should actually land.
|
||||||
|
|
||||||
|
## What Codex flagged (verified against repo)
|
||||||
|
|
||||||
|
### High — wrong sequencing
|
||||||
|
|
||||||
|
`docs/master-plan-status.md` defines the **Now** list:
|
||||||
|
|
||||||
|
1. Observe the enhanced pipeline for a week
|
||||||
|
2. Knowledge density — batch-extract over all 234 interactions, target 100+ memories
|
||||||
|
3. Multi-model triage (Phase 11 entry)
|
||||||
|
4. Fix p04-constraints harness failure
|
||||||
|
|
||||||
|
Engineering V1 appears under **Next** (line 179) as
|
||||||
|
"Engineering V1 implementation sprint — once knowledge density is sufficient and the
|
||||||
|
pipeline feels boring and dependable."
|
||||||
|
|
||||||
|
Claude's plan jumped over all four **Now** items. That was the primary sequencing error.
|
||||||
|
|
||||||
|
### High — wrong predicate set
|
||||||
|
|
||||||
|
`docs/architecture/engineering-ontology-v1.md` already defines a 17-predicate V1
|
||||||
|
ontology across four families:
|
||||||
|
|
||||||
|
- **Structural:** `CONTAINS`, `PART_OF`, `INTERFACES_WITH`
|
||||||
|
- **Intent / logic:** `SATISFIES`, `CONSTRAINED_BY`, `BASED_ON_ASSUMPTION`,
|
||||||
|
`AFFECTED_BY_DECISION`, `SUPERSEDES`
|
||||||
|
- **Validation:** `ANALYZED_BY`, `VALIDATED_BY`, `SUPPORTS`, `CONFLICTS_WITH`,
|
||||||
|
`DEPENDS_ON`
|
||||||
|
- **Artifact / provenance:** `DESCRIBED_BY`, `UPDATED_BY_SESSION`, `EVIDENCED_BY`,
|
||||||
|
`SUMMARIZED_IN`
|
||||||
|
|
||||||
|
Claude's six-predicate set was a gbrain-shaped subset that could not express the V1
|
||||||
|
example statements at lines 141–147 of that doc. Shipping it first would have been
|
||||||
|
schema debt on day one.
|
||||||
|
|
||||||
|
### High — wrong canonical boundary
|
||||||
|
|
||||||
|
`docs/architecture/memory-vs-entities.md` and
|
||||||
|
`docs/architecture/engineering-v1-acceptance.md` establish that V1 is **typed
|
||||||
|
entities plus typed relationships**, with one canonical home per concept, a shared
|
||||||
|
candidate-review / promotion flow, provenance, conflict handling, and mirror
|
||||||
|
generation. Claude's "typed edges on top of wikilinks" framing bypassed the canonical
|
||||||
|
entity contract — it would have produced labelled links over notes without the
|
||||||
|
promotion / canonicalization machinery V1 actually requires.
|
||||||
|
|
||||||
|
### Medium — overstated problem
|
||||||
|
|
||||||
|
Claude described the nightly pipeline as a "monolithic bash script" that needed to be
|
||||||
|
replaced. The actual runtime is API-driven (`src/atocore/api/routes.py:516`,
|
||||||
|
`src/atocore/interactions/service.py:55`), SQLite is already in WAL with a busy
|
||||||
|
timeout (`src/atocore/models/database.py:151`), and the reflection loop is explicit
|
||||||
|
capture / reinforce / extract. The queue argument overstated the current shape.
|
||||||
|
|
||||||
|
## What was right
|
||||||
|
|
||||||
|
- gbrain is genuine validation of the general pattern: **durable background jobs +
|
||||||
|
typed relationship graph compound value**. The gbrain v0.12.0 graph release and
|
||||||
|
Minions benchmark (both 2026-04-18) are evidence, not just inspiration.
|
||||||
|
- Async-ification of extraction with retries, per-job visibility, and SLOs remains a
|
||||||
|
real future win for AtoCore — but **additively, behind flags, after V1**, not as a
|
||||||
|
replacement for the current explicit endpoints.
|
||||||
|
|
||||||
|
## What we will do instead
|
||||||
|
|
||||||
|
1. **Keep to the `master-plan-status.md` Now list.** No leapfrog. Observe the
|
||||||
|
pipeline (including the confidence-decay Step F4 first real run), land knowledge
|
||||||
|
density via full-backlog batch extract, progress multi-model triage, fix
|
||||||
|
p04-constraints.
|
||||||
|
2. **When Engineering V1 is ready to start** (criterion: pipeline feels boring and
|
||||||
|
dependable, knowledge density ≥ 100 active memories), write a V1 foundation plan
|
||||||
|
that follows `engineering-ontology-v1.md`, `engineering-query-catalog.md`,
|
||||||
|
`memory-vs-entities.md`, and `engineering-v1-acceptance.md` — entities +
|
||||||
|
relationships + memory-to-entity bridge + mirror / query surfaces, in that order.
|
||||||
|
3. **Async workerization is optional and later.** Only after V1 is working, and only
|
||||||
|
if observed contention or latency warrants it. Jobs stay in the primary SQLite
|
||||||
|
(WAL already in place). No separate DB unless contention is measured.
|
||||||
|
|
||||||
|
## Lesson for future plans
|
||||||
|
|
||||||
|
A plan built from a **new external reference** (gbrain) without reading the
|
||||||
|
repository's own architecture docs will mis-specify predicates, boundaries, and
|
||||||
|
sequencing — even when the underlying mechanic is valid. Read the four V1
|
||||||
|
architecture docs end-to-end before proposing schema work.
|
||||||
|
|
||||||
|
## References
|
||||||
|
|
||||||
|
- https://github.com/garrytan/gbrain
|
||||||
|
- `docs/master-plan-status.md` (Now / Next / Later)
|
||||||
|
- `docs/architecture/engineering-ontology-v1.md`
|
||||||
|
- `docs/architecture/engineering-query-catalog.md`
|
||||||
|
- `docs/architecture/memory-vs-entities.md`
|
||||||
|
- `docs/architecture/engineering-v1-acceptance.md`
|
||||||
|
- `docs/architecture/llm-client-integration.md`
|
||||||
|
- `docs/architecture/human-mirror-rules.md`
|
||||||
617
docs/plans/engineering-v1-completion-plan.md
Normal file
617
docs/plans/engineering-v1-completion-plan.md
Normal file
@@ -0,0 +1,617 @@
|
|||||||
|
# Engineering V1 Completion Plan
|
||||||
|
|
||||||
|
**Date:** 2026-04-22
|
||||||
|
**Author:** Claude (after reading the four V1 architecture docs + promotion-rules,
|
||||||
|
conflict-model, human-mirror-rules, tool-handoff-boundaries end-to-end)
|
||||||
|
**Status:** Draft, pending Codex review
|
||||||
|
**Replaces:** the rejected "Phase 8 Minions + typed edges" plan (see
|
||||||
|
`docs/decisions/2026-04-22-gbrain-plan-rejection.md`)
|
||||||
|
|
||||||
|
## Position
|
||||||
|
|
||||||
|
This is **not** a plan to start Engineering V1. It is a plan to **finish** V1.
|
||||||
|
|
||||||
|
**Against what criterion?** Each F/Q/O/D item in `engineering-v1-acceptance.md`
|
||||||
|
gets scored individually in the Gap audit table below with exact code/test/doc
|
||||||
|
references. No global percentage. The headline framing from the first draft
|
||||||
|
("50–70% built") is withdrawn — it's either done per-criterion or it isn't.
|
||||||
|
|
||||||
|
The relevant observation is narrower: the entity schema, the full
|
||||||
|
relationship type set, the 4-state lifecycle, basic CRUD and most of the
|
||||||
|
killer-correctness query functions are already implemented in
|
||||||
|
`src/atocore/engineering/*.py` in the Windows working tree at
|
||||||
|
`C:\Users\antoi\ATOCore` (the canonical dev workspace, per
|
||||||
|
`CLAUDE.md`). The recent commits e147ab2, b94f9df, 081c058, 069d155, b1a3dd0
|
||||||
|
are V1 entity-layer work. **Codex auditors working in a different
|
||||||
|
workspace / branch should sync from the canonical dev tree before
|
||||||
|
per-file review** — see the "Workspace note" at the end of this doc.
|
||||||
|
|
||||||
|
The question this plan answers: given the current code state, in what
|
||||||
|
order should the remaining V1 acceptance criteria be closed so that
|
||||||
|
every phase builds on invariants the earlier phases already enforced?
|
||||||
|
|
||||||
|
## Corrected sequencing principle (post-Codex review 2026-04-22)
|
||||||
|
|
||||||
|
The first draft ordered phases F-1 → F-2 → F-3 → F-4 → F-5 → F-6 → F-7 → F-8
|
||||||
|
following the acceptance doc's suggested reading order. Codex rejected
|
||||||
|
that ordering. The correct dependency order, which this revision adopts, is:
|
||||||
|
|
||||||
|
1. **Write-time invariants come first.** Every later phase creates active
|
||||||
|
entities. Provenance-at-write (F-8) and synchronous conflict-detection
|
||||||
|
hooks (F-5 minimal) must be enforced **before** any phase that writes
|
||||||
|
entities at scale (ingest, graduation, or broad query coverage that
|
||||||
|
depends on the model being populated).
|
||||||
|
2. **Query closure sits on top of the schema + invariants**, not ahead of
|
||||||
|
them. A minimum query slice that proves the model is fine early. The
|
||||||
|
full catalog closure waits until after the write paths are invariant-safe.
|
||||||
|
3. **Mirror is a derived consumer** of the entity layer, not a midstream
|
||||||
|
milestone. It comes after the entity layer produces enforced, correct data.
|
||||||
|
4. **Graduation and full conflict-spec compliance** are finishing work that
|
||||||
|
depend on everything above being stable.
|
||||||
|
|
||||||
|
The acceptance criteria are unchanged. Only the order of closing them changes.
|
||||||
|
|
||||||
|
## How this plan respects the rejected-plan lessons
|
||||||
|
|
||||||
|
- **No new predicates.** The V1 ontology in `engineering-ontology-v1.md:112-137`
|
||||||
|
already defines 18 relationship types; `service.py:38-62` already implements
|
||||||
|
them. Nothing added, nothing reshaped.
|
||||||
|
- **No new canonical boundary.** Typed entities + typed relationships with
|
||||||
|
promotion-based candidate flow per `memory-vs-entities.md`. Not
|
||||||
|
edges-over-wikilinks.
|
||||||
|
- **No leapfrog of `master-plan-status.md` Now list.** This plan is **in
|
||||||
|
parallel** with (not ahead of) the Now items because V1 entity work is
|
||||||
|
already happening alongside them. The sequencing section below is explicit.
|
||||||
|
- **Queue/worker infrastructure is explicitly out of scope.** The "flag it for
|
||||||
|
later" note at the end of this doc is the only mention, per
|
||||||
|
`engineering-v1-acceptance.md:378` negative list.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Gap audit against `engineering-v1-acceptance.md`
|
||||||
|
|
||||||
|
Each criterion marked: ✅ done / 🟡 partial / ❌ missing. "Partial" means the
|
||||||
|
capability exists but does not yet match spec shape or coverage.
|
||||||
|
|
||||||
|
### Functional (F-1 through F-8)
|
||||||
|
|
||||||
|
| ID | Criterion | Status | Evidence |
|
||||||
|
|----|-----------|--------|----------|
|
||||||
|
| F-1 | 12 V1 entity types, 4 relationship families, shared header fields, 4-state lifecycle | 🟡 partial (per Codex 2026-04-22 audit) | `service.py:16-36` has 16 types (superset of V1's 12), `service.py:38-62` has 18 relationship types, `service.py:64` statuses, `Entity` dataclass at line 67. **Gaps vs `engineering-v1-acceptance.md:45`**: `extractor_version` missing from dataclass and `entities` table; `canonical_home` missing from dataclass and table; `project` field is the project identifier but not named `project_id` as spec uses — spec says "fields equivalent to" so naming flexibility is allowed but needs an explicit doc note. Remediation lands in V1-0 |
|
||||||
|
| F-2 | All v1-required Q-001 through Q-020 implemented, with provenance where required | 🟡 partial (per Codex 2026-04-22 per-function audit) | **Ground truth from per-function read of `queries.py` + `routes.py:2092+`:** Q-001 partial (`system_map()` returns project-wide tree, not the catalog's subsystem-scoped `GET /entities/Subsystem/<id>?expand=contains` shape per `engineering-query-catalog.md:71`); Q-002 missing; Q-003 missing; Q-004 done (covered by `system_map()`); Q-005 done (`requirements_for()`); Q-006 done (`orphan_requirements()`); Q-007 missing; Q-008 done (`decisions_affecting()`); Q-009 done (`risky_decisions()`); Q-010 missing; Q-011 done (`unsupported_claims()`); Q-012 missing; Q-013 done (`recent_changes()`); Q-014 missing; Q-016 done (`impact_analysis()`); Q-017 done (`evidence_chain()`); Q-018 missing; Q-019 missing; Q-020 missing (mirror route in spec shape). **Net: 9 of 20 v1-required queries done, 1 partial (Q-001), 10 missing.** Q-015 is v1-stretch, out of scope |
|
||||||
|
| F-3 | `POST /ingest/kb-cad/export` and `POST /ingest/kb-fem/export` | ❌ missing | No `/ingest/kb-cad` or `/ingest/kb-fem` route in `api/routes.py`. No schema doc under `docs/architecture/` |
|
||||||
|
| F-4 | Candidate review queue end-to-end (list/promote/reject/edit) | 🟡 partial for entities | Memory side shipped in Phase 9 Commit C. Entity side has `promote_entity`, `supersede_entity`, `invalidate_active_entity` but reject path and editable-before-promote may not match spec shape. Need to verify `GET /entities?status=candidate` returns spec shape |
|
||||||
|
| F-5 | Conflict detector fires synchronously; `POST /conflicts/{id}/resolve` + dismiss | 🟡 partial (per Codex 2026-04-22 audit — schema present, detector+routes divergent) | **Schema is already spec-shaped**: `database.py:190` defines the generic `conflicts` + `conflict_members` tables per `conflict-model.md`; `conflicts.py:154` persists through them. **Divergences are in detection and API, not schema**: (1) `conflicts.py:36` dispatches per-type detectors only (`_check_component_conflicts`, `_check_requirement_conflicts`) — needs generalization to slot-key-driven detection; (2) routes live at `/admin/conflicts/*`, spec says `/conflicts/*` — needs alias + deprecation. **No schema migration needed** |
|
||||||
|
| F-6 | Mirror: `/mirror/{project}/overview`, `/decisions`, `/subsystems/{id}`, `/regenerate`; files under `/srv/storage/atocore/data/mirror/`; disputed + curated markers; deterministic output | 🟡 partial | `mirror.py` has `generate_project_overview` with header/state/system/decisions/requirements/materials/vendors/memories/footer sections. API at `/projects/{project_name}/mirror` and `.html`. **Gaps**: no separate `/mirror/{project}/decisions` or `/mirror/{project}/subsystems/{id}` routes, no `POST /regenerate` endpoint, no debounced-async-on-write, no daily refresh, no `⚠ disputed` markers wired to conflicts, no `(curated)` override annotations verified, no golden-file test for determinism |
|
||||||
|
| F-7 | Memory→entity graduation: `POST /memory/{id}/graduate` + `graduated` status + forward pointer + original preserved | 🟡 partial (per Codex 2026-04-22 third-round audit) | `_graduation_prompt.py` exists; `scripts/graduate_memories.py` creates entity candidates from active memories; `database.py:143-146` adds `graduated_to_entity_id`; `memory.service` already has a `graduated` status; `service.py:354-356,389-451` preserves the original memory and marks it `graduated` with a forward pointer on entity promote; `tests/test_engineering_v1_phase5.py:67-90` covers that flow. **Gaps vs spec**: no direct `POST /memory/{id}/graduate` route yet (current surface is batch/admin-driven via `/admin/graduation/request`); no explicit acceptance tests yet for `adaptation→decision` and `project→requirement`; spec wording `knowledge→Fact` does not match the current ontology (there is no `fact` entity type in `service.py` / `_graduation_prompt.py`) and should be reconciled to an actual V1 type such as `parameter` or another ontology-defined entity. |
|
||||||
|
| F-8 | Every active entity has `source_refs`; Q-017 returns ≥1 row for every active entity | 🟡 partial | `Entity.source_refs` field exists; Q-017 (`evidence_chain`) exists. **Gap**: is provenance enforced at write time (not NULL), or just encouraged? Per spec it must be mandatory |
|
||||||
|
|
||||||
|
### Quality (Q-1 through Q-6)
|
||||||
|
|
||||||
|
| ID | Criterion | Status | Evidence |
|
||||||
|
|----|-----------|--------|----------|
|
||||||
|
| Q-1 | All pre-V1 tests still pass | ✅ presumed | 533 tests passing per DEV-LEDGER line 12 |
|
||||||
|
| Q-2 | Each F criterion has happy-path + error-path test, <10s each, <30s total | 🟡 partial | 16 + 15 + 15 + 12 = 58 tests in engineering/queries/v1-phase5/patch files. Need to verify coverage of each F criterion one-for-one |
|
||||||
|
| Q-3 | Conflict invariants enforced by tests (contradictory imports produce conflict, can't promote both, flag-never-block) | 🟡 partial | Tests likely exist in `test_engineering_v1_phase5.py` — verify explicit coverage of the three invariants |
|
||||||
|
| Q-4 | Trust hierarchy enforced by tests (candidates never in context, active-only reinforcement, no auto-project-state writes) | 🟡 partial | Phase 9 Commit B covered the memory side; verify entity side has equivalent tests |
|
||||||
|
| Q-5 | Mirror has golden-file test, deterministic output | ❌ missing | No golden file seen; mirror output reads wall-clock time inside `_footer()` (`mirror.py:320-327`). Determinism should come from injecting the regenerated timestamp/checksum as inputs to the renderer and pinning them in the golden-file test, not from calling `datetime.now()` inside render code |
|
||||||
|
| Q-6 | Killer correctness queries pass against seeded real-ish data (5 seed cases per Q-006/Q-009/Q-011) | ❌ likely missing | No fixture file named for this seen. The three queries exist but there's no evidence of the single integration test described in Q-6 |
|
||||||
|
|
||||||
|
### Operational (O-1 through O-5)
|
||||||
|
|
||||||
|
| ID | Criterion | Status | Evidence |
|
||||||
|
|----|-----------|--------|----------|
|
||||||
|
| O-1 | Schema migration additive, idempotent, tested against fresh + prod-copy DB | 🟡 presumed | `_apply_migrations` pattern is in use per CLAUDE.md sessions; tables exist. Need one confirmation run against a Dalidou backup copy |
|
||||||
|
| O-2 | Backup includes new tables; full restore drill passes; post-restore Q-001 works | ❌ not done | No evidence a restore drill has been run on V1 entity state. `docs/backup-restore-procedure.md` exists but drill is an explicit V1 prerequisite |
|
||||||
|
| O-3 | Performance bounds: write <100ms p99, query <500ms p99 at 1000 entities, mirror <5s per project | 🟡 unmeasured | 35 entities in system — bounds unmeasured at scale. Spec says "sanity-checked, not benchmarked", so this is a one-off manual check |
|
||||||
|
| O-4 | No new manual ops burden | 🟡 | Mirror regen auto-triggers not wired yet (see F-6 gap) — they must be wired for O-4 to pass |
|
||||||
|
| O-5 | Phase 9 reflection loop unchanged for identity/preference/episodic | ✅ presumed | `memory-vs-entities.md` says these three types don't interact with engineering layer. No recent change to memory extractor for these types |
|
||||||
|
|
||||||
|
### Documentation (D-1 through D-4)
|
||||||
|
|
||||||
|
| ID | Criterion | Status | Evidence |
|
||||||
|
|----|-----------|--------|----------|
|
||||||
|
| D-1 | 12 per-entity-type spec docs under `docs/architecture/entities/` | ❌ missing | No `docs/architecture/entities/` folder |
|
||||||
|
| D-2 | `kb-cad-export-schema.md` + `kb-fem-export-schema.md` | ❌ missing | No such files in `docs/architecture/` |
|
||||||
|
| D-3 | `docs/v1-release-notes.md` | ❌ missing | Not written yet (appropriately — it's written when V1 is done) |
|
||||||
|
| D-4 | `master-plan-status.md` + `current-state.md` updated with V1 completion | ❌ not yet | `master-plan-status.md:179` still has V1 under **Next** |
|
||||||
|
|
||||||
|
### Summary (revised per Codex 2026-04-22 per-file audit)
|
||||||
|
|
||||||
|
- **Functional:** 0/8 ✅, 7/8 🟡 partial (F-1 downgraded from ✅ — two header fields missing; F-2 through F-7 partial), 1/8 ❌ missing (F-3 ingest endpoints) → the entity layer shape is real but not yet spec-clean; write-time invariants come first, then everything builds on stable invariants
|
||||||
|
- **F-2 detail:** 9 of 20 v1-required queries done, 1 partial (Q-001 needs subsystem-scoped variant), 10 missing
|
||||||
|
- **F-5 detail:** generic `conflicts` + `conflict_members` schema already present (no migration needed); detector body + routes diverge from spec
|
||||||
|
- **Quality:** 1/6 ✅, 3/6 🟡 partial, 2/6 ❌ missing → golden file + killer-correctness integration test are the two clear gaps
|
||||||
|
- **Operational:** 0/5 ✅ (none fully verified), 3/5 🟡, 1/5 ❌ → backup drill is the one hard blocker here
|
||||||
|
- **Documentation:** 0/4 ✅, 4/4 ❌ → all 4 docs need writing
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Proposed completion order (revised post-Codex review)
|
||||||
|
|
||||||
|
Seven phases instead of six. The new V1-0 establishes the write-time
|
||||||
|
invariants (provenance enforcement F-8 + synchronous conflict hooks F-5
|
||||||
|
minimal) that every later phase depends on. V1-A becomes a **minimal query
|
||||||
|
slice** that proves the model on one project, not a full catalog closure.
|
||||||
|
Full query catalog closure moves to V1-C. Full F-5 spec compliance (the
|
||||||
|
generic `conflicts`/`conflict_members` slot-key schema) stays in V1-F
|
||||||
|
because that's the final shape, but the *minimal hooks* that fire
|
||||||
|
synchronously on writes land in V1-0.
|
||||||
|
|
||||||
|
Skipped by construction: F-1 core schema (already implemented) and O-5
|
||||||
|
(identity/preference/episodic don't touch the engineering layer).
|
||||||
|
|
||||||
|
### Phase V1-0: Write-time invariants (F-8 + F-5 minimal + F-1 audit)
|
||||||
|
|
||||||
|
**Scope:**
|
||||||
|
- **F-1 remediation (Codex audit 2026-04-22 already completed).** Add
|
||||||
|
the two missing shared-header fields to the `Entity` dataclass
|
||||||
|
(`service.py:67`) and the `entities` table schema:
|
||||||
|
- `extractor_version TEXT` — semver-ish string carrying the extractor
|
||||||
|
module version per `promotion-rules.md:268`. Backfill existing rows
|
||||||
|
with `"0.0.0"` or `NULL` flagged as unknown. Every future
|
||||||
|
write carries the current `EXTRACTOR_VERSION` constant.
|
||||||
|
- `canonical_home TEXT` — which layer is canonical for this concept.
|
||||||
|
For entities, value is always `"entity"`. For future graduation
|
||||||
|
records it may be `"memory"` (frozen pointer). Backfill active
|
||||||
|
rows with `"entity"`.
|
||||||
|
- Additive migration via the existing `_apply_migrations` pattern,
|
||||||
|
idempotent, safe on replay.
|
||||||
|
- Add doc note in `engineering-ontology-v1.md` clarifying that the
|
||||||
|
`project` field IS the `project_id` per spec — "fields equivalent
|
||||||
|
to" wording in the spec allows this, but make it explicit so
|
||||||
|
future readers don't trip on the naming.
|
||||||
|
- **F-8 provenance enforcement.** Add a NOT-NULL invariant at
|
||||||
|
`create_entity` and `promote_entity` that `source_refs` is non-empty
|
||||||
|
OR an explicit `hand_authored=True` flag is set (per
|
||||||
|
`promotion-rules.md:253`). Backfill any existing active entities that
|
||||||
|
fail the invariant — either attach provenance, flag as hand-authored,
|
||||||
|
or invalidate. Every future active entity has provenance by schema,
|
||||||
|
not by discipline.
|
||||||
|
- **F-5 minimal hooks.** Wire synchronous conflict detection into every
|
||||||
|
active-entity write path (`create_entity` with status=active,
|
||||||
|
`promote_entity`, `supersede_entity`). The *detector* can stay in its
|
||||||
|
current per-type form (`_check_component_conflicts`,
|
||||||
|
`_check_requirement_conflicts`); the *hook* must fire on every write.
|
||||||
|
Full generic slot-keyed schema lands in V1-F; the hook shape must be
|
||||||
|
generic enough that V1-F is a detector-body swap, not an API refactor.
|
||||||
|
- **Q-3 "flag never block" test.** The hook must return conflict-id in
|
||||||
|
the response body but never 4xx-block the write. One test per write
|
||||||
|
path demonstrating this.
|
||||||
|
- **Q-4 trust-hierarchy test for candidates.** One test: entity
|
||||||
|
candidates never appear in `/context/build` output. (Full trust tests
|
||||||
|
land in V1-E; this is the one that V1-0 can cover without graduation
|
||||||
|
being ready.)
|
||||||
|
|
||||||
|
**Acceptance:** F-1 ✅ (after `extractor_version` + `canonical_home`
|
||||||
|
land + doc note on `project` naming), F-8 ✅, F-5 hooks ✅, Q-3 ✅,
|
||||||
|
partial Q-4 ✅.
|
||||||
|
|
||||||
|
**Estimated size:** 3 days (two small schema additions + invariant
|
||||||
|
patches + hook wiring + tests; no audit overhead — Codex already did
|
||||||
|
that part).
|
||||||
|
|
||||||
|
**Tests added:** ~10.
|
||||||
|
|
||||||
|
**Why first:** every later phase writes entities. Without F-8 + F-5
|
||||||
|
hooks, V1-A through V1-F can leak invalid state into the store that
|
||||||
|
must then be cleaned up.
|
||||||
|
|
||||||
|
### Phase V1-A: Minimal query slice that proves the model (partial F-2 + Q-6)
|
||||||
|
|
||||||
|
**Scope:**
|
||||||
|
- Pick the **four pillar queries**: Q-001 (subsystem contents),
|
||||||
|
Q-005 (component satisfies requirements), Q-006 (orphan requirements —
|
||||||
|
killer correctness), Q-017 (evidence chain). These exercise structural +
|
||||||
|
intent + killer-correctness + provenance.
|
||||||
|
- **Q-001 needs a shape fix**: Codex's audit confirms the existing
|
||||||
|
`system_map()` returns a project-wide tree, not the spec's
|
||||||
|
subsystem-scoped `GET /entities/Subsystem/<id>?expand=contains`.
|
||||||
|
Add a subsystem-scoped variant (the existing project-wide route stays
|
||||||
|
for Q-004). This is the only shape fix in V1-A; larger query additions
|
||||||
|
move to V1-C.
|
||||||
|
- Q-005, Q-006, Q-017 are already implemented per Codex audit. V1-A
|
||||||
|
verifies them against seeded data; no code changes expected.
|
||||||
|
- Seed p05-interferometer with Q-6 integration data (one satisfying
|
||||||
|
Component + one orphan Requirement + one Decision on flagged
|
||||||
|
Assumption + one supported ValidationClaim + one unsupported
|
||||||
|
ValidationClaim).
|
||||||
|
- All three killer-correctness queries (Q-006, Q-009, Q-011) are
|
||||||
|
**already implemented** per Codex audit. V1-A runs them as a single
|
||||||
|
integration test against the seed data.
|
||||||
|
|
||||||
|
**Acceptance:** Q-001 subsystem-scoped variant + Q-6 integration test.
|
||||||
|
Partial F-2 (remaining 10 missing + 1 partial queries land in V1-C).
|
||||||
|
|
||||||
|
**Estimated size:** 1.5 days (scope shrunk — most pillar queries already
|
||||||
|
work per Codex audit; only Q-001 shape fix + seed data + integration
|
||||||
|
test required).
|
||||||
|
|
||||||
|
**Tests added:** ~4.
|
||||||
|
|
||||||
|
**Why second:** proves the entity layer shape works end-to-end on real
|
||||||
|
data before we start bolting ingest, graduation, or mirror onto it. If
|
||||||
|
the four pillar queries don't work, stopping here is cheap.
|
||||||
|
|
||||||
|
### Phase V1-B: KB-CAD / KB-FEM ingest (F-3) + D-2 schema docs
|
||||||
|
|
||||||
|
**Scope:**
|
||||||
|
- Write `docs/architecture/kb-cad-export-schema.md` and
|
||||||
|
`kb-fem-export-schema.md` (matches D-2).
|
||||||
|
- Implement `POST /ingest/kb-cad/export` and `POST /ingest/kb-fem/export`
|
||||||
|
per `tool-handoff-boundaries.md` sketches. Validator + entity-candidate
|
||||||
|
producer + provenance population **using the F-8 invariant from V1-0**.
|
||||||
|
- Hand-craft one real KB-CAD export for p05-interferometer and
|
||||||
|
round-trip it: export → candidate queue → reviewer promotes → queryable
|
||||||
|
via V1-A's four pillar queries.
|
||||||
|
- Tests: valid export → candidates created; invalid export → 400;
|
||||||
|
duplicate re-export → no duplicate candidates; re-export with changed
|
||||||
|
value → new candidate + conflict row (exercises V1-0's F-5 hook on a
|
||||||
|
real workload).
|
||||||
|
|
||||||
|
**Acceptance:** F-3 ✅, D-2 ✅.
|
||||||
|
|
||||||
|
**Estimated size:** 2 days.
|
||||||
|
|
||||||
|
**Tests added:** ~8.
|
||||||
|
|
||||||
|
**Why third:** ingest is the first real stress test of the V1-0
|
||||||
|
invariants. A re-import that creates a conflict must trigger the V1-0
|
||||||
|
hook; if it doesn't, V1-0 is incomplete and we catch it before going
|
||||||
|
further.
|
||||||
|
|
||||||
|
### Phase V1-C: Close the rest of the query catalog (remaining F-2)
|
||||||
|
|
||||||
|
**Scope:** close the 10 missing queries per Codex's audit. Already-done
|
||||||
|
queries (Q-004/Q-005/Q-006/Q-008/Q-009/Q-011/Q-013/Q-016/Q-017) are
|
||||||
|
verified but not rewritten.
|
||||||
|
- Q-002 (component → parents, inverse of CONTAINS)
|
||||||
|
- Q-003 (subsystem interfaces, Interface as simple string label)
|
||||||
|
- Q-007 (component → constraints via CONSTRAINED_BY)
|
||||||
|
- Q-010 (ValidationClaim → supporting results + AnalysisModel trace)
|
||||||
|
- Q-012 (conflicting results on same claim — exercises V1-0's F-5 hook)
|
||||||
|
- Q-014 (decision log ordered + superseded chain)
|
||||||
|
- Q-018 (`include=superseded` for supersession chains)
|
||||||
|
- Q-019 (Material → components, derived from Component.material field
|
||||||
|
per `engineering-query-catalog.md:266`, no edge needed)
|
||||||
|
- Q-020 (project overview mirror route) — deferred to V1-D where the
|
||||||
|
mirror lands in full.
|
||||||
|
|
||||||
|
**Acceptance:** F-2 ✅ (all 19 of 20 v1-required queries; Q-020 in V1-D).
|
||||||
|
|
||||||
|
**Estimated size:** 2 days (eight new query functions + routes +
|
||||||
|
per-query happy-path tests).
|
||||||
|
|
||||||
|
**Tests added:** ~12.
|
||||||
|
|
||||||
|
**Why fourth:** with the model proven (V1-A) and ingest exercising the
|
||||||
|
write invariants (V1-B), filling in the remaining queries is mechanical.
|
||||||
|
They all sit on top of the same entity store and V1-0 invariants.
|
||||||
|
|
||||||
|
### Phase V1-D: Full Mirror surface (F-6) + determinism golden file (Q-5) + Q-020
|
||||||
|
|
||||||
|
**Scope:**
|
||||||
|
- Split the single `/projects/{project_name}/mirror` route into the three
|
||||||
|
spec routes: `/mirror/{project}/overview` (= Q-020),
|
||||||
|
`/mirror/{project}/decisions`, `/mirror/{project}/subsystems/{subsystem}`.
|
||||||
|
- Add `POST /mirror/{project}/regenerate`.
|
||||||
|
- Move generated files to `/srv/storage/atocore/data/mirror/{project}/`.
|
||||||
|
- **Deterministic output:** inject regenerated timestamp + checksum as
|
||||||
|
renderer inputs (pinned by golden tests), sort every iteration, and
|
||||||
|
remove `dict` / database ordering dependencies. The renderer should
|
||||||
|
not call wall-clock time directly.
|
||||||
|
- `⚠ disputed` markers inline wherever an open conflict touches a
|
||||||
|
rendered field (uses V1-0's F-5 hook output).
|
||||||
|
- `(curated)` annotations where project_state overrides entity state.
|
||||||
|
- Regeneration triggers: synchronous on regenerate, debounced async on
|
||||||
|
entity write (30s window), daily scheduled refresh via existing
|
||||||
|
nightly cron (one new cron line, not a new queue).
|
||||||
|
- `mirror_regeneration_failures` table.
|
||||||
|
- Golden-file test: fixture project state → render → bytes equal.
|
||||||
|
|
||||||
|
**Acceptance:** F-6 ✅, Q-5 ✅, Q-020 ✅, O-4 moves toward ✅.
|
||||||
|
|
||||||
|
**Estimated size:** 3–4 days.
|
||||||
|
|
||||||
|
**Tests added:** ~15.
|
||||||
|
|
||||||
|
**Why fifth:** mirror is a derived consumer. It cannot be correct
|
||||||
|
before the entity store + queries + conflict hooks are correct. It
|
||||||
|
lands after everything it depends on is stable.
|
||||||
|
|
||||||
|
### Phase V1-E: Memory→entity graduation end-to-end (F-7) + remaining Q-4
|
||||||
|
|
||||||
|
**Scope:**
|
||||||
|
- Verify and close F-7 spec gaps:
|
||||||
|
- Add the missing direct `POST /memory/{id}/graduate` route, reusing the
|
||||||
|
same prompt/parser as the batch graduation path.
|
||||||
|
- Keep `/admin/graduation/request` as the bulk lane; direct route is the
|
||||||
|
per-memory acceptance surface.
|
||||||
|
- Preserve current behavior where promote marks source memories
|
||||||
|
`status="graduated"` and sets `graduated_to_entity_id`.
|
||||||
|
- Flow tested for `adaptation` → Decision and `project` → Requirement.
|
||||||
|
- Reconcile the spec's `knowledge` → Fact wording with the actual V1
|
||||||
|
ontology (no `fact` entity type exists today). Prefer doc alignment to
|
||||||
|
an existing typed entity such as `parameter`, rather than adding a vague
|
||||||
|
catch-all `Fact` type late in V1.
|
||||||
|
- Schema is mostly already in place: `graduated` status exists in memory
|
||||||
|
service, `graduated_to_entity_id` column + index exist, and promote
|
||||||
|
preserves the original memory. Remaining work is route surface,
|
||||||
|
ontology/spec reconciliation, and targeted end-to-end tests.
|
||||||
|
- **Q-4 full trust-hierarchy tests**: no auto-write to project_state
|
||||||
|
from any promote path; active-only reinforcement for entities; etc.
|
||||||
|
(The entity-candidates-excluded-from-context test shipped in V1-0.)
|
||||||
|
|
||||||
|
**Acceptance:** F-7 ✅, Q-4 ✅.
|
||||||
|
|
||||||
|
**Estimated size:** 3–4 days.
|
||||||
|
|
||||||
|
**Tests added:** ~8.
|
||||||
|
|
||||||
|
**Why sixth:** graduation touches memory-layer semantics (adds a
|
||||||
|
`graduated` status, flows memory→entity, requires memory-module changes).
|
||||||
|
Doing it after the entity layer is fully invariant-safe + query-complete
|
||||||
|
+ mirror-derived means the memory side only has to deal with one shape:
|
||||||
|
a stable, tested entity layer.
|
||||||
|
|
||||||
|
### Phase V1-F: Full F-5 spec compliance + O-1/O-2/O-3 + D-1/D-3/D-4
|
||||||
|
|
||||||
|
**Scope:**
|
||||||
|
- **F-5 full spec compliance** (Codex 2026-04-22 audit already confirmed
|
||||||
|
the gap shape — schema is spec-compliant, divergence is in detector +
|
||||||
|
routes only).
|
||||||
|
- **Detector generalization.** Replace the per-type dispatch at
|
||||||
|
`conflicts.py:36` (`_check_component_conflicts`,
|
||||||
|
`_check_requirement_conflicts`) with a slot-key-driven generic
|
||||||
|
detector that reads the per-entity-type conflict slot from a
|
||||||
|
registry and queries the already-generic `conflicts` +
|
||||||
|
`conflict_members` tables. The V1-0 hook shape was chosen to make
|
||||||
|
this a detector-body swap, not an API change.
|
||||||
|
- **Route alignment.** Add `/conflicts/*` routes as the canonical
|
||||||
|
surface per `conflict-model.md:187`. Keep `/admin/conflicts/*` as
|
||||||
|
aliases for one release, deprecate in D-3 release notes, remove
|
||||||
|
in V1.1.
|
||||||
|
- **No schema migration needed** (the tables at `database.py:190`
|
||||||
|
already match the spec).
|
||||||
|
- **O-1:** Run the full migration against a Dalidou backup copy.
|
||||||
|
Confirm additive, idempotent, safe to run twice.
|
||||||
|
- **O-2:** Run a full restore drill on the test project per
|
||||||
|
`docs/backup-restore-procedure.md`. Post-restore, Q-001 returns
|
||||||
|
correct shape. `POST /admin/backup` snapshot includes the new tables.
|
||||||
|
- **O-3:** Manual sanity-check of the three performance bounds.
|
||||||
|
- **D-1:** Write 12 short spec docs under `docs/architecture/entities/`
|
||||||
|
(one per V1 entity type).
|
||||||
|
- **D-3:** Write `docs/v1-release-notes.md`.
|
||||||
|
- **D-4:** Update `master-plan-status.md` and `current-state.md` —
|
||||||
|
move engineering V1 from **Next** to **What Is Real Today**.
|
||||||
|
|
||||||
|
**Acceptance:** F-5 ✅, O-1 ✅, O-2 ✅, O-3 ✅, D-1 ✅, D-3 ✅, D-4 ✅ →
|
||||||
|
**V1 is done.**
|
||||||
|
|
||||||
|
**Estimated size:** 3 days (F-5 migration if needed is the main unknown;
|
||||||
|
D-1 entity docs at ~30 min each ≈ 6 hours; verification is fast).
|
||||||
|
|
||||||
|
**Tests added:** ~6 (F-5 spec-shape tests; verification adds no automated
|
||||||
|
tests).
|
||||||
|
|
||||||
|
### Total (revised after Codex 2026-04-22 audit)
|
||||||
|
|
||||||
|
- Phase budgets: V1-0 (3) + V1-A (1.5) + V1-B (2) + V1-C (2) + V1-D (3-4)
|
||||||
|
+ V1-E (3-4) + V1-F (3) ≈ **17.5–19.5 days of focused work**. This is a
|
||||||
|
realistic engineering-effort estimate, but a single-operator calendar
|
||||||
|
plan should still carry context-switch / soak / review buffer on top.
|
||||||
|
- Adds roughly **60 tests** (533 → ~593).
|
||||||
|
- Branch strategy: one branch per phase (V1-0 → V1-F), each squash-merged
|
||||||
|
to main after Codex review. Phases sequential because each builds on
|
||||||
|
the previous. **V1-0 is a hard prerequisite for all later phases** —
|
||||||
|
nothing starts until V1-0 lands.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Sequencing with the `master-plan-status.md` Now list
|
||||||
|
|
||||||
|
The **Now** list from master-plan-status.md:159-169 is:
|
||||||
|
|
||||||
|
1. Observe the enhanced pipeline (1 week soak — first F4 confidence decay
|
||||||
|
run was 2026-04-19 per Trusted State, so soak window ends ~2026-04-26)
|
||||||
|
2. Knowledge density — batch extract over 234 interactions, target 100+
|
||||||
|
active memories (currently 84)
|
||||||
|
3. Multi-model triage (Phase 11 entry)
|
||||||
|
4. Fix p04-constraints harness failure
|
||||||
|
|
||||||
|
**Principle (revised per Codex review):** V1 work and the Now list are
|
||||||
|
**less disjoint than the first draft claimed**. Real collision points:
|
||||||
|
|
||||||
|
| V1 phase | Collides with Now list at |
|
||||||
|
|---|---|
|
||||||
|
| V1-0 provenance enforcement | memory extractor write path if it shares helper functions; context assembly for the Q-4 partial trust test |
|
||||||
|
| V1-0 F-5 hooks | any write path that creates active rows (limited collision; entity writes are separate from memory writes) |
|
||||||
|
| V1-B KB-CAD/FEM ingest | none on the Now list, but adds an ingest surface that becomes operational burden (ties to O-4 "no new manual ops") |
|
||||||
|
| V1-D mirror regen triggers | scheduling / ops behavior that intersects with "boring and dependable" gate — mirror regen failures become an observable that the pipeline soak must accommodate |
|
||||||
|
| V1-E graduation | memory module (new `graduated` status, memory→entity flow); direct collision with memory extractor + triage |
|
||||||
|
| V1-F F-5 migration | conflicts.py touches the write path shared with memory promotion |
|
||||||
|
|
||||||
|
**Recommended schedule (revised):**
|
||||||
|
|
||||||
|
- **This week (2026-04-22 to 2026-04-26):** Pipeline soak continues.
|
||||||
|
Density batch-extract continues. V1 work **waits** — V1-0 would start
|
||||||
|
touching write paths, which is explicitly something we should not do
|
||||||
|
during a soak window. Density target (100+ active memories) and the
|
||||||
|
pipeline soak complete first.
|
||||||
|
- **Week of 2026-04-27:** If soak is clean and density reached, V1-0
|
||||||
|
starts. V1-0 is a hard prerequisite and cannot be skipped or parallelized.
|
||||||
|
- **Weeks of 2026-05-04 and 2026-05-11:** V1-A through V1-D in order.
|
||||||
|
Multi-model triage work (Now list item 3) continues in parallel only
|
||||||
|
if its touch-surface is triage-path-only (memory side). Any memory
|
||||||
|
extractor change pauses V1-E.
|
||||||
|
- **Week of 2026-05-18 approx:** V1-E (graduation). **This phase must
|
||||||
|
not run in parallel with memory extractor changes** — it directly
|
||||||
|
modifies memory module semantics. Multi-model triage should be settled
|
||||||
|
before V1-E starts.
|
||||||
|
- **Week of 2026-05-25:** V1-F.
|
||||||
|
- **End date target:** ~2026-06-01, four weeks later than the first
|
||||||
|
draft's 2026-05-18 soft target. The shift is deliberate — the first
|
||||||
|
draft's "parallel / disjoint" claim understated the real collisions.
|
||||||
|
|
||||||
|
**Pause points (explicit):**
|
||||||
|
|
||||||
|
- Any Now-list item that regresses the pipeline → V1 pauses immediately.
|
||||||
|
- Memory extractor changes in flight → V1-E pauses until they land and
|
||||||
|
soak.
|
||||||
|
- p04-constraints fix requires retrieval ranking changes → V1 does not
|
||||||
|
pause (retrieval is genuinely disjoint from entities).
|
||||||
|
- Multi-model triage work touching the entity extractor path (if one
|
||||||
|
gets prototyped) → V1-0 pauses until the triage decision settles.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Test project
|
||||||
|
|
||||||
|
Per `engineering-v1-acceptance.md:379`, the recommended test bed is
|
||||||
|
**p05-interferometer** — "the optical/structural domain has the cleanest
|
||||||
|
entity model". I agree. Every F-2, F-3, F-6 criterion asserts against this
|
||||||
|
project.
|
||||||
|
|
||||||
|
p06-polisher is the backup test bed if p05 turns out to have data gaps
|
||||||
|
(polisher suite is actively worked and has more content).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## What V1 completion does NOT include
|
||||||
|
|
||||||
|
Per the negative list in `engineering-v1-acceptance.md:351-373`, all of the
|
||||||
|
following are **explicitly out of scope** for this plan:
|
||||||
|
|
||||||
|
- LLM extractor for entities (rule-based is V1)
|
||||||
|
- Auto-promotion of candidates (human-only in V1)
|
||||||
|
- Write-back to KB-CAD / KB-FEM
|
||||||
|
- Multi-user auth
|
||||||
|
- Real-time UI (API + Mirror markdown only)
|
||||||
|
- Cross-project rollups
|
||||||
|
- Time-travel queries (Q-015 stays stretch)
|
||||||
|
- Nightly conflict sweep (synchronous only)
|
||||||
|
- Incremental Chroma snapshots
|
||||||
|
- Retention cleanup script
|
||||||
|
- Backup encryption
|
||||||
|
- Off-Dalidou backup target (already exists at clawdbot per ledger, but
|
||||||
|
not a V1 criterion)
|
||||||
|
- **Async job queue / minions pattern** (the rejected plan's centerpiece —
|
||||||
|
explicitly deferred to post-V1 per the negative list)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Open questions for Codex (post-second-round revision)
|
||||||
|
|
||||||
|
Three of the original eight questions (F-1 field audit, F-2 per-query
|
||||||
|
audit, F-5 schema divergence) were answered by Codex's 2026-04-22 audit
|
||||||
|
and folded into the plan. One open question remains; the rest are now
|
||||||
|
resolved in-plan:
|
||||||
|
|
||||||
|
1. **Parallel schedule vs Now list.** The first-round review correctly
|
||||||
|
softened this from "fully parallel" to "less disjoint than claimed".
|
||||||
|
Is the revised collision table + pause-points section enough, or
|
||||||
|
should specific Now-list items gate specific V1 phases more strictly?
|
||||||
|
|
||||||
|
2. **F-7 graduation gap depth.** Resolved by Codex audit. The schema and
|
||||||
|
preserve-original-memory hook are already in place, so V1-E is not a
|
||||||
|
greenfield build. But the direct `/memory/{id}/graduate` route and the
|
||||||
|
ontology/spec mismatch around `knowledge` → `Fact` are still open, so
|
||||||
|
V1-E is closer to **3–4 days** than 2.
|
||||||
|
|
||||||
|
3. **Mirror determinism — where does `now` go?** Resolved. Keep the
|
||||||
|
regenerated timestamp in the rendered output if desired, but pass it
|
||||||
|
into the renderer as an input value. Golden-file tests pin that input;
|
||||||
|
render code must not read the clock directly.
|
||||||
|
|
||||||
|
4. **`project` field naming.** Resolved. Keep the existing `project`
|
||||||
|
field; add the explicit doc note that it is the project identifier for
|
||||||
|
V1 acceptance purposes. No storage rename needed.
|
||||||
|
|
||||||
|
5. **Velocity calibration.** Resolved. **17.5–19.5 focused days** is a
|
||||||
|
fair engineering-effort estimate after the F-7 audit. For an actual
|
||||||
|
operator schedule, keep additional buffer for context switching, soak,
|
||||||
|
and review rounds.
|
||||||
|
|
||||||
|
6. **Minions/queue as V2 item in D-3.** Resolved. Do not name the
|
||||||
|
rejected "Minions" plan in V1 release notes. If D-3 includes a future
|
||||||
|
work section, refer to it neutrally as "queued background processing /
|
||||||
|
async workers" rather than canonizing a V2 codename before V2 is
|
||||||
|
designed.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Risks
|
||||||
|
|
||||||
|
| Risk | Mitigation |
|
||||||
|
|---|---|
|
||||||
|
| V1 work slows the Now list | V1 pauses on any Now-list blocker. Codex veto on any V1 PR that touches memory extractor, retrieval ranking, or triage paths |
|
||||||
|
| F-5 detector generalization harder than estimated | Codex audit confirmed schema is already spec-compliant; only detector body + routes need work. If detector generalization still slips, keep per-type detectors and document as a V1.1 cleanup (detection correctness is unaffected, only code organization) |
|
||||||
|
| Mirror determinism regresses existing mirror output | Keep `/projects/{project_name}/mirror` alias returning the current shape; new `/mirror/{project}/overview` is the spec-compliant one. Deprecate old in V1 release notes |
|
||||||
|
| Golden file churn as templates evolve | Standard workflow: updating a golden file is a normal part of template work, documented in V1-C commit message |
|
||||||
|
| Backup drill on Dalidou is disruptive | Run against a clone of the Dalidou DB at a safe hour; no production drill required for V1 acceptance |
|
||||||
|
| p05-interferometer data gaps | Fall back to p06-polisher per this plan's test-project section |
|
||||||
|
| Scope creep during V1-A query audit | Any query that isn't in the v1-required set (Q-021 onward) is out of scope, period |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## What this plan is **for**
|
||||||
|
|
||||||
|
1. A checklist Claude can follow to close V1.
|
||||||
|
2. A review target for Codex — every phase has explicit acceptance
|
||||||
|
criteria tied to the acceptance doc.
|
||||||
|
3. A communication artifact for Antoine — "here's what's left, here's why,
|
||||||
|
here's the order, here's the risk."
|
||||||
|
|
||||||
|
## What this plan is **not**
|
||||||
|
|
||||||
|
1. Not a commitment to start tomorrow. Pipeline soak + density density
|
||||||
|
come first in parallel; V1-A can start this week only because it's
|
||||||
|
zero-risk additive work.
|
||||||
|
2. Not a rewrite. Every phase builds on existing code.
|
||||||
|
3. Not an ontology debate. The ontology is fixed in
|
||||||
|
`engineering-ontology-v1.md`. Any desire to change it is V2 material.
|
||||||
|
|
||||||
|
## Workspace note (for Codex audit)
|
||||||
|
|
||||||
|
Codex's first-round review (2026-04-22) flagged that
|
||||||
|
`docs/plans/engineering-v1-completion-plan.md` and `DEV-LEDGER.md` were
|
||||||
|
**not visible** in the Playground workspace they were running against,
|
||||||
|
and that `src/atocore/engineering/` appeared empty there.
|
||||||
|
|
||||||
|
The canonical dev workspace for AtoCore is the Windows path
|
||||||
|
`C:\Users\antoi\ATOCore` (per `CLAUDE.md`). The engineering layer code
|
||||||
|
(`src/atocore/engineering/service.py`, `queries.py`, `conflicts.py`,
|
||||||
|
`mirror.py`, `_graduation_prompt.py`, `wiki.py`, `triage_ui.py`) exists
|
||||||
|
there and is what the recent commits (e147ab2, b94f9df, 081c058, 069d155,
|
||||||
|
b1a3dd0) touched. The Windows working tree is what this plan was written
|
||||||
|
against.
|
||||||
|
|
||||||
|
Before the file-level audit:
|
||||||
|
|
||||||
|
1. Confirm which branch / SHA Codex is reviewing. The Windows working
|
||||||
|
tree has uncommitted changes to this plan + DEV-LEDGER as of
|
||||||
|
2026-04-22; commit will be made only after Antoine approves sync.
|
||||||
|
2. If Codex is reviewing `ATOCore-clean` or a Playground snapshot, that
|
||||||
|
tree may lag the canonical dev tree. Sync or re-clone from the
|
||||||
|
Windows working tree / current `origin/main` before per-file audit.
|
||||||
|
3. The three visible-to-Codex file paths for this plan are:
|
||||||
|
- `docs/plans/engineering-v1-completion-plan.md` (this file)
|
||||||
|
- `docs/decisions/2026-04-22-gbrain-plan-rejection.md` (prior decision)
|
||||||
|
- `DEV-LEDGER.md` (Recent Decisions + Session Log entries 2026-04-22)
|
||||||
|
|
||||||
|
## References
|
||||||
|
|
||||||
|
- `docs/architecture/engineering-ontology-v1.md`
|
||||||
|
- `docs/architecture/engineering-query-catalog.md`
|
||||||
|
- `docs/architecture/memory-vs-entities.md`
|
||||||
|
- `docs/architecture/engineering-v1-acceptance.md`
|
||||||
|
- `docs/architecture/promotion-rules.md`
|
||||||
|
- `docs/architecture/conflict-model.md`
|
||||||
|
- `docs/architecture/human-mirror-rules.md`
|
||||||
|
- `docs/architecture/tool-handoff-boundaries.md`
|
||||||
|
- `docs/master-plan-status.md` (Now/Next/Later list)
|
||||||
|
- `docs/decisions/2026-04-22-gbrain-plan-rejection.md` (the rejected plan)
|
||||||
|
- `src/atocore/engineering/service.py` (current V1 entity service)
|
||||||
|
- `src/atocore/engineering/queries.py` (current V1 query implementations)
|
||||||
|
- `src/atocore/engineering/conflicts.py` (current conflicts module)
|
||||||
|
- `src/atocore/engineering/mirror.py` (current mirror module)
|
||||||
@@ -1,29 +1,40 @@
|
|||||||
# AtoCore Capture Plugin for OpenClaw
|
# AtoCore Capture + Context Plugin for OpenClaw
|
||||||
|
|
||||||
Minimal OpenClaw plugin that mirrors Claude Code's `capture_stop.py` behavior:
|
Two-way bridge between OpenClaw agents and AtoCore:
|
||||||
|
|
||||||
|
**Capture (since v1)**
|
||||||
- watches user-triggered assistant turns
|
- watches user-triggered assistant turns
|
||||||
- POSTs `prompt` + `response` to `POST /interactions`
|
- POSTs `prompt` + `response` to `POST /interactions`
|
||||||
- sets `client="openclaw"`
|
- sets `client="openclaw"`, `reinforce=true`
|
||||||
- sets `reinforce=true`
|
|
||||||
- fails open on network or API errors
|
- fails open on network or API errors
|
||||||
|
|
||||||
## Config
|
**Context injection (Phase 7I, v2+)**
|
||||||
|
- on `before_agent_start`, fetches a context pack from `POST /context/build`
|
||||||
|
- prepends the pack to the agent's prompt so whatever LLM runs underneath
|
||||||
|
(sonnet, opus, codex, local model — whichever OpenClaw delegates to)
|
||||||
|
answers grounded in what AtoCore already knows
|
||||||
|
- original user prompt is still what gets captured later (no recursion)
|
||||||
|
- fails open: context unreachable → agent runs as before
|
||||||
|
|
||||||
Optional plugin config:
|
## Config
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"baseUrl": "http://dalidou:8100",
|
"baseUrl": "http://dalidou:8100",
|
||||||
"minPromptLength": 15,
|
"minPromptLength": 15,
|
||||||
"maxResponseLength": 50000
|
"maxResponseLength": 50000,
|
||||||
|
"injectContext": true,
|
||||||
|
"contextCharBudget": 4000
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
If `baseUrl` is omitted, the plugin uses `ATOCORE_BASE_URL` or defaults to `http://dalidou:8100`.
|
- `baseUrl` — defaults to `ATOCORE_BASE_URL` env or `http://dalidou:8100`
|
||||||
|
- `injectContext` — set to `false` to disable the Phase 7I context injection and make this a pure one-way capture plugin again
|
||||||
|
- `contextCharBudget` — cap on injected context size. `/context/build` respects it too; this is a client-side safety net. Default 4000 chars (~1000 tokens).
|
||||||
|
|
||||||
## Notes
|
## Notes
|
||||||
|
|
||||||
- Project detection is intentionally left empty for now. Unscoped capture is acceptable because AtoCore's extraction pipeline handles unscoped interactions.
|
- Project detection is intentionally left empty — AtoCore's extraction pipeline handles unscoped interactions and infers the project from content.
|
||||||
- Extraction is **not** part of the capture path. This plugin only records interactions and lets AtoCore reinforcement run automatically.
|
- Extraction is **not** part of this plugin. Interactions are captured; batch extraction runs via cron on the AtoCore host.
|
||||||
- The plugin captures only user-triggered turns, not heartbeats or system-only runs.
|
- Context injection only fires for user-triggered turns (not heartbeats or system-only runs).
|
||||||
|
- Timeouts: context fetch is 5s (short so a slow AtoCore never blocks a user turn); capture post is 10s.
|
||||||
|
|||||||
@@ -98,6 +98,14 @@ export default definePluginEntry({
|
|||||||
lastPrompt = null;
|
lastPrompt = null;
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
// Filter cron-initiated agent runs. OpenClaw's scheduled tasks fire
|
||||||
|
// agent sessions with prompts that begin "[cron:<id> ...]". These are
|
||||||
|
// automated polls (DXF email watcher, calendar reminders, etc.), not
|
||||||
|
// real user turns — they're pure noise in the AtoCore capture stream.
|
||||||
|
if (prompt.startsWith("[cron:")) {
|
||||||
|
lastPrompt = null;
|
||||||
|
return;
|
||||||
|
}
|
||||||
lastPrompt = { text: prompt, sessionKey: ctx?.sessionKey || "", ts: Date.now() };
|
lastPrompt = { text: prompt, sessionKey: ctx?.sessionKey || "", ts: Date.now() };
|
||||||
log.info("atocore-capture:prompt_buffered", { len: prompt.length });
|
log.info("atocore-capture:prompt_buffered", { len: prompt.length });
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -3,6 +3,11 @@ import { definePluginEntry } from "openclaw/plugin-sdk/core";
|
|||||||
const DEFAULT_BASE_URL = process.env.ATOCORE_BASE_URL || "http://dalidou:8100";
|
const DEFAULT_BASE_URL = process.env.ATOCORE_BASE_URL || "http://dalidou:8100";
|
||||||
const DEFAULT_MIN_PROMPT_LENGTH = 15;
|
const DEFAULT_MIN_PROMPT_LENGTH = 15;
|
||||||
const DEFAULT_MAX_RESPONSE_LENGTH = 50_000;
|
const DEFAULT_MAX_RESPONSE_LENGTH = 50_000;
|
||||||
|
// Phase 7I — context injection: cap how much AtoCore context we stuff
|
||||||
|
// back into the prompt. The /context/build endpoint respects a budget
|
||||||
|
// parameter too, but we keep a client-side safety net.
|
||||||
|
const DEFAULT_CONTEXT_CHAR_BUDGET = 4_000;
|
||||||
|
const DEFAULT_INJECT_CONTEXT = true;
|
||||||
|
|
||||||
function trimText(value) {
|
function trimText(value) {
|
||||||
return typeof value === "string" ? value.trim() : "";
|
return typeof value === "string" ? value.trim() : "";
|
||||||
@@ -41,6 +46,37 @@ async function postInteraction(baseUrl, payload, logger) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Phase 7I — fetch a context pack for the incoming prompt so the agent
|
||||||
|
// answers grounded in what AtoCore already knows. Fail-open: if the
|
||||||
|
// request times out or errors, we just don't inject; the agent runs as
|
||||||
|
// before. Never block the user's turn on AtoCore availability.
|
||||||
|
async function fetchContextPack(baseUrl, prompt, project, charBudget, logger) {
|
||||||
|
try {
|
||||||
|
const res = await fetch(`${baseUrl.replace(/\/$/, "")}/context/build`, {
|
||||||
|
method: "POST",
|
||||||
|
headers: { "Content-Type": "application/json" },
|
||||||
|
body: JSON.stringify({
|
||||||
|
prompt,
|
||||||
|
project: project || "",
|
||||||
|
char_budget: charBudget
|
||||||
|
}),
|
||||||
|
signal: AbortSignal.timeout(5_000)
|
||||||
|
});
|
||||||
|
if (!res.ok) {
|
||||||
|
logger?.debug?.("atocore_context_fetch_failed", { status: res.status });
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
const data = await res.json();
|
||||||
|
const pack = trimText(data?.formatted_context || "");
|
||||||
|
return pack || null;
|
||||||
|
} catch (error) {
|
||||||
|
logger?.debug?.("atocore_context_fetch_error", {
|
||||||
|
error: error instanceof Error ? error.message : String(error)
|
||||||
|
});
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
export default definePluginEntry({
|
export default definePluginEntry({
|
||||||
register(api) {
|
register(api) {
|
||||||
const logger = api.logger;
|
const logger = api.logger;
|
||||||
@@ -55,6 +91,28 @@ export default definePluginEntry({
|
|||||||
pendingBySession.delete(ctx.sessionId);
|
pendingBySession.delete(ctx.sessionId);
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Phase 7I — inject AtoCore context into the agent's prompt so it
|
||||||
|
// answers grounded in what the brain already knows. Config-gated
|
||||||
|
// (injectContext: false disables). Fail-open.
|
||||||
|
const baseUrl = trimText(config.baseUrl) || DEFAULT_BASE_URL;
|
||||||
|
const injectContext = config.injectContext !== false && DEFAULT_INJECT_CONTEXT;
|
||||||
|
const charBudget = Number(config.contextCharBudget || DEFAULT_CONTEXT_CHAR_BUDGET);
|
||||||
|
if (injectContext && event && typeof event === "object") {
|
||||||
|
const pack = await fetchContextPack(baseUrl, prompt, "", charBudget, logger);
|
||||||
|
if (pack) {
|
||||||
|
// Prepend to the event's prompt so the agent sees grounded info
|
||||||
|
// before the user's question. OpenClaw's agent receives
|
||||||
|
// event.prompt as its primary input; modifying it here grounds
|
||||||
|
// whatever LLM the agent delegates to (sonnet, opus, codex,
|
||||||
|
// local model — doesn't matter).
|
||||||
|
event.prompt = `${pack}\n\n---\n\n${prompt}`;
|
||||||
|
logger?.debug?.("atocore_context_injected", { chars: pack.length });
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Record the ORIGINAL user prompt (not the injected version) so
|
||||||
|
// captured interactions stay clean for later extraction.
|
||||||
pendingBySession.set(ctx.sessionId, {
|
pendingBySession.set(ctx.sessionId, {
|
||||||
prompt,
|
prompt,
|
||||||
sessionId: ctx.sessionId,
|
sessionId: ctx.sessionId,
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
{
|
{
|
||||||
"name": "@atomaste/atocore-openclaw-capture",
|
"name": "@atomaste/atocore-openclaw-capture",
|
||||||
"private": true,
|
"private": true,
|
||||||
"version": "0.0.0",
|
"version": "0.2.0",
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"description": "OpenClaw plugin that captures assistant turns to AtoCore interactions"
|
"description": "OpenClaw plugin: captures assistant turns to AtoCore interactions AND injects AtoCore context into agent prompts before they run (Phase 7I two-way bridge)"
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -17,6 +17,8 @@ dependencies = [
|
|||||||
"pydantic-settings>=2.1.0",
|
"pydantic-settings>=2.1.0",
|
||||||
"structlog>=24.1.0",
|
"structlog>=24.1.0",
|
||||||
"markdown>=3.5.0",
|
"markdown>=3.5.0",
|
||||||
|
"python-multipart>=0.0.9",
|
||||||
|
"Pillow>=10.0.0",
|
||||||
]
|
]
|
||||||
|
|
||||||
[project.optional-dependencies]
|
[project.optional-dependencies]
|
||||||
|
|||||||
@@ -7,3 +7,5 @@ pydantic>=2.6.0
|
|||||||
pydantic-settings>=2.1.0
|
pydantic-settings>=2.1.0
|
||||||
structlog>=24.1.0
|
structlog>=24.1.0
|
||||||
markdown>=3.5.0
|
markdown>=3.5.0
|
||||||
|
python-multipart>=0.0.9
|
||||||
|
Pillow>=10.0.0
|
||||||
|
|||||||
@@ -37,6 +37,17 @@ import urllib.error
|
|||||||
import urllib.parse
|
import urllib.parse
|
||||||
import urllib.request
|
import urllib.request
|
||||||
|
|
||||||
|
# Force UTF-8 on stdio — MCP protocol expects UTF-8 but Windows Python
|
||||||
|
# defaults stdout to cp1252, which crashes on any non-ASCII char (emojis,
|
||||||
|
# ≥, →, etc.) in tool responses. This call is a no-op on Linux/macOS
|
||||||
|
# where UTF-8 is already the default.
|
||||||
|
try:
|
||||||
|
sys.stdin.reconfigure(encoding="utf-8")
|
||||||
|
sys.stdout.reconfigure(encoding="utf-8")
|
||||||
|
sys.stderr.reconfigure(encoding="utf-8")
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
# --- Configuration ---
|
# --- Configuration ---
|
||||||
|
|
||||||
ATOCORE_URL = os.environ.get("ATOCORE_URL", "http://dalidou:8100").rstrip("/")
|
ATOCORE_URL = os.environ.get("ATOCORE_URL", "http://dalidou:8100").rstrip("/")
|
||||||
@@ -232,6 +243,72 @@ def _tool_projects(args: dict) -> str:
|
|||||||
return "\n".join(lines)
|
return "\n".join(lines)
|
||||||
|
|
||||||
|
|
||||||
|
def _tool_remember(args: dict) -> str:
|
||||||
|
"""Phase 6 Part B — universal capture from any Claude session.
|
||||||
|
|
||||||
|
Wraps POST /memory to create a candidate memory tagged with
|
||||||
|
source='mcp-remember'. The existing 3-tier triage is the quality
|
||||||
|
gate: nothing becomes active until sonnet (+ opus if borderline)
|
||||||
|
approves it. Returns the memory id so the caller can reference it
|
||||||
|
in the same session.
|
||||||
|
"""
|
||||||
|
content = (args.get("content") or "").strip()
|
||||||
|
if not content:
|
||||||
|
return "Error: 'content' is required."
|
||||||
|
|
||||||
|
memory_type = (args.get("memory_type") or "knowledge").strip()
|
||||||
|
valid_types = ["identity", "preference", "project", "episodic", "knowledge", "adaptation"]
|
||||||
|
if memory_type not in valid_types:
|
||||||
|
return f"Error: memory_type must be one of {valid_types}."
|
||||||
|
|
||||||
|
project = (args.get("project") or "").strip()
|
||||||
|
try:
|
||||||
|
confidence = float(args.get("confidence") or 0.6)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
confidence = 0.6
|
||||||
|
confidence = max(0.0, min(1.0, confidence))
|
||||||
|
|
||||||
|
valid_until = (args.get("valid_until") or "").strip()
|
||||||
|
tags = args.get("domain_tags") or []
|
||||||
|
if not isinstance(tags, list):
|
||||||
|
tags = []
|
||||||
|
# Normalize tags: lowercase, dedupe, cap at 10
|
||||||
|
clean_tags: list[str] = []
|
||||||
|
for t in tags[:10]:
|
||||||
|
if not isinstance(t, str):
|
||||||
|
continue
|
||||||
|
t = t.strip().lower()
|
||||||
|
if t and t not in clean_tags:
|
||||||
|
clean_tags.append(t)
|
||||||
|
|
||||||
|
payload = {
|
||||||
|
"memory_type": memory_type,
|
||||||
|
"content": content,
|
||||||
|
"project": project,
|
||||||
|
"confidence": confidence,
|
||||||
|
"status": "candidate",
|
||||||
|
}
|
||||||
|
if valid_until:
|
||||||
|
payload["valid_until"] = valid_until
|
||||||
|
if clean_tags:
|
||||||
|
payload["domain_tags"] = clean_tags
|
||||||
|
|
||||||
|
result, err = safe_call(http_post, "/memory", payload)
|
||||||
|
if err:
|
||||||
|
return f"AtoCore remember failed: {err}"
|
||||||
|
|
||||||
|
mid = result.get("id", "?")
|
||||||
|
scope = project if project else "(global)"
|
||||||
|
tag_str = f" tags=[{', '.join(clean_tags)}]" if clean_tags else ""
|
||||||
|
expires = f" valid_until={valid_until}" if valid_until else ""
|
||||||
|
return (
|
||||||
|
f"Remembered as candidate: id={mid}\n"
|
||||||
|
f" type={memory_type} project={scope} confidence={confidence:.2f}{tag_str}{expires}\n"
|
||||||
|
f"Will flow through the standard triage pipeline within 24h "
|
||||||
|
f"(or on next auto-process button click at /admin/triage)."
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def _tool_health(args: dict) -> str:
|
def _tool_health(args: dict) -> str:
|
||||||
"""Check AtoCore service health."""
|
"""Check AtoCore service health."""
|
||||||
result, err = safe_call(http_get, "/health")
|
result, err = safe_call(http_get, "/health")
|
||||||
@@ -516,6 +593,58 @@ TOOLS = [
|
|||||||
},
|
},
|
||||||
"handler": _tool_memory_create,
|
"handler": _tool_memory_create,
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
"name": "atocore_remember",
|
||||||
|
"description": (
|
||||||
|
"Save a durable fact to AtoCore's memory layer from any conversation. "
|
||||||
|
"Use when the user says 'remember this', 'save that for later', "
|
||||||
|
"'don't lose this fact', or when you identify a decision/insight/"
|
||||||
|
"preference worth persisting across future sessions. The fact "
|
||||||
|
"goes through quality review before being consulted in future "
|
||||||
|
"context packs (so durable facts get kept, noise gets rejected). "
|
||||||
|
"Call multiple times if one conversation has multiple distinct "
|
||||||
|
"facts worth remembering — one tool call per atomic fact. "
|
||||||
|
"Prefer 'knowledge' type for cross-project engineering insights, "
|
||||||
|
"'project' for facts specific to one project, 'preference' for "
|
||||||
|
"user work-style notes, 'adaptation' for standing behavioral rules."
|
||||||
|
),
|
||||||
|
"inputSchema": {
|
||||||
|
"type": "object",
|
||||||
|
"properties": {
|
||||||
|
"content": {
|
||||||
|
"type": "string",
|
||||||
|
"description": "The atomic fact to remember. Under 250 chars. Should stand alone without session context.",
|
||||||
|
},
|
||||||
|
"memory_type": {
|
||||||
|
"type": "string",
|
||||||
|
"enum": ["identity", "preference", "project", "episodic", "knowledge", "adaptation"],
|
||||||
|
"default": "knowledge",
|
||||||
|
},
|
||||||
|
"project": {
|
||||||
|
"type": "string",
|
||||||
|
"description": "Project id if scoped. Empty for cross-project. Unregistered names flagged by triage as 'emerging project' proposals.",
|
||||||
|
},
|
||||||
|
"confidence": {
|
||||||
|
"type": "number",
|
||||||
|
"minimum": 0,
|
||||||
|
"maximum": 1,
|
||||||
|
"default": 0.6,
|
||||||
|
"description": "0.5-0.7 typical. 0.8+ only for ratified/committed claims.",
|
||||||
|
},
|
||||||
|
"valid_until": {
|
||||||
|
"type": "string",
|
||||||
|
"description": "ISO date YYYY-MM-DD if time-bounded (e.g. current state, scheduled event, quote expiry). Empty for permanent facts.",
|
||||||
|
},
|
||||||
|
"domain_tags": {
|
||||||
|
"type": "array",
|
||||||
|
"items": {"type": "string"},
|
||||||
|
"description": "Lowercase topical tags (optics, thermal, firmware, procurement, etc.) for cross-project retrieval. 2-5 tags typical.",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
"required": ["content"],
|
||||||
|
},
|
||||||
|
"handler": _tool_remember,
|
||||||
|
},
|
||||||
{
|
{
|
||||||
"name": "atocore_project_state",
|
"name": "atocore_project_state",
|
||||||
"description": (
|
"description": (
|
||||||
|
|||||||
254
scripts/canonicalize_tags.py
Normal file
254
scripts/canonicalize_tags.py
Normal file
@@ -0,0 +1,254 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Phase 7C — tag canonicalization detector.
|
||||||
|
|
||||||
|
Weekly (or on-demand) LLM pass that:
|
||||||
|
1. Fetches the tag distribution across all active memories via HTTP
|
||||||
|
2. Asks claude-p to propose alias→canonical mappings
|
||||||
|
3. AUTO-APPLIES aliases with confidence >= AUTO_APPROVE_CONF (0.8)
|
||||||
|
4. Submits lower-confidence proposals as pending for human review
|
||||||
|
|
||||||
|
Autonomous by default — matches the Phase 7A.1 pattern. Set
|
||||||
|
--no-auto-approve to force every proposal into human review.
|
||||||
|
|
||||||
|
Host-side because claude CLI lives on Dalidou, not the container.
|
||||||
|
Reuses the PYTHONPATH=src pattern from scripts/memory_dedup.py.
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
python3 scripts/canonicalize_tags.py [--base-url URL] [--dry-run] [--no-auto-approve]
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import shutil
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
import time
|
||||||
|
import urllib.error
|
||||||
|
import urllib.request
|
||||||
|
|
||||||
|
_SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
|
||||||
|
_SRC_DIR = os.path.abspath(os.path.join(_SCRIPT_DIR, "..", "src"))
|
||||||
|
if _SRC_DIR not in sys.path:
|
||||||
|
sys.path.insert(0, _SRC_DIR)
|
||||||
|
|
||||||
|
from atocore.memory._tag_canon_prompt import ( # noqa: E402
|
||||||
|
PROTECTED_PROJECT_TOKENS,
|
||||||
|
SYSTEM_PROMPT,
|
||||||
|
TAG_CANON_PROMPT_VERSION,
|
||||||
|
build_user_message,
|
||||||
|
normalize_alias_item,
|
||||||
|
parse_canon_output,
|
||||||
|
)
|
||||||
|
|
||||||
|
DEFAULT_BASE_URL = os.environ.get("ATOCORE_BASE_URL", "http://127.0.0.1:8100")
|
||||||
|
DEFAULT_MODEL = os.environ.get("ATOCORE_TAG_CANON_MODEL", "sonnet")
|
||||||
|
DEFAULT_TIMEOUT_S = float(os.environ.get("ATOCORE_TAG_CANON_TIMEOUT_S", "90"))
|
||||||
|
|
||||||
|
AUTO_APPROVE_CONF = float(os.environ.get("ATOCORE_TAG_CANON_AUTO_APPROVE_CONF", "0.8"))
|
||||||
|
MIN_ALIAS_COUNT = int(os.environ.get("ATOCORE_TAG_CANON_MIN_ALIAS_COUNT", "1"))
|
||||||
|
|
||||||
|
_sandbox_cwd = None
|
||||||
|
|
||||||
|
|
||||||
|
def get_sandbox_cwd() -> str:
|
||||||
|
global _sandbox_cwd
|
||||||
|
if _sandbox_cwd is None:
|
||||||
|
_sandbox_cwd = tempfile.mkdtemp(prefix="ato-tagcanon-")
|
||||||
|
return _sandbox_cwd
|
||||||
|
|
||||||
|
|
||||||
|
def api_get(base_url: str, path: str) -> dict:
|
||||||
|
req = urllib.request.Request(f"{base_url}{path}")
|
||||||
|
with urllib.request.urlopen(req, timeout=30) as resp:
|
||||||
|
return json.loads(resp.read().decode("utf-8"))
|
||||||
|
|
||||||
|
|
||||||
|
def api_post(base_url: str, path: str, body: dict | None = None) -> dict:
|
||||||
|
data = json.dumps(body or {}).encode("utf-8")
|
||||||
|
req = urllib.request.Request(
|
||||||
|
f"{base_url}{path}", method="POST",
|
||||||
|
headers={"Content-Type": "application/json"}, data=data,
|
||||||
|
)
|
||||||
|
with urllib.request.urlopen(req, timeout=30) as resp:
|
||||||
|
return json.loads(resp.read().decode("utf-8"))
|
||||||
|
|
||||||
|
|
||||||
|
def call_claude(user_message: str, model: str, timeout_s: float) -> tuple[str | None, str | None]:
|
||||||
|
if not shutil.which("claude"):
|
||||||
|
return None, "claude CLI not available"
|
||||||
|
args = [
|
||||||
|
"claude", "-p",
|
||||||
|
"--model", model,
|
||||||
|
"--append-system-prompt", SYSTEM_PROMPT,
|
||||||
|
"--disable-slash-commands",
|
||||||
|
user_message,
|
||||||
|
]
|
||||||
|
last_error = ""
|
||||||
|
for attempt in range(3):
|
||||||
|
if attempt > 0:
|
||||||
|
time.sleep(2 ** attempt)
|
||||||
|
try:
|
||||||
|
completed = subprocess.run(
|
||||||
|
args, capture_output=True, text=True,
|
||||||
|
timeout=timeout_s, cwd=get_sandbox_cwd(),
|
||||||
|
encoding="utf-8", errors="replace",
|
||||||
|
)
|
||||||
|
except subprocess.TimeoutExpired:
|
||||||
|
last_error = f"{model} timed out"
|
||||||
|
continue
|
||||||
|
except Exception as exc:
|
||||||
|
last_error = f"subprocess error: {exc}"
|
||||||
|
continue
|
||||||
|
if completed.returncode == 0:
|
||||||
|
return (completed.stdout or "").strip(), None
|
||||||
|
stderr = (completed.stderr or "").strip()[:200]
|
||||||
|
last_error = f"{model} exit {completed.returncode}: {stderr}"
|
||||||
|
return None, last_error
|
||||||
|
|
||||||
|
|
||||||
|
def fetch_tag_distribution(base_url: str) -> dict[str, int]:
|
||||||
|
"""Count tag occurrences across active memories (client-side)."""
|
||||||
|
try:
|
||||||
|
result = api_get(base_url, "/memory?active_only=true&limit=2000")
|
||||||
|
except Exception as e:
|
||||||
|
print(f"ERROR: could not fetch memories: {e}", file=sys.stderr)
|
||||||
|
return {}
|
||||||
|
mems = result.get("memories", [])
|
||||||
|
counts: dict[str, int] = {}
|
||||||
|
for m in mems:
|
||||||
|
tags = m.get("domain_tags") or []
|
||||||
|
if isinstance(tags, str):
|
||||||
|
try:
|
||||||
|
tags = json.loads(tags)
|
||||||
|
except Exception:
|
||||||
|
tags = []
|
||||||
|
if not isinstance(tags, list):
|
||||||
|
continue
|
||||||
|
for t in tags:
|
||||||
|
if not isinstance(t, str):
|
||||||
|
continue
|
||||||
|
key = t.strip().lower()
|
||||||
|
if key:
|
||||||
|
counts[key] = counts.get(key, 0) + 1
|
||||||
|
return counts
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
parser = argparse.ArgumentParser(description="Phase 7C tag canonicalization detector")
|
||||||
|
parser.add_argument("--base-url", default=DEFAULT_BASE_URL)
|
||||||
|
parser.add_argument("--model", default=DEFAULT_MODEL)
|
||||||
|
parser.add_argument("--timeout-s", type=float, default=DEFAULT_TIMEOUT_S)
|
||||||
|
parser.add_argument("--no-auto-approve", action="store_true",
|
||||||
|
help="Disable autonomous apply; all proposals → human queue")
|
||||||
|
parser.add_argument("--dry-run", action="store_true",
|
||||||
|
help="Print decisions without touching state")
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
base = args.base_url.rstrip("/")
|
||||||
|
autonomous = not args.no_auto_approve
|
||||||
|
|
||||||
|
print(
|
||||||
|
f"canonicalize_tags {TAG_CANON_PROMPT_VERSION} | model={args.model} | "
|
||||||
|
f"autonomous={autonomous} | auto-approve conf>={AUTO_APPROVE_CONF}"
|
||||||
|
)
|
||||||
|
|
||||||
|
dist = fetch_tag_distribution(base)
|
||||||
|
print(f"tag distribution: {len(dist)} unique tags, "
|
||||||
|
f"{sum(dist.values())} total references")
|
||||||
|
if not dist:
|
||||||
|
print("no tags found — nothing to canonicalize")
|
||||||
|
return
|
||||||
|
|
||||||
|
user_msg = build_user_message(dist)
|
||||||
|
raw, err = call_claude(user_msg, args.model, args.timeout_s)
|
||||||
|
if err or raw is None:
|
||||||
|
print(f"ERROR: LLM call failed: {err}", file=sys.stderr)
|
||||||
|
return
|
||||||
|
|
||||||
|
aliases_raw = parse_canon_output(raw)
|
||||||
|
print(f"LLM returned {len(aliases_raw)} raw alias proposals")
|
||||||
|
|
||||||
|
auto_applied = 0
|
||||||
|
auto_skipped_missing_canonical = 0
|
||||||
|
proposals_created = 0
|
||||||
|
duplicates_skipped = 0
|
||||||
|
|
||||||
|
for item in aliases_raw:
|
||||||
|
norm = normalize_alias_item(item)
|
||||||
|
if norm is None:
|
||||||
|
continue
|
||||||
|
alias = norm["alias"]
|
||||||
|
canonical = norm["canonical"]
|
||||||
|
confidence = norm["confidence"]
|
||||||
|
|
||||||
|
alias_count = dist.get(alias, 0)
|
||||||
|
canonical_count = dist.get(canonical, 0)
|
||||||
|
|
||||||
|
# Sanity: alias must actually exist in the current distribution
|
||||||
|
if alias_count < MIN_ALIAS_COUNT:
|
||||||
|
print(f" SKIP {alias!r} → {canonical!r}: alias not in distribution")
|
||||||
|
continue
|
||||||
|
if canonical_count == 0:
|
||||||
|
auto_skipped_missing_canonical += 1
|
||||||
|
print(f" SKIP {alias!r} → {canonical!r}: canonical missing from distribution")
|
||||||
|
continue
|
||||||
|
|
||||||
|
label = f"{alias!r} ({alias_count}) → {canonical!r} ({canonical_count}) conf={confidence:.2f}"
|
||||||
|
|
||||||
|
auto_apply = autonomous and confidence >= AUTO_APPROVE_CONF
|
||||||
|
if auto_apply:
|
||||||
|
if args.dry_run:
|
||||||
|
auto_applied += 1
|
||||||
|
print(f" [dry-run] would auto-apply: {label}")
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
result = api_post(base, "/admin/tags/aliases/apply", {
|
||||||
|
"alias": alias, "canonical": canonical,
|
||||||
|
"confidence": confidence, "reason": norm["reason"],
|
||||||
|
"alias_count": alias_count, "canonical_count": canonical_count,
|
||||||
|
"actor": "auto-tag-canon",
|
||||||
|
})
|
||||||
|
touched = result.get("memories_touched", 0)
|
||||||
|
auto_applied += 1
|
||||||
|
print(f" ✅ auto-applied: {label} ({touched} memories)")
|
||||||
|
except Exception as e:
|
||||||
|
print(f" ⚠️ auto-apply failed: {label} — {e}", file=sys.stderr)
|
||||||
|
time.sleep(0.2)
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Lower confidence → human review
|
||||||
|
if args.dry_run:
|
||||||
|
proposals_created += 1
|
||||||
|
print(f" [dry-run] would propose for review: {label}")
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
result = api_post(base, "/admin/tags/aliases/propose", {
|
||||||
|
"alias": alias, "canonical": canonical,
|
||||||
|
"confidence": confidence, "reason": norm["reason"],
|
||||||
|
"alias_count": alias_count, "canonical_count": canonical_count,
|
||||||
|
})
|
||||||
|
if result.get("proposal_id"):
|
||||||
|
proposals_created += 1
|
||||||
|
print(f" → pending proposal: {label}")
|
||||||
|
else:
|
||||||
|
duplicates_skipped += 1
|
||||||
|
print(f" (duplicate pending proposal): {label}")
|
||||||
|
except Exception as e:
|
||||||
|
print(f" ⚠️ propose failed: {label} — {e}", file=sys.stderr)
|
||||||
|
time.sleep(0.2)
|
||||||
|
|
||||||
|
print(
|
||||||
|
f"\nsummary: proposals_seen={len(aliases_raw)} "
|
||||||
|
f"auto_applied={auto_applied} "
|
||||||
|
f"proposals_created={proposals_created} "
|
||||||
|
f"duplicates_skipped={duplicates_skipped} "
|
||||||
|
f"skipped_missing_canonical={auto_skipped_missing_canonical}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
223
scripts/detect_emerging.py
Normal file
223
scripts/detect_emerging.py
Normal file
@@ -0,0 +1,223 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Phase 6 C.1 — Emerging-concepts detector (HTTP-only).
|
||||||
|
|
||||||
|
Scans active + candidate memories via the HTTP API to surface:
|
||||||
|
1. Unregistered projects — project strings appearing on 3+ memories
|
||||||
|
that aren't in the project registry. Surface for one-click
|
||||||
|
registration.
|
||||||
|
2. Emerging categories — top 20 domain_tags by frequency, for
|
||||||
|
"what themes are emerging in my work?" intelligence.
|
||||||
|
3. Reinforced transients — active memories with reference_count >= 5
|
||||||
|
AND valid_until set. These "were temporary but now durable"; a
|
||||||
|
sibling endpoint (/admin/memory/extend-reinforced) actually
|
||||||
|
performs the extension.
|
||||||
|
|
||||||
|
Writes results to project_state under atocore/proposals/* via the API.
|
||||||
|
Runs host-side (cron calls it) so uses stdlib only — no atocore deps.
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
python3 scripts/detect_emerging.py [--base-url URL] [--dry-run]
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
import urllib.error
|
||||||
|
import urllib.request
|
||||||
|
from collections import Counter, defaultdict
|
||||||
|
|
||||||
|
PROJECT_MIN_MEMORIES = int(os.environ.get("ATOCORE_EMERGING_PROJECT_MIN", "3"))
|
||||||
|
PROJECT_ALERT_THRESHOLD = int(os.environ.get("ATOCORE_EMERGING_ALERT_THRESHOLD", "5"))
|
||||||
|
TOP_TAGS_LIMIT = int(os.environ.get("ATOCORE_EMERGING_TOP_TAGS", "20"))
|
||||||
|
|
||||||
|
|
||||||
|
def api_get(base_url: str, path: str, timeout: int = 30) -> dict:
|
||||||
|
req = urllib.request.Request(f"{base_url}{path}")
|
||||||
|
with urllib.request.urlopen(req, timeout=timeout) as resp:
|
||||||
|
return json.loads(resp.read().decode("utf-8"))
|
||||||
|
|
||||||
|
|
||||||
|
def api_post(base_url: str, path: str, body: dict, timeout: int = 10) -> dict:
|
||||||
|
data = json.dumps(body).encode("utf-8")
|
||||||
|
req = urllib.request.Request(
|
||||||
|
f"{base_url}{path}", method="POST",
|
||||||
|
headers={"Content-Type": "application/json"}, data=data,
|
||||||
|
)
|
||||||
|
with urllib.request.urlopen(req, timeout=timeout) as resp:
|
||||||
|
return json.loads(resp.read().decode("utf-8"))
|
||||||
|
|
||||||
|
|
||||||
|
def fetch_registered_project_names(base_url: str) -> set[str]:
|
||||||
|
"""Set of all registered project ids + aliases, lowercased."""
|
||||||
|
try:
|
||||||
|
result = api_get(base_url, "/projects")
|
||||||
|
except Exception as e:
|
||||||
|
print(f"WARN: could not load project registry: {e}", file=sys.stderr)
|
||||||
|
return set()
|
||||||
|
registered = set()
|
||||||
|
for p in result.get("projects", []):
|
||||||
|
pid = (p.get("project_id") or p.get("id") or p.get("name") or "").strip()
|
||||||
|
if pid:
|
||||||
|
registered.add(pid.lower())
|
||||||
|
for alias in p.get("aliases", []) or []:
|
||||||
|
if isinstance(alias, str) and alias.strip():
|
||||||
|
registered.add(alias.strip().lower())
|
||||||
|
return registered
|
||||||
|
|
||||||
|
|
||||||
|
def fetch_memories(base_url: str, status: str, limit: int = 500) -> list[dict]:
|
||||||
|
try:
|
||||||
|
params = f"limit={limit}"
|
||||||
|
if status == "active":
|
||||||
|
params += "&active_only=true"
|
||||||
|
else:
|
||||||
|
params += f"&status={status}"
|
||||||
|
result = api_get(base_url, f"/memory?{params}")
|
||||||
|
return result.get("memories", [])
|
||||||
|
except Exception as e:
|
||||||
|
print(f"WARN: could not fetch {status} memories: {e}", file=sys.stderr)
|
||||||
|
return []
|
||||||
|
|
||||||
|
|
||||||
|
def fetch_previous_proposals(base_url: str) -> list[dict]:
|
||||||
|
"""Read last run's unregistered_projects to diff against this run."""
|
||||||
|
try:
|
||||||
|
result = api_get(base_url, "/project/state/atocore")
|
||||||
|
entries = result.get("entries", result.get("state", []))
|
||||||
|
for e in entries:
|
||||||
|
if e.get("category") == "proposals" and e.get("key") == "unregistered_projects_prev":
|
||||||
|
try:
|
||||||
|
return json.loads(e.get("value") or "[]")
|
||||||
|
except Exception:
|
||||||
|
return []
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
return []
|
||||||
|
|
||||||
|
|
||||||
|
def set_state(base_url: str, category: str, key: str, value: str, source: str = "emerging detector") -> None:
|
||||||
|
api_post(base_url, "/project/state", {
|
||||||
|
"project": "atocore",
|
||||||
|
"category": category,
|
||||||
|
"key": key,
|
||||||
|
"value": value,
|
||||||
|
"source": source,
|
||||||
|
})
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
parser = argparse.ArgumentParser(description="Detect emerging projects + categories")
|
||||||
|
parser.add_argument("--base-url", default=os.environ.get("ATOCORE_BASE_URL", "http://127.0.0.1:8100"))
|
||||||
|
parser.add_argument("--dry-run", action="store_true", help="Report without writing to project state")
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
base = args.base_url.rstrip("/")
|
||||||
|
|
||||||
|
registered = fetch_registered_project_names(base)
|
||||||
|
active = fetch_memories(base, "active")
|
||||||
|
candidates = fetch_memories(base, "candidate")
|
||||||
|
all_mems = active + candidates
|
||||||
|
|
||||||
|
# --- Unregistered projects ---
|
||||||
|
project_mems: dict[str, list] = defaultdict(list)
|
||||||
|
for m in all_mems:
|
||||||
|
proj = (m.get("project") or "").strip().lower()
|
||||||
|
if not proj or proj in registered:
|
||||||
|
continue
|
||||||
|
project_mems[proj].append(m)
|
||||||
|
|
||||||
|
unregistered = []
|
||||||
|
for proj, mems in sorted(project_mems.items()):
|
||||||
|
if len(mems) < PROJECT_MIN_MEMORIES:
|
||||||
|
continue
|
||||||
|
unregistered.append({
|
||||||
|
"project": proj,
|
||||||
|
"count": len(mems),
|
||||||
|
"sample_memory_ids": [m.get("id") for m in mems[:3]],
|
||||||
|
"sample_contents": [(m.get("content") or "")[:150] for m in mems[:3]],
|
||||||
|
})
|
||||||
|
|
||||||
|
# --- Emerging domain_tags (active only) ---
|
||||||
|
tag_counter: Counter = Counter()
|
||||||
|
for m in active:
|
||||||
|
for t in (m.get("domain_tags") or []):
|
||||||
|
if isinstance(t, str) and t.strip():
|
||||||
|
tag_counter[t.strip().lower()] += 1
|
||||||
|
emerging_tags = [{"tag": tag, "count": cnt} for tag, cnt in tag_counter.most_common(TOP_TAGS_LIMIT)]
|
||||||
|
|
||||||
|
# --- Reinforced transients (active, high refs, has expiry) ---
|
||||||
|
reinforced = []
|
||||||
|
for m in active:
|
||||||
|
ref_count = int(m.get("reference_count") or 0)
|
||||||
|
vu = (m.get("valid_until") or "").strip()
|
||||||
|
if ref_count >= 5 and vu:
|
||||||
|
reinforced.append({
|
||||||
|
"memory_id": m.get("id"),
|
||||||
|
"reference_count": ref_count,
|
||||||
|
"valid_until": vu,
|
||||||
|
"content_preview": (m.get("content") or "")[:150],
|
||||||
|
"project": m.get("project") or "",
|
||||||
|
})
|
||||||
|
|
||||||
|
result = {
|
||||||
|
"unregistered_projects": unregistered,
|
||||||
|
"emerging_categories": emerging_tags,
|
||||||
|
"reinforced_transients": reinforced,
|
||||||
|
"counts": {
|
||||||
|
"active_memories": len(active),
|
||||||
|
"candidate_memories": len(candidates),
|
||||||
|
"unregistered_project_count": len(unregistered),
|
||||||
|
"emerging_tag_count": len(emerging_tags),
|
||||||
|
"reinforced_transient_count": len(reinforced),
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
print(json.dumps(result, indent=2))
|
||||||
|
|
||||||
|
if args.dry_run:
|
||||||
|
return
|
||||||
|
|
||||||
|
# --- Persist to project state via HTTP ---
|
||||||
|
try:
|
||||||
|
set_state(base, "proposals", "unregistered_projects", json.dumps(unregistered))
|
||||||
|
set_state(base, "proposals", "emerging_categories", json.dumps(emerging_tags))
|
||||||
|
set_state(base, "proposals", "reinforced_transients", json.dumps(reinforced))
|
||||||
|
except Exception as e:
|
||||||
|
print(f"WARN: failed to persist proposals: {e}", file=sys.stderr)
|
||||||
|
|
||||||
|
# --- Alert on NEW projects crossing the threshold ---
|
||||||
|
try:
|
||||||
|
prev = fetch_previous_proposals(base)
|
||||||
|
prev_names = {p.get("project") for p in prev if isinstance(p, dict)}
|
||||||
|
newly_crossed = [
|
||||||
|
p for p in unregistered
|
||||||
|
if p["count"] >= PROJECT_ALERT_THRESHOLD
|
||||||
|
and p["project"] not in prev_names
|
||||||
|
]
|
||||||
|
if newly_crossed:
|
||||||
|
names = ", ".join(p["project"] for p in newly_crossed)
|
||||||
|
# Use existing alert mechanism via state (Phase 4 infra)
|
||||||
|
try:
|
||||||
|
set_state(base, "alert", "last_warning", json.dumps({
|
||||||
|
"title": f"Emerging project(s) detected: {names}",
|
||||||
|
"message": (
|
||||||
|
f"{len(newly_crossed)} unregistered project(s) crossed "
|
||||||
|
f"the {PROJECT_ALERT_THRESHOLD}-memory threshold. "
|
||||||
|
f"Review at /wiki or /admin/dashboard."
|
||||||
|
),
|
||||||
|
"timestamp": "",
|
||||||
|
}))
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
# Snapshot for next run's diff
|
||||||
|
set_state(base, "proposals", "unregistered_projects_prev", json.dumps(unregistered))
|
||||||
|
except Exception as e:
|
||||||
|
print(f"WARN: alert/state write failed: {e}", file=sys.stderr)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
374
scripts/memory_dedup.py
Normal file
374
scripts/memory_dedup.py
Normal file
@@ -0,0 +1,374 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Phase 7A — semantic memory dedup detector (stdlib-only host script).
|
||||||
|
|
||||||
|
Finds clusters of near-duplicate active memories and writes merge-
|
||||||
|
candidate proposals for human (or autonomous) approval.
|
||||||
|
|
||||||
|
Algorithm:
|
||||||
|
1. POST /admin/memory/dedup-cluster on the AtoCore server — it
|
||||||
|
computes embeddings + transitive clusters under the (project,
|
||||||
|
memory_type) bucket rule (sentence-transformers lives in the
|
||||||
|
container, not on the host)
|
||||||
|
2. For each returned cluster of size >= 2, ask claude-p (host-side
|
||||||
|
CLI) to draft unified content preserving all specifics
|
||||||
|
3. Server-side tiering:
|
||||||
|
- TIER-1 auto-approve: sonnet confidence >= 0.8 AND min_sim >= 0.92
|
||||||
|
AND all sources share project+type → immediately submit and
|
||||||
|
approve (actor="auto-dedup-tier1")
|
||||||
|
- TIER-2 escalation: opus confirms with conf >= 0.8 → auto-approve
|
||||||
|
(actor="auto-dedup-tier2"); opus rejects → skip silently
|
||||||
|
- HUMAN: pending proposal lands in /admin/triage
|
||||||
|
|
||||||
|
Host-only dep: the `claude` CLI. No python packages beyond stdlib.
|
||||||
|
Reuses atocore.memory._dedup_prompt (stdlib-only shared prompt).
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
python3 scripts/memory_dedup.py --base-url http://127.0.0.1:8100 \\
|
||||||
|
--similarity-threshold 0.88 --max-batch 50
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import shutil
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
import time
|
||||||
|
import urllib.error
|
||||||
|
import urllib.request
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
# Make src/ importable for the stdlib-only prompt module.
|
||||||
|
# We DO NOT import anything that pulls in pydantic_settings or
|
||||||
|
# sentence-transformers; those live on the server side.
|
||||||
|
_SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
|
||||||
|
_SRC_DIR = os.path.abspath(os.path.join(_SCRIPT_DIR, "..", "src"))
|
||||||
|
if _SRC_DIR not in sys.path:
|
||||||
|
sys.path.insert(0, _SRC_DIR)
|
||||||
|
|
||||||
|
from atocore.memory._dedup_prompt import ( # noqa: E402
|
||||||
|
DEDUP_PROMPT_VERSION,
|
||||||
|
SYSTEM_PROMPT,
|
||||||
|
TIER2_SYSTEM_PROMPT,
|
||||||
|
build_tier2_user_message,
|
||||||
|
build_user_message,
|
||||||
|
normalize_merge_verdict,
|
||||||
|
parse_merge_verdict,
|
||||||
|
)
|
||||||
|
|
||||||
|
DEFAULT_BASE_URL = os.environ.get("ATOCORE_BASE_URL", "http://127.0.0.1:8100")
|
||||||
|
DEFAULT_MODEL = os.environ.get("ATOCORE_DEDUP_MODEL", "sonnet")
|
||||||
|
DEFAULT_TIER2_MODEL = os.environ.get("ATOCORE_DEDUP_TIER2_MODEL", "opus")
|
||||||
|
DEFAULT_TIMEOUT_S = float(os.environ.get("ATOCORE_DEDUP_TIMEOUT_S", "60"))
|
||||||
|
|
||||||
|
AUTO_APPROVE_CONF = float(os.environ.get("ATOCORE_DEDUP_AUTO_APPROVE_CONF", "0.8"))
|
||||||
|
AUTO_APPROVE_SIM = float(os.environ.get("ATOCORE_DEDUP_AUTO_APPROVE_SIM", "0.92"))
|
||||||
|
TIER2_MIN_CONF = float(os.environ.get("ATOCORE_DEDUP_TIER2_MIN_CONF", "0.5"))
|
||||||
|
TIER2_MIN_SIM = float(os.environ.get("ATOCORE_DEDUP_TIER2_MIN_SIM", "0.85"))
|
||||||
|
|
||||||
|
_sandbox_cwd = None
|
||||||
|
|
||||||
|
|
||||||
|
def get_sandbox_cwd() -> str:
|
||||||
|
global _sandbox_cwd
|
||||||
|
if _sandbox_cwd is None:
|
||||||
|
_sandbox_cwd = tempfile.mkdtemp(prefix="ato-dedup-")
|
||||||
|
return _sandbox_cwd
|
||||||
|
|
||||||
|
|
||||||
|
def api_get(base_url: str, path: str) -> dict:
|
||||||
|
req = urllib.request.Request(f"{base_url}{path}")
|
||||||
|
with urllib.request.urlopen(req, timeout=30) as resp:
|
||||||
|
return json.loads(resp.read().decode("utf-8"))
|
||||||
|
|
||||||
|
|
||||||
|
def api_post(base_url: str, path: str, body: dict | None = None, timeout: int = 60) -> dict:
|
||||||
|
data = json.dumps(body or {}).encode("utf-8")
|
||||||
|
req = urllib.request.Request(
|
||||||
|
f"{base_url}{path}", method="POST",
|
||||||
|
headers={"Content-Type": "application/json"}, data=data,
|
||||||
|
)
|
||||||
|
with urllib.request.urlopen(req, timeout=timeout) as resp:
|
||||||
|
return json.loads(resp.read().decode("utf-8"))
|
||||||
|
|
||||||
|
|
||||||
|
def call_claude(system_prompt: str, user_message: str, model: str, timeout_s: float) -> tuple[str | None, str | None]:
|
||||||
|
if not shutil.which("claude"):
|
||||||
|
return None, "claude CLI not available"
|
||||||
|
args = [
|
||||||
|
"claude", "-p",
|
||||||
|
"--model", model,
|
||||||
|
"--append-system-prompt", system_prompt,
|
||||||
|
"--disable-slash-commands",
|
||||||
|
user_message,
|
||||||
|
]
|
||||||
|
last_error = ""
|
||||||
|
for attempt in range(3):
|
||||||
|
if attempt > 0:
|
||||||
|
time.sleep(2 ** attempt)
|
||||||
|
try:
|
||||||
|
completed = subprocess.run(
|
||||||
|
args, capture_output=True, text=True,
|
||||||
|
timeout=timeout_s, cwd=get_sandbox_cwd(),
|
||||||
|
encoding="utf-8", errors="replace",
|
||||||
|
)
|
||||||
|
except subprocess.TimeoutExpired:
|
||||||
|
last_error = f"{model} timed out"
|
||||||
|
continue
|
||||||
|
except Exception as exc:
|
||||||
|
last_error = f"subprocess error: {exc}"
|
||||||
|
continue
|
||||||
|
if completed.returncode == 0:
|
||||||
|
return (completed.stdout or "").strip(), None
|
||||||
|
stderr = (completed.stderr or "").strip()[:200]
|
||||||
|
last_error = f"{model} exit {completed.returncode}: {stderr}"
|
||||||
|
return None, last_error
|
||||||
|
|
||||||
|
|
||||||
|
def fetch_clusters(base_url: str, project: str, threshold: float, max_clusters: int) -> list[dict]:
|
||||||
|
"""Ask the server to compute near-duplicate clusters. The server
|
||||||
|
owns sentence-transformers; host stays lean."""
|
||||||
|
try:
|
||||||
|
result = api_post(base_url, "/admin/memory/dedup-cluster", {
|
||||||
|
"project": project,
|
||||||
|
"similarity_threshold": threshold,
|
||||||
|
"max_clusters": max_clusters,
|
||||||
|
}, timeout=120)
|
||||||
|
except Exception as e:
|
||||||
|
print(f"ERROR: dedup-cluster fetch failed: {e}", file=sys.stderr)
|
||||||
|
return []
|
||||||
|
clusters = result.get("clusters", [])
|
||||||
|
print(
|
||||||
|
f"server returned {len(clusters)} clusters "
|
||||||
|
f"(total_active={result.get('total_active_scanned')}, "
|
||||||
|
f"buckets={result.get('bucket_count')})"
|
||||||
|
)
|
||||||
|
return clusters
|
||||||
|
|
||||||
|
|
||||||
|
def draft_merge(sources: list[dict], model: str, timeout_s: float) -> dict[str, Any] | None:
|
||||||
|
user_msg = build_user_message(sources)
|
||||||
|
raw, err = call_claude(SYSTEM_PROMPT, user_msg, model, timeout_s)
|
||||||
|
if err:
|
||||||
|
print(f" WARN: claude tier-1 failed: {err}", file=sys.stderr)
|
||||||
|
return None
|
||||||
|
parsed = parse_merge_verdict(raw or "")
|
||||||
|
if parsed is None:
|
||||||
|
print(f" WARN: could not parse tier-1 verdict: {(raw or '')[:200]}", file=sys.stderr)
|
||||||
|
return None
|
||||||
|
return normalize_merge_verdict(parsed)
|
||||||
|
|
||||||
|
|
||||||
|
def tier2_review(sources: list[dict], tier1_verdict: dict, model: str, timeout_s: float) -> dict | None:
|
||||||
|
user_msg = build_tier2_user_message(sources, tier1_verdict)
|
||||||
|
raw, err = call_claude(TIER2_SYSTEM_PROMPT, user_msg, model, timeout_s)
|
||||||
|
if err:
|
||||||
|
print(f" WARN: claude tier-2 failed: {err}", file=sys.stderr)
|
||||||
|
return None
|
||||||
|
parsed = parse_merge_verdict(raw or "")
|
||||||
|
if parsed is None:
|
||||||
|
print(f" WARN: could not parse tier-2 verdict: {(raw or '')[:200]}", file=sys.stderr)
|
||||||
|
return None
|
||||||
|
return normalize_merge_verdict(parsed)
|
||||||
|
|
||||||
|
|
||||||
|
def submit_candidate(base_url: str, memory_ids: list[str], similarity: float, verdict: dict, dry_run: bool) -> str | None:
|
||||||
|
body = {
|
||||||
|
"memory_ids": memory_ids,
|
||||||
|
"similarity": similarity,
|
||||||
|
"proposed_content": verdict["content"],
|
||||||
|
"proposed_memory_type": verdict["memory_type"],
|
||||||
|
"proposed_project": verdict["project"],
|
||||||
|
"proposed_tags": verdict["domain_tags"],
|
||||||
|
"proposed_confidence": verdict["confidence"],
|
||||||
|
"reason": verdict["reason"],
|
||||||
|
}
|
||||||
|
if dry_run:
|
||||||
|
print(f" [dry-run] would POST: {json.dumps(body)[:200]}...")
|
||||||
|
return "dry-run"
|
||||||
|
try:
|
||||||
|
result = api_post(base_url, "/admin/memory/merge-candidates/create", body)
|
||||||
|
return result.get("candidate_id")
|
||||||
|
except urllib.error.HTTPError as e:
|
||||||
|
print(f" ERROR: submit failed: {e.code} {e.read().decode()[:200]}", file=sys.stderr)
|
||||||
|
return None
|
||||||
|
except Exception as e:
|
||||||
|
print(f" ERROR: submit failed: {e}", file=sys.stderr)
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def auto_approve(base_url: str, candidate_id: str, actor: str, dry_run: bool) -> str | None:
|
||||||
|
if dry_run:
|
||||||
|
return "dry-run"
|
||||||
|
try:
|
||||||
|
result = api_post(
|
||||||
|
base_url,
|
||||||
|
f"/admin/memory/merge-candidates/{candidate_id}/approve",
|
||||||
|
{"actor": actor},
|
||||||
|
)
|
||||||
|
return result.get("result_memory_id")
|
||||||
|
except Exception as e:
|
||||||
|
print(f" ERROR: auto-approve failed: {e}", file=sys.stderr)
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
parser = argparse.ArgumentParser(description="Phase 7A semantic dedup detector (tiered, stdlib-only host)")
|
||||||
|
parser.add_argument("--base-url", default=DEFAULT_BASE_URL)
|
||||||
|
parser.add_argument("--project", default="", help="Only scan this project (empty = all)")
|
||||||
|
parser.add_argument("--similarity-threshold", type=float, default=0.88)
|
||||||
|
parser.add_argument("--max-batch", type=int, default=50,
|
||||||
|
help="Max clusters to process per run")
|
||||||
|
parser.add_argument("--model", default=DEFAULT_MODEL)
|
||||||
|
parser.add_argument("--tier2-model", default=DEFAULT_TIER2_MODEL)
|
||||||
|
parser.add_argument("--timeout-s", type=float, default=DEFAULT_TIMEOUT_S)
|
||||||
|
parser.add_argument("--no-auto-approve", action="store_true",
|
||||||
|
help="Disable autonomous merging; all merges land in human triage queue")
|
||||||
|
parser.add_argument("--dry-run", action="store_true")
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
base = args.base_url.rstrip("/")
|
||||||
|
autonomous = not args.no_auto_approve
|
||||||
|
|
||||||
|
print(
|
||||||
|
f"memory_dedup {DEDUP_PROMPT_VERSION} | threshold={args.similarity_threshold} | "
|
||||||
|
f"tier1={args.model} tier2={args.tier2_model} | "
|
||||||
|
f"autonomous={autonomous} | "
|
||||||
|
f"auto-approve: conf>={AUTO_APPROVE_CONF} sim>={AUTO_APPROVE_SIM}"
|
||||||
|
)
|
||||||
|
|
||||||
|
clusters = fetch_clusters(
|
||||||
|
base, args.project, args.similarity_threshold, args.max_batch,
|
||||||
|
)
|
||||||
|
if not clusters:
|
||||||
|
print("no clusters — nothing to dedup")
|
||||||
|
return
|
||||||
|
|
||||||
|
auto_merged_tier1 = 0
|
||||||
|
auto_merged_tier2 = 0
|
||||||
|
human_candidates = 0
|
||||||
|
tier1_rejections = 0
|
||||||
|
tier2_overrides = 0
|
||||||
|
skipped_existing = 0
|
||||||
|
processed = 0
|
||||||
|
|
||||||
|
for cluster in clusters:
|
||||||
|
if processed >= args.max_batch:
|
||||||
|
break
|
||||||
|
processed += 1
|
||||||
|
sources = cluster["sources"]
|
||||||
|
ids = cluster["memory_ids"]
|
||||||
|
min_sim = float(cluster["min_similarity"])
|
||||||
|
proj = cluster.get("project") or "(global)"
|
||||||
|
mtype = cluster.get("memory_type") or "?"
|
||||||
|
|
||||||
|
print(f"\n[{proj}/{mtype}] cluster size={cluster['size']} min_sim={min_sim:.3f} "
|
||||||
|
f"{[s['id'][:8] for s in sources]}")
|
||||||
|
|
||||||
|
tier1 = draft_merge(sources, args.model, args.timeout_s)
|
||||||
|
if tier1 is None:
|
||||||
|
continue
|
||||||
|
if tier1["action"] == "reject":
|
||||||
|
tier1_rejections += 1
|
||||||
|
print(f" TIER-1 rejected: {tier1['reason'][:100]}")
|
||||||
|
continue
|
||||||
|
|
||||||
|
# All sources share the bucket by construction from the server
|
||||||
|
bucket_ok = True
|
||||||
|
tier1_ok = (
|
||||||
|
tier1["confidence"] >= AUTO_APPROVE_CONF
|
||||||
|
and min_sim >= AUTO_APPROVE_SIM
|
||||||
|
and bucket_ok
|
||||||
|
)
|
||||||
|
|
||||||
|
if autonomous and tier1_ok:
|
||||||
|
cid = submit_candidate(base, ids, min_sim, tier1, args.dry_run)
|
||||||
|
if cid == "dry-run":
|
||||||
|
auto_merged_tier1 += 1
|
||||||
|
print(" [dry-run] would auto-merge (tier-1)")
|
||||||
|
elif cid:
|
||||||
|
new_id = auto_approve(base, cid, actor="auto-dedup-tier1", dry_run=args.dry_run)
|
||||||
|
if new_id:
|
||||||
|
auto_merged_tier1 += 1
|
||||||
|
print(f" ✅ auto-merged (tier-1) → {str(new_id)[:8]}")
|
||||||
|
else:
|
||||||
|
human_candidates += 1
|
||||||
|
print(f" ⚠️ tier-1 approve failed; candidate {cid[:8]} pending")
|
||||||
|
else:
|
||||||
|
skipped_existing += 1
|
||||||
|
time.sleep(0.3)
|
||||||
|
continue
|
||||||
|
|
||||||
|
tier2_eligible = (
|
||||||
|
autonomous
|
||||||
|
and min_sim >= TIER2_MIN_SIM
|
||||||
|
and tier1["confidence"] >= TIER2_MIN_CONF
|
||||||
|
)
|
||||||
|
|
||||||
|
if tier2_eligible:
|
||||||
|
print(" → escalating to tier-2 (opus)…")
|
||||||
|
tier2 = tier2_review(sources, tier1, args.tier2_model, args.timeout_s)
|
||||||
|
if tier2 is None:
|
||||||
|
cid = submit_candidate(base, ids, min_sim, tier1, args.dry_run)
|
||||||
|
if cid and cid != "dry-run":
|
||||||
|
human_candidates += 1
|
||||||
|
print(f" → candidate {cid[:8]} (tier-2 errored, human review)")
|
||||||
|
time.sleep(0.5)
|
||||||
|
continue
|
||||||
|
|
||||||
|
if tier2["action"] == "reject":
|
||||||
|
tier2_overrides += 1
|
||||||
|
print(f" ❌ TIER-2 override (reject): {tier2['reason'][:100]}")
|
||||||
|
time.sleep(0.5)
|
||||||
|
continue
|
||||||
|
|
||||||
|
if tier2["confidence"] >= AUTO_APPROVE_CONF:
|
||||||
|
cid = submit_candidate(base, ids, min_sim, tier2, args.dry_run)
|
||||||
|
if cid == "dry-run":
|
||||||
|
auto_merged_tier2 += 1
|
||||||
|
elif cid:
|
||||||
|
new_id = auto_approve(base, cid, actor="auto-dedup-tier2", dry_run=args.dry_run)
|
||||||
|
if new_id:
|
||||||
|
auto_merged_tier2 += 1
|
||||||
|
print(f" ✅ auto-merged (tier-2) → {str(new_id)[:8]}")
|
||||||
|
else:
|
||||||
|
human_candidates += 1
|
||||||
|
else:
|
||||||
|
skipped_existing += 1
|
||||||
|
time.sleep(0.5)
|
||||||
|
continue
|
||||||
|
|
||||||
|
cid = submit_candidate(base, ids, min_sim, tier2, args.dry_run)
|
||||||
|
if cid and cid != "dry-run":
|
||||||
|
human_candidates += 1
|
||||||
|
print(f" → candidate {cid[:8]} (tier-2 low-conf, human review)")
|
||||||
|
time.sleep(0.5)
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Below tier-2 thresholds — human review with tier-1 draft
|
||||||
|
cid = submit_candidate(base, ids, min_sim, tier1, args.dry_run)
|
||||||
|
if cid == "dry-run":
|
||||||
|
human_candidates += 1
|
||||||
|
elif cid:
|
||||||
|
human_candidates += 1
|
||||||
|
print(f" → candidate {cid[:8]} (human review)")
|
||||||
|
else:
|
||||||
|
skipped_existing += 1
|
||||||
|
time.sleep(0.3)
|
||||||
|
|
||||||
|
print(
|
||||||
|
f"\nsummary: clusters_processed={processed} "
|
||||||
|
f"auto_merged_tier1={auto_merged_tier1} "
|
||||||
|
f"auto_merged_tier2={auto_merged_tier2} "
|
||||||
|
f"human_candidates={human_candidates} "
|
||||||
|
f"tier1_rejections={tier1_rejections} "
|
||||||
|
f"tier2_overrides={tier2_overrides} "
|
||||||
|
f"skipped_existing={skipped_existing}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
167
scripts/v1_0_backfill_provenance.py
Normal file
167
scripts/v1_0_backfill_provenance.py
Normal file
@@ -0,0 +1,167 @@
|
|||||||
|
"""V1-0 one-time backfill: flag existing active entities that have no
|
||||||
|
provenance (empty source_refs) as hand_authored=1 so they stop failing
|
||||||
|
the F-8 invariant.
|
||||||
|
|
||||||
|
Runs against the live AtoCore DB. Idempotent: a second run after the
|
||||||
|
first touches nothing because the flagged rows already have
|
||||||
|
hand_authored=1.
|
||||||
|
|
||||||
|
Per the Engineering V1 Completion Plan (V1-0 scope), the three options
|
||||||
|
for an existing active entity without provenance are:
|
||||||
|
|
||||||
|
1. Attach provenance — impossible without human review, not automatable
|
||||||
|
2. Flag hand-authored — safe, additive, this script's default
|
||||||
|
3. Invalidate — destructive, requires operator sign-off
|
||||||
|
|
||||||
|
This script picks option (2) by default. Add --dry-run to see what
|
||||||
|
would change without writing. Add --invalidate-instead to pick option
|
||||||
|
(3) for all rows (not recommended for first run).
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
python scripts/v1_0_backfill_provenance.py --base-url http://dalidou:8100 --dry-run
|
||||||
|
python scripts/v1_0_backfill_provenance.py --base-url http://dalidou:8100
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import sqlite3
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
|
||||||
|
def run(db_path: Path, dry_run: bool, invalidate_instead: bool) -> int:
|
||||||
|
if not db_path.exists():
|
||||||
|
print(f"ERROR: db not found: {db_path}", file=sys.stderr)
|
||||||
|
return 2
|
||||||
|
|
||||||
|
conn = sqlite3.connect(str(db_path))
|
||||||
|
conn.row_factory = sqlite3.Row
|
||||||
|
|
||||||
|
# Verify the V1-0 migration ran: if hand_authored column is missing
|
||||||
|
# the operator hasn't deployed V1-0 yet, and running this script
|
||||||
|
# would crash. Fail loud rather than attempt the ALTER here.
|
||||||
|
cols = {r["name"] for r in conn.execute("PRAGMA table_info(entities)").fetchall()}
|
||||||
|
if "hand_authored" not in cols:
|
||||||
|
print(
|
||||||
|
"ERROR: entities table lacks the hand_authored column. "
|
||||||
|
"Deploy V1-0 migrations first (init_db + init_engineering_schema).",
|
||||||
|
file=sys.stderr,
|
||||||
|
)
|
||||||
|
return 2
|
||||||
|
|
||||||
|
# Scope differs by mode:
|
||||||
|
# - Default (flag hand_authored=1): safe/additive, applies to active
|
||||||
|
# AND superseded rows so the historical trail is consistent.
|
||||||
|
# - --invalidate-instead: destructive — scope to ACTIVE rows only.
|
||||||
|
# Invalidating already-superseded history would collapse the audit
|
||||||
|
# trail, which the plan's remediation scope never intended
|
||||||
|
# (V1-0 talks about existing active no-provenance entities).
|
||||||
|
if invalidate_instead:
|
||||||
|
scope_sql = "status = 'active'"
|
||||||
|
else:
|
||||||
|
scope_sql = "status IN ('active', 'superseded')"
|
||||||
|
rows = conn.execute(
|
||||||
|
f"SELECT id, entity_type, name, project, status, source_refs, hand_authored "
|
||||||
|
f"FROM entities WHERE {scope_sql} AND hand_authored = 0"
|
||||||
|
).fetchall()
|
||||||
|
|
||||||
|
needs_fix = []
|
||||||
|
for row in rows:
|
||||||
|
refs_raw = row["source_refs"] or "[]"
|
||||||
|
try:
|
||||||
|
refs = json.loads(refs_raw)
|
||||||
|
except Exception:
|
||||||
|
refs = []
|
||||||
|
if not refs:
|
||||||
|
needs_fix.append(row)
|
||||||
|
|
||||||
|
print(f"found {len(needs_fix)} active/superseded entities with no provenance")
|
||||||
|
for row in needs_fix:
|
||||||
|
print(
|
||||||
|
f" - {row['id'][:8]} [{row['entity_type']}] "
|
||||||
|
f"{row['name']!r} project={row['project']!r} status={row['status']}"
|
||||||
|
)
|
||||||
|
|
||||||
|
if dry_run:
|
||||||
|
print("--dry-run: no changes written")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
if not needs_fix:
|
||||||
|
print("nothing to do")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
action = "invalidate" if invalidate_instead else "flag hand_authored=1"
|
||||||
|
print(f"applying: {action}")
|
||||||
|
|
||||||
|
cur = conn.cursor()
|
||||||
|
for row in needs_fix:
|
||||||
|
if invalidate_instead:
|
||||||
|
cur.execute(
|
||||||
|
"UPDATE entities SET status = 'invalid', "
|
||||||
|
"updated_at = CURRENT_TIMESTAMP WHERE id = ?",
|
||||||
|
(row["id"],),
|
||||||
|
)
|
||||||
|
cur.execute(
|
||||||
|
"INSERT INTO memory_audit "
|
||||||
|
"(id, memory_id, action, actor, before_json, after_json, note, entity_kind) "
|
||||||
|
"VALUES (?, ?, 'invalidated', 'v1_0_backfill', ?, ?, ?, 'entity')",
|
||||||
|
(
|
||||||
|
f"v10bf-{row['id'][:8]}-inv",
|
||||||
|
row["id"],
|
||||||
|
json.dumps({"status": row["status"]}),
|
||||||
|
json.dumps({"status": "invalid"}),
|
||||||
|
"V1-0 backfill: invalidated, no provenance",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
cur.execute(
|
||||||
|
"UPDATE entities SET hand_authored = 1, "
|
||||||
|
"updated_at = CURRENT_TIMESTAMP WHERE id = ?",
|
||||||
|
(row["id"],),
|
||||||
|
)
|
||||||
|
cur.execute(
|
||||||
|
"INSERT INTO memory_audit "
|
||||||
|
"(id, memory_id, action, actor, before_json, after_json, note, entity_kind) "
|
||||||
|
"VALUES (?, ?, 'hand_authored_flagged', 'v1_0_backfill', ?, ?, ?, 'entity')",
|
||||||
|
(
|
||||||
|
f"v10bf-{row['id'][:8]}-ha",
|
||||||
|
row["id"],
|
||||||
|
json.dumps({"hand_authored": False}),
|
||||||
|
json.dumps({"hand_authored": True}),
|
||||||
|
"V1-0 backfill: flagged hand_authored since source_refs empty",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
conn.commit()
|
||||||
|
print(f"done: updated {len(needs_fix)} entities")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
parser = argparse.ArgumentParser(description=__doc__)
|
||||||
|
parser.add_argument(
|
||||||
|
"--db",
|
||||||
|
type=Path,
|
||||||
|
default=Path("data/db/atocore.db"),
|
||||||
|
help="Path to the SQLite database (default: data/db/atocore.db)",
|
||||||
|
)
|
||||||
|
parser.add_argument("--dry-run", action="store_true", help="Report only; no writes")
|
||||||
|
parser.add_argument(
|
||||||
|
"--invalidate-instead",
|
||||||
|
action="store_true",
|
||||||
|
help=(
|
||||||
|
"DESTRUCTIVE. Invalidate active rows with no provenance instead "
|
||||||
|
"of flagging them hand_authored. Scoped to status='active' only "
|
||||||
|
"(superseded rows are left alone to preserve audit history). "
|
||||||
|
"Not recommended for first run — start with --dry-run, then "
|
||||||
|
"the default hand_authored flag path."
|
||||||
|
),
|
||||||
|
)
|
||||||
|
args = parser.parse_args()
|
||||||
|
return run(args.db, args.dry_run, args.invalidate_instead)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
File diff suppressed because it is too large
Load Diff
31
src/atocore/assets/__init__.py
Normal file
31
src/atocore/assets/__init__.py
Normal file
@@ -0,0 +1,31 @@
|
|||||||
|
"""Binary asset store (Issue F — visual evidence)."""
|
||||||
|
|
||||||
|
from atocore.assets.service import (
|
||||||
|
ALLOWED_MIME_TYPES,
|
||||||
|
Asset,
|
||||||
|
AssetError,
|
||||||
|
AssetNotFound,
|
||||||
|
AssetTooLarge,
|
||||||
|
AssetTypeNotAllowed,
|
||||||
|
get_asset,
|
||||||
|
get_asset_binary,
|
||||||
|
get_thumbnail,
|
||||||
|
invalidate_asset,
|
||||||
|
list_orphan_assets,
|
||||||
|
store_asset,
|
||||||
|
)
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"ALLOWED_MIME_TYPES",
|
||||||
|
"Asset",
|
||||||
|
"AssetError",
|
||||||
|
"AssetNotFound",
|
||||||
|
"AssetTooLarge",
|
||||||
|
"AssetTypeNotAllowed",
|
||||||
|
"get_asset",
|
||||||
|
"get_asset_binary",
|
||||||
|
"get_thumbnail",
|
||||||
|
"invalidate_asset",
|
||||||
|
"list_orphan_assets",
|
||||||
|
"store_asset",
|
||||||
|
]
|
||||||
367
src/atocore/assets/service.py
Normal file
367
src/atocore/assets/service.py
Normal file
@@ -0,0 +1,367 @@
|
|||||||
|
"""Binary asset storage with hash-dedup and on-demand thumbnails.
|
||||||
|
|
||||||
|
Issue F — visual evidence. Stores uploaded images / PDFs / CAD exports
|
||||||
|
under ``<assets_dir>/<hash[:2]>/<hash>.<ext>``. Re-uploads are idempotent
|
||||||
|
on SHA-256. Thumbnails are generated on first request and cached under
|
||||||
|
``<assets_dir>/.thumbnails/<size>/<hash>.jpg``.
|
||||||
|
|
||||||
|
Kept deliberately small: no authentication, no background jobs, no
|
||||||
|
image transformations beyond thumbnailing. Callers (API layer) own
|
||||||
|
MIME validation and size caps.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
import uuid
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
from io import BytesIO
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import atocore.config as _config
|
||||||
|
from atocore.models.database import get_connection
|
||||||
|
from atocore.observability.logger import get_logger
|
||||||
|
|
||||||
|
log = get_logger("assets")
|
||||||
|
|
||||||
|
|
||||||
|
# Whitelisted mime types. Start conservative; extend when a real use
|
||||||
|
# case lands rather than speculatively.
|
||||||
|
ALLOWED_MIME_TYPES: dict[str, str] = {
|
||||||
|
"image/png": "png",
|
||||||
|
"image/jpeg": "jpg",
|
||||||
|
"image/webp": "webp",
|
||||||
|
"image/gif": "gif",
|
||||||
|
"application/pdf": "pdf",
|
||||||
|
"model/step": "step",
|
||||||
|
"model/iges": "iges",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
class AssetError(Exception):
|
||||||
|
"""Base class for asset errors."""
|
||||||
|
|
||||||
|
|
||||||
|
class AssetTooLarge(AssetError):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
class AssetTypeNotAllowed(AssetError):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
class AssetNotFound(AssetError):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class Asset:
|
||||||
|
id: str
|
||||||
|
hash_sha256: str
|
||||||
|
mime_type: str
|
||||||
|
size_bytes: int
|
||||||
|
stored_path: str
|
||||||
|
width: int | None = None
|
||||||
|
height: int | None = None
|
||||||
|
original_filename: str = ""
|
||||||
|
project: str = ""
|
||||||
|
caption: str = ""
|
||||||
|
source_refs: list[str] = field(default_factory=list)
|
||||||
|
status: str = "active"
|
||||||
|
created_at: str = ""
|
||||||
|
updated_at: str = ""
|
||||||
|
|
||||||
|
def to_dict(self) -> dict:
|
||||||
|
return {
|
||||||
|
"id": self.id,
|
||||||
|
"hash_sha256": self.hash_sha256,
|
||||||
|
"mime_type": self.mime_type,
|
||||||
|
"size_bytes": self.size_bytes,
|
||||||
|
"width": self.width,
|
||||||
|
"height": self.height,
|
||||||
|
"stored_path": self.stored_path,
|
||||||
|
"original_filename": self.original_filename,
|
||||||
|
"project": self.project,
|
||||||
|
"caption": self.caption,
|
||||||
|
"source_refs": self.source_refs,
|
||||||
|
"status": self.status,
|
||||||
|
"created_at": self.created_at,
|
||||||
|
"updated_at": self.updated_at,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _assets_root() -> Path:
|
||||||
|
root = _config.settings.resolved_assets_dir
|
||||||
|
root.mkdir(parents=True, exist_ok=True)
|
||||||
|
return root
|
||||||
|
|
||||||
|
|
||||||
|
def _blob_path(hash_sha256: str, ext: str) -> Path:
|
||||||
|
root = _assets_root()
|
||||||
|
return root / hash_sha256[:2] / f"{hash_sha256}.{ext}"
|
||||||
|
|
||||||
|
|
||||||
|
def _thumbnails_root() -> Path:
|
||||||
|
return _assets_root() / ".thumbnails"
|
||||||
|
|
||||||
|
|
||||||
|
def _thumbnail_path(hash_sha256: str, size: int) -> Path:
|
||||||
|
return _thumbnails_root() / str(size) / f"{hash_sha256}.jpg"
|
||||||
|
|
||||||
|
|
||||||
|
def _image_dimensions(data: bytes, mime_type: str) -> tuple[int | None, int | None]:
|
||||||
|
if not mime_type.startswith("image/"):
|
||||||
|
return None, None
|
||||||
|
try:
|
||||||
|
from PIL import Image
|
||||||
|
except Exception:
|
||||||
|
return None, None
|
||||||
|
try:
|
||||||
|
with Image.open(BytesIO(data)) as img:
|
||||||
|
return img.width, img.height
|
||||||
|
except Exception as e:
|
||||||
|
log.warning("asset_dimension_probe_failed", error=str(e))
|
||||||
|
return None, None
|
||||||
|
|
||||||
|
|
||||||
|
def store_asset(
|
||||||
|
data: bytes,
|
||||||
|
mime_type: str,
|
||||||
|
original_filename: str = "",
|
||||||
|
project: str = "",
|
||||||
|
caption: str = "",
|
||||||
|
source_refs: list[str] | None = None,
|
||||||
|
) -> Asset:
|
||||||
|
"""Persist a binary blob and return the catalog row.
|
||||||
|
|
||||||
|
Idempotent on SHA-256 — a re-upload returns the existing asset row
|
||||||
|
without rewriting the blob or creating a duplicate catalog entry.
|
||||||
|
Caption / project / source_refs on re-upload are ignored; update
|
||||||
|
those via the owning entity's properties instead.
|
||||||
|
"""
|
||||||
|
max_bytes = _config.settings.assets_max_upload_bytes
|
||||||
|
if len(data) > max_bytes:
|
||||||
|
raise AssetTooLarge(
|
||||||
|
f"Upload is {len(data)} bytes; limit is {max_bytes} bytes"
|
||||||
|
)
|
||||||
|
if mime_type not in ALLOWED_MIME_TYPES:
|
||||||
|
raise AssetTypeNotAllowed(
|
||||||
|
f"mime_type {mime_type!r} not in allowlist. "
|
||||||
|
f"Allowed: {sorted(ALLOWED_MIME_TYPES)}"
|
||||||
|
)
|
||||||
|
|
||||||
|
hash_sha256 = hashlib.sha256(data).hexdigest()
|
||||||
|
ext = ALLOWED_MIME_TYPES[mime_type]
|
||||||
|
|
||||||
|
# Idempotency — if we already have this hash, return the existing row.
|
||||||
|
existing = _fetch_by_hash(hash_sha256)
|
||||||
|
if existing is not None:
|
||||||
|
log.info("asset_dedup_hit", asset_id=existing.id, hash=hash_sha256[:12])
|
||||||
|
return existing
|
||||||
|
|
||||||
|
width, height = _image_dimensions(data, mime_type)
|
||||||
|
|
||||||
|
blob_path = _blob_path(hash_sha256, ext)
|
||||||
|
blob_path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
blob_path.write_bytes(data)
|
||||||
|
|
||||||
|
asset_id = str(uuid.uuid4())
|
||||||
|
now = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S")
|
||||||
|
refs = source_refs or []
|
||||||
|
|
||||||
|
with get_connection() as conn:
|
||||||
|
conn.execute(
|
||||||
|
"""INSERT INTO assets
|
||||||
|
(id, hash_sha256, mime_type, size_bytes, width, height,
|
||||||
|
stored_path, original_filename, project, caption,
|
||||||
|
source_refs, status, created_at, updated_at)
|
||||||
|
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 'active', ?, ?)""",
|
||||||
|
(
|
||||||
|
asset_id, hash_sha256, mime_type, len(data), width, height,
|
||||||
|
str(blob_path), original_filename, project, caption,
|
||||||
|
json.dumps(refs), now, now,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
log.info(
|
||||||
|
"asset_stored", asset_id=asset_id, hash=hash_sha256[:12],
|
||||||
|
mime_type=mime_type, size_bytes=len(data),
|
||||||
|
)
|
||||||
|
return Asset(
|
||||||
|
id=asset_id, hash_sha256=hash_sha256, mime_type=mime_type,
|
||||||
|
size_bytes=len(data), width=width, height=height,
|
||||||
|
stored_path=str(blob_path), original_filename=original_filename,
|
||||||
|
project=project, caption=caption, source_refs=refs,
|
||||||
|
status="active", created_at=now, updated_at=now,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _fetch_by_hash(hash_sha256: str) -> Asset | None:
|
||||||
|
with get_connection() as conn:
|
||||||
|
row = conn.execute(
|
||||||
|
"SELECT * FROM assets WHERE hash_sha256 = ? AND status != 'invalid'",
|
||||||
|
(hash_sha256,),
|
||||||
|
).fetchone()
|
||||||
|
return _row_to_asset(row) if row else None
|
||||||
|
|
||||||
|
|
||||||
|
def get_asset(asset_id: str) -> Asset | None:
|
||||||
|
with get_connection() as conn:
|
||||||
|
row = conn.execute(
|
||||||
|
"SELECT * FROM assets WHERE id = ?", (asset_id,)
|
||||||
|
).fetchone()
|
||||||
|
return _row_to_asset(row) if row else None
|
||||||
|
|
||||||
|
|
||||||
|
def get_asset_binary(asset_id: str) -> tuple[Asset, bytes]:
|
||||||
|
"""Return (metadata, raw bytes). Raises AssetNotFound."""
|
||||||
|
asset = get_asset(asset_id)
|
||||||
|
if asset is None or asset.status == "invalid":
|
||||||
|
raise AssetNotFound(f"Asset not found: {asset_id}")
|
||||||
|
path = Path(asset.stored_path)
|
||||||
|
if not path.exists():
|
||||||
|
raise AssetNotFound(
|
||||||
|
f"Asset {asset_id} row exists but blob is missing at {path}"
|
||||||
|
)
|
||||||
|
return asset, path.read_bytes()
|
||||||
|
|
||||||
|
|
||||||
|
def get_thumbnail(asset_id: str, size: int = 240) -> tuple[Asset, bytes]:
|
||||||
|
"""Return (metadata, thumbnail JPEG bytes).
|
||||||
|
|
||||||
|
Thumbnails are only generated for image mime types. For non-images
|
||||||
|
the caller should render a placeholder instead. Generated thumbs
|
||||||
|
are cached on disk at ``<assets_dir>/.thumbnails/<size>/<hash>.jpg``.
|
||||||
|
"""
|
||||||
|
asset = get_asset(asset_id)
|
||||||
|
if asset is None or asset.status == "invalid":
|
||||||
|
raise AssetNotFound(f"Asset not found: {asset_id}")
|
||||||
|
if not asset.mime_type.startswith("image/"):
|
||||||
|
raise AssetError(
|
||||||
|
f"Thumbnails are only supported for images; "
|
||||||
|
f"{asset.mime_type!r} is not an image"
|
||||||
|
)
|
||||||
|
|
||||||
|
size = max(16, min(int(size), 2048))
|
||||||
|
thumb_path = _thumbnail_path(asset.hash_sha256, size)
|
||||||
|
if thumb_path.exists():
|
||||||
|
return asset, thumb_path.read_bytes()
|
||||||
|
|
||||||
|
try:
|
||||||
|
from PIL import Image
|
||||||
|
except Exception as e:
|
||||||
|
raise AssetError(f"Pillow not available for thumbnailing: {e}")
|
||||||
|
|
||||||
|
src_path = Path(asset.stored_path)
|
||||||
|
if not src_path.exists():
|
||||||
|
raise AssetNotFound(
|
||||||
|
f"Asset {asset_id} row exists but blob is missing at {src_path}"
|
||||||
|
)
|
||||||
|
|
||||||
|
thumb_path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
with Image.open(src_path) as img:
|
||||||
|
img = img.convert("RGB") if img.mode not in ("RGB", "L") else img
|
||||||
|
img.thumbnail((size, size))
|
||||||
|
buf = BytesIO()
|
||||||
|
img.save(buf, format="JPEG", quality=85, optimize=True)
|
||||||
|
jpeg_bytes = buf.getvalue()
|
||||||
|
thumb_path.write_bytes(jpeg_bytes)
|
||||||
|
return asset, jpeg_bytes
|
||||||
|
|
||||||
|
|
||||||
|
def list_orphan_assets(limit: int = 200) -> list[Asset]:
|
||||||
|
"""Assets not referenced by any active entity or memory.
|
||||||
|
|
||||||
|
"Referenced" means: an active entity has ``properties.asset_id``
|
||||||
|
pointing at this asset, OR any active entity / memory's
|
||||||
|
source_refs contains ``asset:<id>``.
|
||||||
|
"""
|
||||||
|
with get_connection() as conn:
|
||||||
|
asset_rows = conn.execute(
|
||||||
|
"SELECT * FROM assets WHERE status = 'active' "
|
||||||
|
"ORDER BY created_at DESC LIMIT ?",
|
||||||
|
(min(limit, 1000),),
|
||||||
|
).fetchall()
|
||||||
|
|
||||||
|
entities_with_asset = set()
|
||||||
|
rows = conn.execute(
|
||||||
|
"SELECT properties, source_refs FROM entities "
|
||||||
|
"WHERE status = 'active'"
|
||||||
|
).fetchall()
|
||||||
|
for r in rows:
|
||||||
|
try:
|
||||||
|
props = json.loads(r["properties"] or "{}")
|
||||||
|
aid = props.get("asset_id")
|
||||||
|
if aid:
|
||||||
|
entities_with_asset.add(aid)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
try:
|
||||||
|
refs = json.loads(r["source_refs"] or "[]")
|
||||||
|
for ref in refs:
|
||||||
|
if isinstance(ref, str) and ref.startswith("asset:"):
|
||||||
|
entities_with_asset.add(ref.split(":", 1)[1])
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
# Memories don't have a properties dict, but source_refs may carry
|
||||||
|
# asset:<id> after Issue F lands for memory-level evidence.
|
||||||
|
# The memories table has no source_refs column today — skip here
|
||||||
|
# and extend once that lands.
|
||||||
|
|
||||||
|
return [
|
||||||
|
_row_to_asset(r)
|
||||||
|
for r in asset_rows
|
||||||
|
if r["id"] not in entities_with_asset
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def invalidate_asset(asset_id: str, actor: str = "api", note: str = "") -> bool:
|
||||||
|
"""Tombstone an asset. No-op if still referenced.
|
||||||
|
|
||||||
|
Returns True on success, False if the asset is missing or still
|
||||||
|
referenced by an active entity (caller should get a 409 in that
|
||||||
|
case). The blob file stays on disk until a future gc pass sweeps
|
||||||
|
orphaned blobs — this function only flips the catalog status.
|
||||||
|
"""
|
||||||
|
asset = get_asset(asset_id)
|
||||||
|
if asset is None:
|
||||||
|
return False
|
||||||
|
orphans = list_orphan_assets(limit=1000)
|
||||||
|
if asset.id not in {o.id for o in orphans} and asset.status == "active":
|
||||||
|
log.info("asset_invalidate_blocked_referenced", asset_id=asset_id)
|
||||||
|
return False
|
||||||
|
|
||||||
|
now = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S")
|
||||||
|
with get_connection() as conn:
|
||||||
|
conn.execute(
|
||||||
|
"UPDATE assets SET status = 'invalid', updated_at = ? WHERE id = ?",
|
||||||
|
(now, asset_id),
|
||||||
|
)
|
||||||
|
log.info("asset_invalidated", asset_id=asset_id, actor=actor, note=note[:80])
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
def _row_to_asset(row) -> Asset:
|
||||||
|
try:
|
||||||
|
refs = json.loads(row["source_refs"] or "[]")
|
||||||
|
except Exception:
|
||||||
|
refs = []
|
||||||
|
return Asset(
|
||||||
|
id=row["id"],
|
||||||
|
hash_sha256=row["hash_sha256"],
|
||||||
|
mime_type=row["mime_type"],
|
||||||
|
size_bytes=row["size_bytes"],
|
||||||
|
width=row["width"],
|
||||||
|
height=row["height"],
|
||||||
|
stored_path=row["stored_path"],
|
||||||
|
original_filename=row["original_filename"] or "",
|
||||||
|
project=row["project"] or "",
|
||||||
|
caption=row["caption"] or "",
|
||||||
|
source_refs=refs,
|
||||||
|
status=row["status"],
|
||||||
|
created_at=row["created_at"] or "",
|
||||||
|
updated_at=row["updated_at"] or "",
|
||||||
|
)
|
||||||
@@ -22,6 +22,8 @@ class Settings(BaseSettings):
|
|||||||
backup_dir: Path = Path("./backups")
|
backup_dir: Path = Path("./backups")
|
||||||
run_dir: Path = Path("./run")
|
run_dir: Path = Path("./run")
|
||||||
project_registry_path: Path = Path("./config/project-registry.json")
|
project_registry_path: Path = Path("./config/project-registry.json")
|
||||||
|
assets_dir: Path | None = None
|
||||||
|
assets_max_upload_bytes: int = 20 * 1024 * 1024 # 20 MB per upload
|
||||||
host: str = "127.0.0.1"
|
host: str = "127.0.0.1"
|
||||||
port: int = 8100
|
port: int = 8100
|
||||||
db_busy_timeout_ms: int = 5000
|
db_busy_timeout_ms: int = 5000
|
||||||
@@ -76,6 +78,10 @@ class Settings(BaseSettings):
|
|||||||
def resolved_data_dir(self) -> Path:
|
def resolved_data_dir(self) -> Path:
|
||||||
return self._resolve_path(self.data_dir)
|
return self._resolve_path(self.data_dir)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def resolved_assets_dir(self) -> Path:
|
||||||
|
return self._resolve_path(self.assets_dir or (self.resolved_data_dir / "assets"))
|
||||||
|
|
||||||
@property
|
@property
|
||||||
def resolved_db_dir(self) -> Path:
|
def resolved_db_dir(self) -> Path:
|
||||||
return self._resolve_path(self.db_dir or (self.resolved_data_dir / "db"))
|
return self._resolve_path(self.db_dir or (self.resolved_data_dir / "db"))
|
||||||
@@ -132,6 +138,7 @@ class Settings(BaseSettings):
|
|||||||
self.resolved_backup_dir,
|
self.resolved_backup_dir,
|
||||||
self.resolved_run_dir,
|
self.resolved_run_dir,
|
||||||
self.resolved_project_registry_path.parent,
|
self.resolved_project_registry_path.parent,
|
||||||
|
self.resolved_assets_dir,
|
||||||
]
|
]
|
||||||
|
|
||||||
@property
|
@property
|
||||||
|
|||||||
@@ -29,6 +29,10 @@ ENTITY_TYPES = [
|
|||||||
"validation_claim",
|
"validation_claim",
|
||||||
"vendor",
|
"vendor",
|
||||||
"process",
|
"process",
|
||||||
|
# Issue F (visual evidence): images, PDFs, CAD exports attached to
|
||||||
|
# other entities via EVIDENCED_BY. properties carries kind +
|
||||||
|
# asset_id + caption + capture_context.
|
||||||
|
"artifact",
|
||||||
]
|
]
|
||||||
|
|
||||||
RELATIONSHIP_TYPES = [
|
RELATIONSHIP_TYPES = [
|
||||||
@@ -59,6 +63,12 @@ RELATIONSHIP_TYPES = [
|
|||||||
|
|
||||||
ENTITY_STATUSES = ["candidate", "active", "superseded", "invalid"]
|
ENTITY_STATUSES = ["candidate", "active", "superseded", "invalid"]
|
||||||
|
|
||||||
|
# V1-0: extractor version this module writes into new entity rows.
|
||||||
|
# Per promotion-rules.md:268, every candidate must record the version of
|
||||||
|
# the extractor that produced it so later re-evaluation is auditable.
|
||||||
|
# Bump this when extraction logic materially changes.
|
||||||
|
EXTRACTOR_VERSION = "v1.0.0"
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
class Entity:
|
class Entity:
|
||||||
@@ -73,6 +83,10 @@ class Entity:
|
|||||||
source_refs: list[str] = field(default_factory=list)
|
source_refs: list[str] = field(default_factory=list)
|
||||||
created_at: str = ""
|
created_at: str = ""
|
||||||
updated_at: str = ""
|
updated_at: str = ""
|
||||||
|
# V1-0 shared-header fields per engineering-v1-acceptance.md:45.
|
||||||
|
extractor_version: str = ""
|
||||||
|
canonical_home: str = "entity"
|
||||||
|
hand_authored: bool = False
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
@@ -99,10 +113,25 @@ def init_engineering_schema() -> None:
|
|||||||
status TEXT NOT NULL DEFAULT 'active',
|
status TEXT NOT NULL DEFAULT 'active',
|
||||||
confidence REAL NOT NULL DEFAULT 1.0,
|
confidence REAL NOT NULL DEFAULT 1.0,
|
||||||
source_refs TEXT NOT NULL DEFAULT '[]',
|
source_refs TEXT NOT NULL DEFAULT '[]',
|
||||||
|
extractor_version TEXT NOT NULL DEFAULT '',
|
||||||
|
canonical_home TEXT NOT NULL DEFAULT 'entity',
|
||||||
|
hand_authored INTEGER NOT NULL DEFAULT 0,
|
||||||
created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||||
updated_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP
|
updated_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP
|
||||||
)
|
)
|
||||||
""")
|
""")
|
||||||
|
# V1-0 (Engineering V1 completion): the three shared-header fields
|
||||||
|
# per engineering-v1-acceptance.md:45. Idempotent ALTERs for
|
||||||
|
# databases created before V1-0 land these columns without a full
|
||||||
|
# migration. Fresh DBs get them via the CREATE TABLE above; the
|
||||||
|
# ALTERs below are a no-op there.
|
||||||
|
from atocore.models.database import _column_exists # late import; avoids cycle
|
||||||
|
if not _column_exists(conn, "entities", "extractor_version"):
|
||||||
|
conn.execute("ALTER TABLE entities ADD COLUMN extractor_version TEXT DEFAULT ''")
|
||||||
|
if not _column_exists(conn, "entities", "canonical_home"):
|
||||||
|
conn.execute("ALTER TABLE entities ADD COLUMN canonical_home TEXT DEFAULT 'entity'")
|
||||||
|
if not _column_exists(conn, "entities", "hand_authored"):
|
||||||
|
conn.execute("ALTER TABLE entities ADD COLUMN hand_authored INTEGER DEFAULT 0")
|
||||||
conn.execute("""
|
conn.execute("""
|
||||||
CREATE TABLE IF NOT EXISTS relationships (
|
CREATE TABLE IF NOT EXISTS relationships (
|
||||||
id TEXT PRIMARY KEY,
|
id TEXT PRIMARY KEY,
|
||||||
@@ -145,6 +174,8 @@ def create_entity(
|
|||||||
confidence: float = 1.0,
|
confidence: float = 1.0,
|
||||||
source_refs: list[str] | None = None,
|
source_refs: list[str] | None = None,
|
||||||
actor: str = "api",
|
actor: str = "api",
|
||||||
|
hand_authored: bool = False,
|
||||||
|
extractor_version: str | None = None,
|
||||||
) -> Entity:
|
) -> Entity:
|
||||||
if entity_type not in ENTITY_TYPES:
|
if entity_type not in ENTITY_TYPES:
|
||||||
raise ValueError(f"Invalid entity type: {entity_type}. Must be one of {ENTITY_TYPES}")
|
raise ValueError(f"Invalid entity type: {entity_type}. Must be one of {ENTITY_TYPES}")
|
||||||
@@ -153,6 +184,21 @@ def create_entity(
|
|||||||
if not name or not name.strip():
|
if not name or not name.strip():
|
||||||
raise ValueError("Entity name must be non-empty")
|
raise ValueError("Entity name must be non-empty")
|
||||||
|
|
||||||
|
refs = list(source_refs) if source_refs else []
|
||||||
|
|
||||||
|
# V1-0 (F-8 provenance enforcement, engineering-v1-acceptance.md:147):
|
||||||
|
# every new entity row must carry non-empty source_refs OR be explicitly
|
||||||
|
# flagged hand_authored. This is the non-negotiable invariant every
|
||||||
|
# later V1 phase depends on — without it, active entities can escape
|
||||||
|
# into the graph with no traceable origin. Raises at the write seam so
|
||||||
|
# the bug is impossible to introduce silently.
|
||||||
|
if not refs and not hand_authored:
|
||||||
|
raise ValueError(
|
||||||
|
"source_refs required: every entity must carry provenance "
|
||||||
|
"(source_chunk_id / source_interaction_id / kb_cad_export_id / ...) "
|
||||||
|
"or set hand_authored=True to explicitly flag a direct human write"
|
||||||
|
)
|
||||||
|
|
||||||
# Phase 5: enforce project canonicalization contract at the write seam.
|
# Phase 5: enforce project canonicalization contract at the write seam.
|
||||||
# Aliases like "p04" become "p04-gigabit" so downstream reads stay
|
# Aliases like "p04" become "p04-gigabit" so downstream reads stay
|
||||||
# consistent with the registry.
|
# consistent with the registry.
|
||||||
@@ -161,18 +207,22 @@ def create_entity(
|
|||||||
entity_id = str(uuid.uuid4())
|
entity_id = str(uuid.uuid4())
|
||||||
now = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S")
|
now = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S")
|
||||||
props = properties or {}
|
props = properties or {}
|
||||||
refs = source_refs or []
|
ev = extractor_version if extractor_version is not None else EXTRACTOR_VERSION
|
||||||
|
|
||||||
with get_connection() as conn:
|
with get_connection() as conn:
|
||||||
conn.execute(
|
conn.execute(
|
||||||
"""INSERT INTO entities
|
"""INSERT INTO entities
|
||||||
(id, entity_type, name, project, description, properties,
|
(id, entity_type, name, project, description, properties,
|
||||||
status, confidence, source_refs, created_at, updated_at)
|
status, confidence, source_refs,
|
||||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
|
extractor_version, canonical_home, hand_authored,
|
||||||
|
created_at, updated_at)
|
||||||
|
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
|
||||||
(
|
(
|
||||||
entity_id, entity_type, name.strip(), project,
|
entity_id, entity_type, name.strip(), project,
|
||||||
description, json.dumps(props), status, confidence,
|
description, json.dumps(props), status, confidence,
|
||||||
json.dumps(refs), now, now,
|
json.dumps(refs),
|
||||||
|
ev, "entity", 1 if hand_authored else 0,
|
||||||
|
now, now,
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -190,14 +240,31 @@ def create_entity(
|
|||||||
"project": project,
|
"project": project,
|
||||||
"status": status,
|
"status": status,
|
||||||
"confidence": confidence,
|
"confidence": confidence,
|
||||||
|
"hand_authored": hand_authored,
|
||||||
|
"extractor_version": ev,
|
||||||
},
|
},
|
||||||
)
|
)
|
||||||
|
|
||||||
|
# V1-0 (F-5 hook, engineering-v1-acceptance.md:99): synchronous
|
||||||
|
# conflict detection on any active-entity write. The promote path
|
||||||
|
# already had this hook (see promote_entity below); V1-0 adds it to
|
||||||
|
# direct-active creates so every active row — however it got that
|
||||||
|
# way — is checked. Fail-open per "flag, never block" rule in
|
||||||
|
# conflict-model.md:256: detector errors log but never fail the write.
|
||||||
|
if status == "active":
|
||||||
|
try:
|
||||||
|
from atocore.engineering.conflicts import detect_conflicts_for_entity
|
||||||
|
detect_conflicts_for_entity(entity_id)
|
||||||
|
except Exception as e:
|
||||||
|
log.warning("conflict_detection_failed", entity_id=entity_id, error=str(e))
|
||||||
|
|
||||||
return Entity(
|
return Entity(
|
||||||
id=entity_id, entity_type=entity_type, name=name.strip(),
|
id=entity_id, entity_type=entity_type, name=name.strip(),
|
||||||
project=project, description=description, properties=props,
|
project=project, description=description, properties=props,
|
||||||
status=status, confidence=confidence, source_refs=refs,
|
status=status, confidence=confidence, source_refs=refs,
|
||||||
created_at=now, updated_at=now,
|
created_at=now, updated_at=now,
|
||||||
|
extractor_version=ev, canonical_home="entity",
|
||||||
|
hand_authored=hand_authored,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -333,9 +400,20 @@ def _set_entity_status(
|
|||||||
return True
|
return True
|
||||||
|
|
||||||
|
|
||||||
def promote_entity(entity_id: str, actor: str = "api", note: str = "") -> bool:
|
def promote_entity(
|
||||||
|
entity_id: str,
|
||||||
|
actor: str = "api",
|
||||||
|
note: str = "",
|
||||||
|
target_project: str | None = None,
|
||||||
|
) -> bool:
|
||||||
"""Promote a candidate entity to active.
|
"""Promote a candidate entity to active.
|
||||||
|
|
||||||
|
When ``target_project`` is provided (Issue C), also retarget the
|
||||||
|
entity's project before flipping the status. Use this to graduate an
|
||||||
|
inbox/global lead into a real project (e.g. when a vendor quote
|
||||||
|
becomes a contract). ``target_project`` is canonicalized through the
|
||||||
|
registry; reserved ids (``inbox``) and ``""`` are accepted verbatim.
|
||||||
|
|
||||||
Phase 5F graduation hook: if this entity has source_refs pointing at
|
Phase 5F graduation hook: if this entity has source_refs pointing at
|
||||||
memories (format "memory:<uuid>"), mark those source memories as
|
memories (format "memory:<uuid>"), mark those source memories as
|
||||||
``status=graduated`` and set their ``graduated_to_entity_id`` forward
|
``status=graduated`` and set their ``graduated_to_entity_id`` forward
|
||||||
@@ -346,6 +424,41 @@ def promote_entity(entity_id: str, actor: str = "api", note: str = "") -> bool:
|
|||||||
if entity is None or entity.status != "candidate":
|
if entity is None or entity.status != "candidate":
|
||||||
return False
|
return False
|
||||||
|
|
||||||
|
# V1-0 (F-8 provenance re-check at promote). The invariant must hold at
|
||||||
|
# BOTH create_entity AND promote_entity per the plan, because candidate
|
||||||
|
# rows can exist in the DB from before V1-0 (no enforcement at their
|
||||||
|
# create time) or can be inserted by code paths that bypass the service
|
||||||
|
# layer. Block any candidate with empty source_refs that is NOT flagged
|
||||||
|
# hand_authored from ever becoming active. Same error shape as the
|
||||||
|
# create-side check for symmetry.
|
||||||
|
if not (entity.source_refs or []) and not entity.hand_authored:
|
||||||
|
raise ValueError(
|
||||||
|
"source_refs required: cannot promote a candidate with no "
|
||||||
|
"provenance. Attach source_refs via PATCH /entities/{id}, "
|
||||||
|
"or flag hand_authored=true before promoting."
|
||||||
|
)
|
||||||
|
|
||||||
|
if target_project is not None:
|
||||||
|
new_project = (
|
||||||
|
resolve_project_name(target_project) if target_project else ""
|
||||||
|
)
|
||||||
|
if new_project != entity.project:
|
||||||
|
now = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S")
|
||||||
|
with get_connection() as conn:
|
||||||
|
conn.execute(
|
||||||
|
"UPDATE entities SET project = ?, updated_at = ? "
|
||||||
|
"WHERE id = ?",
|
||||||
|
(new_project, now, entity_id),
|
||||||
|
)
|
||||||
|
_audit_entity(
|
||||||
|
entity_id=entity_id,
|
||||||
|
action="retargeted",
|
||||||
|
actor=actor,
|
||||||
|
before={"project": entity.project},
|
||||||
|
after={"project": new_project},
|
||||||
|
note=note,
|
||||||
|
)
|
||||||
|
|
||||||
ok = _set_entity_status(entity_id, "active", actor=actor, note=note)
|
ok = _set_entity_status(entity_id, "active", actor=actor, note=note)
|
||||||
if not ok:
|
if not ok:
|
||||||
return False
|
return False
|
||||||
@@ -426,9 +539,192 @@ def reject_entity_candidate(entity_id: str, actor: str = "api", note: str = "")
|
|||||||
return _set_entity_status(entity_id, "invalid", actor=actor, note=note)
|
return _set_entity_status(entity_id, "invalid", actor=actor, note=note)
|
||||||
|
|
||||||
|
|
||||||
def supersede_entity(entity_id: str, actor: str = "api", note: str = "") -> bool:
|
def supersede_entity(
|
||||||
"""Mark an active entity as superseded by a newer one."""
|
entity_id: str,
|
||||||
return _set_entity_status(entity_id, "superseded", actor=actor, note=note)
|
actor: str = "api",
|
||||||
|
note: str = "",
|
||||||
|
superseded_by: str | None = None,
|
||||||
|
) -> bool:
|
||||||
|
"""Mark an active entity as superseded.
|
||||||
|
|
||||||
|
When ``superseded_by`` names a real entity, also create a
|
||||||
|
``supersedes`` relationship from the new entity to the old one
|
||||||
|
(semantics: ``new SUPERSEDES old``). This keeps the graph
|
||||||
|
navigable without the caller remembering to make that edge.
|
||||||
|
"""
|
||||||
|
if superseded_by:
|
||||||
|
new_entity = get_entity(superseded_by)
|
||||||
|
if new_entity is None:
|
||||||
|
raise ValueError(
|
||||||
|
f"superseded_by entity not found: {superseded_by}"
|
||||||
|
)
|
||||||
|
if new_entity.id == entity_id:
|
||||||
|
raise ValueError("entity cannot supersede itself")
|
||||||
|
|
||||||
|
ok = _set_entity_status(entity_id, "superseded", actor=actor, note=note)
|
||||||
|
if not ok:
|
||||||
|
return False
|
||||||
|
|
||||||
|
if superseded_by:
|
||||||
|
try:
|
||||||
|
create_relationship(
|
||||||
|
source_entity_id=superseded_by,
|
||||||
|
target_entity_id=entity_id,
|
||||||
|
relationship_type="supersedes",
|
||||||
|
source_refs=[f"supersede-api:{actor}"],
|
||||||
|
)
|
||||||
|
except Exception as e:
|
||||||
|
log.warning(
|
||||||
|
"supersede_relationship_create_failed",
|
||||||
|
entity_id=entity_id,
|
||||||
|
superseded_by=superseded_by,
|
||||||
|
error=str(e),
|
||||||
|
)
|
||||||
|
|
||||||
|
# V1-0 (F-5 hook on supersede, per plan's "every active-entity
|
||||||
|
# write path"). Supersede demotes `entity_id` AND adds a
|
||||||
|
# `supersedes` relationship rooted at the already-active
|
||||||
|
# `superseded_by`. That new edge can create a conflict the
|
||||||
|
# detector should catch synchronously. Fail-open per
|
||||||
|
# conflict-model.md:256.
|
||||||
|
try:
|
||||||
|
from atocore.engineering.conflicts import detect_conflicts_for_entity
|
||||||
|
detect_conflicts_for_entity(superseded_by)
|
||||||
|
except Exception as e:
|
||||||
|
log.warning(
|
||||||
|
"conflict_detection_failed",
|
||||||
|
entity_id=superseded_by,
|
||||||
|
error=str(e),
|
||||||
|
)
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
def invalidate_active_entity(
|
||||||
|
entity_id: str,
|
||||||
|
actor: str = "api",
|
||||||
|
reason: str = "",
|
||||||
|
) -> tuple[bool, str]:
|
||||||
|
"""Mark an active entity as invalid (Issue E — retraction path).
|
||||||
|
|
||||||
|
Returns (success, status_code) where status_code is one of:
|
||||||
|
- "invalidated" — happy path
|
||||||
|
- "not_found" — no such entity
|
||||||
|
- "already_invalid" — already invalid (idempotent)
|
||||||
|
- "not_active" — entity is candidate/superseded; use the
|
||||||
|
appropriate other endpoint
|
||||||
|
|
||||||
|
This is the public retraction API distinct from
|
||||||
|
``reject_entity_candidate`` (which only handles candidate→invalid).
|
||||||
|
"""
|
||||||
|
entity = get_entity(entity_id)
|
||||||
|
if entity is None:
|
||||||
|
return False, "not_found"
|
||||||
|
if entity.status == "invalid":
|
||||||
|
return True, "already_invalid"
|
||||||
|
if entity.status != "active":
|
||||||
|
return False, "not_active"
|
||||||
|
ok = _set_entity_status(entity_id, "invalid", actor=actor, note=reason)
|
||||||
|
return ok, "invalidated" if ok else "not_active"
|
||||||
|
|
||||||
|
|
||||||
|
def update_entity(
|
||||||
|
entity_id: str,
|
||||||
|
*,
|
||||||
|
description: str | None = None,
|
||||||
|
properties_patch: dict | None = None,
|
||||||
|
confidence: float | None = None,
|
||||||
|
append_source_refs: list[str] | None = None,
|
||||||
|
actor: str = "api",
|
||||||
|
note: str = "",
|
||||||
|
) -> Entity | None:
|
||||||
|
"""Update mutable fields on an existing entity (Issue E follow-up).
|
||||||
|
|
||||||
|
Field rules (kept narrow on purpose):
|
||||||
|
|
||||||
|
- ``description``: replaces the current value when provided.
|
||||||
|
- ``properties_patch``: merged into the existing ``properties`` dict,
|
||||||
|
shallow. Pass ``None`` as a value to delete a key; pass a new
|
||||||
|
value to overwrite it.
|
||||||
|
- ``confidence``: replaces when provided. Must be in [0, 1].
|
||||||
|
- ``append_source_refs``: appended verbatim to the existing list
|
||||||
|
(duplicates are filtered out, order preserved).
|
||||||
|
|
||||||
|
What you cannot change via this path:
|
||||||
|
|
||||||
|
- ``entity_type`` — requires supersede+create (a new type is a new
|
||||||
|
thing).
|
||||||
|
- ``project`` — use ``promote_entity`` with ``target_project`` for
|
||||||
|
inbox→project graduation, or supersede+create for anything else.
|
||||||
|
- ``name`` — renames are destructive to cross-references;
|
||||||
|
supersede+create.
|
||||||
|
- ``status`` — use the dedicated promote/reject/invalidate/supersede
|
||||||
|
endpoints.
|
||||||
|
|
||||||
|
Returns the updated entity, or None if no such entity exists.
|
||||||
|
"""
|
||||||
|
entity = get_entity(entity_id)
|
||||||
|
if entity is None:
|
||||||
|
return None
|
||||||
|
if confidence is not None and not (0.0 <= confidence <= 1.0):
|
||||||
|
raise ValueError("confidence must be in [0, 1]")
|
||||||
|
|
||||||
|
before = {
|
||||||
|
"description": entity.description,
|
||||||
|
"properties": dict(entity.properties or {}),
|
||||||
|
"confidence": entity.confidence,
|
||||||
|
"source_refs": list(entity.source_refs or []),
|
||||||
|
}
|
||||||
|
|
||||||
|
new_description = entity.description if description is None else description
|
||||||
|
new_confidence = entity.confidence if confidence is None else confidence
|
||||||
|
new_properties = dict(entity.properties or {})
|
||||||
|
if properties_patch:
|
||||||
|
for key, value in properties_patch.items():
|
||||||
|
if value is None:
|
||||||
|
new_properties.pop(key, None)
|
||||||
|
else:
|
||||||
|
new_properties[key] = value
|
||||||
|
new_refs = list(entity.source_refs or [])
|
||||||
|
if append_source_refs:
|
||||||
|
existing = set(new_refs)
|
||||||
|
for ref in append_source_refs:
|
||||||
|
if ref and ref not in existing:
|
||||||
|
new_refs.append(ref)
|
||||||
|
existing.add(ref)
|
||||||
|
|
||||||
|
now = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S")
|
||||||
|
with get_connection() as conn:
|
||||||
|
conn.execute(
|
||||||
|
"""UPDATE entities
|
||||||
|
SET description = ?, properties = ?, confidence = ?,
|
||||||
|
source_refs = ?, updated_at = ?
|
||||||
|
WHERE id = ?""",
|
||||||
|
(
|
||||||
|
new_description,
|
||||||
|
json.dumps(new_properties),
|
||||||
|
new_confidence,
|
||||||
|
json.dumps(new_refs),
|
||||||
|
now,
|
||||||
|
entity_id,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
after = {
|
||||||
|
"description": new_description,
|
||||||
|
"properties": new_properties,
|
||||||
|
"confidence": new_confidence,
|
||||||
|
"source_refs": new_refs,
|
||||||
|
}
|
||||||
|
_audit_entity(
|
||||||
|
entity_id=entity_id,
|
||||||
|
action="updated",
|
||||||
|
actor=actor,
|
||||||
|
before=before,
|
||||||
|
after=after,
|
||||||
|
note=note,
|
||||||
|
)
|
||||||
|
log.info("entity_updated", entity_id=entity_id, actor=actor)
|
||||||
|
return get_entity(entity_id)
|
||||||
|
|
||||||
|
|
||||||
def get_entity_audit(entity_id: str, limit: int = 100) -> list[dict]:
|
def get_entity_audit(entity_id: str, limit: int = 100) -> list[dict]:
|
||||||
@@ -470,7 +766,24 @@ def get_entities(
|
|||||||
status: str = "active",
|
status: str = "active",
|
||||||
name_contains: str | None = None,
|
name_contains: str | None = None,
|
||||||
limit: int = 100,
|
limit: int = 100,
|
||||||
|
scope_only: bool = False,
|
||||||
) -> list[Entity]:
|
) -> list[Entity]:
|
||||||
|
"""List entities with optional filters.
|
||||||
|
|
||||||
|
Project scoping rules (Issue C — inbox + cross-project):
|
||||||
|
|
||||||
|
- ``project=None``: no project filter, return everything matching status.
|
||||||
|
- ``project=""``: return only cross-project (global) entities.
|
||||||
|
- ``project="inbox"``: return only inbox entities.
|
||||||
|
- ``project="<real>"`` and ``scope_only=False`` (default): return entities
|
||||||
|
scoped to that project PLUS cross-project (``project=""``) entities.
|
||||||
|
- ``project="<real>"`` and ``scope_only=True``: return only that project,
|
||||||
|
without the cross-project bleed.
|
||||||
|
"""
|
||||||
|
from atocore.projects.registry import (
|
||||||
|
INBOX_PROJECT, GLOBAL_PROJECT, is_reserved_project,
|
||||||
|
)
|
||||||
|
|
||||||
query = "SELECT * FROM entities WHERE status = ?"
|
query = "SELECT * FROM entities WHERE status = ?"
|
||||||
params: list = [status]
|
params: list = [status]
|
||||||
|
|
||||||
@@ -478,8 +791,14 @@ def get_entities(
|
|||||||
query += " AND entity_type = ?"
|
query += " AND entity_type = ?"
|
||||||
params.append(entity_type)
|
params.append(entity_type)
|
||||||
if project is not None:
|
if project is not None:
|
||||||
|
p = (project or "").strip()
|
||||||
|
if p == GLOBAL_PROJECT or is_reserved_project(p) or scope_only:
|
||||||
query += " AND project = ?"
|
query += " AND project = ?"
|
||||||
params.append(project)
|
params.append(p)
|
||||||
|
else:
|
||||||
|
# Real project — include cross-project entities by default.
|
||||||
|
query += " AND (project = ? OR project = ?)"
|
||||||
|
params.extend([p, GLOBAL_PROJECT])
|
||||||
if name_contains:
|
if name_contains:
|
||||||
query += " AND name LIKE ?"
|
query += " AND name LIKE ?"
|
||||||
params.append(f"%{name_contains}%")
|
params.append(f"%{name_contains}%")
|
||||||
@@ -548,6 +867,15 @@ def get_entity_with_context(entity_id: str) -> dict | None:
|
|||||||
|
|
||||||
|
|
||||||
def _row_to_entity(row) -> Entity:
|
def _row_to_entity(row) -> Entity:
|
||||||
|
# V1-0 shared-header fields are optional on read — rows that predate
|
||||||
|
# V1-0 migration have NULL / missing values, so defaults kick in and
|
||||||
|
# older tests that build Entity() without the new fields keep passing.
|
||||||
|
# `row.keys()` lets us tolerate SQLite rows that lack the columns
|
||||||
|
# entirely (pre-migration sqlite3.Row).
|
||||||
|
keys = set(row.keys())
|
||||||
|
extractor_version = (row["extractor_version"] or "") if "extractor_version" in keys else ""
|
||||||
|
canonical_home = (row["canonical_home"] or "entity") if "canonical_home" in keys else "entity"
|
||||||
|
hand_authored = bool(row["hand_authored"]) if "hand_authored" in keys and row["hand_authored"] is not None else False
|
||||||
return Entity(
|
return Entity(
|
||||||
id=row["id"],
|
id=row["id"],
|
||||||
entity_type=row["entity_type"],
|
entity_type=row["entity_type"],
|
||||||
@@ -560,6 +888,9 @@ def _row_to_entity(row) -> Entity:
|
|||||||
source_refs=json.loads(row["source_refs"] or "[]"),
|
source_refs=json.loads(row["source_refs"] or "[]"),
|
||||||
created_at=row["created_at"] or "",
|
created_at=row["created_at"] or "",
|
||||||
updated_at=row["updated_at"] or "",
|
updated_at=row["updated_at"] or "",
|
||||||
|
extractor_version=extractor_version,
|
||||||
|
canonical_home=canonical_home,
|
||||||
|
hand_authored=hand_authored,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -377,6 +377,177 @@ _ENTITY_TRIAGE_CSS = """
|
|||||||
"""
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------
|
||||||
|
# Phase 7A — Merge candidates (semantic dedup)
|
||||||
|
# ---------------------------------------------------------------------
|
||||||
|
|
||||||
|
_MERGE_TRIAGE_CSS = """
|
||||||
|
<style>
|
||||||
|
.cand-merge { border-left: 3px solid #8b5cf6; }
|
||||||
|
.merge-type { background: #8b5cf6; color: white; padding: 0.1rem 0.5rem; border-radius: 3px; font-size: 0.75rem; }
|
||||||
|
.merge-sources { margin: 0.5rem 0 0.8rem 0; display: flex; flex-direction: column; gap: 0.35rem; }
|
||||||
|
.merge-source { background: var(--bg); border: 1px dashed var(--border); border-radius: 4px; padding: 0.4rem 0.6rem; font-size: 0.85rem; }
|
||||||
|
.merge-source-meta { font-family: monospace; font-size: 0.72rem; opacity: 0.7; margin-bottom: 0.2rem; }
|
||||||
|
.merge-arrow { text-align: center; font-size: 1.1rem; opacity: 0.5; margin: 0.3rem 0; }
|
||||||
|
.merge-proposed { background: var(--card); border: 1px solid #8b5cf6; border-radius: 4px; padding: 0.5rem; }
|
||||||
|
.btn-merge-approve { background: #8b5cf6; color: white; border-color: #8b5cf6; }
|
||||||
|
.btn-merge-approve:hover { background: #7c3aed; }
|
||||||
|
</style>
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
def _render_merge_card(cand: dict) -> str:
|
||||||
|
import json as _json
|
||||||
|
cid = _escape(cand.get("id", ""))
|
||||||
|
sim = cand.get("similarity") or 0.0
|
||||||
|
sources = cand.get("sources") or []
|
||||||
|
proposed_content = cand.get("proposed_content") or ""
|
||||||
|
proposed_tags = cand.get("proposed_tags") or []
|
||||||
|
proposed_project = cand.get("proposed_project") or ""
|
||||||
|
reason = cand.get("reason") or ""
|
||||||
|
|
||||||
|
src_html = "".join(
|
||||||
|
f"""
|
||||||
|
<div class="merge-source">
|
||||||
|
<div class="merge-source-meta">
|
||||||
|
{_escape(s.get('id','')[:8])} · [{_escape(s.get('memory_type',''))}]
|
||||||
|
· {_escape(s.get('project','') or '(global)')}
|
||||||
|
· conf {float(s.get('confidence',0)):.2f}
|
||||||
|
· refs {int(s.get('reference_count',0))}
|
||||||
|
</div>
|
||||||
|
<div>{_escape((s.get('content') or '')[:300])}</div>
|
||||||
|
</div>
|
||||||
|
"""
|
||||||
|
for s in sources
|
||||||
|
)
|
||||||
|
tags_str = ", ".join(proposed_tags)
|
||||||
|
return f"""
|
||||||
|
<div class="cand cand-merge" id="mcand-{cid}" data-merge-id="{cid}">
|
||||||
|
<div class="cand-head">
|
||||||
|
<span class="cand-type merge-type">[merge · {len(sources)} sources]</span>
|
||||||
|
<span class="cand-project">{_escape(proposed_project or '(global)')}</span>
|
||||||
|
<span class="cand-meta">sim ≥ {sim:.2f}</span>
|
||||||
|
</div>
|
||||||
|
<div class="merge-sources">{src_html}</div>
|
||||||
|
<div class="merge-arrow">↓ merged into ↓</div>
|
||||||
|
<div class="merge-proposed">
|
||||||
|
<textarea class="cand-content" id="mcontent-{cid}">{_escape(proposed_content)}</textarea>
|
||||||
|
<div class="cand-meta-row">
|
||||||
|
<label class="cand-field-label">Tags:
|
||||||
|
<input type="text" class="cand-tags-input" id="mtags-{cid}" value="{_escape(tags_str)}" placeholder="tag1, tag2">
|
||||||
|
</label>
|
||||||
|
</div>
|
||||||
|
{f'<div class="auto-triage-msg" style="margin-top:0.4rem;">💡 {_escape(reason)}</div>' if reason else ''}
|
||||||
|
</div>
|
||||||
|
<div class="cand-actions">
|
||||||
|
<button class="btn-merge-approve" data-merge-id="{cid}" title="Approve merge">✅ Approve Merge</button>
|
||||||
|
<button class="btn-reject" data-merge-id="{cid}" data-merge-reject="1" title="Keep separate">❌ Keep Separate</button>
|
||||||
|
</div>
|
||||||
|
<div class="cand-status" id="mstatus-{cid}"></div>
|
||||||
|
</div>
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
_MERGE_TRIAGE_SCRIPT = """
|
||||||
|
<script>
|
||||||
|
async function mergeApprove(id) {
|
||||||
|
const st = document.getElementById('mstatus-' + id);
|
||||||
|
st.textContent = 'Merging…';
|
||||||
|
st.className = 'cand-status ok';
|
||||||
|
const content = document.getElementById('mcontent-' + id).value;
|
||||||
|
const tagsRaw = document.getElementById('mtags-' + id).value;
|
||||||
|
const tags = tagsRaw.split(',').map(t => t.trim()).filter(Boolean);
|
||||||
|
const r = await fetch('/admin/memory/merge-candidates/' + encodeURIComponent(id) + '/approve', {
|
||||||
|
method: 'POST',
|
||||||
|
headers: {'Content-Type': 'application/json'},
|
||||||
|
body: JSON.stringify({actor: 'human-triage', content: content, domain_tags: tags}),
|
||||||
|
});
|
||||||
|
if (r.ok) {
|
||||||
|
const data = await r.json();
|
||||||
|
st.textContent = '✅ Merged → ' + (data.result_memory_id || '').slice(0, 8);
|
||||||
|
setTimeout(() => {
|
||||||
|
const card = document.getElementById('mcand-' + id);
|
||||||
|
if (card) { card.style.opacity = '0'; setTimeout(() => card.remove(), 300); }
|
||||||
|
}, 600);
|
||||||
|
} else {
|
||||||
|
const err = await r.text();
|
||||||
|
st.textContent = '❌ ' + r.status + ': ' + err.slice(0, 120);
|
||||||
|
st.className = 'cand-status err';
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function mergeReject(id) {
|
||||||
|
const st = document.getElementById('mstatus-' + id);
|
||||||
|
st.textContent = 'Rejecting…';
|
||||||
|
st.className = 'cand-status ok';
|
||||||
|
const r = await fetch('/admin/memory/merge-candidates/' + encodeURIComponent(id) + '/reject', {
|
||||||
|
method: 'POST',
|
||||||
|
headers: {'Content-Type': 'application/json'},
|
||||||
|
body: JSON.stringify({actor: 'human-triage'}),
|
||||||
|
});
|
||||||
|
if (r.ok) {
|
||||||
|
st.textContent = '❌ Kept separate';
|
||||||
|
setTimeout(() => {
|
||||||
|
const card = document.getElementById('mcand-' + id);
|
||||||
|
if (card) { card.style.opacity = '0'; setTimeout(() => card.remove(), 300); }
|
||||||
|
}, 400);
|
||||||
|
} else st.textContent = '❌ ' + r.status;
|
||||||
|
}
|
||||||
|
|
||||||
|
document.addEventListener('click', (e) => {
|
||||||
|
const mid = e.target.dataset?.mergeId;
|
||||||
|
if (!mid) return;
|
||||||
|
if (e.target.classList.contains('btn-merge-approve')) mergeApprove(mid);
|
||||||
|
else if (e.target.dataset?.mergeReject) mergeReject(mid);
|
||||||
|
});
|
||||||
|
|
||||||
|
async function requestDedupScan() {
|
||||||
|
const btn = document.getElementById('dedup-btn');
|
||||||
|
const status = document.getElementById('dedup-status');
|
||||||
|
btn.disabled = true;
|
||||||
|
btn.textContent = 'Queuing…';
|
||||||
|
status.textContent = '';
|
||||||
|
status.className = 'auto-triage-msg';
|
||||||
|
const threshold = parseFloat(document.getElementById('dedup-threshold').value || '0.88');
|
||||||
|
const r = await fetch('/admin/memory/dedup-scan', {
|
||||||
|
method: 'POST',
|
||||||
|
headers: {'Content-Type': 'application/json'},
|
||||||
|
body: JSON.stringify({project: '', similarity_threshold: threshold, max_batch: 50}),
|
||||||
|
});
|
||||||
|
if (r.ok) {
|
||||||
|
status.textContent = `✓ Queued dedup scan at threshold ${threshold}. Host watcher runs every 2 min; refresh in ~3 min to see merge candidates.`;
|
||||||
|
status.className = 'auto-triage-msg ok';
|
||||||
|
} else {
|
||||||
|
status.textContent = '✗ ' + r.status;
|
||||||
|
status.className = 'auto-triage-msg err';
|
||||||
|
}
|
||||||
|
setTimeout(() => {
|
||||||
|
btn.disabled = false;
|
||||||
|
btn.textContent = '🔗 Scan for duplicates';
|
||||||
|
}, 2000);
|
||||||
|
}
|
||||||
|
</script>
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
def _render_dedup_bar() -> str:
|
||||||
|
return """
|
||||||
|
<div class="auto-triage-bar">
|
||||||
|
<button id="dedup-btn" onclick="requestDedupScan()" title="Run semantic dedup scan on Dalidou host">
|
||||||
|
🔗 Scan for duplicates
|
||||||
|
</button>
|
||||||
|
<label class="cand-field-label" style="margin:0 0.5rem;">
|
||||||
|
Threshold:
|
||||||
|
<input id="dedup-threshold" type="number" min="0.70" max="0.99" step="0.01" value="0.88"
|
||||||
|
style="width:70px; padding:0.25rem; background:var(--bg); color:var(--text); border:1px solid var(--border); border-radius:3px;">
|
||||||
|
</label>
|
||||||
|
<span id="dedup-status" class="auto-triage-msg">
|
||||||
|
Finds semantically near-duplicate active memories and proposes LLM-drafted merges for review. Source memories become <code>superseded</code> on approve; nothing is deleted.
|
||||||
|
</span>
|
||||||
|
</div>
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
def _render_graduation_bar() -> str:
|
def _render_graduation_bar() -> str:
|
||||||
"""The 'Graduate memories → entity candidates' control bar."""
|
"""The 'Graduate memories → entity candidates' control bar."""
|
||||||
from atocore.projects.registry import load_project_registry
|
from atocore.projects.registry import load_project_registry
|
||||||
@@ -478,26 +649,51 @@ def render_triage_page(limit: int = 100) -> str:
|
|||||||
except Exception as e:
|
except Exception as e:
|
||||||
entity_candidates = []
|
entity_candidates = []
|
||||||
|
|
||||||
total = len(mem_candidates) + len(entity_candidates)
|
try:
|
||||||
|
from atocore.memory.service import get_merge_candidates
|
||||||
|
merge_candidates = get_merge_candidates(status="pending", limit=limit)
|
||||||
|
except Exception:
|
||||||
|
merge_candidates = []
|
||||||
|
|
||||||
|
total = len(mem_candidates) + len(entity_candidates) + len(merge_candidates)
|
||||||
graduation_bar = _render_graduation_bar()
|
graduation_bar = _render_graduation_bar()
|
||||||
|
dedup_bar = _render_dedup_bar()
|
||||||
|
|
||||||
if total == 0:
|
if total == 0:
|
||||||
body = _TRIAGE_CSS + _ENTITY_TRIAGE_CSS + f"""
|
body = _TRIAGE_CSS + _ENTITY_TRIAGE_CSS + _MERGE_TRIAGE_CSS + f"""
|
||||||
<div class="triage-header">
|
<div class="triage-header">
|
||||||
<h1>Triage Queue</h1>
|
<h1>Triage Queue</h1>
|
||||||
</div>
|
</div>
|
||||||
{graduation_bar}
|
{graduation_bar}
|
||||||
|
{dedup_bar}
|
||||||
<div class="empty">
|
<div class="empty">
|
||||||
<p>🎉 No candidates to review.</p>
|
<p>🎉 No candidates to review.</p>
|
||||||
<p>The auto-triage pipeline keeps this queue empty unless something needs your judgment.</p>
|
<p>The auto-triage pipeline keeps this queue empty unless something needs your judgment.</p>
|
||||||
<p>Use the 🎓 Graduate memories button above to propose new entity candidates from existing memories.</p>
|
<p>Use 🎓 Graduate memories to propose entity candidates, or 🔗 Scan for duplicates to find near-duplicate memories to merge.</p>
|
||||||
</div>
|
</div>
|
||||||
""" + _GRADUATION_SCRIPT
|
""" + _GRADUATION_SCRIPT + _MERGE_TRIAGE_SCRIPT
|
||||||
return render_html("Triage — AtoCore", body, breadcrumbs=[("Wiki", "/wiki"), ("Triage", "")])
|
return render_html("Triage — AtoCore", body, breadcrumbs=[("Wiki", "/wiki"), ("Triage", "")])
|
||||||
|
|
||||||
# Memory cards
|
# Memory cards
|
||||||
mem_cards = "".join(_render_candidate_card(c) for c in mem_candidates)
|
mem_cards = "".join(_render_candidate_card(c) for c in mem_candidates)
|
||||||
|
|
||||||
|
# Merge cards (Phase 7A)
|
||||||
|
merge_cards_html = ""
|
||||||
|
if merge_candidates:
|
||||||
|
merge_cards = "".join(_render_merge_card(c) for c in merge_candidates)
|
||||||
|
merge_cards_html = f"""
|
||||||
|
<div class="section-break">
|
||||||
|
<h2>🔗 Merge Candidates ({len(merge_candidates)})</h2>
|
||||||
|
<p class="auto-triage-msg">
|
||||||
|
Semantically near-duplicate active memories. Approving merges the sources
|
||||||
|
into the proposed unified memory; sources become <code>superseded</code>
|
||||||
|
(not deleted — still queryable). You can edit the draft content and tags
|
||||||
|
before approving.
|
||||||
|
</p>
|
||||||
|
</div>
|
||||||
|
{merge_cards}
|
||||||
|
"""
|
||||||
|
|
||||||
# Entity cards
|
# Entity cards
|
||||||
ent_cards_html = ""
|
ent_cards_html = ""
|
||||||
if entity_candidates:
|
if entity_candidates:
|
||||||
@@ -513,11 +709,12 @@ def render_triage_page(limit: int = 100) -> str:
|
|||||||
{ent_cards}
|
{ent_cards}
|
||||||
"""
|
"""
|
||||||
|
|
||||||
body = _TRIAGE_CSS + _ENTITY_TRIAGE_CSS + f"""
|
body = _TRIAGE_CSS + _ENTITY_TRIAGE_CSS + _MERGE_TRIAGE_CSS + f"""
|
||||||
<div class="triage-header">
|
<div class="triage-header">
|
||||||
<h1>Triage Queue</h1>
|
<h1>Triage Queue</h1>
|
||||||
<span class="count">
|
<span class="count">
|
||||||
<span id="cand-count">{len(mem_candidates)}</span> memory ·
|
<span id="cand-count">{len(mem_candidates)}</span> memory ·
|
||||||
|
{len(merge_candidates)} merge ·
|
||||||
{len(entity_candidates)} entity
|
{len(entity_candidates)} entity
|
||||||
</span>
|
</span>
|
||||||
</div>
|
</div>
|
||||||
@@ -536,10 +733,12 @@ def render_triage_page(limit: int = 100) -> str:
|
|||||||
</span>
|
</span>
|
||||||
</div>
|
</div>
|
||||||
{graduation_bar}
|
{graduation_bar}
|
||||||
|
{dedup_bar}
|
||||||
<h2>📝 Memory Candidates ({len(mem_candidates)})</h2>
|
<h2>📝 Memory Candidates ({len(mem_candidates)})</h2>
|
||||||
{mem_cards}
|
{mem_cards}
|
||||||
|
{merge_cards_html}
|
||||||
{ent_cards_html}
|
{ent_cards_html}
|
||||||
""" + _TRIAGE_SCRIPT + _ENTITY_TRIAGE_SCRIPT + _GRADUATION_SCRIPT
|
""" + _TRIAGE_SCRIPT + _ENTITY_TRIAGE_SCRIPT + _GRADUATION_SCRIPT + _MERGE_TRIAGE_SCRIPT
|
||||||
|
|
||||||
return render_html(
|
return render_html(
|
||||||
"Triage — AtoCore",
|
"Triage — AtoCore",
|
||||||
|
|||||||
@@ -23,11 +23,32 @@ from atocore.engineering.service import (
|
|||||||
get_relationships,
|
get_relationships,
|
||||||
)
|
)
|
||||||
from atocore.memory.service import get_memories
|
from atocore.memory.service import get_memories
|
||||||
from atocore.projects.registry import load_project_registry
|
from atocore.projects.registry import (
|
||||||
|
GLOBAL_PROJECT,
|
||||||
|
INBOX_PROJECT,
|
||||||
|
load_project_registry,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def render_html(title: str, body_html: str, breadcrumbs: list[tuple[str, str]] | None = None) -> str:
|
_TOP_NAV_LINKS = [
|
||||||
nav = ""
|
("🏠 Home", "/wiki"),
|
||||||
|
("📡 Activity", "/wiki/activity"),
|
||||||
|
("🔀 Triage", "/admin/triage"),
|
||||||
|
("📊 Dashboard", "/admin/dashboard"),
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def _render_topnav(active_path: str = "") -> str:
|
||||||
|
items = []
|
||||||
|
for label, href in _TOP_NAV_LINKS:
|
||||||
|
cls = "topnav-item active" if href == active_path else "topnav-item"
|
||||||
|
items.append(f'<a href="{href}" class="{cls}">{label}</a>')
|
||||||
|
return f'<nav class="topnav">{" ".join(items)}</nav>'
|
||||||
|
|
||||||
|
|
||||||
|
def render_html(title: str, body_html: str, breadcrumbs: list[tuple[str, str]] | None = None, active_path: str = "") -> str:
|
||||||
|
topnav = _render_topnav(active_path)
|
||||||
|
crumbs = ""
|
||||||
if breadcrumbs:
|
if breadcrumbs:
|
||||||
parts = []
|
parts = []
|
||||||
for label, href in breadcrumbs:
|
for label, href in breadcrumbs:
|
||||||
@@ -35,8 +56,9 @@ def render_html(title: str, body_html: str, breadcrumbs: list[tuple[str, str]] |
|
|||||||
parts.append(f'<a href="{href}">{label}</a>')
|
parts.append(f'<a href="{href}">{label}</a>')
|
||||||
else:
|
else:
|
||||||
parts.append(f"<span>{label}</span>")
|
parts.append(f"<span>{label}</span>")
|
||||||
nav = f'<nav class="breadcrumbs">{" / ".join(parts)}</nav>'
|
crumbs = f'<nav class="breadcrumbs">{" / ".join(parts)}</nav>'
|
||||||
|
|
||||||
|
nav = topnav + crumbs
|
||||||
return _TEMPLATE.replace("{{title}}", title).replace("{{nav}}", nav).replace("{{body}}", body_html)
|
return _TEMPLATE.replace("{{title}}", title).replace("{{nav}}", nav).replace("{{body}}", body_html)
|
||||||
|
|
||||||
|
|
||||||
@@ -100,6 +122,72 @@ def render_homepage() -> str:
|
|||||||
lines.append('<button type="submit">Search</button>')
|
lines.append('<button type="submit">Search</button>')
|
||||||
lines.append('</form>')
|
lines.append('</form>')
|
||||||
|
|
||||||
|
# What's happening — autonomous activity snippet
|
||||||
|
try:
|
||||||
|
from atocore.memory.service import get_recent_audit
|
||||||
|
recent = get_recent_audit(limit=30)
|
||||||
|
by_action: dict[str, int] = {}
|
||||||
|
by_actor: dict[str, int] = {}
|
||||||
|
for a in recent:
|
||||||
|
by_action[a["action"]] = by_action.get(a["action"], 0) + 1
|
||||||
|
by_actor[a["actor"]] = by_actor.get(a["actor"], 0) + 1
|
||||||
|
# Surface autonomous actors specifically
|
||||||
|
auto_actors = {k: v for k, v in by_actor.items()
|
||||||
|
if k.startswith("auto-") or k == "confidence-decay"
|
||||||
|
or k == "phase10-auto-promote" or k == "transient-to-durable"}
|
||||||
|
if recent:
|
||||||
|
lines.append('<div class="activity-snippet">')
|
||||||
|
lines.append('<h3>📡 What the brain is doing</h3>')
|
||||||
|
top_actions = sorted(by_action.items(), key=lambda x: -x[1])[:6]
|
||||||
|
lines.append('<div class="stat-row">' +
|
||||||
|
"".join(f'<span>{a}: {n}</span>' for a, n in top_actions) +
|
||||||
|
'</div>')
|
||||||
|
if auto_actors:
|
||||||
|
lines.append(f'<p style="font-size:0.9rem; margin:0.3rem 0;">Autonomous actors: ' +
|
||||||
|
" · ".join(f'<code>{k}</code> ({v})' for k, v in auto_actors.items()) +
|
||||||
|
'</p>')
|
||||||
|
lines.append('<p style="font-size:0.85rem; margin:0;"><a href="/wiki/activity">Full timeline →</a></p>')
|
||||||
|
lines.append('</div>')
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
# Issue C: Inbox + Global pseudo-projects alongside registered projects.
|
||||||
|
# scope_only=True keeps real-project entities out of these counts.
|
||||||
|
try:
|
||||||
|
inbox_count = len(get_entities(
|
||||||
|
project=INBOX_PROJECT, scope_only=True, limit=500,
|
||||||
|
))
|
||||||
|
global_count = len(get_entities(
|
||||||
|
project=GLOBAL_PROJECT, scope_only=True, limit=500,
|
||||||
|
))
|
||||||
|
except Exception:
|
||||||
|
inbox_count = 0
|
||||||
|
global_count = 0
|
||||||
|
|
||||||
|
lines.append('<h2>📥 Inbox & Global</h2>')
|
||||||
|
lines.append(
|
||||||
|
'<p class="emerging-intro">Entities that don\'t belong to a specific '
|
||||||
|
'project yet. <strong>Inbox</strong> holds pre-project leads and quotes. '
|
||||||
|
'<strong>Global</strong> holds cross-project facts (material properties, '
|
||||||
|
'vendor capabilities) that apply everywhere.</p>'
|
||||||
|
)
|
||||||
|
lines.append('<div class="card-grid">')
|
||||||
|
lines.append(
|
||||||
|
f'<a href="/entities?project=inbox&scope_only=true" class="card">'
|
||||||
|
f'<h3>📥 Inbox</h3>'
|
||||||
|
f'<p>Pre-project leads, quotes, early conversations.</p>'
|
||||||
|
f'<div class="stats">{inbox_count} entities</div>'
|
||||||
|
f'</a>'
|
||||||
|
)
|
||||||
|
lines.append(
|
||||||
|
f'<a href="/entities?project=&scope_only=true" class="card">'
|
||||||
|
f'<h3>🌐 Global</h3>'
|
||||||
|
f'<p>Cross-project facts: materials, vendors, shared knowledge.</p>'
|
||||||
|
f'<div class="stats">{global_count} entities</div>'
|
||||||
|
f'</a>'
|
||||||
|
)
|
||||||
|
lines.append('</div>')
|
||||||
|
|
||||||
for bucket_name, items in buckets.items():
|
for bucket_name, items in buckets.items():
|
||||||
if not items:
|
if not items:
|
||||||
continue
|
continue
|
||||||
@@ -116,6 +204,40 @@ def render_homepage() -> str:
|
|||||||
lines.append('</a>')
|
lines.append('</a>')
|
||||||
lines.append('</div>')
|
lines.append('</div>')
|
||||||
|
|
||||||
|
# Phase 6 C.2: Emerging projects section
|
||||||
|
try:
|
||||||
|
import json as _json
|
||||||
|
emerging_projects = []
|
||||||
|
state_entries = get_state("atocore")
|
||||||
|
for e in state_entries:
|
||||||
|
if e.category == "proposals" and e.key == "unregistered_projects":
|
||||||
|
try:
|
||||||
|
emerging_projects = _json.loads(e.value)
|
||||||
|
except Exception:
|
||||||
|
emerging_projects = []
|
||||||
|
break
|
||||||
|
if emerging_projects:
|
||||||
|
lines.append('<h2>📋 Emerging</h2>')
|
||||||
|
lines.append('<p class="emerging-intro">Projects that appear in memories but aren\'t yet registered. '
|
||||||
|
'One click to promote them to first-class projects.</p>')
|
||||||
|
lines.append('<div class="emerging-grid">')
|
||||||
|
for ep in emerging_projects[:10]:
|
||||||
|
name = ep.get("project", "?")
|
||||||
|
count = ep.get("count", 0)
|
||||||
|
samples = ep.get("sample_contents", [])
|
||||||
|
samples_html = "".join(f'<li>{s[:120]}</li>' for s in samples[:2])
|
||||||
|
lines.append(
|
||||||
|
f'<div class="emerging-card">'
|
||||||
|
f'<h3>{name}</h3>'
|
||||||
|
f'<div class="emerging-count">{count} memories</div>'
|
||||||
|
f'<ul class="emerging-samples">{samples_html}</ul>'
|
||||||
|
f'<button class="btn-register-emerging" onclick="registerEmerging({name!r})">📌 Register as project</button>'
|
||||||
|
f'</div>'
|
||||||
|
)
|
||||||
|
lines.append('</div>')
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
# Quick stats
|
# Quick stats
|
||||||
all_entities = get_entities(limit=500)
|
all_entities = get_entities(limit=500)
|
||||||
all_memories = get_memories(active_only=True, limit=500)
|
all_memories = get_memories(active_only=True, limit=500)
|
||||||
@@ -133,13 +255,17 @@ def render_homepage() -> str:
|
|||||||
|
|
||||||
lines.append(f'<p><a href="/admin/triage">Triage Queue</a> · <a href="/admin/dashboard">API Dashboard (JSON)</a> · <a href="/health">Health Check</a></p>')
|
lines.append(f'<p><a href="/admin/triage">Triage Queue</a> · <a href="/admin/dashboard">API Dashboard (JSON)</a> · <a href="/health">Health Check</a></p>')
|
||||||
|
|
||||||
return render_html("AtoCore Wiki", "\n".join(lines))
|
return render_html("AtoCore Wiki", "\n".join(lines), active_path="/wiki")
|
||||||
|
|
||||||
|
|
||||||
def render_project(project: str) -> str:
|
def render_project(project: str) -> str:
|
||||||
from atocore.engineering.mirror import generate_project_overview
|
from atocore.engineering.mirror import generate_project_overview
|
||||||
|
|
||||||
markdown_content = generate_project_overview(project)
|
markdown_content = generate_project_overview(project)
|
||||||
|
# Resolve [[Wikilinks]] before markdown so redlinks / cross-project
|
||||||
|
# indicators appear in the rendered HTML. (Issue B)
|
||||||
|
markdown_content = _wikilink_transform(markdown_content, current_project=project)
|
||||||
|
|
||||||
# Convert entity names to links
|
# Convert entity names to links
|
||||||
entities = get_entities(project=project, limit=200)
|
entities = get_entities(project=project, limit=200)
|
||||||
html_body = md.markdown(markdown_content, extensions=["tables", "fenced_code"])
|
html_body = md.markdown(markdown_content, extensions=["tables", "fenced_code"])
|
||||||
@@ -155,6 +281,256 @@ def render_project(project: str) -> str:
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
import re as _re
|
||||||
|
|
||||||
|
_WIKILINK_PATTERN = _re.compile(r"\[\[([^\[\]|]+?)(?:\|([^\[\]]+?))?\]\]")
|
||||||
|
|
||||||
|
|
||||||
|
def _resolve_wikilink(target: str, current_project: str | None) -> tuple[str, str, str]:
|
||||||
|
"""Resolve a ``[[Name]]`` target to ``(href, css_class, extra_suffix)``.
|
||||||
|
|
||||||
|
Resolution order (Issue B):
|
||||||
|
|
||||||
|
1. Same-project exact name match → live link (class ``wikilink``).
|
||||||
|
2. Other-project exact name match → live link with ``(in project X)``
|
||||||
|
suffix (class ``wikilink wikilink-cross``).
|
||||||
|
3. No match → redlink pointing at ``/wiki/new?name=...`` so clicking
|
||||||
|
opens a pre-filled "create this entity" form (class ``redlink``).
|
||||||
|
"""
|
||||||
|
needle = target.strip()
|
||||||
|
if not needle:
|
||||||
|
return ("/wiki/new", "redlink", "")
|
||||||
|
|
||||||
|
same_project = None
|
||||||
|
cross_project = None
|
||||||
|
try:
|
||||||
|
candidates = get_entities(name_contains=needle, limit=200)
|
||||||
|
except Exception:
|
||||||
|
candidates = []
|
||||||
|
|
||||||
|
lowered = needle.lower()
|
||||||
|
for ent in candidates:
|
||||||
|
if ent.name.lower() != lowered:
|
||||||
|
continue
|
||||||
|
if current_project and ent.project == current_project:
|
||||||
|
same_project = ent
|
||||||
|
break
|
||||||
|
if cross_project is None:
|
||||||
|
cross_project = ent
|
||||||
|
|
||||||
|
if same_project is not None:
|
||||||
|
return (f"/wiki/entities/{same_project.id}", "wikilink", "")
|
||||||
|
if cross_project is not None:
|
||||||
|
suffix = (
|
||||||
|
f' <span class="wikilink-scope">(in {cross_project.project})</span>'
|
||||||
|
if cross_project.project
|
||||||
|
else ' <span class="wikilink-scope">(global)</span>'
|
||||||
|
)
|
||||||
|
return (f"/wiki/entities/{cross_project.id}", "wikilink wikilink-cross", suffix)
|
||||||
|
|
||||||
|
from urllib.parse import quote
|
||||||
|
href = f"/wiki/new?name={quote(needle)}"
|
||||||
|
if current_project:
|
||||||
|
href += f"&project={quote(current_project)}"
|
||||||
|
return (href, "redlink", "")
|
||||||
|
|
||||||
|
|
||||||
|
def _wikilink_transform(text: str, current_project: str | None) -> str:
|
||||||
|
"""Replace ``[[Name]]`` / ``[[Name|Display]]`` tokens with HTML anchors.
|
||||||
|
|
||||||
|
Runs before markdown rendering. Emits raw HTML which python-markdown
|
||||||
|
preserves unchanged.
|
||||||
|
"""
|
||||||
|
if not text or "[[" not in text:
|
||||||
|
return text
|
||||||
|
|
||||||
|
def _sub(match: _re.Match) -> str:
|
||||||
|
target = match.group(1)
|
||||||
|
display = (match.group(2) or target).strip()
|
||||||
|
href, cls, suffix = _resolve_wikilink(target, current_project)
|
||||||
|
title = "create this entity" if cls == "redlink" else target.strip()
|
||||||
|
return (
|
||||||
|
f'<a href="{href}" class="{cls}" title="{_escape_attr(title)}">'
|
||||||
|
f'{_escape_html(display)}</a>{suffix}'
|
||||||
|
)
|
||||||
|
|
||||||
|
return _WIKILINK_PATTERN.sub(_sub, text)
|
||||||
|
|
||||||
|
|
||||||
|
def render_new_entity_form(name: str = "", project: str = "") -> str:
|
||||||
|
"""Issue B — "create this entity" form targeted by redlinks."""
|
||||||
|
from atocore.engineering.service import ENTITY_TYPES
|
||||||
|
|
||||||
|
safe_name = _escape_attr(name or "")
|
||||||
|
safe_project = _escape_attr(project or "")
|
||||||
|
opts = "".join(
|
||||||
|
f'<option value="{t}">{t}</option>' for t in ENTITY_TYPES
|
||||||
|
)
|
||||||
|
lines = [
|
||||||
|
'<h1>Create entity</h1>',
|
||||||
|
('<p>This entity was referenced via a wikilink but does not exist yet. '
|
||||||
|
'Fill in the details to create it — the wiki link will resolve on reload.</p>'),
|
||||||
|
'<form id="new-entity-form" class="new-entity-form">',
|
||||||
|
f'<label>Name<br><input type="text" name="name" value="{safe_name}" required></label>',
|
||||||
|
f'<label>Entity type<br><select name="entity_type" required>{opts}</select></label>',
|
||||||
|
f'<label>Project<br><input type="text" name="project" value="{safe_project}" '
|
||||||
|
f'placeholder="leave blank for cross-project / global"></label>',
|
||||||
|
'<label>Description<br><textarea name="description" rows="4"></textarea></label>',
|
||||||
|
'<button type="submit">Create</button>',
|
||||||
|
'<div id="new-entity-result" class="new-entity-result"></div>',
|
||||||
|
'</form>',
|
||||||
|
"""<script>
|
||||||
|
(function() {
|
||||||
|
const form = document.getElementById('new-entity-form');
|
||||||
|
const out = document.getElementById('new-entity-result');
|
||||||
|
form.addEventListener('submit', async (ev) => {
|
||||||
|
ev.preventDefault();
|
||||||
|
const fd = new FormData(form);
|
||||||
|
const body = {
|
||||||
|
name: fd.get('name'),
|
||||||
|
entity_type: fd.get('entity_type'),
|
||||||
|
project: fd.get('project') || '',
|
||||||
|
description: fd.get('description') || '',
|
||||||
|
// V1-0: human writes via the wiki form are hand_authored by definition.
|
||||||
|
hand_authored: true,
|
||||||
|
};
|
||||||
|
try {
|
||||||
|
const r = await fetch('/v1/entities', {
|
||||||
|
method: 'POST',
|
||||||
|
headers: {'Content-Type': 'application/json'},
|
||||||
|
body: JSON.stringify(body),
|
||||||
|
});
|
||||||
|
const j = await r.json();
|
||||||
|
if (r.ok) {
|
||||||
|
out.innerHTML = 'Created: <a href="/wiki/entities/' + j.id + '">' +
|
||||||
|
(j.name || 'new entity') + '</a>';
|
||||||
|
setTimeout(() => { window.location.href = '/wiki/entities/' + j.id; }, 800);
|
||||||
|
} else {
|
||||||
|
out.textContent = 'Error: ' + (j.detail || JSON.stringify(j));
|
||||||
|
}
|
||||||
|
} catch (e) {
|
||||||
|
out.textContent = 'Network error: ' + e;
|
||||||
|
}
|
||||||
|
});
|
||||||
|
})();
|
||||||
|
</script>""",
|
||||||
|
]
|
||||||
|
return render_html(
|
||||||
|
f"Create {name}" if name else "Create entity",
|
||||||
|
"\n".join(lines),
|
||||||
|
breadcrumbs=[("Wiki", "/wiki"), ("Create entity", "")],
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _render_visual_evidence(entity_id: str, ctx: dict) -> str:
|
||||||
|
"""Render EVIDENCED_BY → artifact links as an inline thumbnail strip."""
|
||||||
|
from atocore.assets import get_asset
|
||||||
|
|
||||||
|
artifacts = []
|
||||||
|
for rel in ctx["relationships"]:
|
||||||
|
if rel.source_entity_id != entity_id or rel.relationship_type != "evidenced_by":
|
||||||
|
continue
|
||||||
|
target = ctx["related_entities"].get(rel.target_entity_id)
|
||||||
|
if target is None or target.entity_type != "artifact":
|
||||||
|
continue
|
||||||
|
artifacts.append(target)
|
||||||
|
|
||||||
|
if not artifacts:
|
||||||
|
return ""
|
||||||
|
|
||||||
|
tiles = []
|
||||||
|
for art in artifacts:
|
||||||
|
props = art.properties or {}
|
||||||
|
kind = props.get("kind", "other")
|
||||||
|
caption = props.get("caption", art.name)
|
||||||
|
asset_id = props.get("asset_id")
|
||||||
|
asset = get_asset(asset_id) if asset_id else None
|
||||||
|
detail_href = f"/wiki/entities/{art.id}"
|
||||||
|
|
||||||
|
if kind == "image" and asset and asset.mime_type.startswith("image/"):
|
||||||
|
full_href = f"/assets/{asset.id}"
|
||||||
|
thumb = f"/assets/{asset.id}/thumbnail?size=240"
|
||||||
|
tiles.append(
|
||||||
|
f'<figure class="evidence-tile">'
|
||||||
|
f'<a href="{full_href}" target="_blank" rel="noopener">'
|
||||||
|
f'<img src="{thumb}" alt="{_escape_attr(caption)}" loading="lazy">'
|
||||||
|
f'</a>'
|
||||||
|
f'<figcaption><a href="{detail_href}">{_escape_html(caption)}</a></figcaption>'
|
||||||
|
f'</figure>'
|
||||||
|
)
|
||||||
|
elif kind == "pdf" and asset:
|
||||||
|
full_href = f"/assets/{asset.id}"
|
||||||
|
tiles.append(
|
||||||
|
f'<div class="evidence-tile evidence-pdf">'
|
||||||
|
f'<a href="{full_href}" target="_blank" rel="noopener">'
|
||||||
|
f'📄 PDF: {_escape_html(caption)}</a>'
|
||||||
|
f' · <a href="{detail_href}">details</a>'
|
||||||
|
f'</div>'
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
tiles.append(
|
||||||
|
f'<div class="evidence-tile evidence-other">'
|
||||||
|
f'<a href="{detail_href}">📎 {_escape_html(caption)}</a>'
|
||||||
|
f' <span class="tag">{kind}</span>'
|
||||||
|
f'</div>'
|
||||||
|
)
|
||||||
|
|
||||||
|
return (
|
||||||
|
'<h2>Visual evidence</h2>'
|
||||||
|
f'<div class="evidence-strip">{"".join(tiles)}</div>'
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _render_artifact_body(ent) -> list[str]:
|
||||||
|
"""Render an artifact entity's own image/pdf/caption."""
|
||||||
|
from atocore.assets import get_asset
|
||||||
|
|
||||||
|
props = ent.properties or {}
|
||||||
|
kind = props.get("kind", "other")
|
||||||
|
caption = props.get("caption", "")
|
||||||
|
capture_context = props.get("capture_context", "")
|
||||||
|
asset_id = props.get("asset_id")
|
||||||
|
asset = get_asset(asset_id) if asset_id else None
|
||||||
|
|
||||||
|
out: list[str] = []
|
||||||
|
if kind == "image" and asset and asset.mime_type.startswith("image/"):
|
||||||
|
out.append(
|
||||||
|
f'<figure class="artifact-full">'
|
||||||
|
f'<a href="/assets/{asset.id}" target="_blank" rel="noopener">'
|
||||||
|
f'<img src="/assets/{asset.id}/thumbnail?size=1024" '
|
||||||
|
f'alt="{_escape_attr(caption or ent.name)}">'
|
||||||
|
f'</a>'
|
||||||
|
f'<figcaption>{_escape_html(caption)}</figcaption>'
|
||||||
|
f'</figure>'
|
||||||
|
)
|
||||||
|
elif kind == "pdf" and asset:
|
||||||
|
out.append(
|
||||||
|
f'<p>📄 <a href="/assets/{asset.id}" target="_blank" rel="noopener">'
|
||||||
|
f'Open PDF ({asset.size_bytes // 1024} KB)</a></p>'
|
||||||
|
)
|
||||||
|
elif asset_id:
|
||||||
|
out.append(f'<p class="meta">asset_id: <code>{asset_id}</code> — blob missing</p>')
|
||||||
|
|
||||||
|
if capture_context:
|
||||||
|
out.append('<h2>Capture context</h2>')
|
||||||
|
out.append(f'<blockquote>{_escape_html(capture_context)}</blockquote>')
|
||||||
|
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def _escape_html(s: str) -> str:
|
||||||
|
if s is None:
|
||||||
|
return ""
|
||||||
|
return (str(s)
|
||||||
|
.replace("&", "&")
|
||||||
|
.replace("<", "<")
|
||||||
|
.replace(">", ">"))
|
||||||
|
|
||||||
|
|
||||||
|
def _escape_attr(s: str) -> str:
|
||||||
|
return _escape_html(s).replace('"', """)
|
||||||
|
|
||||||
|
|
||||||
def render_entity(entity_id: str) -> str | None:
|
def render_entity(entity_id: str) -> str | None:
|
||||||
ctx = get_entity_with_context(entity_id)
|
ctx = get_entity_with_context(entity_id)
|
||||||
if ctx is None:
|
if ctx is None:
|
||||||
@@ -166,7 +542,8 @@ def render_entity(entity_id: str) -> str | None:
|
|||||||
if ent.project:
|
if ent.project:
|
||||||
lines.append(f'<p>Project: <a href="/wiki/projects/{ent.project}">{ent.project}</a></p>')
|
lines.append(f'<p>Project: <a href="/wiki/projects/{ent.project}">{ent.project}</a></p>')
|
||||||
if ent.description:
|
if ent.description:
|
||||||
lines.append(f'<p>{ent.description}</p>')
|
desc_html = _wikilink_transform(ent.description, current_project=ent.project)
|
||||||
|
lines.append(f'<p>{desc_html}</p>')
|
||||||
if ent.properties:
|
if ent.properties:
|
||||||
lines.append('<h2>Properties</h2><ul>')
|
lines.append('<h2>Properties</h2><ul>')
|
||||||
for k, v in ent.properties.items():
|
for k, v in ent.properties.items():
|
||||||
@@ -175,6 +552,15 @@ def render_entity(entity_id: str) -> str | None:
|
|||||||
|
|
||||||
lines.append(f'<p class="meta">confidence: {ent.confidence} · status: {ent.status} · created: {ent.created_at}</p>')
|
lines.append(f'<p class="meta">confidence: {ent.confidence} · status: {ent.status} · created: {ent.created_at}</p>')
|
||||||
|
|
||||||
|
# Issue F: artifact entities render their own image inline; other
|
||||||
|
# entities render their EVIDENCED_BY artifacts as a visual strip.
|
||||||
|
if ent.entity_type == "artifact":
|
||||||
|
lines.extend(_render_artifact_body(ent))
|
||||||
|
else:
|
||||||
|
evidence_html = _render_visual_evidence(ent.id, ctx)
|
||||||
|
if evidence_html:
|
||||||
|
lines.append(evidence_html)
|
||||||
|
|
||||||
if ctx["relationships"]:
|
if ctx["relationships"]:
|
||||||
lines.append('<h2>Relationships</h2><ul>')
|
lines.append('<h2>Relationships</h2><ul>')
|
||||||
for rel in ctx["relationships"]:
|
for rel in ctx["relationships"]:
|
||||||
@@ -254,6 +640,381 @@ def render_search(query: str) -> str:
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------
|
||||||
|
# /wiki/capture — DEPRECATED emergency paste-in form.
|
||||||
|
# Kept as an endpoint because POST /interactions is public anyway, but
|
||||||
|
# REMOVED from the topnav so it's not promoted as the capture path.
|
||||||
|
# The sanctioned surfaces are Claude Code (Stop + UserPromptSubmit
|
||||||
|
# hooks) and OpenClaw (capture plugin with 7I context injection).
|
||||||
|
# This form is explicitly a last-resort for when someone has to feed
|
||||||
|
# in an external log and can't get the normal hooks to reach it.
|
||||||
|
# ---------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def render_capture() -> str:
|
||||||
|
lines = ['<h1>📥 Manual capture (fallback only)</h1>']
|
||||||
|
lines.append(
|
||||||
|
'<div class="triage-warning"><strong>This is not the capture path.</strong> '
|
||||||
|
'The sanctioned capture surfaces are Claude Code (Stop hook auto-captures every turn) '
|
||||||
|
'and OpenClaw (plugin auto-captures + injects AtoCore context on every agent turn). '
|
||||||
|
'This form exists only as a last resort for external logs you can\'t get into the normal pipeline.</div>'
|
||||||
|
)
|
||||||
|
lines.append(
|
||||||
|
'<p>If you\'re reaching for this page because you had a chat somewhere AtoCore didn\'t see, '
|
||||||
|
'fix the capture surface instead — don\'t paste. The deliberate scope is Claude Code + OpenClaw.</p>'
|
||||||
|
)
|
||||||
|
lines.append('<p class="meta">Your prompt + the assistant\'s response. Project is optional — '
|
||||||
|
'the extractor infers it from content.</p>')
|
||||||
|
lines.append("""
|
||||||
|
<form id="capture-form" style="display:flex; flex-direction:column; gap:0.8rem; margin-top:1rem;">
|
||||||
|
<label><strong>Your prompt / question</strong>
|
||||||
|
<textarea id="cap-prompt" required rows="4"
|
||||||
|
style="width:100%; padding:0.6rem; background:var(--bg); color:var(--text); border:1px solid var(--border); border-radius:6px; font-family:inherit; font-size:0.95rem;"
|
||||||
|
placeholder="Paste what you asked…"></textarea>
|
||||||
|
</label>
|
||||||
|
<label><strong>Assistant response</strong>
|
||||||
|
<textarea id="cap-response" required rows="10"
|
||||||
|
style="width:100%; padding:0.6rem; background:var(--bg); color:var(--text); border:1px solid var(--border); border-radius:6px; font-family:inherit; font-size:0.95rem;"
|
||||||
|
placeholder="Paste the full assistant response…"></textarea>
|
||||||
|
</label>
|
||||||
|
<div style="display:flex; gap:0.5rem; align-items:center; flex-wrap:wrap;">
|
||||||
|
<label style="display:flex; gap:0.35rem; align-items:center;">Project (optional):
|
||||||
|
<input type="text" id="cap-project" placeholder="auto-detect"
|
||||||
|
style="padding:0.35rem 0.6rem; background:var(--bg); color:var(--text); border:1px solid var(--border); border-radius:4px; font-family:monospace; width:180px;">
|
||||||
|
</label>
|
||||||
|
<label style="display:flex; gap:0.35rem; align-items:center;">Source:
|
||||||
|
<select id="cap-source" style="padding:0.35rem; background:var(--bg); color:var(--text); border:1px solid var(--border); border-radius:4px;">
|
||||||
|
<option value="claude-desktop">Claude Desktop</option>
|
||||||
|
<option value="claude-web">Claude.ai web</option>
|
||||||
|
<option value="claude-mobile">Claude mobile</option>
|
||||||
|
<option value="chatgpt">ChatGPT</option>
|
||||||
|
<option value="other">Other</option>
|
||||||
|
</select>
|
||||||
|
</label>
|
||||||
|
</div>
|
||||||
|
<button type="submit"
|
||||||
|
style="padding:0.6rem 1.2rem; background:var(--accent); color:white; border:none; border-radius:6px; cursor:pointer; font-size:1rem; font-weight:600; align-self:flex-start;">
|
||||||
|
Save to AtoCore
|
||||||
|
</button>
|
||||||
|
</form>
|
||||||
|
<div id="cap-status" style="margin-top:1rem; font-size:0.9rem; min-height:1.5em;"></div>
|
||||||
|
|
||||||
|
<script>
|
||||||
|
document.getElementById('capture-form').addEventListener('submit', async (e) => {
|
||||||
|
e.preventDefault();
|
||||||
|
const prompt = document.getElementById('cap-prompt').value.trim();
|
||||||
|
const response = document.getElementById('cap-response').value.trim();
|
||||||
|
const project = document.getElementById('cap-project').value.trim();
|
||||||
|
const source = document.getElementById('cap-source').value;
|
||||||
|
const status = document.getElementById('cap-status');
|
||||||
|
if (!prompt || !response) { status.textContent = 'Need both prompt and response.'; return; }
|
||||||
|
status.textContent = 'Saving…';
|
||||||
|
try {
|
||||||
|
const r = await fetch('/interactions', {
|
||||||
|
method: 'POST',
|
||||||
|
headers: {'Content-Type': 'application/json'},
|
||||||
|
body: JSON.stringify({
|
||||||
|
prompt: prompt, response: response,
|
||||||
|
client: source, project: project, reinforce: true
|
||||||
|
})
|
||||||
|
});
|
||||||
|
if (r.ok) {
|
||||||
|
const data = await r.json();
|
||||||
|
status.innerHTML = '✅ Saved — interaction ' + (data.interaction_id || '?').slice(0,8) +
|
||||||
|
'. Runs through extraction + triage within the hour.<br>' +
|
||||||
|
'<a href="/interactions/' + (data.interaction_id || '') + '">view</a>';
|
||||||
|
document.getElementById('capture-form').reset();
|
||||||
|
} else {
|
||||||
|
status.textContent = '❌ ' + r.status + ': ' + (await r.text()).slice(0, 200);
|
||||||
|
}
|
||||||
|
} catch (err) { status.textContent = '❌ ' + err.message; }
|
||||||
|
});
|
||||||
|
</script>
|
||||||
|
""")
|
||||||
|
lines.append(
|
||||||
|
'<h2>How this works</h2>'
|
||||||
|
'<ul>'
|
||||||
|
'<li><strong>Claude Code</strong> → auto-captured via Stop hook</li>'
|
||||||
|
'<li><strong>OpenClaw</strong> → auto-captured + gets AtoCore context injected on prompt start (Phase 7I)</li>'
|
||||||
|
'<li><strong>Anything else</strong> (Claude Desktop, mobile, web, ChatGPT) → paste here</li>'
|
||||||
|
'</ul>'
|
||||||
|
'<p>The extractor is aggressive about capturing signal — don\'t hand-filter. '
|
||||||
|
'If the conversation had nothing durable, triage will auto-reject.</p>'
|
||||||
|
)
|
||||||
|
|
||||||
|
return render_html(
|
||||||
|
"Capture — AtoCore",
|
||||||
|
"\n".join(lines),
|
||||||
|
breadcrumbs=[("Wiki", "/wiki"), ("Capture", "")],
|
||||||
|
active_path="/wiki/capture",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------
|
||||||
|
# Phase 7E — /wiki/memories/{id}: memory detail page
|
||||||
|
# ---------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def render_memory_detail(memory_id: str) -> str | None:
|
||||||
|
"""Full view of a single memory: content, audit trail, source refs,
|
||||||
|
neighbors, graduation status. Fills the drill-down gap the list
|
||||||
|
views can't."""
|
||||||
|
from atocore.memory.service import get_memory_audit
|
||||||
|
from atocore.models.database import get_connection
|
||||||
|
|
||||||
|
with get_connection() as conn:
|
||||||
|
row = conn.execute("SELECT * FROM memories WHERE id = ?", (memory_id,)).fetchone()
|
||||||
|
if row is None:
|
||||||
|
return None
|
||||||
|
|
||||||
|
import json as _json
|
||||||
|
mem = dict(row)
|
||||||
|
try:
|
||||||
|
tags = _json.loads(mem.get("domain_tags") or "[]") or []
|
||||||
|
except Exception:
|
||||||
|
tags = []
|
||||||
|
|
||||||
|
lines = [f'<h1>{mem["memory_type"]}: <span style="color:var(--text);">{mem["content"][:80]}</span></h1>']
|
||||||
|
if len(mem["content"]) > 80:
|
||||||
|
lines.append(f'<blockquote><p>{mem["content"]}</p></blockquote>')
|
||||||
|
|
||||||
|
# Metadata row
|
||||||
|
meta_items = [
|
||||||
|
f'<span class="tag">{mem["status"]}</span>',
|
||||||
|
f'<strong>{mem["memory_type"]}</strong>',
|
||||||
|
]
|
||||||
|
if mem.get("project"):
|
||||||
|
meta_items.append(f'<a href="/wiki/projects/{mem["project"]}">{mem["project"]}</a>')
|
||||||
|
meta_items.append(f'confidence: <strong>{float(mem.get("confidence") or 0):.2f}</strong>')
|
||||||
|
meta_items.append(f'refs: <strong>{int(mem.get("reference_count") or 0)}</strong>')
|
||||||
|
if mem.get("valid_until"):
|
||||||
|
meta_items.append(f'<span class="mem-expiry">valid until {str(mem["valid_until"])[:10]}</span>')
|
||||||
|
lines.append(f'<p>{" · ".join(meta_items)}</p>')
|
||||||
|
|
||||||
|
if tags:
|
||||||
|
tag_links = " ".join(f'<a href="/wiki/domains/{t}" class="tag-badge">{t}</a>' for t in tags)
|
||||||
|
lines.append(f'<p><span class="mem-tags">{tag_links}</span></p>')
|
||||||
|
|
||||||
|
lines.append(f'<p class="meta">id: <code>{mem["id"]}</code> · created: {mem["created_at"]}'
|
||||||
|
f' · updated: {mem.get("updated_at", "?")}'
|
||||||
|
+ (f' · last referenced: {mem["last_referenced_at"]}' if mem.get("last_referenced_at") else '')
|
||||||
|
+ '</p>')
|
||||||
|
|
||||||
|
# Graduation
|
||||||
|
if mem.get("graduated_to_entity_id"):
|
||||||
|
eid = mem["graduated_to_entity_id"]
|
||||||
|
lines.append(
|
||||||
|
f'<h2>🎓 Graduated</h2>'
|
||||||
|
f'<p>This memory was promoted to a typed entity: '
|
||||||
|
f'<a href="/wiki/entities/{eid}">{eid[:8]}</a></p>'
|
||||||
|
)
|
||||||
|
|
||||||
|
# Source chunk
|
||||||
|
if mem.get("source_chunk_id"):
|
||||||
|
lines.append(f'<h2>Source chunk</h2><p><code>{mem["source_chunk_id"]}</code></p>')
|
||||||
|
|
||||||
|
# Audit trail
|
||||||
|
audit = get_memory_audit(memory_id, limit=50)
|
||||||
|
if audit:
|
||||||
|
lines.append(f'<h2>Audit trail ({len(audit)} events)</h2><ul>')
|
||||||
|
for a in audit:
|
||||||
|
note = f' — {a["note"]}' if a.get("note") else ""
|
||||||
|
lines.append(
|
||||||
|
f'<li><code>{a["timestamp"]}</code> '
|
||||||
|
f'<strong>{a["action"]}</strong> '
|
||||||
|
f'<em>{a["actor"]}</em>{note}</li>'
|
||||||
|
)
|
||||||
|
lines.append('</ul>')
|
||||||
|
|
||||||
|
# Neighbors by shared tag
|
||||||
|
if tags:
|
||||||
|
from atocore.memory.service import get_memories as _get_memories
|
||||||
|
neighbors = []
|
||||||
|
for t in tags[:3]:
|
||||||
|
for other in _get_memories(active_only=True, limit=30):
|
||||||
|
if other.id == memory_id:
|
||||||
|
continue
|
||||||
|
if any(ot == t for ot in (other.domain_tags or [])):
|
||||||
|
neighbors.append(other)
|
||||||
|
# Dedupe
|
||||||
|
seen = set()
|
||||||
|
uniq = []
|
||||||
|
for n in neighbors:
|
||||||
|
if n.id in seen:
|
||||||
|
continue
|
||||||
|
seen.add(n.id)
|
||||||
|
uniq.append(n)
|
||||||
|
if uniq:
|
||||||
|
lines.append(f'<h2>Related (by tag)</h2><ul>')
|
||||||
|
for n in uniq[:10]:
|
||||||
|
lines.append(
|
||||||
|
f'<li><a href="/wiki/memories/{n.id}">[{n.memory_type}] '
|
||||||
|
f'{n.content[:120]}</a>'
|
||||||
|
+ (f' <span class="tag">{n.project}</span>' if n.project else '')
|
||||||
|
+ '</li>'
|
||||||
|
)
|
||||||
|
lines.append('</ul>')
|
||||||
|
|
||||||
|
return render_html(
|
||||||
|
f"Memory {memory_id[:8]}",
|
||||||
|
"\n".join(lines),
|
||||||
|
breadcrumbs=[("Wiki", "/wiki"), ("Memory", "")],
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------
|
||||||
|
# Phase 7F — /wiki/domains/{tag}: cross-project domain view
|
||||||
|
# ---------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def render_domain(tag: str) -> str:
|
||||||
|
"""All memories + entities carrying a given domain_tag, grouped by project.
|
||||||
|
Answers 'what does the brain know about optics, across all projects?'"""
|
||||||
|
tag = (tag or "").strip().lower()
|
||||||
|
if not tag:
|
||||||
|
return render_html("Domain", "<p>No tag specified.</p>",
|
||||||
|
breadcrumbs=[("Wiki", "/wiki"), ("Domains", "")])
|
||||||
|
|
||||||
|
all_mems = get_memories(active_only=True, limit=500)
|
||||||
|
matching = [m for m in all_mems
|
||||||
|
if any((t or "").lower() == tag for t in (m.domain_tags or []))]
|
||||||
|
|
||||||
|
# Group by project
|
||||||
|
by_project: dict[str, list] = {}
|
||||||
|
for m in matching:
|
||||||
|
by_project.setdefault(m.project or "(global)", []).append(m)
|
||||||
|
|
||||||
|
lines = [f'<h1>Domain: <code>{tag}</code></h1>']
|
||||||
|
lines.append(f'<p class="meta">{len(matching)} active memories across {len(by_project)} projects</p>')
|
||||||
|
|
||||||
|
if not matching:
|
||||||
|
lines.append(
|
||||||
|
f'<p>No memories currently carry the tag <code>{tag}</code>.</p>'
|
||||||
|
'<p>Domain tags are assigned by the extractor when it identifies '
|
||||||
|
'the topical scope of a memory. They update over time.</p>'
|
||||||
|
)
|
||||||
|
return render_html(
|
||||||
|
f"Domain: {tag}",
|
||||||
|
"\n".join(lines),
|
||||||
|
breadcrumbs=[("Wiki", "/wiki"), ("Domains", ""), (tag, "")],
|
||||||
|
)
|
||||||
|
|
||||||
|
# Sort projects by count descending, (global) last
|
||||||
|
def sort_key(item: tuple[str, list]) -> tuple[int, int]:
|
||||||
|
proj, mems = item
|
||||||
|
return (1 if proj == "(global)" else 0, -len(mems))
|
||||||
|
|
||||||
|
for proj, mems in sorted(by_project.items(), key=sort_key):
|
||||||
|
proj_link = proj if proj == "(global)" else f'<a href="/wiki/projects/{proj}">{proj}</a>'
|
||||||
|
lines.append(f'<h2>{proj_link} ({len(mems)})</h2><ul>')
|
||||||
|
for m in mems:
|
||||||
|
other_tags = [t for t in (m.domain_tags or []) if t != tag][:3]
|
||||||
|
other_tags_html = ""
|
||||||
|
if other_tags:
|
||||||
|
other_tags_html = ' <span class="mem-tags">' + " ".join(
|
||||||
|
f'<a href="/wiki/domains/{t}" class="tag-badge">{t}</a>' for t in other_tags
|
||||||
|
) + '</span>'
|
||||||
|
lines.append(
|
||||||
|
f'<li><a href="/wiki/memories/{m.id}">[{m.memory_type}] '
|
||||||
|
f'{m.content[:200]}</a>'
|
||||||
|
f' <span class="meta">conf {m.confidence:.2f} · refs {m.reference_count}</span>'
|
||||||
|
f'{other_tags_html}</li>'
|
||||||
|
)
|
||||||
|
lines.append('</ul>')
|
||||||
|
|
||||||
|
# Entities with this tag (if any have tags — currently they might not)
|
||||||
|
try:
|
||||||
|
all_entities = get_entities(limit=500)
|
||||||
|
ent_matching = []
|
||||||
|
for e in all_entities:
|
||||||
|
tags = e.properties.get("domain_tags") if e.properties else []
|
||||||
|
if isinstance(tags, list) and tag in [str(t).lower() for t in tags]:
|
||||||
|
ent_matching.append(e)
|
||||||
|
if ent_matching:
|
||||||
|
lines.append(f'<h2>🔧 Entities ({len(ent_matching)})</h2><ul>')
|
||||||
|
for e in ent_matching:
|
||||||
|
lines.append(
|
||||||
|
f'<li><a href="/wiki/entities/{e.id}">[{e.entity_type}] {e.name}</a>'
|
||||||
|
+ (f' <span class="tag">{e.project}</span>' if e.project else '')
|
||||||
|
+ '</li>'
|
||||||
|
)
|
||||||
|
lines.append('</ul>')
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
return render_html(
|
||||||
|
f"Domain: {tag}",
|
||||||
|
"\n".join(lines),
|
||||||
|
breadcrumbs=[("Wiki", "/wiki"), ("Domains", ""), (tag, "")],
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------
|
||||||
|
# /wiki/activity — autonomous-activity feed
|
||||||
|
# ---------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def render_activity(hours: int = 48, limit: int = 100) -> str:
|
||||||
|
"""Timeline of what the autonomous pipeline did recently. Answers
|
||||||
|
'what has the brain been doing while I was away?'"""
|
||||||
|
from atocore.memory.service import get_recent_audit
|
||||||
|
|
||||||
|
audit = get_recent_audit(limit=limit)
|
||||||
|
|
||||||
|
# Group events by category for summary
|
||||||
|
by_action: dict[str, int] = {}
|
||||||
|
by_actor: dict[str, int] = {}
|
||||||
|
for a in audit:
|
||||||
|
by_action[a["action"]] = by_action.get(a["action"], 0) + 1
|
||||||
|
by_actor[a["actor"]] = by_actor.get(a["actor"], 0) + 1
|
||||||
|
|
||||||
|
lines = [f'<h1>📡 Activity Feed</h1>']
|
||||||
|
lines.append(f'<p class="meta">Last {len(audit)} events in the memory audit log</p>')
|
||||||
|
|
||||||
|
# Summary chips
|
||||||
|
if by_action or by_actor:
|
||||||
|
lines.append('<h2>Summary</h2>')
|
||||||
|
lines.append('<p><strong>By action:</strong> ' +
|
||||||
|
" · ".join(f'{k}: {v}' for k, v in sorted(by_action.items(), key=lambda x: -x[1])) +
|
||||||
|
'</p>')
|
||||||
|
lines.append('<p><strong>By actor:</strong> ' +
|
||||||
|
" · ".join(f'<code>{k}</code>: {v}' for k, v in sorted(by_actor.items(), key=lambda x: -x[1])) +
|
||||||
|
'</p>')
|
||||||
|
|
||||||
|
# Action-type color/emoji
|
||||||
|
action_emoji = {
|
||||||
|
"created": "➕", "promoted": "✅", "rejected": "❌", "invalidated": "🚫",
|
||||||
|
"superseded": "🔀", "reinforced": "🔁", "updated": "✏️",
|
||||||
|
"auto_promoted": "⚡", "created_via_merge": "🔗",
|
||||||
|
"valid_until_extended": "⏳", "tag_canonicalized": "🏷️",
|
||||||
|
}
|
||||||
|
|
||||||
|
lines.append('<h2>Timeline</h2><ul>')
|
||||||
|
for a in audit:
|
||||||
|
emoji = action_emoji.get(a["action"], "•")
|
||||||
|
preview = a.get("content_preview") or ""
|
||||||
|
ts_short = a["timestamp"][:16] if a.get("timestamp") else "?"
|
||||||
|
mid_short = (a.get("memory_id") or "")[:8]
|
||||||
|
note = f' — <em>{a["note"]}</em>' if a.get("note") else ""
|
||||||
|
lines.append(
|
||||||
|
f'<li>{emoji} <code>{ts_short}</code> '
|
||||||
|
f'<strong>{a["action"]}</strong> '
|
||||||
|
f'<em>{a["actor"]}</em> '
|
||||||
|
f'<a href="/wiki/memories/{a["memory_id"]}">{mid_short}</a>'
|
||||||
|
f'{note}'
|
||||||
|
+ (f'<br><span style="opacity:0.6; font-size:0.85rem; margin-left:1.5rem;">{preview[:140]}</span>' if preview else '')
|
||||||
|
+ '</li>'
|
||||||
|
)
|
||||||
|
lines.append('</ul>')
|
||||||
|
|
||||||
|
return render_html(
|
||||||
|
"Activity — AtoCore",
|
||||||
|
"\n".join(lines),
|
||||||
|
breadcrumbs=[("Wiki", "/wiki"), ("Activity", "")],
|
||||||
|
active_path="/wiki/activity",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
_TEMPLATE = """<!DOCTYPE html>
|
_TEMPLATE = """<!DOCTYPE html>
|
||||||
<html lang="en">
|
<html lang="en">
|
||||||
<head>
|
<head>
|
||||||
@@ -290,7 +1051,37 @@ _TEMPLATE = """<!DOCTYPE html>
|
|||||||
hr { border: none; border-top: 1px solid var(--border); margin: 2rem 0; }
|
hr { border: none; border-top: 1px solid var(--border); margin: 2rem 0; }
|
||||||
.breadcrumbs { margin-bottom: 1.5rem; font-size: 0.85em; opacity: 0.7; }
|
.breadcrumbs { margin-bottom: 1.5rem; font-size: 0.85em; opacity: 0.7; }
|
||||||
.breadcrumbs a { opacity: 0.8; }
|
.breadcrumbs a { opacity: 0.8; }
|
||||||
|
.topnav { display: flex; gap: 0.25rem; flex-wrap: wrap; margin-bottom: 1rem; padding-bottom: 0.8rem; border-bottom: 1px solid var(--border); }
|
||||||
|
.topnav-item { padding: 0.35rem 0.8rem; background: var(--card); border: 1px solid var(--border); border-radius: 6px; font-size: 0.88rem; color: var(--text); opacity: 0.75; text-decoration: none; }
|
||||||
|
.topnav-item:hover { opacity: 1; background: var(--hover); text-decoration: none; }
|
||||||
|
.topnav-item.active { background: var(--accent); color: white; border-color: var(--accent); opacity: 1; }
|
||||||
|
.topnav-item.active:hover { background: var(--accent); }
|
||||||
|
.activity-snippet { background: var(--card); border: 1px solid var(--border); border-radius: 8px; padding: 1rem; margin: 1rem 0; }
|
||||||
|
.activity-snippet h3 { color: var(--accent); margin-bottom: 0.4rem; }
|
||||||
|
.activity-snippet ul { margin: 0.3rem 0 0 1.2rem; font-size: 0.9rem; }
|
||||||
|
.activity-snippet li { margin-bottom: 0.2rem; }
|
||||||
|
.stat-row { display: flex; gap: 1rem; flex-wrap: wrap; font-size: 0.9rem; margin: 0.4rem 0; }
|
||||||
|
.stat-row span { padding: 0.1rem 0.4rem; background: var(--hover); border-radius: 4px; }
|
||||||
.meta { font-size: 0.8em; opacity: 0.5; margin-top: 0.5rem; }
|
.meta { font-size: 0.8em; opacity: 0.5; margin-top: 0.5rem; }
|
||||||
|
.wikilink { color: var(--accent); text-decoration: none; border-bottom: 1px dashed transparent; }
|
||||||
|
.wikilink:hover { border-bottom-color: var(--accent); }
|
||||||
|
.wikilink-cross { border-bottom-style: dotted; }
|
||||||
|
.wikilink-scope { font-size: 0.75em; opacity: 0.6; font-style: italic; }
|
||||||
|
.redlink { color: #d0473d; text-decoration: none; font-style: italic; border-bottom: 1px dashed #d0473d; }
|
||||||
|
.redlink:hover { background: rgba(208, 71, 61, 0.08); }
|
||||||
|
.new-entity-form { display: flex; flex-direction: column; gap: 0.9rem; max-width: 520px; background: var(--card); border: 1px solid var(--border); border-radius: 8px; padding: 1rem 1.2rem; }
|
||||||
|
.new-entity-form label { display: flex; flex-direction: column; font-size: 0.88rem; opacity: 0.8; }
|
||||||
|
.new-entity-form input, .new-entity-form select, .new-entity-form textarea { margin-top: 0.3rem; padding: 0.45rem 0.6rem; background: var(--bg); border: 1px solid var(--border); border-radius: 4px; color: var(--text); font-size: 0.95rem; }
|
||||||
|
.new-entity-form button { align-self: flex-start; padding: 0.5rem 1.1rem; background: var(--accent); color: white; border: none; border-radius: 6px; cursor: pointer; }
|
||||||
|
.new-entity-result { font-size: 0.9rem; opacity: 0.85; min-height: 1em; }
|
||||||
|
.evidence-strip { display: flex; flex-wrap: wrap; gap: 0.75rem; margin: 0.75rem 0 1.25rem; }
|
||||||
|
.evidence-tile { margin: 0; background: var(--card); border: 1px solid var(--border); border-radius: 6px; padding: 0.4rem; max-width: 270px; }
|
||||||
|
.evidence-tile img { display: block; max-width: 100%; height: auto; border-radius: 3px; }
|
||||||
|
.evidence-tile figcaption { font-size: 0.8rem; margin-top: 0.35rem; opacity: 0.85; }
|
||||||
|
.evidence-pdf, .evidence-other { padding: 0.6rem 0.8rem; font-size: 0.9rem; }
|
||||||
|
.artifact-full figure, .artifact-full { margin: 0 0 1rem; }
|
||||||
|
.artifact-full img { display: block; max-width: 100%; height: auto; border: 1px solid var(--border); border-radius: 4px; }
|
||||||
|
.artifact-full figcaption { font-size: 0.9rem; margin-top: 0.5rem; opacity: 0.85; }
|
||||||
.tag { background: var(--accent); color: var(--bg); padding: 0.1rem 0.4rem; border-radius: 3px; font-size: 0.75em; margin-left: 0.3rem; }
|
.tag { background: var(--accent); color: var(--bg); padding: 0.1rem 0.4rem; border-radius: 3px; font-size: 0.75em; margin-left: 0.3rem; }
|
||||||
.search-box { display: flex; gap: 0.5rem; margin: 1.5rem 0; }
|
.search-box { display: flex; gap: 0.5rem; margin: 1.5rem 0; }
|
||||||
.search-box input {
|
.search-box input {
|
||||||
@@ -324,7 +1115,41 @@ _TEMPLATE = """<!DOCTYPE html>
|
|||||||
.tag-badge:hover { opacity: 0.85; text-decoration: none; }
|
.tag-badge:hover { opacity: 0.85; text-decoration: none; }
|
||||||
.mem-expiry { font-size: 0.75rem; color: #d97706; font-style: italic; margin-left: 0.4rem; }
|
.mem-expiry { font-size: 0.75rem; color: #d97706; font-style: italic; margin-left: 0.4rem; }
|
||||||
@media (prefers-color-scheme: dark) { .mem-expiry { color: #fbbf24; } }
|
@media (prefers-color-scheme: dark) { .mem-expiry { color: #fbbf24; } }
|
||||||
|
/* Phase 6 C.2 — Emerging projects section */
|
||||||
|
.emerging-intro { font-size: 0.9rem; opacity: 0.75; margin-bottom: 0.8rem; }
|
||||||
|
.emerging-grid { display: grid; grid-template-columns: repeat(auto-fill, minmax(280px, 1fr)); gap: 1rem; margin-bottom: 1rem; }
|
||||||
|
.emerging-card { background: var(--card); border: 1px dashed var(--accent); border-radius: 8px; padding: 1rem; }
|
||||||
|
.emerging-card h3 { margin: 0 0 0.3rem 0; color: var(--accent); font-family: monospace; font-size: 1rem; }
|
||||||
|
.emerging-count { font-size: 0.8rem; opacity: 0.6; margin-bottom: 0.5rem; }
|
||||||
|
.emerging-samples { font-size: 0.85rem; margin: 0.5rem 0; padding-left: 1.2rem; opacity: 0.8; }
|
||||||
|
.emerging-samples li { margin-bottom: 0.25rem; }
|
||||||
|
.btn-register-emerging { width: 100%; padding: 0.45rem 0.9rem; background: var(--accent); color: white; border: 1px solid var(--accent); border-radius: 4px; cursor: pointer; font-size: 0.88rem; font-weight: 500; margin-top: 0.5rem; }
|
||||||
|
.btn-register-emerging:hover { opacity: 0.9; }
|
||||||
</style>
|
</style>
|
||||||
|
<script>
|
||||||
|
async function registerEmerging(projectId) {
|
||||||
|
if (!confirm(`Register "${projectId}" as a first-class project?\n\nThis creates:\n• /wiki/projects/${projectId} page\n• System map + gaps + killer queries\n• Triage + graduation support\n\nIngest root defaults to vault:incoming/projects/${projectId}/`)) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
const r = await fetch('/admin/projects/register-emerging', {
|
||||||
|
method: 'POST',
|
||||||
|
headers: {'Content-Type': 'application/json'},
|
||||||
|
body: JSON.stringify({project_id: projectId}),
|
||||||
|
});
|
||||||
|
if (r.ok) {
|
||||||
|
const data = await r.json();
|
||||||
|
alert(data.message || `Registered ${projectId}`);
|
||||||
|
window.location.reload();
|
||||||
|
} else {
|
||||||
|
const err = await r.text();
|
||||||
|
alert(`Registration failed: ${r.status}\n${err.substring(0, 300)}`);
|
||||||
|
}
|
||||||
|
} catch (e) {
|
||||||
|
alert(`Network error: ${e.message}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
</script>
|
||||||
</head>
|
</head>
|
||||||
<body>
|
<body>
|
||||||
{{nav}}
|
{{nav}}
|
||||||
|
|||||||
@@ -2,7 +2,8 @@
|
|||||||
|
|
||||||
from contextlib import asynccontextmanager
|
from contextlib import asynccontextmanager
|
||||||
|
|
||||||
from fastapi import FastAPI
|
from fastapi import APIRouter, FastAPI
|
||||||
|
from fastapi.routing import APIRoute
|
||||||
|
|
||||||
from atocore import __version__
|
from atocore import __version__
|
||||||
from atocore.api.routes import router
|
from atocore.api.routes import router
|
||||||
@@ -53,6 +54,79 @@ app = FastAPI(
|
|||||||
app.include_router(router)
|
app.include_router(router)
|
||||||
|
|
||||||
|
|
||||||
|
# Public API v1 — stable contract for external clients (AKC, OpenClaw, etc.).
|
||||||
|
# Paths listed here are re-mounted under /v1 as aliases of the existing
|
||||||
|
# unversioned handlers. Unversioned paths continue to work; new endpoints
|
||||||
|
# land at the latest version; breaking schema changes bump the prefix.
|
||||||
|
_V1_PUBLIC_PATHS = {
|
||||||
|
"/entities",
|
||||||
|
"/entities/{entity_id}",
|
||||||
|
"/entities/{entity_id}/promote",
|
||||||
|
"/entities/{entity_id}/reject",
|
||||||
|
"/entities/{entity_id}/invalidate",
|
||||||
|
"/entities/{entity_id}/supersede",
|
||||||
|
"/entities/{entity_id}/audit",
|
||||||
|
"/relationships",
|
||||||
|
"/ingest",
|
||||||
|
"/ingest/sources",
|
||||||
|
"/context/build",
|
||||||
|
"/query",
|
||||||
|
"/projects",
|
||||||
|
"/projects/{project_name}",
|
||||||
|
"/projects/{project_name}/refresh",
|
||||||
|
"/projects/{project_name}/mirror",
|
||||||
|
"/projects/{project_name}/mirror.html",
|
||||||
|
"/memory",
|
||||||
|
"/memory/{memory_id}",
|
||||||
|
"/memory/{memory_id}/audit",
|
||||||
|
"/memory/{memory_id}/promote",
|
||||||
|
"/memory/{memory_id}/reject",
|
||||||
|
"/memory/{memory_id}/invalidate",
|
||||||
|
"/memory/{memory_id}/supersede",
|
||||||
|
"/project/state",
|
||||||
|
"/project/state/{project_name}",
|
||||||
|
"/interactions",
|
||||||
|
"/interactions/{interaction_id}",
|
||||||
|
"/interactions/{interaction_id}/reinforce",
|
||||||
|
"/interactions/{interaction_id}/extract",
|
||||||
|
"/health",
|
||||||
|
"/sources",
|
||||||
|
"/stats",
|
||||||
|
# Issue F: asset store + evidence query
|
||||||
|
"/assets",
|
||||||
|
"/assets/{asset_id}",
|
||||||
|
"/assets/{asset_id}/thumbnail",
|
||||||
|
"/assets/{asset_id}/meta",
|
||||||
|
"/entities/{entity_id}/evidence",
|
||||||
|
# Issue D: engineering query surface (decisions, systems, components,
|
||||||
|
# gaps, evidence, impact, changes)
|
||||||
|
"/engineering/projects/{project_name}/systems",
|
||||||
|
"/engineering/decisions",
|
||||||
|
"/engineering/components/{component_id}/requirements",
|
||||||
|
"/engineering/changes",
|
||||||
|
"/engineering/gaps",
|
||||||
|
"/engineering/gaps/orphan-requirements",
|
||||||
|
"/engineering/gaps/risky-decisions",
|
||||||
|
"/engineering/gaps/unsupported-claims",
|
||||||
|
"/engineering/impact",
|
||||||
|
"/engineering/evidence",
|
||||||
|
}
|
||||||
|
|
||||||
|
_v1_router = APIRouter(prefix="/v1", tags=["v1"])
|
||||||
|
for _route in list(router.routes):
|
||||||
|
if isinstance(_route, APIRoute) and _route.path in _V1_PUBLIC_PATHS:
|
||||||
|
_v1_router.add_api_route(
|
||||||
|
_route.path,
|
||||||
|
_route.endpoint,
|
||||||
|
methods=list(_route.methods),
|
||||||
|
response_model=_route.response_model,
|
||||||
|
response_class=_route.response_class,
|
||||||
|
name=f"v1_{_route.name}",
|
||||||
|
include_in_schema=True,
|
||||||
|
)
|
||||||
|
app.include_router(_v1_router)
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
import uvicorn
|
import uvicorn
|
||||||
|
|
||||||
|
|||||||
200
src/atocore/memory/_dedup_prompt.py
Normal file
200
src/atocore/memory/_dedup_prompt.py
Normal file
@@ -0,0 +1,200 @@
|
|||||||
|
"""Shared LLM prompt + parser for memory dedup (Phase 7A).
|
||||||
|
|
||||||
|
Stdlib-only — must be importable from both the in-container service
|
||||||
|
layer (when a user clicks "scan for duplicates" in the UI) and the
|
||||||
|
host-side batch script (``scripts/memory_dedup.py``), which runs on
|
||||||
|
Dalidou where the container's Python deps are not available.
|
||||||
|
|
||||||
|
The prompt instructs the model to draft a UNIFIED memory that
|
||||||
|
preserves every specific detail from the sources. We never want a
|
||||||
|
merge to lose information — if two memories disagree on a number, the
|
||||||
|
merged content should surface both with context.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
DEDUP_PROMPT_VERSION = "dedup-0.1.0"
|
||||||
|
MAX_CONTENT_CHARS = 1000
|
||||||
|
MAX_SOURCES = 8 # cluster size cap — bigger clusters are suspicious
|
||||||
|
|
||||||
|
SYSTEM_PROMPT = """You consolidate near-duplicate memories for AtoCore, a personal context engine.
|
||||||
|
|
||||||
|
Given 2-8 memories that a semantic-similarity scan flagged as likely duplicates, draft a UNIFIED replacement that preserves every specific detail from every source.
|
||||||
|
|
||||||
|
CORE PRINCIPLE: information never gets lost. If the sources disagree on a number, date, vendor, or spec, surface BOTH with attribution (e.g., "quoted at $3.2k on 2026-03-01, revised to $3.8k on 2026-04-10"). If one source is more specific than another, keep the specificity. If they say the same thing differently, pick the clearer wording.
|
||||||
|
|
||||||
|
YOU MUST:
|
||||||
|
- Produce content under 500 characters that reads as a single coherent statement
|
||||||
|
- Keep all project/vendor/person/part names that appear in any source
|
||||||
|
- Keep all numbers, dates, and identifiers
|
||||||
|
- Keep the strongest claim wording ("ratified", "decided", "committed") if any source has it
|
||||||
|
- Propose domain_tags as a UNION of the sources' tags (lowercase, deduped, cap 6)
|
||||||
|
- Return valid_until = latest non-null valid_until across sources, or null if any source has null (permanent beats transient)
|
||||||
|
|
||||||
|
REFUSE TO MERGE (return action="reject") if:
|
||||||
|
- The memories are actually about DIFFERENT subjects that just share vocabulary (e.g., "p04 mirror" and "p05 mirror" — same project bucket means same project, but different components)
|
||||||
|
- One memory CONTRADICTS another and you cannot reconcile them — flag for contradiction review instead
|
||||||
|
- The sources span different time snapshots of a changing state that should stay as a timeline, not be collapsed
|
||||||
|
|
||||||
|
OUTPUT — raw JSON, no prose, no markdown fences:
|
||||||
|
{
|
||||||
|
"action": "merge" | "reject",
|
||||||
|
"content": "the unified memory content",
|
||||||
|
"memory_type": "knowledge|project|preference|adaptation|episodic|identity",
|
||||||
|
"project": "project-slug or empty",
|
||||||
|
"domain_tags": ["tag1", "tag2"],
|
||||||
|
"confidence": 0.5,
|
||||||
|
"reason": "one sentence explaining the merge (or the rejection)"
|
||||||
|
}
|
||||||
|
|
||||||
|
On action=reject, still fill content with a short explanation and set confidence=0."""
|
||||||
|
|
||||||
|
|
||||||
|
TIER2_SYSTEM_PROMPT = """You are the second-opinion reviewer for AtoCore's memory-consolidation pipeline.
|
||||||
|
|
||||||
|
A tier-1 model (cheaper, faster) already drafted a unified memory from N near-duplicate source memories. Your job is to either CONFIRM the merge (refining the content if you see a clearer phrasing) or OVERRIDE with action="reject" if the tier-1 missed something important.
|
||||||
|
|
||||||
|
You must be STRICTER than tier-1. Specifically, REJECT if:
|
||||||
|
- The sources are about different subjects that share vocabulary (e.g., different components within the same project)
|
||||||
|
- The tier-1 draft dropped specifics that existed in the sources (numbers, dates, vendors, people, part IDs)
|
||||||
|
- One source contradicts another and the draft glossed over it
|
||||||
|
- The sources span a timeline of a changing state (should be preserved as a sequence, not collapsed)
|
||||||
|
|
||||||
|
If you CONFIRM, you may polish the content — but preserve every specific from every source.
|
||||||
|
|
||||||
|
Same output schema as tier-1:
|
||||||
|
{
|
||||||
|
"action": "merge" | "reject",
|
||||||
|
"content": "the unified memory content",
|
||||||
|
"memory_type": "knowledge|project|preference|adaptation|episodic|identity",
|
||||||
|
"project": "project-slug or empty",
|
||||||
|
"domain_tags": ["tag1", "tag2"],
|
||||||
|
"confidence": 0.5,
|
||||||
|
"reason": "one sentence — what you confirmed or why you overrode"
|
||||||
|
}
|
||||||
|
|
||||||
|
Raw JSON only, no prose, no markdown fences."""
|
||||||
|
|
||||||
|
|
||||||
|
def build_tier2_user_message(sources: list[dict[str, Any]], tier1_verdict: dict[str, Any]) -> str:
|
||||||
|
"""Format tier-2 review payload: same sources + tier-1's draft."""
|
||||||
|
base = build_user_message(sources)
|
||||||
|
draft_summary = (
|
||||||
|
f"\n\n--- TIER-1 DRAFT (for your review) ---\n"
|
||||||
|
f"action: {tier1_verdict.get('action')}\n"
|
||||||
|
f"confidence: {tier1_verdict.get('confidence', 0):.2f}\n"
|
||||||
|
f"proposed content: {(tier1_verdict.get('content') or '')[:600]}\n"
|
||||||
|
f"proposed memory_type: {tier1_verdict.get('memory_type', '')}\n"
|
||||||
|
f"proposed project: {tier1_verdict.get('project', '')}\n"
|
||||||
|
f"proposed tags: {tier1_verdict.get('domain_tags', [])}\n"
|
||||||
|
f"tier-1 reason: {tier1_verdict.get('reason', '')[:300]}\n"
|
||||||
|
f"---\n\n"
|
||||||
|
f"Return your JSON verdict now. Confirm or override."
|
||||||
|
)
|
||||||
|
return base.replace("Return the JSON object now.", "").rstrip() + draft_summary
|
||||||
|
|
||||||
|
|
||||||
|
def build_user_message(sources: list[dict[str, Any]]) -> str:
|
||||||
|
"""Format N source memories for the model to consolidate.
|
||||||
|
|
||||||
|
Each source dict should carry id, content, project, memory_type,
|
||||||
|
domain_tags, confidence, valid_until, reference_count.
|
||||||
|
"""
|
||||||
|
lines = [f"You have {len(sources)} source memories in the same (project, memory_type) bucket:\n"]
|
||||||
|
for i, src in enumerate(sources[:MAX_SOURCES], start=1):
|
||||||
|
tags = src.get("domain_tags") or []
|
||||||
|
if isinstance(tags, str):
|
||||||
|
try:
|
||||||
|
tags = json.loads(tags)
|
||||||
|
except Exception:
|
||||||
|
tags = []
|
||||||
|
lines.append(
|
||||||
|
f"--- Source {i} (id={src.get('id','?')[:8]}, "
|
||||||
|
f"refs={src.get('reference_count',0)}, "
|
||||||
|
f"conf={src.get('confidence',0):.2f}, "
|
||||||
|
f"valid_until={src.get('valid_until') or 'permanent'}) ---"
|
||||||
|
)
|
||||||
|
lines.append(f"project: {src.get('project','')}")
|
||||||
|
lines.append(f"type: {src.get('memory_type','')}")
|
||||||
|
lines.append(f"tags: {tags}")
|
||||||
|
lines.append(f"content: {(src.get('content') or '')[:MAX_CONTENT_CHARS]}")
|
||||||
|
lines.append("")
|
||||||
|
lines.append("Return the JSON object now.")
|
||||||
|
return "\n".join(lines)
|
||||||
|
|
||||||
|
|
||||||
|
def parse_merge_verdict(raw_output: str) -> dict[str, Any] | None:
|
||||||
|
"""Strip markdown fences / leading prose and return the parsed JSON
|
||||||
|
object. Returns None on parse failure."""
|
||||||
|
text = (raw_output or "").strip()
|
||||||
|
if text.startswith("```"):
|
||||||
|
text = text.strip("`")
|
||||||
|
nl = text.find("\n")
|
||||||
|
if nl >= 0:
|
||||||
|
text = text[nl + 1:]
|
||||||
|
if text.endswith("```"):
|
||||||
|
text = text[:-3]
|
||||||
|
text = text.strip()
|
||||||
|
|
||||||
|
if not text.lstrip().startswith("{"):
|
||||||
|
start = text.find("{")
|
||||||
|
end = text.rfind("}")
|
||||||
|
if start >= 0 and end > start:
|
||||||
|
text = text[start:end + 1]
|
||||||
|
|
||||||
|
try:
|
||||||
|
parsed = json.loads(text)
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
return None
|
||||||
|
if not isinstance(parsed, dict):
|
||||||
|
return None
|
||||||
|
return parsed
|
||||||
|
|
||||||
|
|
||||||
|
def normalize_merge_verdict(verdict: dict[str, Any]) -> dict[str, Any] | None:
|
||||||
|
"""Validate + normalize a raw merge verdict. Returns None if the
|
||||||
|
verdict is unusable (no content, unknown action)."""
|
||||||
|
action = str(verdict.get("action") or "").strip().lower()
|
||||||
|
if action not in ("merge", "reject"):
|
||||||
|
return None
|
||||||
|
|
||||||
|
content = str(verdict.get("content") or "").strip()
|
||||||
|
if not content:
|
||||||
|
return None
|
||||||
|
|
||||||
|
memory_type = str(verdict.get("memory_type") or "knowledge").strip().lower()
|
||||||
|
project = str(verdict.get("project") or "").strip()
|
||||||
|
|
||||||
|
raw_tags = verdict.get("domain_tags") or []
|
||||||
|
if isinstance(raw_tags, str):
|
||||||
|
raw_tags = [t.strip() for t in raw_tags.split(",") if t.strip()]
|
||||||
|
if not isinstance(raw_tags, list):
|
||||||
|
raw_tags = []
|
||||||
|
tags: list[str] = []
|
||||||
|
for t in raw_tags[:6]:
|
||||||
|
if not isinstance(t, str):
|
||||||
|
continue
|
||||||
|
tt = t.strip().lower()
|
||||||
|
if tt and tt not in tags:
|
||||||
|
tags.append(tt)
|
||||||
|
|
||||||
|
try:
|
||||||
|
confidence = float(verdict.get("confidence", 0.5))
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
confidence = 0.5
|
||||||
|
confidence = max(0.0, min(1.0, confidence))
|
||||||
|
|
||||||
|
reason = str(verdict.get("reason") or "").strip()[:500]
|
||||||
|
|
||||||
|
return {
|
||||||
|
"action": action,
|
||||||
|
"content": content[:1000],
|
||||||
|
"memory_type": memory_type,
|
||||||
|
"project": project,
|
||||||
|
"domain_tags": tags,
|
||||||
|
"confidence": confidence,
|
||||||
|
"reason": reason,
|
||||||
|
}
|
||||||
@@ -21,7 +21,7 @@ from __future__ import annotations
|
|||||||
import json
|
import json
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
LLM_EXTRACTOR_VERSION = "llm-0.5.0"
|
LLM_EXTRACTOR_VERSION = "llm-0.6.0" # bolder unknown-project tagging
|
||||||
MAX_RESPONSE_CHARS = 8000
|
MAX_RESPONSE_CHARS = 8000
|
||||||
MAX_PROMPT_CHARS = 2000
|
MAX_PROMPT_CHARS = 2000
|
||||||
MEMORY_TYPES = {"identity", "preference", "project", "episodic", "knowledge", "adaptation"}
|
MEMORY_TYPES = {"identity", "preference", "project", "episodic", "knowledge", "adaptation"}
|
||||||
@@ -30,7 +30,24 @@ SYSTEM_PROMPT = """You extract memory candidates from LLM conversation turns for
|
|||||||
|
|
||||||
AtoCore is the brain for Atomaste's engineering work. Known projects:
|
AtoCore is the brain for Atomaste's engineering work. Known projects:
|
||||||
p04-gigabit, p05-interferometer, p06-polisher, atomizer-v2, atocore,
|
p04-gigabit, p05-interferometer, p06-polisher, atomizer-v2, atocore,
|
||||||
abb-space. Unknown project names — still tag them, the system auto-detects.
|
abb-space.
|
||||||
|
|
||||||
|
UNKNOWN PROJECT/TOOL DETECTION (important): when a memory is clearly
|
||||||
|
about a named tool, product, project, or system that is NOT in the
|
||||||
|
known list above, use a slugified version of that name as the project
|
||||||
|
tag (e.g., "apm" for "Atomaste Part Manager", "foo-bar" for "Foo Bar
|
||||||
|
System"). DO NOT default to a nearest registered match just because
|
||||||
|
APM isn't listed — that's misattribution. The system's Living
|
||||||
|
Taxonomy detector scans for these unregistered tags and surfaces them
|
||||||
|
for one-click registration once they appear in ≥3 memories. Your job
|
||||||
|
is to be honest about scope, not to squeeze everything into existing
|
||||||
|
buckets.
|
||||||
|
|
||||||
|
Exception: if the memory is about a registered project that merely
|
||||||
|
uses or integrates with an unknown tool (e.g., "p04 parts are missing
|
||||||
|
materials in APM"), tag with the registered project (p04-gigabit) and
|
||||||
|
mention the tool in content. Only use an unknown tool as the project
|
||||||
|
tag when the tool itself is the primary subject.
|
||||||
|
|
||||||
Your job is to emit SIGNALS that matter for future context. Be aggressive:
|
Your job is to emit SIGNALS that matter for future context. Be aggressive:
|
||||||
err on the side of capturing useful signal. Triage filters noise downstream.
|
err on the side of capturing useful signal. Triage filters noise downstream.
|
||||||
|
|||||||
158
src/atocore/memory/_tag_canon_prompt.py
Normal file
158
src/atocore/memory/_tag_canon_prompt.py
Normal file
@@ -0,0 +1,158 @@
|
|||||||
|
"""Shared LLM prompt + parser for tag canonicalization (Phase 7C).
|
||||||
|
|
||||||
|
Stdlib-only, importable from both the in-container service layer and the
|
||||||
|
host-side batch script that shells out to ``claude -p``.
|
||||||
|
|
||||||
|
The prompt instructs the model to propose a map of domain_tag aliases
|
||||||
|
to their canonical form. Confidence is key here — we AUTO-APPLY high-
|
||||||
|
confidence aliases; low-confidence go to human review. Over-merging
|
||||||
|
distinct concepts ("optics" vs "optical" — sometimes equivalent,
|
||||||
|
sometimes not) destroys cross-cutting retrieval, so the model is
|
||||||
|
instructed to err conservative.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
TAG_CANON_PROMPT_VERSION = "tagcanon-0.1.0"
|
||||||
|
MAX_TAGS_IN_PROMPT = 100
|
||||||
|
|
||||||
|
SYSTEM_PROMPT = """You canonicalize domain tags for AtoCore's memory layer.
|
||||||
|
|
||||||
|
Input: a distribution of lowercase domain tags (keyword → usage count across active memories). Examples: "firmware: 23", "fw: 5", "firmware-control: 3", "optics: 18", "optical: 2".
|
||||||
|
|
||||||
|
Your job: identify aliases — distinct strings that refer to the SAME concept — and map them to a single canonical form. The canonical should be the clearest / most-used / most-descriptive variant.
|
||||||
|
|
||||||
|
STRICT RULES:
|
||||||
|
|
||||||
|
1. ONLY propose aliases that are UNAMBIGUOUSLY equivalent. Examples:
|
||||||
|
- "fw" → "firmware" (abbreviation)
|
||||||
|
- "firmware-control" → "firmware" (compound narrowing — only if usage context makes it clear the narrower one is never used to DISTINGUISH from firmware-in-general)
|
||||||
|
- "py" → "python"
|
||||||
|
- "ml" → "machine-learning"
|
||||||
|
Do NOT merge:
|
||||||
|
- "optics" vs "optical" — these CAN diverge ("optics" = subsystem/product domain; "optical" = adjective used in non-optics contexts)
|
||||||
|
- "p04" vs "p04-gigabit" — project ids are their own namespace, never canonicalize
|
||||||
|
- "thermal" vs "temperature" — related but distinct
|
||||||
|
- Anything where you're not sure — skip it, human review will catch real aliases next week
|
||||||
|
|
||||||
|
2. Confidence scale:
|
||||||
|
0.9+ obvious abbreviation, very high usage disparity, no plausible alternative meaning
|
||||||
|
0.7-0.9 likely alias, one-word-diff or standard contraction
|
||||||
|
0.5-0.7 plausible but requires context — low count on alias side
|
||||||
|
<0.5 DO NOT PROPOSE — if you're under 0.5, skip the pair entirely
|
||||||
|
AtoCore auto-applies aliases at confidence >= 0.8; anything below goes to human review.
|
||||||
|
|
||||||
|
3. The CANONICAL must actually appear in the input list (don't invent a new term).
|
||||||
|
|
||||||
|
4. Never propose `alias == canonical`. Never propose circular mappings.
|
||||||
|
|
||||||
|
5. Project tags (p04, p05, p06, abb-space, atomizer-v2, atocore, apm) are OFF LIMITS — they are project identifiers, not concepts. Leave them alone entirely.
|
||||||
|
|
||||||
|
OUTPUT — raw JSON, no prose, no markdown fences:
|
||||||
|
{
|
||||||
|
"aliases": [
|
||||||
|
{"alias": "fw", "canonical": "firmware", "confidence": 0.95, "reason": "fw is a standard abbreviation of firmware; 5 uses vs 23"},
|
||||||
|
{"alias": "ml", "canonical": "machine-learning", "confidence": 0.90, "reason": "ml is the universal abbreviation"}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
|
||||||
|
Empty aliases list is fine if nothing in the distribution is a clear alias. Err conservative — one false merge can pollute retrieval for hundreds of memories."""
|
||||||
|
|
||||||
|
|
||||||
|
def build_user_message(tag_distribution: dict[str, int]) -> str:
|
||||||
|
"""Format the tag distribution for the model.
|
||||||
|
|
||||||
|
Limited to MAX_TAGS_IN_PROMPT entries, sorted by count descending
|
||||||
|
so high-usage tags appear first (the LLM uses them as anchor points
|
||||||
|
for canonical selection).
|
||||||
|
"""
|
||||||
|
if not tag_distribution:
|
||||||
|
return "Empty tag distribution — return {\"aliases\": []}."
|
||||||
|
|
||||||
|
sorted_tags = sorted(tag_distribution.items(), key=lambda x: x[1], reverse=True)
|
||||||
|
top = sorted_tags[:MAX_TAGS_IN_PROMPT]
|
||||||
|
lines = [f"{tag}: {count}" for tag, count in top]
|
||||||
|
return (
|
||||||
|
f"Tag distribution across {sum(tag_distribution.values())} total tag references "
|
||||||
|
f"(showing top {len(top)} of {len(tag_distribution)} unique tags):\n\n"
|
||||||
|
+ "\n".join(lines)
|
||||||
|
+ "\n\nReturn the JSON aliases map now. Only propose UNAMBIGUOUS equivalents."
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def parse_canon_output(raw_output: str) -> list[dict[str, Any]]:
|
||||||
|
"""Strip markdown fences / prose and return the parsed aliases list."""
|
||||||
|
text = (raw_output or "").strip()
|
||||||
|
if text.startswith("```"):
|
||||||
|
text = text.strip("`")
|
||||||
|
nl = text.find("\n")
|
||||||
|
if nl >= 0:
|
||||||
|
text = text[nl + 1:]
|
||||||
|
if text.endswith("```"):
|
||||||
|
text = text[:-3]
|
||||||
|
text = text.strip()
|
||||||
|
|
||||||
|
if not text.lstrip().startswith("{"):
|
||||||
|
start = text.find("{")
|
||||||
|
end = text.rfind("}")
|
||||||
|
if start >= 0 and end > start:
|
||||||
|
text = text[start:end + 1]
|
||||||
|
|
||||||
|
try:
|
||||||
|
parsed = json.loads(text)
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
return []
|
||||||
|
|
||||||
|
if not isinstance(parsed, dict):
|
||||||
|
return []
|
||||||
|
aliases = parsed.get("aliases") or []
|
||||||
|
if not isinstance(aliases, list):
|
||||||
|
return []
|
||||||
|
return [a for a in aliases if isinstance(a, dict)]
|
||||||
|
|
||||||
|
|
||||||
|
# Project tokens that must never be canonicalized — they're project ids,
|
||||||
|
# not concepts. Keep this list in sync with the registered projects.
|
||||||
|
# Safe to be over-inclusive; extra entries just skip canonicalization.
|
||||||
|
PROTECTED_PROJECT_TOKENS = frozenset({
|
||||||
|
"p04", "p04-gigabit",
|
||||||
|
"p05", "p05-interferometer",
|
||||||
|
"p06", "p06-polisher",
|
||||||
|
"p08", "abb-space",
|
||||||
|
"atomizer", "atomizer-v2",
|
||||||
|
"atocore", "apm",
|
||||||
|
})
|
||||||
|
|
||||||
|
|
||||||
|
def normalize_alias_item(item: dict[str, Any]) -> dict[str, Any] | None:
|
||||||
|
"""Validate one raw alias proposal. Returns None if unusable.
|
||||||
|
|
||||||
|
Filters: non-strings, empty strings, identity mappings, protected
|
||||||
|
project tokens on either side.
|
||||||
|
"""
|
||||||
|
alias = str(item.get("alias") or "").strip().lower()
|
||||||
|
canonical = str(item.get("canonical") or "").strip().lower()
|
||||||
|
if not alias or not canonical:
|
||||||
|
return None
|
||||||
|
if alias == canonical:
|
||||||
|
return None
|
||||||
|
if alias in PROTECTED_PROJECT_TOKENS or canonical in PROTECTED_PROJECT_TOKENS:
|
||||||
|
return None
|
||||||
|
|
||||||
|
try:
|
||||||
|
confidence = float(item.get("confidence", 0.0))
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
confidence = 0.0
|
||||||
|
confidence = max(0.0, min(1.0, confidence))
|
||||||
|
|
||||||
|
reason = str(item.get("reason") or "").strip()[:300]
|
||||||
|
|
||||||
|
return {
|
||||||
|
"alias": alias,
|
||||||
|
"canonical": canonical,
|
||||||
|
"confidence": confidence,
|
||||||
|
"reason": reason,
|
||||||
|
}
|
||||||
@@ -456,14 +456,26 @@ def update_memory(
|
|||||||
return False
|
return False
|
||||||
|
|
||||||
|
|
||||||
def invalidate_memory(memory_id: str, actor: str = "api") -> bool:
|
def invalidate_memory(
|
||||||
"""Mark a memory as invalid (error correction)."""
|
memory_id: str,
|
||||||
return update_memory(memory_id, status="invalid", actor=actor)
|
actor: str = "api",
|
||||||
|
reason: str = "",
|
||||||
|
) -> bool:
|
||||||
|
"""Mark a memory as invalid (error correction).
|
||||||
|
|
||||||
|
``reason`` lands in the audit row's ``note`` column so future
|
||||||
|
review has a record of why this memory was retracted.
|
||||||
|
"""
|
||||||
|
return update_memory(memory_id, status="invalid", actor=actor, note=reason)
|
||||||
|
|
||||||
|
|
||||||
def supersede_memory(memory_id: str, actor: str = "api") -> bool:
|
def supersede_memory(
|
||||||
|
memory_id: str,
|
||||||
|
actor: str = "api",
|
||||||
|
reason: str = "",
|
||||||
|
) -> bool:
|
||||||
"""Mark a memory as superseded (replaced by newer info)."""
|
"""Mark a memory as superseded (replaced by newer info)."""
|
||||||
return update_memory(memory_id, status="superseded", actor=actor)
|
return update_memory(memory_id, status="superseded", actor=actor, note=reason)
|
||||||
|
|
||||||
|
|
||||||
def promote_memory(memory_id: str, actor: str = "api", note: str = "") -> bool:
|
def promote_memory(memory_id: str, actor: str = "api", note: str = "") -> bool:
|
||||||
@@ -604,6 +616,204 @@ def auto_promote_reinforced(
|
|||||||
return promoted
|
return promoted
|
||||||
|
|
||||||
|
|
||||||
|
def extend_reinforced_valid_until(
|
||||||
|
min_reference_count: int = 5,
|
||||||
|
permanent_reference_count: int = 10,
|
||||||
|
extension_days: int = 90,
|
||||||
|
imminent_expiry_days: int = 30,
|
||||||
|
) -> list[dict]:
|
||||||
|
"""Phase 6 C.3 — transient-to-durable auto-extension.
|
||||||
|
|
||||||
|
For active memories with valid_until within the next N days AND
|
||||||
|
reference_count >= min_reference_count: extend valid_until by
|
||||||
|
extension_days. If reference_count >= permanent_reference_count,
|
||||||
|
clear valid_until entirely (becomes permanent).
|
||||||
|
|
||||||
|
Matches the user's intuition: "something transient becomes important
|
||||||
|
if you keep coming back to it". The system watches reinforcement
|
||||||
|
signals and extends expiry so context packs keep seeing durable
|
||||||
|
facts instead of letting them decay out.
|
||||||
|
|
||||||
|
Returns a list of {memory_id, action, old, new} dicts for each
|
||||||
|
memory touched.
|
||||||
|
"""
|
||||||
|
from datetime import timedelta
|
||||||
|
|
||||||
|
now = datetime.now(timezone.utc)
|
||||||
|
horizon = (now + timedelta(days=imminent_expiry_days)).strftime("%Y-%m-%d")
|
||||||
|
new_expiry = (now + timedelta(days=extension_days)).strftime("%Y-%m-%d")
|
||||||
|
now_str = now.strftime("%Y-%m-%d %H:%M:%S")
|
||||||
|
|
||||||
|
extended: list[dict] = []
|
||||||
|
|
||||||
|
with get_connection() as conn:
|
||||||
|
rows = conn.execute(
|
||||||
|
"SELECT id, valid_until, reference_count FROM memories "
|
||||||
|
"WHERE status = 'active' "
|
||||||
|
"AND valid_until IS NOT NULL AND valid_until != '' "
|
||||||
|
"AND substr(valid_until, 1, 10) <= ? "
|
||||||
|
"AND COALESCE(reference_count, 0) >= ?",
|
||||||
|
(horizon, min_reference_count),
|
||||||
|
).fetchall()
|
||||||
|
|
||||||
|
for r in rows:
|
||||||
|
mid = r["id"]
|
||||||
|
old_vu = r["valid_until"]
|
||||||
|
ref_count = int(r["reference_count"] or 0)
|
||||||
|
|
||||||
|
if ref_count >= permanent_reference_count:
|
||||||
|
# Permanent promotion
|
||||||
|
conn.execute(
|
||||||
|
"UPDATE memories SET valid_until = NULL, updated_at = ? WHERE id = ?",
|
||||||
|
(now_str, mid),
|
||||||
|
)
|
||||||
|
extended.append({
|
||||||
|
"memory_id": mid, "action": "made_permanent",
|
||||||
|
"old_valid_until": old_vu, "new_valid_until": None,
|
||||||
|
"reference_count": ref_count,
|
||||||
|
})
|
||||||
|
else:
|
||||||
|
# 90-day extension
|
||||||
|
conn.execute(
|
||||||
|
"UPDATE memories SET valid_until = ?, updated_at = ? WHERE id = ?",
|
||||||
|
(new_expiry, now_str, mid),
|
||||||
|
)
|
||||||
|
extended.append({
|
||||||
|
"memory_id": mid, "action": "extended",
|
||||||
|
"old_valid_until": old_vu, "new_valid_until": new_expiry,
|
||||||
|
"reference_count": ref_count,
|
||||||
|
})
|
||||||
|
|
||||||
|
# Audit rows via the shared framework (fail-open)
|
||||||
|
for ex in extended:
|
||||||
|
try:
|
||||||
|
_audit_memory(
|
||||||
|
memory_id=ex["memory_id"],
|
||||||
|
action="valid_until_extended",
|
||||||
|
actor="transient-to-durable",
|
||||||
|
before={"valid_until": ex["old_valid_until"]},
|
||||||
|
after={"valid_until": ex["new_valid_until"]},
|
||||||
|
note=f"reinforced {ex['reference_count']}x; {ex['action']}",
|
||||||
|
)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
if extended:
|
||||||
|
log.info("reinforced_valid_until_extended", count=len(extended))
|
||||||
|
return extended
|
||||||
|
|
||||||
|
|
||||||
|
def decay_unreferenced_memories(
|
||||||
|
idle_days_threshold: int = 30,
|
||||||
|
daily_decay_factor: float = 0.97,
|
||||||
|
supersede_confidence_floor: float = 0.30,
|
||||||
|
actor: str = "confidence-decay",
|
||||||
|
) -> dict[str, list]:
|
||||||
|
"""Phase 7D — daily confidence decay on cold memories.
|
||||||
|
|
||||||
|
For every active, non-graduated memory with ``reference_count == 0``
|
||||||
|
AND whose last activity (``last_referenced_at`` if set, else
|
||||||
|
``created_at``) is older than ``idle_days_threshold``: multiply
|
||||||
|
confidence by ``daily_decay_factor`` (0.97/day ≈ 2-month half-life).
|
||||||
|
|
||||||
|
If the decayed confidence falls below ``supersede_confidence_floor``,
|
||||||
|
auto-supersede the memory with note "decayed, no references".
|
||||||
|
Supersession is non-destructive — the row stays queryable via
|
||||||
|
``status='superseded'`` for audit.
|
||||||
|
|
||||||
|
Reinforcement already bumps confidence back up, so a decayed memory
|
||||||
|
that later gets referenced reverses its trajectory naturally.
|
||||||
|
|
||||||
|
The job is idempotent-per-day: running it multiple times in one day
|
||||||
|
decays extra, but the cron runs once/day so this stays on-policy.
|
||||||
|
If a day's cron gets skipped, we under-decay (safe direction —
|
||||||
|
memories age slower, not faster, than the policy).
|
||||||
|
|
||||||
|
Returns {"decayed": [...], "superseded": [...]} with per-memory
|
||||||
|
before/after snapshots for audit/observability.
|
||||||
|
"""
|
||||||
|
from datetime import timedelta
|
||||||
|
|
||||||
|
if not (0.0 < daily_decay_factor < 1.0):
|
||||||
|
raise ValueError("daily_decay_factor must be between 0 and 1 (exclusive)")
|
||||||
|
if not (0.0 <= supersede_confidence_floor <= 1.0):
|
||||||
|
raise ValueError("supersede_confidence_floor must be in [0,1]")
|
||||||
|
|
||||||
|
cutoff_dt = datetime.now(timezone.utc) - timedelta(days=idle_days_threshold)
|
||||||
|
cutoff_str = cutoff_dt.strftime("%Y-%m-%d %H:%M:%S")
|
||||||
|
now_str = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S")
|
||||||
|
|
||||||
|
decayed: list[dict] = []
|
||||||
|
superseded: list[dict] = []
|
||||||
|
|
||||||
|
with get_connection() as conn:
|
||||||
|
# COALESCE(last_referenced_at, created_at) is the effective "last
|
||||||
|
# activity" — if a memory was never reinforced, we measure age
|
||||||
|
# from creation. "IS NOT status graduated" is enforced to keep
|
||||||
|
# graduated memories (which are frozen pointers to entities)
|
||||||
|
# out of the decay pool.
|
||||||
|
rows = conn.execute(
|
||||||
|
"SELECT id, confidence, last_referenced_at, created_at "
|
||||||
|
"FROM memories "
|
||||||
|
"WHERE status = 'active' "
|
||||||
|
"AND COALESCE(reference_count, 0) = 0 "
|
||||||
|
"AND COALESCE(last_referenced_at, created_at) < ?",
|
||||||
|
(cutoff_str,),
|
||||||
|
).fetchall()
|
||||||
|
|
||||||
|
for r in rows:
|
||||||
|
mid = r["id"]
|
||||||
|
old_conf = float(r["confidence"])
|
||||||
|
new_conf = max(0.0, old_conf * daily_decay_factor)
|
||||||
|
|
||||||
|
if new_conf < supersede_confidence_floor:
|
||||||
|
# Auto-supersede
|
||||||
|
conn.execute(
|
||||||
|
"UPDATE memories SET status = 'superseded', "
|
||||||
|
"confidence = ?, updated_at = ? WHERE id = ?",
|
||||||
|
(new_conf, now_str, mid),
|
||||||
|
)
|
||||||
|
superseded.append({
|
||||||
|
"memory_id": mid,
|
||||||
|
"old_confidence": old_conf,
|
||||||
|
"new_confidence": new_conf,
|
||||||
|
})
|
||||||
|
else:
|
||||||
|
conn.execute(
|
||||||
|
"UPDATE memories SET confidence = ?, updated_at = ? WHERE id = ?",
|
||||||
|
(new_conf, now_str, mid),
|
||||||
|
)
|
||||||
|
decayed.append({
|
||||||
|
"memory_id": mid,
|
||||||
|
"old_confidence": old_conf,
|
||||||
|
"new_confidence": new_conf,
|
||||||
|
})
|
||||||
|
|
||||||
|
# Audit rows outside the transaction. We skip per-decay audit because
|
||||||
|
# it would be too chatty (potentially hundreds of rows/day for no
|
||||||
|
# human value); supersessions ARE audited because those are
|
||||||
|
# status-changing events humans may want to review.
|
||||||
|
for entry in superseded:
|
||||||
|
_audit_memory(
|
||||||
|
memory_id=entry["memory_id"],
|
||||||
|
action="superseded",
|
||||||
|
actor=actor,
|
||||||
|
before={"status": "active", "confidence": entry["old_confidence"]},
|
||||||
|
after={"status": "superseded", "confidence": entry["new_confidence"]},
|
||||||
|
note=f"decayed below floor {supersede_confidence_floor}, no references",
|
||||||
|
)
|
||||||
|
|
||||||
|
if decayed or superseded:
|
||||||
|
log.info(
|
||||||
|
"confidence_decay_run",
|
||||||
|
decayed=len(decayed),
|
||||||
|
superseded=len(superseded),
|
||||||
|
idle_days_threshold=idle_days_threshold,
|
||||||
|
daily_decay_factor=daily_decay_factor,
|
||||||
|
)
|
||||||
|
return {"decayed": decayed, "superseded": superseded}
|
||||||
|
|
||||||
|
|
||||||
def expire_stale_candidates(
|
def expire_stale_candidates(
|
||||||
max_age_days: int = 14,
|
max_age_days: int = 14,
|
||||||
) -> list[str]:
|
) -> list[str]:
|
||||||
@@ -838,3 +1048,545 @@ def _row_to_memory(row) -> Memory:
|
|||||||
def _validate_confidence(confidence: float) -> None:
|
def _validate_confidence(confidence: float) -> None:
|
||||||
if not 0.0 <= confidence <= 1.0:
|
if not 0.0 <= confidence <= 1.0:
|
||||||
raise ValueError("Confidence must be between 0.0 and 1.0")
|
raise ValueError("Confidence must be between 0.0 and 1.0")
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------
|
||||||
|
# Phase 7C — Tag canonicalization
|
||||||
|
# ---------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def get_tag_distribution(
|
||||||
|
active_only: bool = True,
|
||||||
|
min_count: int = 1,
|
||||||
|
) -> dict[str, int]:
|
||||||
|
"""Return {tag: occurrence_count} across memories for LLM input.
|
||||||
|
|
||||||
|
Used by the canonicalization detector to spot alias clusters like
|
||||||
|
{firmware: 23, fw: 5, firmware-control: 3}. Only counts memories
|
||||||
|
in the requested status (active by default) so superseded/invalid
|
||||||
|
rows don't bias the distribution.
|
||||||
|
"""
|
||||||
|
import json as _json
|
||||||
|
counts: dict[str, int] = {}
|
||||||
|
query = "SELECT domain_tags FROM memories"
|
||||||
|
if active_only:
|
||||||
|
query += " WHERE status = 'active'"
|
||||||
|
with get_connection() as conn:
|
||||||
|
rows = conn.execute(query).fetchall()
|
||||||
|
for r in rows:
|
||||||
|
tags_raw = r["domain_tags"]
|
||||||
|
try:
|
||||||
|
tags = _json.loads(tags_raw) if tags_raw else []
|
||||||
|
except Exception:
|
||||||
|
tags = []
|
||||||
|
if not isinstance(tags, list):
|
||||||
|
continue
|
||||||
|
for t in tags:
|
||||||
|
if not isinstance(t, str):
|
||||||
|
continue
|
||||||
|
key = t.strip().lower()
|
||||||
|
if key:
|
||||||
|
counts[key] = counts.get(key, 0) + 1
|
||||||
|
if min_count > 1:
|
||||||
|
counts = {k: v for k, v in counts.items() if v >= min_count}
|
||||||
|
return counts
|
||||||
|
|
||||||
|
|
||||||
|
def apply_tag_alias(
|
||||||
|
alias: str,
|
||||||
|
canonical: str,
|
||||||
|
actor: str = "tag-canon",
|
||||||
|
) -> dict:
|
||||||
|
"""Rewrite every active memory's domain_tags: alias → canonical.
|
||||||
|
|
||||||
|
Atomic per-memory. Dedupes within each memory's tag list (so if a
|
||||||
|
memory already has both alias AND canonical, we drop the alias and
|
||||||
|
keep canonical without duplicating). Writes one audit row per
|
||||||
|
touched memory with action="tag_canonicalized" so the full trail
|
||||||
|
is recoverable.
|
||||||
|
|
||||||
|
Returns {"memories_touched": int, "alias": ..., "canonical": ...}.
|
||||||
|
"""
|
||||||
|
import json as _json
|
||||||
|
alias = (alias or "").strip().lower()
|
||||||
|
canonical = (canonical or "").strip().lower()
|
||||||
|
if not alias or not canonical:
|
||||||
|
raise ValueError("alias and canonical must be non-empty")
|
||||||
|
if alias == canonical:
|
||||||
|
raise ValueError("alias cannot equal canonical")
|
||||||
|
|
||||||
|
touched: list[tuple[str, list[str], list[str]]] = []
|
||||||
|
|
||||||
|
with get_connection() as conn:
|
||||||
|
rows = conn.execute(
|
||||||
|
"SELECT id, domain_tags FROM memories WHERE status = 'active'"
|
||||||
|
).fetchall()
|
||||||
|
|
||||||
|
for r in rows:
|
||||||
|
raw = r["domain_tags"]
|
||||||
|
try:
|
||||||
|
tags = _json.loads(raw) if raw else []
|
||||||
|
except Exception:
|
||||||
|
tags = []
|
||||||
|
if not isinstance(tags, list):
|
||||||
|
continue
|
||||||
|
if alias not in tags:
|
||||||
|
continue
|
||||||
|
|
||||||
|
old_tags = [t for t in tags if isinstance(t, str)]
|
||||||
|
new_tags: list[str] = []
|
||||||
|
for t in old_tags:
|
||||||
|
rewritten = canonical if t == alias else t
|
||||||
|
if rewritten not in new_tags:
|
||||||
|
new_tags.append(rewritten)
|
||||||
|
if new_tags == old_tags:
|
||||||
|
continue
|
||||||
|
conn.execute(
|
||||||
|
"UPDATE memories SET domain_tags = ?, updated_at = CURRENT_TIMESTAMP "
|
||||||
|
"WHERE id = ?",
|
||||||
|
(_json.dumps(new_tags), r["id"]),
|
||||||
|
)
|
||||||
|
touched.append((r["id"], old_tags, new_tags))
|
||||||
|
|
||||||
|
# Audit rows outside the transaction
|
||||||
|
for mem_id, old_tags, new_tags in touched:
|
||||||
|
_audit_memory(
|
||||||
|
memory_id=mem_id,
|
||||||
|
action="tag_canonicalized",
|
||||||
|
actor=actor,
|
||||||
|
before={"domain_tags": old_tags},
|
||||||
|
after={"domain_tags": new_tags},
|
||||||
|
note=f"{alias} → {canonical}",
|
||||||
|
)
|
||||||
|
|
||||||
|
if touched:
|
||||||
|
log.info("tag_alias_applied", alias=alias, canonical=canonical, memories_touched=len(touched))
|
||||||
|
return {
|
||||||
|
"memories_touched": len(touched),
|
||||||
|
"alias": alias,
|
||||||
|
"canonical": canonical,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def create_tag_alias_proposal(
|
||||||
|
alias: str,
|
||||||
|
canonical: str,
|
||||||
|
confidence: float,
|
||||||
|
alias_count: int = 0,
|
||||||
|
canonical_count: int = 0,
|
||||||
|
reason: str = "",
|
||||||
|
) -> str | None:
|
||||||
|
"""Insert a tag_aliases row in status=pending.
|
||||||
|
|
||||||
|
Idempotent: if a pending proposal for (alias, canonical) already
|
||||||
|
exists, returns None.
|
||||||
|
"""
|
||||||
|
import json as _json # noqa: F401 — kept for parity with other helpers
|
||||||
|
alias = (alias or "").strip().lower()
|
||||||
|
canonical = (canonical or "").strip().lower()
|
||||||
|
if not alias or not canonical or alias == canonical:
|
||||||
|
return None
|
||||||
|
confidence = max(0.0, min(1.0, float(confidence)))
|
||||||
|
|
||||||
|
proposal_id = str(uuid.uuid4())
|
||||||
|
with get_connection() as conn:
|
||||||
|
existing = conn.execute(
|
||||||
|
"SELECT id FROM tag_aliases WHERE alias = ? AND canonical = ? "
|
||||||
|
"AND status = 'pending'",
|
||||||
|
(alias, canonical),
|
||||||
|
).fetchone()
|
||||||
|
if existing:
|
||||||
|
return None
|
||||||
|
|
||||||
|
conn.execute(
|
||||||
|
"INSERT INTO tag_aliases (id, alias, canonical, status, confidence, "
|
||||||
|
"alias_count, canonical_count, reason) "
|
||||||
|
"VALUES (?, ?, ?, 'pending', ?, ?, ?, ?)",
|
||||||
|
(proposal_id, alias, canonical, confidence,
|
||||||
|
int(alias_count), int(canonical_count), reason[:500]),
|
||||||
|
)
|
||||||
|
log.info(
|
||||||
|
"tag_alias_proposed",
|
||||||
|
proposal_id=proposal_id,
|
||||||
|
alias=alias,
|
||||||
|
canonical=canonical,
|
||||||
|
confidence=round(confidence, 3),
|
||||||
|
)
|
||||||
|
return proposal_id
|
||||||
|
|
||||||
|
|
||||||
|
def get_tag_alias_proposals(status: str = "pending", limit: int = 100) -> list[dict]:
|
||||||
|
"""List tag alias proposals."""
|
||||||
|
with get_connection() as conn:
|
||||||
|
rows = conn.execute(
|
||||||
|
"SELECT * FROM tag_aliases WHERE status = ? "
|
||||||
|
"ORDER BY confidence DESC, created_at DESC LIMIT ?",
|
||||||
|
(status, limit),
|
||||||
|
).fetchall()
|
||||||
|
return [dict(r) for r in rows]
|
||||||
|
|
||||||
|
|
||||||
|
def approve_tag_alias(
|
||||||
|
proposal_id: str,
|
||||||
|
actor: str = "human-triage",
|
||||||
|
) -> dict | None:
|
||||||
|
"""Apply the alias rewrite + mark the proposal approved.
|
||||||
|
|
||||||
|
Returns the apply_tag_alias result dict, or None if the proposal
|
||||||
|
is not found or already resolved.
|
||||||
|
"""
|
||||||
|
now_str = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S")
|
||||||
|
with get_connection() as conn:
|
||||||
|
row = conn.execute(
|
||||||
|
"SELECT alias, canonical, status FROM tag_aliases WHERE id = ?",
|
||||||
|
(proposal_id,),
|
||||||
|
).fetchone()
|
||||||
|
if row is None or row["status"] != "pending":
|
||||||
|
return None
|
||||||
|
alias, canonical = row["alias"], row["canonical"]
|
||||||
|
|
||||||
|
result = apply_tag_alias(alias, canonical, actor=actor)
|
||||||
|
|
||||||
|
with get_connection() as conn:
|
||||||
|
conn.execute(
|
||||||
|
"UPDATE tag_aliases SET status = 'approved', resolved_at = ?, "
|
||||||
|
"resolved_by = ?, applied_to_memories = ? WHERE id = ?",
|
||||||
|
(now_str, actor, result["memories_touched"], proposal_id),
|
||||||
|
)
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def reject_tag_alias(proposal_id: str, actor: str = "human-triage") -> bool:
|
||||||
|
"""Mark a tag alias proposal as rejected without applying it."""
|
||||||
|
now_str = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S")
|
||||||
|
with get_connection() as conn:
|
||||||
|
result = conn.execute(
|
||||||
|
"UPDATE tag_aliases SET status = 'rejected', resolved_at = ?, "
|
||||||
|
"resolved_by = ? WHERE id = ? AND status = 'pending'",
|
||||||
|
(now_str, actor, proposal_id),
|
||||||
|
)
|
||||||
|
return result.rowcount > 0
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------
|
||||||
|
# Phase 7A — Memory Consolidation: merge-candidate lifecycle
|
||||||
|
# ---------------------------------------------------------------------
|
||||||
|
#
|
||||||
|
# The detector (scripts/memory_dedup.py) writes proposals into
|
||||||
|
# memory_merge_candidates. The triage UI lists pending rows, a human
|
||||||
|
# reviews, and on approve we execute the merge here — never at detect
|
||||||
|
# time. This keeps the audit trail clean: every mutation is a human
|
||||||
|
# decision.
|
||||||
|
|
||||||
|
|
||||||
|
def create_merge_candidate(
|
||||||
|
memory_ids: list[str],
|
||||||
|
similarity: float,
|
||||||
|
proposed_content: str,
|
||||||
|
proposed_memory_type: str,
|
||||||
|
proposed_project: str,
|
||||||
|
proposed_tags: list[str] | None = None,
|
||||||
|
proposed_confidence: float = 0.6,
|
||||||
|
reason: str = "",
|
||||||
|
) -> str | None:
|
||||||
|
"""Insert a merge-candidate row. Returns the new row id, or None if
|
||||||
|
a pending candidate already covers this exact set of memory ids
|
||||||
|
(idempotent scan — re-running the detector doesn't double-create)."""
|
||||||
|
import json as _json
|
||||||
|
|
||||||
|
if not memory_ids or len(memory_ids) < 2:
|
||||||
|
raise ValueError("merge candidate requires at least 2 memory_ids")
|
||||||
|
|
||||||
|
memory_ids_sorted = sorted(set(memory_ids))
|
||||||
|
memory_ids_json = _json.dumps(memory_ids_sorted)
|
||||||
|
tags_json = _json.dumps(_normalize_tags(proposed_tags))
|
||||||
|
candidate_id = str(uuid.uuid4())
|
||||||
|
|
||||||
|
with get_connection() as conn:
|
||||||
|
# Idempotency: same sorted-id set already pending? skip.
|
||||||
|
existing = conn.execute(
|
||||||
|
"SELECT id FROM memory_merge_candidates "
|
||||||
|
"WHERE status = 'pending' AND memory_ids = ?",
|
||||||
|
(memory_ids_json,),
|
||||||
|
).fetchone()
|
||||||
|
if existing:
|
||||||
|
return None
|
||||||
|
|
||||||
|
conn.execute(
|
||||||
|
"INSERT INTO memory_merge_candidates "
|
||||||
|
"(id, status, memory_ids, similarity, proposed_content, "
|
||||||
|
"proposed_memory_type, proposed_project, proposed_tags, "
|
||||||
|
"proposed_confidence, reason) "
|
||||||
|
"VALUES (?, 'pending', ?, ?, ?, ?, ?, ?, ?, ?)",
|
||||||
|
(
|
||||||
|
candidate_id, memory_ids_json, float(similarity or 0.0),
|
||||||
|
(proposed_content or "")[:2000],
|
||||||
|
(proposed_memory_type or "knowledge")[:50],
|
||||||
|
(proposed_project or "")[:100],
|
||||||
|
tags_json,
|
||||||
|
max(0.0, min(1.0, float(proposed_confidence))),
|
||||||
|
(reason or "")[:500],
|
||||||
|
),
|
||||||
|
)
|
||||||
|
log.info(
|
||||||
|
"merge_candidate_created",
|
||||||
|
candidate_id=candidate_id,
|
||||||
|
memory_count=len(memory_ids_sorted),
|
||||||
|
similarity=round(similarity, 4),
|
||||||
|
)
|
||||||
|
return candidate_id
|
||||||
|
|
||||||
|
|
||||||
|
def get_merge_candidates(status: str = "pending", limit: int = 100) -> list[dict]:
|
||||||
|
"""List merge candidates with their source memories inlined."""
|
||||||
|
import json as _json
|
||||||
|
|
||||||
|
with get_connection() as conn:
|
||||||
|
rows = conn.execute(
|
||||||
|
"SELECT * FROM memory_merge_candidates "
|
||||||
|
"WHERE status = ? ORDER BY created_at DESC LIMIT ?",
|
||||||
|
(status, limit),
|
||||||
|
).fetchall()
|
||||||
|
|
||||||
|
out = []
|
||||||
|
for r in rows:
|
||||||
|
try:
|
||||||
|
mem_ids = _json.loads(r["memory_ids"] or "[]")
|
||||||
|
except Exception:
|
||||||
|
mem_ids = []
|
||||||
|
try:
|
||||||
|
tags = _json.loads(r["proposed_tags"] or "[]")
|
||||||
|
except Exception:
|
||||||
|
tags = []
|
||||||
|
|
||||||
|
sources = []
|
||||||
|
for mid in mem_ids:
|
||||||
|
srow = conn.execute(
|
||||||
|
"SELECT id, memory_type, content, project, confidence, "
|
||||||
|
"status, reference_count, domain_tags, valid_until "
|
||||||
|
"FROM memories WHERE id = ?",
|
||||||
|
(mid,),
|
||||||
|
).fetchone()
|
||||||
|
if srow:
|
||||||
|
try:
|
||||||
|
stags = _json.loads(srow["domain_tags"] or "[]")
|
||||||
|
except Exception:
|
||||||
|
stags = []
|
||||||
|
sources.append({
|
||||||
|
"id": srow["id"],
|
||||||
|
"memory_type": srow["memory_type"],
|
||||||
|
"content": srow["content"],
|
||||||
|
"project": srow["project"] or "",
|
||||||
|
"confidence": srow["confidence"],
|
||||||
|
"status": srow["status"],
|
||||||
|
"reference_count": int(srow["reference_count"] or 0),
|
||||||
|
"domain_tags": stags,
|
||||||
|
"valid_until": srow["valid_until"] or "",
|
||||||
|
})
|
||||||
|
|
||||||
|
out.append({
|
||||||
|
"id": r["id"],
|
||||||
|
"status": r["status"],
|
||||||
|
"memory_ids": mem_ids,
|
||||||
|
"similarity": r["similarity"],
|
||||||
|
"proposed_content": r["proposed_content"] or "",
|
||||||
|
"proposed_memory_type": r["proposed_memory_type"] or "knowledge",
|
||||||
|
"proposed_project": r["proposed_project"] or "",
|
||||||
|
"proposed_tags": tags,
|
||||||
|
"proposed_confidence": r["proposed_confidence"],
|
||||||
|
"reason": r["reason"] or "",
|
||||||
|
"created_at": r["created_at"],
|
||||||
|
"resolved_at": r["resolved_at"],
|
||||||
|
"resolved_by": r["resolved_by"],
|
||||||
|
"result_memory_id": r["result_memory_id"],
|
||||||
|
"sources": sources,
|
||||||
|
})
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def reject_merge_candidate(candidate_id: str, actor: str = "human-triage", note: str = "") -> bool:
|
||||||
|
"""Mark a merge candidate as rejected. Source memories stay untouched."""
|
||||||
|
now_str = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S")
|
||||||
|
with get_connection() as conn:
|
||||||
|
result = conn.execute(
|
||||||
|
"UPDATE memory_merge_candidates "
|
||||||
|
"SET status = 'rejected', resolved_at = ?, resolved_by = ? "
|
||||||
|
"WHERE id = ? AND status = 'pending'",
|
||||||
|
(now_str, actor, candidate_id),
|
||||||
|
)
|
||||||
|
if result.rowcount == 0:
|
||||||
|
return False
|
||||||
|
log.info("merge_candidate_rejected", candidate_id=candidate_id, actor=actor, note=note[:100])
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
def merge_memories(
|
||||||
|
candidate_id: str,
|
||||||
|
actor: str = "human-triage",
|
||||||
|
override_content: str | None = None,
|
||||||
|
override_tags: list[str] | None = None,
|
||||||
|
) -> str | None:
|
||||||
|
"""Execute an approved merge candidate.
|
||||||
|
|
||||||
|
1. Validate all source memories still status=active
|
||||||
|
2. Create the new merged memory (status=active)
|
||||||
|
3. Mark each source status=superseded with an audit row pointing at
|
||||||
|
the new merged id
|
||||||
|
4. Mark the candidate status=approved, record result_memory_id
|
||||||
|
5. Write a consolidated audit row on the new memory
|
||||||
|
|
||||||
|
Returns the new merged memory's id, or None if the candidate cannot
|
||||||
|
be executed (already resolved, source tampered, etc.).
|
||||||
|
|
||||||
|
``override_content`` and ``override_tags`` let the UI pass the human's
|
||||||
|
edits before clicking approve.
|
||||||
|
"""
|
||||||
|
import json as _json
|
||||||
|
|
||||||
|
now_str = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S")
|
||||||
|
|
||||||
|
with get_connection() as conn:
|
||||||
|
row = conn.execute(
|
||||||
|
"SELECT * FROM memory_merge_candidates WHERE id = ?",
|
||||||
|
(candidate_id,),
|
||||||
|
).fetchone()
|
||||||
|
if row is None or row["status"] != "pending":
|
||||||
|
log.warning("merge_candidate_not_pending", candidate_id=candidate_id)
|
||||||
|
return None
|
||||||
|
|
||||||
|
try:
|
||||||
|
mem_ids = _json.loads(row["memory_ids"] or "[]")
|
||||||
|
except Exception:
|
||||||
|
mem_ids = []
|
||||||
|
if not mem_ids or len(mem_ids) < 2:
|
||||||
|
log.warning("merge_candidate_invalid_memory_ids", candidate_id=candidate_id)
|
||||||
|
return None
|
||||||
|
|
||||||
|
# Snapshot sources + validate all active
|
||||||
|
source_rows = []
|
||||||
|
for mid in mem_ids:
|
||||||
|
srow = conn.execute(
|
||||||
|
"SELECT * FROM memories WHERE id = ?", (mid,)
|
||||||
|
).fetchone()
|
||||||
|
if srow is None or srow["status"] != "active":
|
||||||
|
log.warning(
|
||||||
|
"merge_source_not_active",
|
||||||
|
candidate_id=candidate_id,
|
||||||
|
memory_id=mid,
|
||||||
|
actual_status=(srow["status"] if srow else "missing"),
|
||||||
|
)
|
||||||
|
return None
|
||||||
|
source_rows.append(srow)
|
||||||
|
|
||||||
|
# Build merged memory fields — prefer human overrides, then proposed
|
||||||
|
content = (override_content or row["proposed_content"] or "").strip()
|
||||||
|
if not content:
|
||||||
|
log.warning("merge_candidate_empty_content", candidate_id=candidate_id)
|
||||||
|
return None
|
||||||
|
|
||||||
|
merged_type = (row["proposed_memory_type"] or source_rows[0]["memory_type"]).lower()
|
||||||
|
if merged_type not in MEMORY_TYPES:
|
||||||
|
merged_type = source_rows[0]["memory_type"]
|
||||||
|
|
||||||
|
merged_project = row["proposed_project"] or source_rows[0]["project"] or ""
|
||||||
|
merged_project = resolve_project_name(merged_project)
|
||||||
|
|
||||||
|
# Tags: override wins, else proposed, else union of sources
|
||||||
|
if override_tags is not None:
|
||||||
|
merged_tags = _normalize_tags(override_tags)
|
||||||
|
else:
|
||||||
|
try:
|
||||||
|
proposed_tags = _json.loads(row["proposed_tags"] or "[]")
|
||||||
|
except Exception:
|
||||||
|
proposed_tags = []
|
||||||
|
if proposed_tags:
|
||||||
|
merged_tags = _normalize_tags(proposed_tags)
|
||||||
|
else:
|
||||||
|
union: list[str] = []
|
||||||
|
for srow in source_rows:
|
||||||
|
try:
|
||||||
|
stags = _json.loads(srow["domain_tags"] or "[]")
|
||||||
|
except Exception:
|
||||||
|
stags = []
|
||||||
|
for t in stags:
|
||||||
|
if isinstance(t, str) and t and t not in union:
|
||||||
|
union.append(t)
|
||||||
|
merged_tags = union
|
||||||
|
|
||||||
|
# confidence = max; reference_count = sum
|
||||||
|
merged_confidence = max(float(s["confidence"]) for s in source_rows)
|
||||||
|
total_refs = sum(int(s["reference_count"] or 0) for s in source_rows)
|
||||||
|
|
||||||
|
# valid_until: if any source is permanent (None/empty), merged is permanent.
|
||||||
|
# Otherwise take the latest (lexical compare on ISO dates works).
|
||||||
|
merged_vu: str | None = "" # placeholder
|
||||||
|
has_permanent = any(not (s["valid_until"] or "").strip() for s in source_rows)
|
||||||
|
if has_permanent:
|
||||||
|
merged_vu = None
|
||||||
|
else:
|
||||||
|
merged_vu = max((s["valid_until"] or "").strip() for s in source_rows) or None
|
||||||
|
|
||||||
|
new_id = str(uuid.uuid4())
|
||||||
|
tags_json = _json.dumps(merged_tags)
|
||||||
|
|
||||||
|
conn.execute(
|
||||||
|
"INSERT INTO memories (id, memory_type, content, project, "
|
||||||
|
"source_chunk_id, confidence, status, domain_tags, valid_until, "
|
||||||
|
"reference_count, last_referenced_at) "
|
||||||
|
"VALUES (?, ?, ?, ?, NULL, ?, 'active', ?, ?, ?, ?)",
|
||||||
|
(
|
||||||
|
new_id, merged_type, content[:2000], merged_project,
|
||||||
|
merged_confidence, tags_json, merged_vu, total_refs, now_str,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
# Mark sources superseded
|
||||||
|
for srow in source_rows:
|
||||||
|
conn.execute(
|
||||||
|
"UPDATE memories SET status = 'superseded', updated_at = ? "
|
||||||
|
"WHERE id = ?",
|
||||||
|
(now_str, srow["id"]),
|
||||||
|
)
|
||||||
|
|
||||||
|
# Mark candidate approved
|
||||||
|
conn.execute(
|
||||||
|
"UPDATE memory_merge_candidates SET status = 'approved', "
|
||||||
|
"resolved_at = ?, resolved_by = ?, result_memory_id = ? WHERE id = ?",
|
||||||
|
(now_str, actor, new_id, candidate_id),
|
||||||
|
)
|
||||||
|
|
||||||
|
# Audit rows (out of the transaction; fail-open via _audit_memory)
|
||||||
|
_audit_memory(
|
||||||
|
memory_id=new_id,
|
||||||
|
action="created_via_merge",
|
||||||
|
actor=actor,
|
||||||
|
after={
|
||||||
|
"memory_type": merged_type,
|
||||||
|
"content": content,
|
||||||
|
"project": merged_project,
|
||||||
|
"confidence": merged_confidence,
|
||||||
|
"domain_tags": merged_tags,
|
||||||
|
"reference_count": total_refs,
|
||||||
|
"merged_from": list(mem_ids),
|
||||||
|
"merge_candidate_id": candidate_id,
|
||||||
|
},
|
||||||
|
note=f"merged {len(mem_ids)} sources via candidate {candidate_id[:8]}",
|
||||||
|
)
|
||||||
|
for srow in source_rows:
|
||||||
|
_audit_memory(
|
||||||
|
memory_id=srow["id"],
|
||||||
|
action="superseded",
|
||||||
|
actor=actor,
|
||||||
|
before={"status": "active", "content": srow["content"]},
|
||||||
|
after={"status": "superseded", "superseded_by": new_id},
|
||||||
|
note=f"merged into {new_id}",
|
||||||
|
)
|
||||||
|
|
||||||
|
log.info(
|
||||||
|
"merge_executed",
|
||||||
|
candidate_id=candidate_id,
|
||||||
|
result_memory_id=new_id,
|
||||||
|
source_count=len(source_rows),
|
||||||
|
actor=actor,
|
||||||
|
)
|
||||||
|
return new_id
|
||||||
|
|||||||
88
src/atocore/memory/similarity.py
Normal file
88
src/atocore/memory/similarity.py
Normal file
@@ -0,0 +1,88 @@
|
|||||||
|
"""Phase 7A (Memory Consolidation): semantic similarity helpers.
|
||||||
|
|
||||||
|
Thin wrapper over ``atocore.retrieval.embeddings`` that exposes
|
||||||
|
pairwise + batch cosine similarity on normalized embeddings. Used by
|
||||||
|
the dedup detector to cluster near-duplicate active memories.
|
||||||
|
|
||||||
|
Embeddings from ``embed_texts()`` are already L2-normalized, so cosine
|
||||||
|
similarity reduces to a dot product — no extra normalization needed.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from atocore.retrieval.embeddings import embed_texts
|
||||||
|
|
||||||
|
|
||||||
|
def _dot(a: list[float], b: list[float]) -> float:
|
||||||
|
return sum(x * y for x, y in zip(a, b))
|
||||||
|
|
||||||
|
|
||||||
|
def cosine(a: list[float], b: list[float]) -> float:
|
||||||
|
"""Cosine similarity on already-normalized vectors. Clamped to [0,1]
|
||||||
|
(embeddings use paraphrase-multilingual-MiniLM which is unit-norm,
|
||||||
|
and we never want negative values leaking into thresholds)."""
|
||||||
|
return max(0.0, min(1.0, _dot(a, b)))
|
||||||
|
|
||||||
|
|
||||||
|
def compute_memory_similarity(text_a: str, text_b: str) -> float:
|
||||||
|
"""Return cosine similarity of two memory contents in [0,1].
|
||||||
|
|
||||||
|
Convenience helper for one-off checks + tests. For batch work (the
|
||||||
|
dedup detector), use ``embed_texts()`` directly and compute the
|
||||||
|
similarity matrix yourself to avoid re-embedding shared texts.
|
||||||
|
"""
|
||||||
|
if not text_a or not text_b:
|
||||||
|
return 0.0
|
||||||
|
vecs = embed_texts([text_a, text_b])
|
||||||
|
return cosine(vecs[0], vecs[1])
|
||||||
|
|
||||||
|
|
||||||
|
def similarity_matrix(texts: list[str]) -> list[list[float]]:
|
||||||
|
"""N×N cosine similarity matrix. Diagonal is 1.0, symmetric."""
|
||||||
|
if not texts:
|
||||||
|
return []
|
||||||
|
vecs = embed_texts(texts)
|
||||||
|
n = len(vecs)
|
||||||
|
matrix = [[0.0] * n for _ in range(n)]
|
||||||
|
for i in range(n):
|
||||||
|
matrix[i][i] = 1.0
|
||||||
|
for j in range(i + 1, n):
|
||||||
|
s = cosine(vecs[i], vecs[j])
|
||||||
|
matrix[i][j] = s
|
||||||
|
matrix[j][i] = s
|
||||||
|
return matrix
|
||||||
|
|
||||||
|
|
||||||
|
def cluster_by_threshold(texts: list[str], threshold: float) -> list[list[int]]:
|
||||||
|
"""Greedy transitive clustering: if sim(i,j) >= threshold, merge.
|
||||||
|
|
||||||
|
Returns a list of clusters, each a list of indices into ``texts``.
|
||||||
|
Singletons are included. Used by the dedup detector to collapse
|
||||||
|
A~B~C into one merge proposal rather than three pair proposals.
|
||||||
|
"""
|
||||||
|
if not texts:
|
||||||
|
return []
|
||||||
|
matrix = similarity_matrix(texts)
|
||||||
|
n = len(texts)
|
||||||
|
parent = list(range(n))
|
||||||
|
|
||||||
|
def find(x: int) -> int:
|
||||||
|
while parent[x] != x:
|
||||||
|
parent[x] = parent[parent[x]]
|
||||||
|
x = parent[x]
|
||||||
|
return x
|
||||||
|
|
||||||
|
def union(x: int, y: int) -> None:
|
||||||
|
rx, ry = find(x), find(y)
|
||||||
|
if rx != ry:
|
||||||
|
parent[rx] = ry
|
||||||
|
|
||||||
|
for i in range(n):
|
||||||
|
for j in range(i + 1, n):
|
||||||
|
if matrix[i][j] >= threshold:
|
||||||
|
union(i, j)
|
||||||
|
|
||||||
|
groups: dict[int, list[int]] = {}
|
||||||
|
for i in range(n):
|
||||||
|
groups.setdefault(find(i), []).append(i)
|
||||||
|
return list(groups.values())
|
||||||
@@ -146,6 +146,28 @@ def _apply_migrations(conn: sqlite3.Connection) -> None:
|
|||||||
"CREATE INDEX IF NOT EXISTS idx_memories_graduated ON memories(graduated_to_entity_id)"
|
"CREATE INDEX IF NOT EXISTS idx_memories_graduated ON memories(graduated_to_entity_id)"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
# V1-0 (Engineering V1 completion): shared header fields per
|
||||||
|
# engineering-v1-acceptance.md:45. Three columns on `entities`:
|
||||||
|
# - extractor_version: which extractor produced this row. Lets old
|
||||||
|
# candidates be re-evaluated with a newer extractor per
|
||||||
|
# promotion-rules.md:268.
|
||||||
|
# - canonical_home: which layer holds the canonical record. Always
|
||||||
|
# "entity" for rows written via create_entity; reserved for future
|
||||||
|
# cross-layer bookkeeping.
|
||||||
|
# - hand_authored: 1 when the row was created directly by a human
|
||||||
|
# without source provenance. Enforced by the write path so every
|
||||||
|
# non-hand-authored row must carry non-empty source_refs (F-8).
|
||||||
|
# The entities table itself is created by init_engineering_schema
|
||||||
|
# (see engineering/service.py); these ALTERs cover existing DBs
|
||||||
|
# where the original CREATE TABLE predates V1-0.
|
||||||
|
if _table_exists(conn, "entities"):
|
||||||
|
if not _column_exists(conn, "entities", "extractor_version"):
|
||||||
|
conn.execute("ALTER TABLE entities ADD COLUMN extractor_version TEXT DEFAULT ''")
|
||||||
|
if not _column_exists(conn, "entities", "canonical_home"):
|
||||||
|
conn.execute("ALTER TABLE entities ADD COLUMN canonical_home TEXT DEFAULT 'entity'")
|
||||||
|
if not _column_exists(conn, "entities", "hand_authored"):
|
||||||
|
conn.execute("ALTER TABLE entities ADD COLUMN hand_authored INTEGER DEFAULT 0")
|
||||||
|
|
||||||
# Phase 4 (Robustness V1): append-only audit log for memory mutations.
|
# Phase 4 (Robustness V1): append-only audit log for memory mutations.
|
||||||
# Every create/update/promote/reject/supersede/invalidate/reinforce/expire/
|
# Every create/update/promote/reject/supersede/invalidate/reinforce/expire/
|
||||||
# auto_promote writes one row here. before/after are JSON snapshots of the
|
# auto_promote writes one row here. before/after are JSON snapshots of the
|
||||||
@@ -251,12 +273,115 @@ def _apply_migrations(conn: sqlite3.Connection) -> None:
|
|||||||
"CREATE INDEX IF NOT EXISTS idx_interactions_created_at ON interactions(created_at)"
|
"CREATE INDEX IF NOT EXISTS idx_interactions_created_at ON interactions(created_at)"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
# Phase 7A (Memory Consolidation — "sleep cycle"): merge candidates.
|
||||||
|
# When the dedup detector finds a cluster of semantically similar active
|
||||||
|
# memories within the same (project, memory_type) bucket, it drafts a
|
||||||
|
# unified content via LLM and writes a proposal here. The triage UI
|
||||||
|
# surfaces these for human approval. On approve, source memories become
|
||||||
|
# status=superseded and a new merged memory is created.
|
||||||
|
# memory_ids is a JSON array (length >= 2) of the source memory ids.
|
||||||
|
# proposed_* hold the LLM's draft; a human can edit before approve.
|
||||||
|
# result_memory_id is filled on approve with the new merged memory's id.
|
||||||
|
conn.execute(
|
||||||
|
"""
|
||||||
|
CREATE TABLE IF NOT EXISTS memory_merge_candidates (
|
||||||
|
id TEXT PRIMARY KEY,
|
||||||
|
status TEXT DEFAULT 'pending',
|
||||||
|
memory_ids TEXT NOT NULL,
|
||||||
|
similarity REAL,
|
||||||
|
proposed_content TEXT,
|
||||||
|
proposed_memory_type TEXT,
|
||||||
|
proposed_project TEXT,
|
||||||
|
proposed_tags TEXT DEFAULT '[]',
|
||||||
|
proposed_confidence REAL,
|
||||||
|
reason TEXT DEFAULT '',
|
||||||
|
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
resolved_at DATETIME,
|
||||||
|
resolved_by TEXT,
|
||||||
|
result_memory_id TEXT
|
||||||
|
)
|
||||||
|
"""
|
||||||
|
)
|
||||||
|
conn.execute(
|
||||||
|
"CREATE INDEX IF NOT EXISTS idx_mmc_status ON memory_merge_candidates(status)"
|
||||||
|
)
|
||||||
|
conn.execute(
|
||||||
|
"CREATE INDEX IF NOT EXISTS idx_mmc_created_at ON memory_merge_candidates(created_at)"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Phase 7C (Memory Consolidation — tag canonicalization): alias → canonical
|
||||||
|
# map for domain_tags. A weekly LLM pass proposes rows here; high-confidence
|
||||||
|
# ones auto-apply (rewrite domain_tags across all memories), low-confidence
|
||||||
|
# ones stay pending for human approval. Immutable history: resolved rows
|
||||||
|
# keep status=approved/rejected; the same alias can re-appear with a new
|
||||||
|
# id if the tag reaches a different canonical later.
|
||||||
|
conn.execute(
|
||||||
|
"""
|
||||||
|
CREATE TABLE IF NOT EXISTS tag_aliases (
|
||||||
|
id TEXT PRIMARY KEY,
|
||||||
|
alias TEXT NOT NULL,
|
||||||
|
canonical TEXT NOT NULL,
|
||||||
|
status TEXT DEFAULT 'pending',
|
||||||
|
confidence REAL DEFAULT 0.0,
|
||||||
|
alias_count INTEGER DEFAULT 0,
|
||||||
|
canonical_count INTEGER DEFAULT 0,
|
||||||
|
reason TEXT DEFAULT '',
|
||||||
|
applied_to_memories INTEGER DEFAULT 0,
|
||||||
|
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
resolved_at DATETIME,
|
||||||
|
resolved_by TEXT
|
||||||
|
)
|
||||||
|
"""
|
||||||
|
)
|
||||||
|
conn.execute("CREATE INDEX IF NOT EXISTS idx_tag_aliases_status ON tag_aliases(status)")
|
||||||
|
conn.execute("CREATE INDEX IF NOT EXISTS idx_tag_aliases_alias ON tag_aliases(alias)")
|
||||||
|
|
||||||
|
# Issue F (visual evidence): binary asset store. One row per unique
|
||||||
|
# content hash — re-uploading the same file is idempotent. The blob
|
||||||
|
# itself lives on disk under stored_path; this table is the catalog.
|
||||||
|
# width/height are populated for image mime types (NULL otherwise).
|
||||||
|
# source_refs is a JSON array of free-form provenance pointers
|
||||||
|
# (e.g. "session:<id>", "interaction:<id>") that survive independent
|
||||||
|
# of the EVIDENCED_BY graph. status=invalid tombstones an asset
|
||||||
|
# without dropping the row so audit trails stay intact.
|
||||||
|
conn.execute(
|
||||||
|
"""
|
||||||
|
CREATE TABLE IF NOT EXISTS assets (
|
||||||
|
id TEXT PRIMARY KEY,
|
||||||
|
hash_sha256 TEXT UNIQUE NOT NULL,
|
||||||
|
mime_type TEXT NOT NULL,
|
||||||
|
size_bytes INTEGER NOT NULL,
|
||||||
|
width INTEGER,
|
||||||
|
height INTEGER,
|
||||||
|
stored_path TEXT NOT NULL,
|
||||||
|
original_filename TEXT DEFAULT '',
|
||||||
|
project TEXT DEFAULT '',
|
||||||
|
caption TEXT DEFAULT '',
|
||||||
|
source_refs TEXT DEFAULT '[]',
|
||||||
|
status TEXT DEFAULT 'active',
|
||||||
|
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
updated_at DATETIME DEFAULT CURRENT_TIMESTAMP
|
||||||
|
)
|
||||||
|
"""
|
||||||
|
)
|
||||||
|
conn.execute("CREATE INDEX IF NOT EXISTS idx_assets_hash ON assets(hash_sha256)")
|
||||||
|
conn.execute("CREATE INDEX IF NOT EXISTS idx_assets_project ON assets(project)")
|
||||||
|
conn.execute("CREATE INDEX IF NOT EXISTS idx_assets_status ON assets(status)")
|
||||||
|
|
||||||
|
|
||||||
def _column_exists(conn: sqlite3.Connection, table: str, column: str) -> bool:
|
def _column_exists(conn: sqlite3.Connection, table: str, column: str) -> bool:
|
||||||
rows = conn.execute(f"PRAGMA table_info({table})").fetchall()
|
rows = conn.execute(f"PRAGMA table_info({table})").fetchall()
|
||||||
return any(row["name"] == column for row in rows)
|
return any(row["name"] == column for row in rows)
|
||||||
|
|
||||||
|
|
||||||
|
def _table_exists(conn: sqlite3.Connection, table: str) -> bool:
|
||||||
|
row = conn.execute(
|
||||||
|
"SELECT name FROM sqlite_master WHERE type='table' AND name=?",
|
||||||
|
(table,),
|
||||||
|
).fetchone()
|
||||||
|
return row is not None
|
||||||
|
|
||||||
|
|
||||||
@contextmanager
|
@contextmanager
|
||||||
def get_connection() -> Generator[sqlite3.Connection, None, None]:
|
def get_connection() -> Generator[sqlite3.Connection, None, None]:
|
||||||
"""Get a database connection with row factory."""
|
"""Get a database connection with row factory."""
|
||||||
|
|||||||
@@ -11,6 +11,20 @@ import atocore.config as _config
|
|||||||
from atocore.ingestion.pipeline import ingest_folder
|
from atocore.ingestion.pipeline import ingest_folder
|
||||||
|
|
||||||
|
|
||||||
|
# Reserved pseudo-projects. `inbox` holds pre-project / lead / quote
|
||||||
|
# entities that don't yet belong to a real project. `""` (empty) is the
|
||||||
|
# cross-project bucket for facts that apply to every project (material
|
||||||
|
# properties, vendor capabilities). Neither may be registered, renamed,
|
||||||
|
# or deleted via the normal registry CRUD.
|
||||||
|
INBOX_PROJECT = "inbox"
|
||||||
|
GLOBAL_PROJECT = ""
|
||||||
|
_RESERVED_PROJECT_IDS = {INBOX_PROJECT}
|
||||||
|
|
||||||
|
|
||||||
|
def is_reserved_project(name: str) -> bool:
|
||||||
|
return (name or "").strip().lower() in _RESERVED_PROJECT_IDS
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
@dataclass(frozen=True)
|
||||||
class ProjectSourceRef:
|
class ProjectSourceRef:
|
||||||
source: str
|
source: str
|
||||||
@@ -56,8 +70,17 @@ def build_project_registration_proposal(
|
|||||||
normalized_id = project_id.strip()
|
normalized_id = project_id.strip()
|
||||||
if not normalized_id:
|
if not normalized_id:
|
||||||
raise ValueError("Project id must be non-empty")
|
raise ValueError("Project id must be non-empty")
|
||||||
|
if is_reserved_project(normalized_id):
|
||||||
|
raise ValueError(
|
||||||
|
f"Project id {normalized_id!r} is reserved and cannot be registered"
|
||||||
|
)
|
||||||
|
|
||||||
normalized_aliases = _normalize_aliases(aliases or [])
|
normalized_aliases = _normalize_aliases(aliases or [])
|
||||||
|
for alias in normalized_aliases:
|
||||||
|
if is_reserved_project(alias):
|
||||||
|
raise ValueError(
|
||||||
|
f"Alias {alias!r} is reserved and cannot be used"
|
||||||
|
)
|
||||||
normalized_roots = _normalize_ingest_roots(ingest_roots or [])
|
normalized_roots = _normalize_ingest_roots(ingest_roots or [])
|
||||||
if not normalized_roots:
|
if not normalized_roots:
|
||||||
raise ValueError("At least one ingest root is required")
|
raise ValueError("At least one ingest root is required")
|
||||||
@@ -129,6 +152,10 @@ def update_project(
|
|||||||
ingest_roots: list[dict] | tuple[dict, ...] | None = None,
|
ingest_roots: list[dict] | tuple[dict, ...] | None = None,
|
||||||
) -> dict:
|
) -> dict:
|
||||||
"""Update an existing project registration in the registry file."""
|
"""Update an existing project registration in the registry file."""
|
||||||
|
if is_reserved_project(project_name):
|
||||||
|
raise ValueError(
|
||||||
|
f"Project {project_name!r} is reserved and cannot be modified"
|
||||||
|
)
|
||||||
existing = get_registered_project(project_name)
|
existing = get_registered_project(project_name)
|
||||||
if existing is None:
|
if existing is None:
|
||||||
raise ValueError(f"Unknown project: {project_name}")
|
raise ValueError(f"Unknown project: {project_name}")
|
||||||
@@ -272,6 +299,8 @@ def resolve_project_name(name: str | None) -> str:
|
|||||||
"""
|
"""
|
||||||
if not name:
|
if not name:
|
||||||
return name or ""
|
return name or ""
|
||||||
|
if is_reserved_project(name):
|
||||||
|
return name.strip().lower()
|
||||||
project = get_registered_project(name)
|
project = get_registered_project(name)
|
||||||
if project is not None:
|
if project is not None:
|
||||||
return project.project_id
|
return project.project_id
|
||||||
|
|||||||
@@ -16,6 +16,36 @@ os.environ["ATOCORE_DATA_DIR"] = _default_test_dir
|
|||||||
os.environ["ATOCORE_DEBUG"] = "true"
|
os.environ["ATOCORE_DEBUG"] = "true"
|
||||||
|
|
||||||
|
|
||||||
|
# V1-0: every entity created in a test is "hand authored" by the test
|
||||||
|
# author — fixture data, not extracted content. Rather than rewrite 100+
|
||||||
|
# existing test call sites, wrap create_entity so that tests which don't
|
||||||
|
# provide source_refs get hand_authored=True automatically. Tests that
|
||||||
|
# explicitly pass source_refs or hand_authored are unaffected. This keeps
|
||||||
|
# the F-8 invariant enforced in production (the API, the wiki form, and
|
||||||
|
# graduation scripts all go through the unwrapped function) while leaving
|
||||||
|
# the existing test corpus intact.
|
||||||
|
def _patch_create_entity_for_tests():
|
||||||
|
from atocore.engineering import service as _svc
|
||||||
|
|
||||||
|
_original = _svc.create_entity
|
||||||
|
|
||||||
|
def _create_entity_test(*args, **kwargs):
|
||||||
|
# Only auto-flag when hand_authored isn't explicitly specified.
|
||||||
|
# Tests that want to exercise the F-8 raise path pass
|
||||||
|
# hand_authored=False explicitly and should hit the error.
|
||||||
|
if (
|
||||||
|
not kwargs.get("source_refs")
|
||||||
|
and "hand_authored" not in kwargs
|
||||||
|
):
|
||||||
|
kwargs["hand_authored"] = True
|
||||||
|
return _original(*args, **kwargs)
|
||||||
|
|
||||||
|
_svc.create_entity = _create_entity_test
|
||||||
|
|
||||||
|
|
||||||
|
_patch_create_entity_for_tests()
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
@pytest.fixture
|
||||||
def tmp_data_dir(tmp_path):
|
def tmp_data_dir(tmp_path):
|
||||||
"""Provide a temporary data directory for tests."""
|
"""Provide a temporary data directory for tests."""
|
||||||
|
|||||||
257
tests/test_assets.py
Normal file
257
tests/test_assets.py
Normal file
@@ -0,0 +1,257 @@
|
|||||||
|
"""Issue F — binary asset store + artifact entity + wiki rendering."""
|
||||||
|
|
||||||
|
from io import BytesIO
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from fastapi.testclient import TestClient
|
||||||
|
from PIL import Image
|
||||||
|
|
||||||
|
from atocore.assets import (
|
||||||
|
AssetTooLarge,
|
||||||
|
AssetTypeNotAllowed,
|
||||||
|
get_asset,
|
||||||
|
get_asset_binary,
|
||||||
|
get_thumbnail,
|
||||||
|
invalidate_asset,
|
||||||
|
list_orphan_assets,
|
||||||
|
store_asset,
|
||||||
|
)
|
||||||
|
from atocore.engineering.service import (
|
||||||
|
ENTITY_TYPES,
|
||||||
|
create_entity,
|
||||||
|
create_relationship,
|
||||||
|
init_engineering_schema,
|
||||||
|
)
|
||||||
|
from atocore.main import app
|
||||||
|
from atocore.models.database import init_db
|
||||||
|
|
||||||
|
|
||||||
|
def _png_bytes(color=(255, 0, 0), size=(64, 48)) -> bytes:
|
||||||
|
buf = BytesIO()
|
||||||
|
Image.new("RGB", size, color).save(buf, format="PNG")
|
||||||
|
return buf.getvalue()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def assets_env(tmp_data_dir, tmp_path, monkeypatch):
|
||||||
|
registry_path = tmp_path / "test-registry.json"
|
||||||
|
registry_path.write_text('{"projects": []}', encoding="utf-8")
|
||||||
|
monkeypatch.setenv("ATOCORE_PROJECT_REGISTRY_PATH", str(registry_path))
|
||||||
|
from atocore import config
|
||||||
|
config.settings = config.Settings()
|
||||||
|
|
||||||
|
init_db()
|
||||||
|
init_engineering_schema()
|
||||||
|
yield tmp_data_dir
|
||||||
|
|
||||||
|
|
||||||
|
def test_artifact_is_in_entity_types():
|
||||||
|
assert "artifact" in ENTITY_TYPES
|
||||||
|
|
||||||
|
|
||||||
|
def test_store_asset_happy_path(assets_env):
|
||||||
|
data = _png_bytes()
|
||||||
|
asset = store_asset(data=data, mime_type="image/png", caption="red square")
|
||||||
|
assert asset.hash_sha256
|
||||||
|
assert asset.size_bytes == len(data)
|
||||||
|
assert asset.width == 64
|
||||||
|
assert asset.height == 48
|
||||||
|
assert asset.mime_type == "image/png"
|
||||||
|
from pathlib import Path
|
||||||
|
assert Path(asset.stored_path).exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_store_asset_is_idempotent_on_hash(assets_env):
|
||||||
|
data = _png_bytes()
|
||||||
|
a = store_asset(data=data, mime_type="image/png")
|
||||||
|
b = store_asset(data=data, mime_type="image/png", caption="different caption")
|
||||||
|
assert a.id == b.id, "same content should dedup to the same asset id"
|
||||||
|
|
||||||
|
|
||||||
|
def test_store_asset_rejects_unknown_mime(assets_env):
|
||||||
|
with pytest.raises(AssetTypeNotAllowed):
|
||||||
|
store_asset(data=b"hello", mime_type="text/plain")
|
||||||
|
|
||||||
|
|
||||||
|
def test_store_asset_rejects_oversize(assets_env, monkeypatch):
|
||||||
|
monkeypatch.setattr(
|
||||||
|
"atocore.config.settings.assets_max_upload_bytes",
|
||||||
|
10,
|
||||||
|
raising=False,
|
||||||
|
)
|
||||||
|
with pytest.raises(AssetTooLarge):
|
||||||
|
store_asset(data=_png_bytes(), mime_type="image/png")
|
||||||
|
|
||||||
|
|
||||||
|
def test_get_asset_binary_roundtrip(assets_env):
|
||||||
|
data = _png_bytes(color=(0, 255, 0))
|
||||||
|
asset = store_asset(data=data, mime_type="image/png")
|
||||||
|
_, roundtrip = get_asset_binary(asset.id)
|
||||||
|
assert roundtrip == data
|
||||||
|
|
||||||
|
|
||||||
|
def test_thumbnail_generates_and_caches(assets_env):
|
||||||
|
data = _png_bytes(size=(800, 600))
|
||||||
|
asset = store_asset(data=data, mime_type="image/png")
|
||||||
|
_, thumb1 = get_thumbnail(asset.id, size=120)
|
||||||
|
_, thumb2 = get_thumbnail(asset.id, size=120)
|
||||||
|
assert thumb1 == thumb2
|
||||||
|
# Must be a valid JPEG and smaller than the source
|
||||||
|
assert thumb1[:3] == b"\xff\xd8\xff"
|
||||||
|
assert len(thumb1) < len(data)
|
||||||
|
|
||||||
|
|
||||||
|
def test_orphan_list_excludes_referenced(assets_env):
|
||||||
|
referenced = store_asset(data=_png_bytes((1, 1, 1)), mime_type="image/png")
|
||||||
|
lonely = store_asset(data=_png_bytes((2, 2, 2)), mime_type="image/png")
|
||||||
|
create_entity(
|
||||||
|
entity_type="artifact",
|
||||||
|
name="ref-test",
|
||||||
|
properties={"kind": "image", "asset_id": referenced.id},
|
||||||
|
)
|
||||||
|
orphan_ids = {o.id for o in list_orphan_assets()}
|
||||||
|
assert lonely.id in orphan_ids
|
||||||
|
assert referenced.id not in orphan_ids
|
||||||
|
|
||||||
|
|
||||||
|
def test_invalidate_refuses_referenced_asset(assets_env):
|
||||||
|
asset = store_asset(data=_png_bytes((3, 3, 3)), mime_type="image/png")
|
||||||
|
create_entity(
|
||||||
|
entity_type="artifact",
|
||||||
|
name="pinned",
|
||||||
|
properties={"kind": "image", "asset_id": asset.id},
|
||||||
|
)
|
||||||
|
assert invalidate_asset(asset.id) is False
|
||||||
|
assert get_asset(asset.id).status == "active"
|
||||||
|
|
||||||
|
|
||||||
|
def test_invalidate_orphan_succeeds(assets_env):
|
||||||
|
asset = store_asset(data=_png_bytes((4, 4, 4)), mime_type="image/png")
|
||||||
|
assert invalidate_asset(asset.id) is True
|
||||||
|
assert get_asset(asset.id).status == "invalid"
|
||||||
|
|
||||||
|
|
||||||
|
def test_api_upload_and_fetch(assets_env):
|
||||||
|
client = TestClient(app)
|
||||||
|
png = _png_bytes((7, 7, 7))
|
||||||
|
r = client.post(
|
||||||
|
"/assets",
|
||||||
|
files={"file": ("red.png", png, "image/png")},
|
||||||
|
data={"project": "p05", "caption": "unit test upload"},
|
||||||
|
)
|
||||||
|
assert r.status_code == 200, r.text
|
||||||
|
body = r.json()
|
||||||
|
assert body["mime_type"] == "image/png"
|
||||||
|
assert body["caption"] == "unit test upload"
|
||||||
|
asset_id = body["id"]
|
||||||
|
|
||||||
|
r2 = client.get(f"/assets/{asset_id}")
|
||||||
|
assert r2.status_code == 200
|
||||||
|
assert r2.headers["content-type"].startswith("image/png")
|
||||||
|
assert r2.content == png
|
||||||
|
|
||||||
|
r3 = client.get(f"/assets/{asset_id}/thumbnail?size=100")
|
||||||
|
assert r3.status_code == 200
|
||||||
|
assert r3.headers["content-type"].startswith("image/jpeg")
|
||||||
|
|
||||||
|
r4 = client.get(f"/assets/{asset_id}/meta")
|
||||||
|
assert r4.status_code == 200
|
||||||
|
assert r4.json()["id"] == asset_id
|
||||||
|
|
||||||
|
|
||||||
|
def test_api_upload_rejects_bad_mime(assets_env):
|
||||||
|
client = TestClient(app)
|
||||||
|
r = client.post(
|
||||||
|
"/assets",
|
||||||
|
files={"file": ("notes.txt", b"hello", "text/plain")},
|
||||||
|
)
|
||||||
|
assert r.status_code == 415
|
||||||
|
|
||||||
|
|
||||||
|
def test_api_get_entity_evidence_returns_artifacts(assets_env):
|
||||||
|
asset = store_asset(data=_png_bytes((9, 9, 9)), mime_type="image/png")
|
||||||
|
artifact = create_entity(
|
||||||
|
entity_type="artifact",
|
||||||
|
name="cap-001",
|
||||||
|
properties={
|
||||||
|
"kind": "image",
|
||||||
|
"asset_id": asset.id,
|
||||||
|
"caption": "tower base",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
tower = create_entity(entity_type="component", name="tower")
|
||||||
|
create_relationship(
|
||||||
|
source_entity_id=tower.id,
|
||||||
|
target_entity_id=artifact.id,
|
||||||
|
relationship_type="evidenced_by",
|
||||||
|
)
|
||||||
|
|
||||||
|
client = TestClient(app)
|
||||||
|
r = client.get(f"/entities/{tower.id}/evidence")
|
||||||
|
assert r.status_code == 200
|
||||||
|
body = r.json()
|
||||||
|
assert body["count"] == 1
|
||||||
|
ev = body["evidence"][0]
|
||||||
|
assert ev["kind"] == "image"
|
||||||
|
assert ev["caption"] == "tower base"
|
||||||
|
assert ev["asset"]["id"] == asset.id
|
||||||
|
|
||||||
|
|
||||||
|
def test_v1_assets_aliases_present(assets_env):
|
||||||
|
client = TestClient(app)
|
||||||
|
spec = client.get("/openapi.json").json()
|
||||||
|
paths = spec["paths"]
|
||||||
|
for p in (
|
||||||
|
"/v1/assets",
|
||||||
|
"/v1/assets/{asset_id}",
|
||||||
|
"/v1/assets/{asset_id}/thumbnail",
|
||||||
|
"/v1/assets/{asset_id}/meta",
|
||||||
|
"/v1/entities/{entity_id}/evidence",
|
||||||
|
):
|
||||||
|
assert p in paths, f"{p} missing from /v1 alias set"
|
||||||
|
|
||||||
|
|
||||||
|
def test_wiki_renders_evidence_strip(assets_env):
|
||||||
|
from atocore.engineering.wiki import render_entity
|
||||||
|
|
||||||
|
asset = store_asset(data=_png_bytes((10, 10, 10)), mime_type="image/png")
|
||||||
|
artifact = create_entity(
|
||||||
|
entity_type="artifact",
|
||||||
|
name="cap-ev-01",
|
||||||
|
properties={
|
||||||
|
"kind": "image",
|
||||||
|
"asset_id": asset.id,
|
||||||
|
"caption": "viewport",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
tower = create_entity(entity_type="component", name="tower-wiki")
|
||||||
|
create_relationship(
|
||||||
|
source_entity_id=tower.id,
|
||||||
|
target_entity_id=artifact.id,
|
||||||
|
relationship_type="evidenced_by",
|
||||||
|
)
|
||||||
|
|
||||||
|
html = render_entity(tower.id)
|
||||||
|
assert "Visual evidence" in html
|
||||||
|
assert f"/assets/{asset.id}/thumbnail" in html
|
||||||
|
assert "viewport" in html
|
||||||
|
|
||||||
|
|
||||||
|
def test_wiki_renders_artifact_full_image(assets_env):
|
||||||
|
from atocore.engineering.wiki import render_entity
|
||||||
|
|
||||||
|
asset = store_asset(data=_png_bytes((11, 11, 11)), mime_type="image/png")
|
||||||
|
artifact = create_entity(
|
||||||
|
entity_type="artifact",
|
||||||
|
name="cap-full-01",
|
||||||
|
properties={
|
||||||
|
"kind": "image",
|
||||||
|
"asset_id": asset.id,
|
||||||
|
"caption": "detail shot",
|
||||||
|
"capture_context": "narrator: here's the base plate close-up",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
html = render_entity(artifact.id)
|
||||||
|
assert f"/assets/{asset.id}/thumbnail?size=1024" in html
|
||||||
|
assert "Capture context" in html
|
||||||
|
assert "narrator" in html
|
||||||
251
tests/test_confidence_decay.py
Normal file
251
tests/test_confidence_decay.py
Normal file
@@ -0,0 +1,251 @@
|
|||||||
|
"""Phase 7D — confidence decay tests.
|
||||||
|
|
||||||
|
Covers:
|
||||||
|
- idle unreferenced memories decay at the expected rate
|
||||||
|
- fresh / reinforced memories are untouched
|
||||||
|
- below floor → auto-supersede with audit
|
||||||
|
- graduated memories exempt
|
||||||
|
- reinforcement reverses decay (integration with Phase 9 Commit B)
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from datetime import datetime, timedelta, timezone
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from atocore.memory.service import (
|
||||||
|
create_memory,
|
||||||
|
decay_unreferenced_memories,
|
||||||
|
get_memory_audit,
|
||||||
|
reinforce_memory,
|
||||||
|
)
|
||||||
|
from atocore.models.database import get_connection, init_db
|
||||||
|
|
||||||
|
|
||||||
|
def _force_old(mem_id: str, days_ago: int) -> None:
|
||||||
|
"""Force last_referenced_at and created_at to N days in the past."""
|
||||||
|
ts = (datetime.now(timezone.utc) - timedelta(days=days_ago)).strftime("%Y-%m-%d %H:%M:%S")
|
||||||
|
with get_connection() as conn:
|
||||||
|
conn.execute(
|
||||||
|
"UPDATE memories SET last_referenced_at = ?, created_at = ? WHERE id = ?",
|
||||||
|
(ts, ts, mem_id),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _set_confidence(mem_id: str, c: float) -> None:
|
||||||
|
with get_connection() as conn:
|
||||||
|
conn.execute("UPDATE memories SET confidence = ? WHERE id = ?", (c, mem_id))
|
||||||
|
|
||||||
|
|
||||||
|
def _set_reference_count(mem_id: str, n: int) -> None:
|
||||||
|
with get_connection() as conn:
|
||||||
|
conn.execute("UPDATE memories SET reference_count = ? WHERE id = ?", (n, mem_id))
|
||||||
|
|
||||||
|
|
||||||
|
def _get(mem_id: str) -> dict:
|
||||||
|
with get_connection() as conn:
|
||||||
|
row = conn.execute("SELECT * FROM memories WHERE id = ?", (mem_id,)).fetchone()
|
||||||
|
return dict(row) if row else {}
|
||||||
|
|
||||||
|
|
||||||
|
def _set_status(mem_id: str, status: str) -> None:
|
||||||
|
with get_connection() as conn:
|
||||||
|
conn.execute("UPDATE memories SET status = ? WHERE id = ?", (status, mem_id))
|
||||||
|
|
||||||
|
|
||||||
|
# --- Basic decay mechanics ---
|
||||||
|
|
||||||
|
|
||||||
|
def test_decay_applies_to_idle_unreferenced(tmp_data_dir):
|
||||||
|
init_db()
|
||||||
|
m = create_memory("knowledge", "cold fact", confidence=0.8)
|
||||||
|
_force_old(m.id, days_ago=60)
|
||||||
|
_set_reference_count(m.id, 0)
|
||||||
|
|
||||||
|
result = decay_unreferenced_memories()
|
||||||
|
assert len(result["decayed"]) == 1
|
||||||
|
assert result["decayed"][0]["memory_id"] == m.id
|
||||||
|
|
||||||
|
row = _get(m.id)
|
||||||
|
# 0.8 * 0.97 = 0.776
|
||||||
|
assert row["confidence"] == pytest.approx(0.776)
|
||||||
|
assert row["status"] == "active" # still above floor
|
||||||
|
|
||||||
|
|
||||||
|
def test_decay_skips_fresh_memory(tmp_data_dir):
|
||||||
|
"""A memory created today shouldn't decay even if reference_count=0."""
|
||||||
|
init_db()
|
||||||
|
m = create_memory("knowledge", "just-created fact", confidence=0.8)
|
||||||
|
# Don't force old — it's fresh
|
||||||
|
result = decay_unreferenced_memories()
|
||||||
|
assert not any(e["memory_id"] == m.id for e in result["decayed"])
|
||||||
|
assert not any(e["memory_id"] == m.id for e in result["superseded"])
|
||||||
|
|
||||||
|
row = _get(m.id)
|
||||||
|
assert row["confidence"] == pytest.approx(0.8)
|
||||||
|
|
||||||
|
|
||||||
|
def test_decay_skips_reinforced_memory(tmp_data_dir):
|
||||||
|
"""Any reinforcement protects the memory from decay."""
|
||||||
|
init_db()
|
||||||
|
m = create_memory("knowledge", "referenced fact", confidence=0.8)
|
||||||
|
_force_old(m.id, days_ago=90)
|
||||||
|
_set_reference_count(m.id, 1) # just one reference is enough
|
||||||
|
|
||||||
|
result = decay_unreferenced_memories()
|
||||||
|
assert not any(e["memory_id"] == m.id for e in result["decayed"])
|
||||||
|
|
||||||
|
row = _get(m.id)
|
||||||
|
assert row["confidence"] == pytest.approx(0.8)
|
||||||
|
|
||||||
|
|
||||||
|
# --- Auto-supersede at floor ---
|
||||||
|
|
||||||
|
|
||||||
|
def test_decay_supersedes_below_floor(tmp_data_dir):
|
||||||
|
init_db()
|
||||||
|
m = create_memory("knowledge", "very cold fact", confidence=0.31)
|
||||||
|
_force_old(m.id, days_ago=60)
|
||||||
|
_set_reference_count(m.id, 0)
|
||||||
|
|
||||||
|
# 0.31 * 0.97 = 0.3007 which is still above the default floor 0.30.
|
||||||
|
# Drop it a hair lower to cross the floor in one step.
|
||||||
|
_set_confidence(m.id, 0.305)
|
||||||
|
|
||||||
|
result = decay_unreferenced_memories(supersede_confidence_floor=0.30)
|
||||||
|
# 0.305 * 0.97 = 0.29585 → below 0.30, supersede
|
||||||
|
assert len(result["superseded"]) == 1
|
||||||
|
assert result["superseded"][0]["memory_id"] == m.id
|
||||||
|
|
||||||
|
row = _get(m.id)
|
||||||
|
assert row["status"] == "superseded"
|
||||||
|
assert row["confidence"] < 0.30
|
||||||
|
|
||||||
|
|
||||||
|
def test_supersede_writes_audit_row(tmp_data_dir):
|
||||||
|
init_db()
|
||||||
|
m = create_memory("knowledge", "will decay out", confidence=0.305)
|
||||||
|
_force_old(m.id, days_ago=60)
|
||||||
|
_set_reference_count(m.id, 0)
|
||||||
|
|
||||||
|
decay_unreferenced_memories(supersede_confidence_floor=0.30)
|
||||||
|
|
||||||
|
audit = get_memory_audit(m.id)
|
||||||
|
actions = [a["action"] for a in audit]
|
||||||
|
assert "superseded" in actions
|
||||||
|
entry = next(a for a in audit if a["action"] == "superseded")
|
||||||
|
assert entry["actor"] == "confidence-decay"
|
||||||
|
assert "decayed below floor" in entry["note"]
|
||||||
|
|
||||||
|
|
||||||
|
# --- Exemptions ---
|
||||||
|
|
||||||
|
|
||||||
|
def test_decay_skips_graduated_memory(tmp_data_dir):
|
||||||
|
"""Graduated memories are frozen pointers to entities — never decay."""
|
||||||
|
init_db()
|
||||||
|
m = create_memory("knowledge", "graduated fact", confidence=0.8)
|
||||||
|
_force_old(m.id, days_ago=90)
|
||||||
|
_set_reference_count(m.id, 0)
|
||||||
|
_set_status(m.id, "graduated")
|
||||||
|
|
||||||
|
result = decay_unreferenced_memories()
|
||||||
|
assert not any(e["memory_id"] == m.id for e in result["decayed"])
|
||||||
|
|
||||||
|
row = _get(m.id)
|
||||||
|
assert row["confidence"] == pytest.approx(0.8) # unchanged
|
||||||
|
|
||||||
|
|
||||||
|
def test_decay_skips_superseded_memory(tmp_data_dir):
|
||||||
|
"""Already superseded memories don't decay further."""
|
||||||
|
init_db()
|
||||||
|
m = create_memory("knowledge", "old news", confidence=0.5)
|
||||||
|
_force_old(m.id, days_ago=90)
|
||||||
|
_set_reference_count(m.id, 0)
|
||||||
|
_set_status(m.id, "superseded")
|
||||||
|
|
||||||
|
result = decay_unreferenced_memories()
|
||||||
|
assert not any(e["memory_id"] == m.id for e in result["decayed"])
|
||||||
|
|
||||||
|
|
||||||
|
# --- Reversibility ---
|
||||||
|
|
||||||
|
|
||||||
|
def test_reinforcement_reverses_decay(tmp_data_dir):
|
||||||
|
"""A memory that decayed then got reinforced comes back up."""
|
||||||
|
init_db()
|
||||||
|
m = create_memory("knowledge", "will come back", confidence=0.8)
|
||||||
|
_force_old(m.id, days_ago=60)
|
||||||
|
_set_reference_count(m.id, 0)
|
||||||
|
|
||||||
|
decay_unreferenced_memories()
|
||||||
|
# Now at 0.776
|
||||||
|
reinforce_memory(m.id, confidence_delta=0.05)
|
||||||
|
row = _get(m.id)
|
||||||
|
assert row["confidence"] == pytest.approx(0.826)
|
||||||
|
assert row["reference_count"] >= 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_reinforced_memory_no_longer_decays(tmp_data_dir):
|
||||||
|
"""Once reinforce_memory bumps reference_count, decay skips it."""
|
||||||
|
init_db()
|
||||||
|
m = create_memory("knowledge", "protected", confidence=0.8)
|
||||||
|
_force_old(m.id, days_ago=90)
|
||||||
|
# Simulate reinforcement
|
||||||
|
reinforce_memory(m.id)
|
||||||
|
|
||||||
|
result = decay_unreferenced_memories()
|
||||||
|
assert not any(e["memory_id"] == m.id for e in result["decayed"])
|
||||||
|
|
||||||
|
|
||||||
|
# --- Parameter validation ---
|
||||||
|
|
||||||
|
|
||||||
|
def test_decay_rejects_invalid_factor(tmp_data_dir):
|
||||||
|
init_db()
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
decay_unreferenced_memories(daily_decay_factor=1.0)
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
decay_unreferenced_memories(daily_decay_factor=0.0)
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
decay_unreferenced_memories(daily_decay_factor=-0.5)
|
||||||
|
|
||||||
|
|
||||||
|
def test_decay_rejects_invalid_floor(tmp_data_dir):
|
||||||
|
init_db()
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
decay_unreferenced_memories(supersede_confidence_floor=1.5)
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
decay_unreferenced_memories(supersede_confidence_floor=-0.1)
|
||||||
|
|
||||||
|
|
||||||
|
# --- Threshold tuning ---
|
||||||
|
|
||||||
|
|
||||||
|
def test_decay_threshold_tight_excludes_newer(tmp_data_dir):
|
||||||
|
"""With idle_days_threshold=90, a 60-day-old memory should NOT decay."""
|
||||||
|
init_db()
|
||||||
|
m = create_memory("knowledge", "60-day-old", confidence=0.8)
|
||||||
|
_force_old(m.id, days_ago=60)
|
||||||
|
_set_reference_count(m.id, 0)
|
||||||
|
|
||||||
|
result = decay_unreferenced_memories(idle_days_threshold=90)
|
||||||
|
assert not any(e["memory_id"] == m.id for e in result["decayed"])
|
||||||
|
|
||||||
|
|
||||||
|
# --- Idempotency-ish (multiple runs apply additional decay) ---
|
||||||
|
|
||||||
|
|
||||||
|
def test_decay_stacks_across_runs(tmp_data_dir):
|
||||||
|
"""Running decay twice (simulating two days) compounds the factor."""
|
||||||
|
init_db()
|
||||||
|
m = create_memory("knowledge", "aging fact", confidence=0.8)
|
||||||
|
_force_old(m.id, days_ago=60)
|
||||||
|
_set_reference_count(m.id, 0)
|
||||||
|
|
||||||
|
decay_unreferenced_memories()
|
||||||
|
decay_unreferenced_memories()
|
||||||
|
row = _get(m.id)
|
||||||
|
# 0.8 * 0.97 * 0.97 = 0.75272
|
||||||
|
assert row["confidence"] == pytest.approx(0.75272, rel=1e-4)
|
||||||
@@ -143,8 +143,11 @@ def test_requirement_name_conflict_detected(tmp_data_dir):
|
|||||||
r2 = create_entity("requirement", "Surface figure < 25nm",
|
r2 = create_entity("requirement", "Surface figure < 25nm",
|
||||||
project="p-test", description="Different interpretation")
|
project="p-test", description="Different interpretation")
|
||||||
|
|
||||||
detected = detect_conflicts_for_entity(r2.id)
|
# V1-0 synchronous hook: the conflict is already detected at r2's
|
||||||
assert len(detected) == 1
|
# create-time, so a redundant detect call returns [] due to
|
||||||
|
# _record_conflict dedup. Assert on list_open_conflicts instead —
|
||||||
|
# that's what the intent of this test really tests: duplicate
|
||||||
|
# active requirements surface as an open conflict.
|
||||||
conflicts = list_open_conflicts(project="p-test")
|
conflicts = list_open_conflicts(project="p-test")
|
||||||
assert any(c["slot_kind"] == "requirement.name" for c in conflicts)
|
assert any(c["slot_kind"] == "requirement.name" for c in conflicts)
|
||||||
|
|
||||||
@@ -191,8 +194,12 @@ def test_conflict_resolution_dismiss_leaves_entities_alone(tmp_data_dir):
|
|||||||
description="first meaning")
|
description="first meaning")
|
||||||
r2 = create_entity("requirement", "Dup req", project="p-test",
|
r2 = create_entity("requirement", "Dup req", project="p-test",
|
||||||
description="second meaning")
|
description="second meaning")
|
||||||
detected = detect_conflicts_for_entity(r2.id)
|
# V1-0 synchronous hook already recorded the conflict at r2's
|
||||||
conflict_id = detected[0]
|
# create-time. Look it up via list_open_conflicts rather than
|
||||||
|
# calling the detector again (which returns [] due to dedup).
|
||||||
|
open_list = list_open_conflicts(project="p-test")
|
||||||
|
assert open_list, "expected conflict recorded by create-time hook"
|
||||||
|
conflict_id = open_list[0]["id"]
|
||||||
|
|
||||||
assert resolve_conflict(conflict_id, "dismiss")
|
assert resolve_conflict(conflict_id, "dismiss")
|
||||||
# Both still active — dismiss just clears the conflict marker
|
# Both still active — dismiss just clears the conflict marker
|
||||||
|
|||||||
202
tests/test_inbox_crossproject.py
Normal file
202
tests/test_inbox_crossproject.py
Normal file
@@ -0,0 +1,202 @@
|
|||||||
|
"""Issue C — inbox pseudo-project + cross-project (project="") entities."""
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from fastapi.testclient import TestClient
|
||||||
|
|
||||||
|
from atocore.engineering.service import (
|
||||||
|
create_entity,
|
||||||
|
get_entities,
|
||||||
|
init_engineering_schema,
|
||||||
|
promote_entity,
|
||||||
|
)
|
||||||
|
from atocore.main import app
|
||||||
|
from atocore.projects.registry import (
|
||||||
|
GLOBAL_PROJECT,
|
||||||
|
INBOX_PROJECT,
|
||||||
|
is_reserved_project,
|
||||||
|
register_project,
|
||||||
|
resolve_project_name,
|
||||||
|
update_project,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def seeded_db(tmp_data_dir, tmp_path, monkeypatch):
|
||||||
|
# Isolate the project registry so "p05" etc. don't canonicalize
|
||||||
|
# to aliases inherited from the host registry.
|
||||||
|
registry_path = tmp_path / "test-registry.json"
|
||||||
|
registry_path.write_text('{"projects": []}', encoding="utf-8")
|
||||||
|
monkeypatch.setenv("ATOCORE_PROJECT_REGISTRY_PATH", str(registry_path))
|
||||||
|
from atocore import config
|
||||||
|
config.settings = config.Settings()
|
||||||
|
|
||||||
|
init_engineering_schema()
|
||||||
|
# Audit table lives in the memory schema — bring it up so audit rows
|
||||||
|
# don't spam warnings during retargeting tests.
|
||||||
|
from atocore.models.database import init_db
|
||||||
|
init_db()
|
||||||
|
yield tmp_data_dir
|
||||||
|
|
||||||
|
|
||||||
|
def test_inbox_is_reserved():
|
||||||
|
assert is_reserved_project("inbox") is True
|
||||||
|
assert is_reserved_project("INBOX") is True
|
||||||
|
assert is_reserved_project("p05-interferometer") is False
|
||||||
|
assert is_reserved_project("") is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_resolve_project_name_preserves_inbox():
|
||||||
|
assert resolve_project_name("inbox") == "inbox"
|
||||||
|
assert resolve_project_name("INBOX") == "inbox"
|
||||||
|
assert resolve_project_name("") == ""
|
||||||
|
|
||||||
|
|
||||||
|
def test_cannot_register_inbox(tmp_path, monkeypatch):
|
||||||
|
monkeypatch.setenv(
|
||||||
|
"ATOCORE_PROJECT_REGISTRY_PATH",
|
||||||
|
str(tmp_path / "registry.json"),
|
||||||
|
)
|
||||||
|
from atocore import config
|
||||||
|
config.settings = config.Settings()
|
||||||
|
|
||||||
|
with pytest.raises(ValueError, match="reserved"):
|
||||||
|
register_project(
|
||||||
|
project_id="inbox",
|
||||||
|
ingest_roots=[{"source": "vault", "subpath": "incoming/inbox"}],
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_cannot_update_inbox(tmp_path, monkeypatch):
|
||||||
|
monkeypatch.setenv(
|
||||||
|
"ATOCORE_PROJECT_REGISTRY_PATH",
|
||||||
|
str(tmp_path / "registry.json"),
|
||||||
|
)
|
||||||
|
from atocore import config
|
||||||
|
config.settings = config.Settings()
|
||||||
|
|
||||||
|
with pytest.raises(ValueError, match="reserved"):
|
||||||
|
update_project(project_name="inbox", description="hijack attempt")
|
||||||
|
|
||||||
|
|
||||||
|
def test_create_entity_with_empty_project_is_global(seeded_db):
|
||||||
|
e = create_entity(entity_type="material", name="Invar", project="")
|
||||||
|
assert e.project == ""
|
||||||
|
|
||||||
|
|
||||||
|
def test_create_entity_in_inbox(seeded_db):
|
||||||
|
e = create_entity(entity_type="vendor", name="Zygo", project="inbox")
|
||||||
|
assert e.project == "inbox"
|
||||||
|
|
||||||
|
|
||||||
|
def test_get_entities_inbox_scope(seeded_db):
|
||||||
|
create_entity(entity_type="vendor", name="Zygo", project="inbox")
|
||||||
|
create_entity(entity_type="material", name="Invar", project="")
|
||||||
|
create_entity(entity_type="component", name="Mirror", project="p05")
|
||||||
|
|
||||||
|
inbox = get_entities(project=INBOX_PROJECT, scope_only=True)
|
||||||
|
assert {e.name for e in inbox} == {"Zygo"}
|
||||||
|
|
||||||
|
|
||||||
|
def test_get_entities_global_scope(seeded_db):
|
||||||
|
create_entity(entity_type="vendor", name="Zygo", project="inbox")
|
||||||
|
create_entity(entity_type="material", name="Invar", project="")
|
||||||
|
create_entity(entity_type="component", name="Mirror", project="p05")
|
||||||
|
|
||||||
|
globals_ = get_entities(project=GLOBAL_PROJECT, scope_only=True)
|
||||||
|
assert {e.name for e in globals_} == {"Invar"}
|
||||||
|
|
||||||
|
|
||||||
|
def test_real_project_includes_global_by_default(seeded_db):
|
||||||
|
create_entity(entity_type="material", name="Invar", project="")
|
||||||
|
create_entity(entity_type="component", name="Mirror", project="p05")
|
||||||
|
create_entity(entity_type="component", name="Other", project="p06")
|
||||||
|
|
||||||
|
p05 = get_entities(project="p05")
|
||||||
|
names = {e.name for e in p05}
|
||||||
|
assert "Mirror" in names
|
||||||
|
assert "Invar" in names, "cross-project material should bleed in by default"
|
||||||
|
assert "Other" not in names
|
||||||
|
|
||||||
|
|
||||||
|
def test_real_project_scope_only_excludes_global(seeded_db):
|
||||||
|
create_entity(entity_type="material", name="Invar", project="")
|
||||||
|
create_entity(entity_type="component", name="Mirror", project="p05")
|
||||||
|
|
||||||
|
p05 = get_entities(project="p05", scope_only=True)
|
||||||
|
assert {e.name for e in p05} == {"Mirror"}
|
||||||
|
|
||||||
|
|
||||||
|
def test_api_post_entity_with_null_project_stores_global(seeded_db):
|
||||||
|
client = TestClient(app)
|
||||||
|
r = client.post("/entities", json={
|
||||||
|
"entity_type": "material",
|
||||||
|
"name": "Titanium",
|
||||||
|
"project": None,
|
||||||
|
"hand_authored": True, # V1-0 F-8: test fixture, no source_refs
|
||||||
|
})
|
||||||
|
assert r.status_code == 200
|
||||||
|
|
||||||
|
globals_ = get_entities(project=GLOBAL_PROJECT, scope_only=True)
|
||||||
|
assert any(e.name == "Titanium" for e in globals_)
|
||||||
|
|
||||||
|
|
||||||
|
def test_api_get_entities_scope_only(seeded_db):
|
||||||
|
create_entity(entity_type="material", name="Invar", project="")
|
||||||
|
create_entity(entity_type="component", name="Mirror", project="p05")
|
||||||
|
|
||||||
|
client = TestClient(app)
|
||||||
|
mixed = client.get("/entities?project=p05").json()
|
||||||
|
scoped = client.get("/entities?project=p05&scope_only=true").json()
|
||||||
|
|
||||||
|
assert mixed["count"] == 2
|
||||||
|
assert scoped["count"] == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_promote_with_target_project_retargets(seeded_db):
|
||||||
|
e = create_entity(
|
||||||
|
entity_type="vendor",
|
||||||
|
name="ZygoLead",
|
||||||
|
project="inbox",
|
||||||
|
status="candidate",
|
||||||
|
)
|
||||||
|
ok = promote_entity(e.id, target_project="p05")
|
||||||
|
assert ok is True
|
||||||
|
|
||||||
|
from atocore.engineering.service import get_entity
|
||||||
|
promoted = get_entity(e.id)
|
||||||
|
assert promoted.status == "active"
|
||||||
|
assert promoted.project == "p05"
|
||||||
|
|
||||||
|
|
||||||
|
def test_promote_without_target_project_keeps_project(seeded_db):
|
||||||
|
e = create_entity(
|
||||||
|
entity_type="vendor",
|
||||||
|
name="ZygoStay",
|
||||||
|
project="inbox",
|
||||||
|
status="candidate",
|
||||||
|
)
|
||||||
|
ok = promote_entity(e.id)
|
||||||
|
assert ok is True
|
||||||
|
|
||||||
|
from atocore.engineering.service import get_entity
|
||||||
|
promoted = get_entity(e.id)
|
||||||
|
assert promoted.status == "active"
|
||||||
|
assert promoted.project == "inbox"
|
||||||
|
|
||||||
|
|
||||||
|
def test_api_promote_with_target_project(seeded_db):
|
||||||
|
e = create_entity(
|
||||||
|
entity_type="vendor",
|
||||||
|
name="ZygoApi",
|
||||||
|
project="inbox",
|
||||||
|
status="candidate",
|
||||||
|
)
|
||||||
|
client = TestClient(app)
|
||||||
|
r = client.post(
|
||||||
|
f"/entities/{e.id}/promote",
|
||||||
|
json={"target_project": "p05"},
|
||||||
|
)
|
||||||
|
assert r.status_code == 200
|
||||||
|
body = r.json()
|
||||||
|
assert body["status"] == "promoted"
|
||||||
|
assert body["target_project"] == "p05"
|
||||||
198
tests/test_inject_context_hook.py
Normal file
198
tests/test_inject_context_hook.py
Normal file
@@ -0,0 +1,198 @@
|
|||||||
|
"""Tests for deploy/hooks/inject_context.py — Claude Code UserPromptSubmit hook.
|
||||||
|
|
||||||
|
These are process-level tests: we run the actual script with subprocess,
|
||||||
|
feed it stdin, and check the exit code + stdout shape. The hook must:
|
||||||
|
- always exit 0 (never block a user prompt)
|
||||||
|
- emit valid hookSpecificOutput JSON on success
|
||||||
|
- fail open (empty output) on network errors, bad stdin, kill-switch
|
||||||
|
- respect the short-prompt filter
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
HOOK = Path(__file__).resolve().parent.parent / "deploy" / "hooks" / "inject_context.py"
|
||||||
|
|
||||||
|
|
||||||
|
def _run_hook(stdin_json: dict | str, env_overrides: dict | None = None, timeout: float = 10) -> tuple[int, str, str]:
|
||||||
|
env = os.environ.copy()
|
||||||
|
# Force kill switch off unless the test overrides
|
||||||
|
env.pop("ATOCORE_CONTEXT_DISABLED", None)
|
||||||
|
if env_overrides:
|
||||||
|
env.update(env_overrides)
|
||||||
|
stdin = stdin_json if isinstance(stdin_json, str) else json.dumps(stdin_json)
|
||||||
|
proc = subprocess.run(
|
||||||
|
[sys.executable, str(HOOK)],
|
||||||
|
input=stdin, text=True,
|
||||||
|
capture_output=True, timeout=timeout,
|
||||||
|
env=env,
|
||||||
|
)
|
||||||
|
return proc.returncode, proc.stdout, proc.stderr
|
||||||
|
|
||||||
|
|
||||||
|
def test_hook_exit_0_on_success_or_failure():
|
||||||
|
"""Canonical contract: the hook never blocks a prompt. Even with a
|
||||||
|
bogus URL we must exit 0 with empty stdout (fail-open)."""
|
||||||
|
code, stdout, stderr = _run_hook(
|
||||||
|
{
|
||||||
|
"prompt": "What's the p04-gigabit current status?",
|
||||||
|
"cwd": "/tmp",
|
||||||
|
"session_id": "t",
|
||||||
|
"hook_event_name": "UserPromptSubmit",
|
||||||
|
},
|
||||||
|
env_overrides={"ATOCORE_URL": "http://127.0.0.1:1", # unreachable
|
||||||
|
"ATOCORE_CONTEXT_TIMEOUT": "1"},
|
||||||
|
)
|
||||||
|
assert code == 0
|
||||||
|
# stdout is empty (fail-open) — no hookSpecificOutput emitted
|
||||||
|
assert stdout.strip() == ""
|
||||||
|
assert "atocore unreachable" in stderr or "request failed" in stderr
|
||||||
|
|
||||||
|
|
||||||
|
def test_hook_kill_switch():
|
||||||
|
code, stdout, stderr = _run_hook(
|
||||||
|
{"prompt": "hello world is this a thing", "cwd": "", "session_id": "t"},
|
||||||
|
env_overrides={"ATOCORE_CONTEXT_DISABLED": "1"},
|
||||||
|
)
|
||||||
|
assert code == 0
|
||||||
|
assert stdout.strip() == ""
|
||||||
|
|
||||||
|
|
||||||
|
def test_hook_ignores_short_prompt():
|
||||||
|
code, stdout, _ = _run_hook(
|
||||||
|
{"prompt": "ok", "cwd": "", "session_id": "t"},
|
||||||
|
env_overrides={"ATOCORE_URL": "http://127.0.0.1:1"},
|
||||||
|
)
|
||||||
|
assert code == 0
|
||||||
|
# No network call attempted; empty output
|
||||||
|
assert stdout.strip() == ""
|
||||||
|
|
||||||
|
|
||||||
|
def test_hook_ignores_xml_prompt():
|
||||||
|
"""System/meta prompts starting with '<' should be skipped."""
|
||||||
|
code, stdout, _ = _run_hook(
|
||||||
|
{"prompt": "<system>do something</system>", "cwd": "", "session_id": "t"},
|
||||||
|
env_overrides={"ATOCORE_URL": "http://127.0.0.1:1"},
|
||||||
|
)
|
||||||
|
assert code == 0
|
||||||
|
assert stdout.strip() == ""
|
||||||
|
|
||||||
|
|
||||||
|
def test_hook_handles_bad_stdin():
|
||||||
|
code, stdout, stderr = _run_hook("not-json-at-all")
|
||||||
|
assert code == 0
|
||||||
|
assert stdout.strip() == ""
|
||||||
|
assert "bad stdin" in stderr
|
||||||
|
|
||||||
|
|
||||||
|
def test_hook_handles_empty_stdin():
|
||||||
|
code, stdout, _ = _run_hook("")
|
||||||
|
assert code == 0
|
||||||
|
assert stdout.strip() == ""
|
||||||
|
|
||||||
|
|
||||||
|
def test_hook_success_shape_with_mock_server(monkeypatch, tmp_path):
|
||||||
|
"""When the API returns a pack, the hook emits valid
|
||||||
|
hookSpecificOutput JSON wrapping it."""
|
||||||
|
# Start a tiny HTTP server on localhost that returns a fake pack
|
||||||
|
import http.server
|
||||||
|
import json as _json
|
||||||
|
import threading
|
||||||
|
|
||||||
|
pack = "Trusted State: foo=bar"
|
||||||
|
|
||||||
|
class Handler(http.server.BaseHTTPRequestHandler):
|
||||||
|
def do_POST(self): # noqa: N802
|
||||||
|
self.rfile.read(int(self.headers.get("Content-Length", 0)))
|
||||||
|
body = _json.dumps({"formatted_context": pack}).encode()
|
||||||
|
self.send_response(200)
|
||||||
|
self.send_header("Content-Type", "application/json")
|
||||||
|
self.send_header("Content-Length", str(len(body)))
|
||||||
|
self.end_headers()
|
||||||
|
self.wfile.write(body)
|
||||||
|
|
||||||
|
def log_message(self, *a, **kw):
|
||||||
|
pass
|
||||||
|
|
||||||
|
server = http.server.HTTPServer(("127.0.0.1", 0), Handler)
|
||||||
|
port = server.server_address[1]
|
||||||
|
t = threading.Thread(target=server.serve_forever, daemon=True)
|
||||||
|
t.start()
|
||||||
|
try:
|
||||||
|
code, stdout, stderr = _run_hook(
|
||||||
|
{
|
||||||
|
"prompt": "What do we know about p04?",
|
||||||
|
"cwd": "",
|
||||||
|
"session_id": "t",
|
||||||
|
"hook_event_name": "UserPromptSubmit",
|
||||||
|
},
|
||||||
|
env_overrides={
|
||||||
|
"ATOCORE_URL": f"http://127.0.0.1:{port}",
|
||||||
|
"ATOCORE_CONTEXT_TIMEOUT": "5",
|
||||||
|
},
|
||||||
|
timeout=15,
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
server.shutdown()
|
||||||
|
|
||||||
|
assert code == 0, stderr
|
||||||
|
assert stdout.strip(), "expected JSON output with context"
|
||||||
|
out = json.loads(stdout)
|
||||||
|
hso = out.get("hookSpecificOutput", {})
|
||||||
|
assert hso.get("hookEventName") == "UserPromptSubmit"
|
||||||
|
assert pack in hso.get("additionalContext", "")
|
||||||
|
assert "AtoCore-injected context" in hso.get("additionalContext", "")
|
||||||
|
|
||||||
|
|
||||||
|
def test_hook_project_inference_from_cwd(monkeypatch):
|
||||||
|
"""The hook should map a known cwd to a project slug and send it in
|
||||||
|
the /context/build payload."""
|
||||||
|
import http.server
|
||||||
|
import json as _json
|
||||||
|
import threading
|
||||||
|
|
||||||
|
captured_body: dict = {}
|
||||||
|
|
||||||
|
class Handler(http.server.BaseHTTPRequestHandler):
|
||||||
|
def do_POST(self): # noqa: N802
|
||||||
|
n = int(self.headers.get("Content-Length", 0))
|
||||||
|
body = self.rfile.read(n)
|
||||||
|
captured_body.update(_json.loads(body.decode()))
|
||||||
|
out = _json.dumps({"formatted_context": "ok"}).encode()
|
||||||
|
self.send_response(200)
|
||||||
|
self.send_header("Content-Length", str(len(out)))
|
||||||
|
self.end_headers()
|
||||||
|
self.wfile.write(out)
|
||||||
|
|
||||||
|
def log_message(self, *a, **kw):
|
||||||
|
pass
|
||||||
|
|
||||||
|
server = http.server.HTTPServer(("127.0.0.1", 0), Handler)
|
||||||
|
port = server.server_address[1]
|
||||||
|
t = threading.Thread(target=server.serve_forever, daemon=True)
|
||||||
|
t.start()
|
||||||
|
try:
|
||||||
|
_run_hook(
|
||||||
|
{
|
||||||
|
"prompt": "Is this being tested properly",
|
||||||
|
"cwd": "C:\\Users\\antoi\\ATOCore",
|
||||||
|
"session_id": "t",
|
||||||
|
},
|
||||||
|
env_overrides={
|
||||||
|
"ATOCORE_URL": f"http://127.0.0.1:{port}",
|
||||||
|
"ATOCORE_CONTEXT_TIMEOUT": "5",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
server.shutdown()
|
||||||
|
|
||||||
|
# Hook should have inferred project="atocore" from the ATOCore cwd
|
||||||
|
assert captured_body.get("project") == "atocore"
|
||||||
|
assert captured_body.get("prompt", "").startswith("Is this being tested")
|
||||||
194
tests/test_invalidate_supersede.py
Normal file
194
tests/test_invalidate_supersede.py
Normal file
@@ -0,0 +1,194 @@
|
|||||||
|
"""Issue E — /invalidate + /supersede for active entities and memories."""
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from fastapi.testclient import TestClient
|
||||||
|
|
||||||
|
from atocore.engineering.service import (
|
||||||
|
create_entity,
|
||||||
|
get_entity,
|
||||||
|
get_relationships,
|
||||||
|
init_engineering_schema,
|
||||||
|
invalidate_active_entity,
|
||||||
|
supersede_entity,
|
||||||
|
)
|
||||||
|
from atocore.main import app
|
||||||
|
from atocore.memory.service import create_memory, get_memories
|
||||||
|
|
||||||
|
|
||||||
|
def _get_memory(memory_id):
|
||||||
|
for status in ("active", "candidate", "invalid", "superseded"):
|
||||||
|
for m in get_memories(status=status, active_only=False, limit=5000):
|
||||||
|
if m.id == memory_id:
|
||||||
|
return m
|
||||||
|
return None
|
||||||
|
from atocore.models.database import init_db
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def env(tmp_data_dir, tmp_path, monkeypatch):
|
||||||
|
registry_path = tmp_path / "test-registry.json"
|
||||||
|
registry_path.write_text('{"projects": []}', encoding="utf-8")
|
||||||
|
monkeypatch.setenv("ATOCORE_PROJECT_REGISTRY_PATH", str(registry_path))
|
||||||
|
from atocore import config
|
||||||
|
config.settings = config.Settings()
|
||||||
|
init_db()
|
||||||
|
init_engineering_schema()
|
||||||
|
yield tmp_data_dir
|
||||||
|
|
||||||
|
|
||||||
|
def test_invalidate_active_entity_transitions_to_invalid(env):
|
||||||
|
e = create_entity(entity_type="component", name="tower-to-kill")
|
||||||
|
ok, code = invalidate_active_entity(e.id, reason="duplicate")
|
||||||
|
assert ok is True
|
||||||
|
assert code == "invalidated"
|
||||||
|
assert get_entity(e.id).status == "invalid"
|
||||||
|
|
||||||
|
|
||||||
|
def test_invalidate_on_candidate_is_409(env):
|
||||||
|
e = create_entity(entity_type="component", name="still-candidate", status="candidate")
|
||||||
|
ok, code = invalidate_active_entity(e.id)
|
||||||
|
assert ok is False
|
||||||
|
assert code == "not_active"
|
||||||
|
|
||||||
|
|
||||||
|
def test_invalidate_is_idempotent_on_invalid(env):
|
||||||
|
e = create_entity(entity_type="component", name="already-gone")
|
||||||
|
invalidate_active_entity(e.id)
|
||||||
|
ok, code = invalidate_active_entity(e.id)
|
||||||
|
assert ok is True
|
||||||
|
assert code == "already_invalid"
|
||||||
|
|
||||||
|
|
||||||
|
def test_supersede_creates_relationship(env):
|
||||||
|
old = create_entity(entity_type="component", name="old-tower")
|
||||||
|
new = create_entity(entity_type="component", name="new-tower")
|
||||||
|
ok = supersede_entity(old.id, superseded_by=new.id, note="replaced")
|
||||||
|
assert ok is True
|
||||||
|
assert get_entity(old.id).status == "superseded"
|
||||||
|
|
||||||
|
rels = get_relationships(new.id, direction="outgoing")
|
||||||
|
assert any(
|
||||||
|
r.relationship_type == "supersedes" and r.target_entity_id == old.id
|
||||||
|
for r in rels
|
||||||
|
), "supersedes relationship must be auto-created"
|
||||||
|
|
||||||
|
|
||||||
|
def test_supersede_rejects_self():
|
||||||
|
# no db needed — validation is pre-write. Using a fresh env anyway.
|
||||||
|
pass # covered below via API test
|
||||||
|
|
||||||
|
|
||||||
|
def test_api_invalidate_entity(env):
|
||||||
|
e = create_entity(entity_type="component", name="api-kill")
|
||||||
|
client = TestClient(app)
|
||||||
|
r = client.post(
|
||||||
|
f"/entities/{e.id}/invalidate",
|
||||||
|
json={"reason": "test cleanup"},
|
||||||
|
)
|
||||||
|
assert r.status_code == 200
|
||||||
|
assert r.json()["status"] == "invalidated"
|
||||||
|
|
||||||
|
|
||||||
|
def test_api_invalidate_entity_idempotent(env):
|
||||||
|
e = create_entity(entity_type="component", name="api-kill-2")
|
||||||
|
client = TestClient(app)
|
||||||
|
client.post(f"/entities/{e.id}/invalidate", json={"reason": "first"})
|
||||||
|
r = client.post(f"/entities/{e.id}/invalidate", json={"reason": "second"})
|
||||||
|
assert r.status_code == 200
|
||||||
|
assert r.json()["status"] == "already_invalid"
|
||||||
|
|
||||||
|
|
||||||
|
def test_api_invalidate_unknown_entity_is_404(env):
|
||||||
|
client = TestClient(app)
|
||||||
|
r = client.post(
|
||||||
|
"/entities/nonexistent-id/invalidate",
|
||||||
|
json={"reason": "missing"},
|
||||||
|
)
|
||||||
|
assert r.status_code == 404
|
||||||
|
|
||||||
|
|
||||||
|
def test_api_invalidate_candidate_entity_is_409(env):
|
||||||
|
e = create_entity(entity_type="component", name="cand", status="candidate")
|
||||||
|
client = TestClient(app)
|
||||||
|
r = client.post(f"/entities/{e.id}/invalidate", json={"reason": "x"})
|
||||||
|
assert r.status_code == 409
|
||||||
|
|
||||||
|
|
||||||
|
def test_api_supersede_entity(env):
|
||||||
|
old = create_entity(entity_type="component", name="api-old-tower")
|
||||||
|
new = create_entity(entity_type="component", name="api-new-tower")
|
||||||
|
client = TestClient(app)
|
||||||
|
r = client.post(
|
||||||
|
f"/entities/{old.id}/supersede",
|
||||||
|
json={"superseded_by": new.id, "reason": "dedup"},
|
||||||
|
)
|
||||||
|
assert r.status_code == 200
|
||||||
|
body = r.json()
|
||||||
|
assert body["status"] == "superseded"
|
||||||
|
assert body["superseded_by"] == new.id
|
||||||
|
assert get_entity(old.id).status == "superseded"
|
||||||
|
|
||||||
|
|
||||||
|
def test_api_supersede_self_is_400(env):
|
||||||
|
e = create_entity(entity_type="component", name="self-sup")
|
||||||
|
client = TestClient(app)
|
||||||
|
r = client.post(
|
||||||
|
f"/entities/{e.id}/supersede",
|
||||||
|
json={"superseded_by": e.id, "reason": "oops"},
|
||||||
|
)
|
||||||
|
assert r.status_code == 400
|
||||||
|
|
||||||
|
|
||||||
|
def test_api_supersede_missing_replacement_is_400(env):
|
||||||
|
old = create_entity(entity_type="component", name="orphan-old")
|
||||||
|
client = TestClient(app)
|
||||||
|
r = client.post(
|
||||||
|
f"/entities/{old.id}/supersede",
|
||||||
|
json={"superseded_by": "does-not-exist", "reason": "missing"},
|
||||||
|
)
|
||||||
|
assert r.status_code == 400
|
||||||
|
|
||||||
|
|
||||||
|
def test_api_invalidate_memory(env):
|
||||||
|
m = create_memory(
|
||||||
|
memory_type="project",
|
||||||
|
content="memory to retract",
|
||||||
|
project="p05",
|
||||||
|
)
|
||||||
|
client = TestClient(app)
|
||||||
|
r = client.post(
|
||||||
|
f"/memory/{m.id}/invalidate",
|
||||||
|
json={"reason": "outdated"},
|
||||||
|
)
|
||||||
|
assert r.status_code == 200
|
||||||
|
assert r.json()["status"] == "invalidated"
|
||||||
|
assert _get_memory(m.id).status == "invalid"
|
||||||
|
|
||||||
|
|
||||||
|
def test_api_supersede_memory(env):
|
||||||
|
m = create_memory(
|
||||||
|
memory_type="project",
|
||||||
|
content="memory to supersede",
|
||||||
|
project="p05",
|
||||||
|
)
|
||||||
|
client = TestClient(app)
|
||||||
|
r = client.post(
|
||||||
|
f"/memory/{m.id}/supersede",
|
||||||
|
json={"reason": "replaced by newer fact"},
|
||||||
|
)
|
||||||
|
assert r.status_code == 200
|
||||||
|
assert r.json()["status"] == "superseded"
|
||||||
|
assert _get_memory(m.id).status == "superseded"
|
||||||
|
|
||||||
|
|
||||||
|
def test_v1_aliases_present(env):
|
||||||
|
client = TestClient(app)
|
||||||
|
spec = client.get("/openapi.json").json()
|
||||||
|
paths = spec["paths"]
|
||||||
|
for p in (
|
||||||
|
"/v1/entities/{entity_id}/invalidate",
|
||||||
|
"/v1/entities/{entity_id}/supersede",
|
||||||
|
"/v1/memory/{memory_id}/invalidate",
|
||||||
|
"/v1/memory/{memory_id}/supersede",
|
||||||
|
):
|
||||||
|
assert p in paths, f"{p} missing"
|
||||||
501
tests/test_memory_dedup.py
Normal file
501
tests/test_memory_dedup.py
Normal file
@@ -0,0 +1,501 @@
|
|||||||
|
"""Phase 7A — memory consolidation tests.
|
||||||
|
|
||||||
|
Covers:
|
||||||
|
- similarity helpers (cosine bounds, matrix symmetry, clustering)
|
||||||
|
- _dedup_prompt parser / normalizer robustness
|
||||||
|
- create_merge_candidate idempotency
|
||||||
|
- get_merge_candidates inlines source memories
|
||||||
|
- merge_memories end-to-end happy path (sources → superseded,
|
||||||
|
new merged memory active, audit rows, result_memory_id)
|
||||||
|
- reject_merge_candidate leaves sources untouched
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from atocore.memory._dedup_prompt import (
|
||||||
|
TIER2_SYSTEM_PROMPT,
|
||||||
|
build_tier2_user_message,
|
||||||
|
normalize_merge_verdict,
|
||||||
|
parse_merge_verdict,
|
||||||
|
)
|
||||||
|
from atocore.memory.service import (
|
||||||
|
create_memory,
|
||||||
|
create_merge_candidate,
|
||||||
|
get_memory_audit,
|
||||||
|
get_merge_candidates,
|
||||||
|
merge_memories,
|
||||||
|
reject_merge_candidate,
|
||||||
|
)
|
||||||
|
from atocore.memory.similarity import (
|
||||||
|
cluster_by_threshold,
|
||||||
|
cosine,
|
||||||
|
compute_memory_similarity,
|
||||||
|
similarity_matrix,
|
||||||
|
)
|
||||||
|
from atocore.models.database import get_connection, init_db
|
||||||
|
|
||||||
|
|
||||||
|
# --- Similarity helpers ---
|
||||||
|
|
||||||
|
|
||||||
|
def test_cosine_bounds():
|
||||||
|
assert cosine([1.0, 0.0], [1.0, 0.0]) == pytest.approx(1.0)
|
||||||
|
assert cosine([1.0, 0.0], [0.0, 1.0]) == pytest.approx(0.0)
|
||||||
|
# Negative dot product clamped to 0
|
||||||
|
assert cosine([1.0, 0.0], [-1.0, 0.0]) == 0.0
|
||||||
|
|
||||||
|
|
||||||
|
def test_compute_memory_similarity_identical_high():
|
||||||
|
s = compute_memory_similarity("the sky is blue", "the sky is blue")
|
||||||
|
assert 0.99 <= s <= 1.0
|
||||||
|
|
||||||
|
|
||||||
|
def test_compute_memory_similarity_unrelated_low():
|
||||||
|
s = compute_memory_similarity(
|
||||||
|
"APM integrates with NX via a Python bridge",
|
||||||
|
"the polisher firmware must use USB SSD not SD card",
|
||||||
|
)
|
||||||
|
assert 0.0 <= s < 0.7
|
||||||
|
|
||||||
|
|
||||||
|
def test_similarity_matrix_symmetric():
|
||||||
|
texts = ["alpha beta gamma", "alpha beta gamma", "completely unrelated text"]
|
||||||
|
m = similarity_matrix(texts)
|
||||||
|
assert len(m) == 3 and all(len(r) == 3 for r in m)
|
||||||
|
for i in range(3):
|
||||||
|
assert m[i][i] == pytest.approx(1.0)
|
||||||
|
for i in range(3):
|
||||||
|
for j in range(3):
|
||||||
|
assert m[i][j] == pytest.approx(m[j][i])
|
||||||
|
|
||||||
|
|
||||||
|
def test_cluster_by_threshold_transitive():
|
||||||
|
# Three near-paraphrases should land in one cluster
|
||||||
|
texts = [
|
||||||
|
"Antoine prefers OAuth over API keys",
|
||||||
|
"Antoine's preference is OAuth, not API keys",
|
||||||
|
"the polisher firmware uses USB SSD storage",
|
||||||
|
]
|
||||||
|
clusters = cluster_by_threshold(texts, threshold=0.7)
|
||||||
|
# At least one cluster of size 2+ containing the paraphrases
|
||||||
|
big = [c for c in clusters if len(c) >= 2]
|
||||||
|
assert big, f"expected at least one multi-member cluster, got {clusters}"
|
||||||
|
assert 0 in big[0] and 1 in big[0]
|
||||||
|
|
||||||
|
|
||||||
|
# --- Prompt parser robustness ---
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_merge_verdict_strips_fences():
|
||||||
|
raw = "```json\n{\"action\":\"merge\",\"content\":\"x\"}\n```"
|
||||||
|
parsed = parse_merge_verdict(raw)
|
||||||
|
assert parsed == {"action": "merge", "content": "x"}
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_merge_verdict_handles_prose_prefix():
|
||||||
|
raw = "Sure! Here's the result:\n{\"action\":\"reject\",\"content\":\"no\"}"
|
||||||
|
parsed = parse_merge_verdict(raw)
|
||||||
|
assert parsed is not None
|
||||||
|
assert parsed["action"] == "reject"
|
||||||
|
|
||||||
|
|
||||||
|
def test_normalize_merge_verdict_fills_defaults():
|
||||||
|
v = normalize_merge_verdict({
|
||||||
|
"action": "merge",
|
||||||
|
"content": "unified text",
|
||||||
|
})
|
||||||
|
assert v is not None
|
||||||
|
assert v["memory_type"] == "knowledge"
|
||||||
|
assert v["project"] == ""
|
||||||
|
assert v["domain_tags"] == []
|
||||||
|
assert v["confidence"] == 0.5
|
||||||
|
|
||||||
|
|
||||||
|
def test_normalize_merge_verdict_rejects_empty_content():
|
||||||
|
assert normalize_merge_verdict({"action": "merge", "content": ""}) is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_normalize_merge_verdict_rejects_unknown_action():
|
||||||
|
assert normalize_merge_verdict({"action": "?", "content": "x"}) is None
|
||||||
|
|
||||||
|
|
||||||
|
# --- Tier-2 (Phase 7A.1) ---
|
||||||
|
|
||||||
|
|
||||||
|
def test_tier2_prompt_is_stricter():
|
||||||
|
# The tier-2 system prompt must explicitly instruct the model to be
|
||||||
|
# stricter than tier-1 — that's the whole point of escalation.
|
||||||
|
assert "STRICTER" in TIER2_SYSTEM_PROMPT
|
||||||
|
assert "REJECT" in TIER2_SYSTEM_PROMPT
|
||||||
|
|
||||||
|
|
||||||
|
def test_build_tier2_user_message_includes_tier1_draft():
|
||||||
|
sources = [{
|
||||||
|
"id": "abc12345", "content": "source text A",
|
||||||
|
"memory_type": "knowledge", "project": "p04",
|
||||||
|
"domain_tags": ["optics"], "confidence": 0.6,
|
||||||
|
"valid_until": "", "reference_count": 2,
|
||||||
|
}, {
|
||||||
|
"id": "def67890", "content": "source text B",
|
||||||
|
"memory_type": "knowledge", "project": "p04",
|
||||||
|
"domain_tags": ["optics"], "confidence": 0.7,
|
||||||
|
"valid_until": "", "reference_count": 1,
|
||||||
|
}]
|
||||||
|
tier1 = {
|
||||||
|
"action": "merge",
|
||||||
|
"content": "unified draft by tier1",
|
||||||
|
"memory_type": "knowledge",
|
||||||
|
"project": "p04",
|
||||||
|
"domain_tags": ["optics"],
|
||||||
|
"confidence": 0.65,
|
||||||
|
"reason": "near-paraphrase",
|
||||||
|
}
|
||||||
|
msg = build_tier2_user_message(sources, tier1)
|
||||||
|
assert "source text A" in msg
|
||||||
|
assert "source text B" in msg
|
||||||
|
assert "TIER-1 DRAFT" in msg
|
||||||
|
assert "unified draft by tier1" in msg
|
||||||
|
assert "near-paraphrase" in msg
|
||||||
|
# Should end asking for a verdict
|
||||||
|
assert "verdict" in msg.lower()
|
||||||
|
|
||||||
|
|
||||||
|
# --- Host script is stdlib-only (Phase 7A architecture rule) ---
|
||||||
|
|
||||||
|
|
||||||
|
def test_memory_dedup_script_is_stdlib_only():
|
||||||
|
"""The host-side scripts/memory_dedup.py must NOT import anything
|
||||||
|
that pulls pydantic_settings, sentence-transformers, torch, etc.
|
||||||
|
into the host Python. The only atocore-land module allowed is the
|
||||||
|
stdlib-only prompt helper at atocore.memory._dedup_prompt.
|
||||||
|
|
||||||
|
This regression test prevents re-introducing the bug where the
|
||||||
|
dedup-watcher on Dalidou host crashed with ModuleNotFoundError
|
||||||
|
because someone imported atocore.memory.similarity (which pulls
|
||||||
|
in atocore.retrieval.embeddings → sentence_transformers)."""
|
||||||
|
import importlib.util
|
||||||
|
import sys as _sys
|
||||||
|
|
||||||
|
before = set(_sys.modules.keys())
|
||||||
|
spec = importlib.util.spec_from_file_location(
|
||||||
|
"memory_dedup_for_test", "scripts/memory_dedup.py",
|
||||||
|
)
|
||||||
|
mod = importlib.util.module_from_spec(spec)
|
||||||
|
spec.loader.exec_module(mod)
|
||||||
|
after = set(_sys.modules.keys())
|
||||||
|
|
||||||
|
new_atocore = sorted(m for m in (after - before) if m.startswith("atocore"))
|
||||||
|
# Only the stdlib-only shared prompt module is allowed to load
|
||||||
|
allowed = {"atocore", "atocore.memory", "atocore.memory._dedup_prompt"}
|
||||||
|
disallowed = [m for m in new_atocore if m not in allowed]
|
||||||
|
assert not disallowed, (
|
||||||
|
f"scripts/memory_dedup.py pulled non-stdlib atocore modules "
|
||||||
|
f"(will break host Python without ML deps): {disallowed}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# --- Server-side clustering (still in atocore.memory.similarity) ---
|
||||||
|
|
||||||
|
|
||||||
|
def test_similarity_module_server_side():
|
||||||
|
"""similarity.py stays server-side for ML deps. These helpers are
|
||||||
|
only invoked via the /admin/memory/dedup-cluster endpoint."""
|
||||||
|
from atocore.memory.similarity import cluster_by_threshold
|
||||||
|
clusters = cluster_by_threshold(
|
||||||
|
["duplicate fact A", "duplicate fact A slightly reworded",
|
||||||
|
"totally unrelated fact about firmware"],
|
||||||
|
threshold=0.7,
|
||||||
|
)
|
||||||
|
multi = [c for c in clusters if len(c) >= 2]
|
||||||
|
assert multi, "expected at least one multi-member cluster"
|
||||||
|
|
||||||
|
|
||||||
|
def test_cluster_endpoint_returns_groups(tmp_data_dir):
|
||||||
|
"""POST /admin/memory/dedup-cluster shape test — we just verify the
|
||||||
|
service layer produces the expected output. Full HTTP is
|
||||||
|
integration-tested by the live scan."""
|
||||||
|
from atocore.models.database import init_db
|
||||||
|
init_db()
|
||||||
|
from atocore.memory.service import create_memory, get_memories
|
||||||
|
create_memory("knowledge", "APM uses NX bridge for DXF to STL conversion",
|
||||||
|
project="apm")
|
||||||
|
create_memory("knowledge", "APM uses the NX Python bridge for DXF-to-STL",
|
||||||
|
project="apm")
|
||||||
|
create_memory("knowledge", "The polisher firmware requires USB SSD storage",
|
||||||
|
project="p06-polisher")
|
||||||
|
|
||||||
|
# Mirror the server code path
|
||||||
|
from atocore.memory.similarity import cluster_by_threshold
|
||||||
|
mems = get_memories(project="apm", active_only=True, limit=100)
|
||||||
|
texts = [m.content for m in mems]
|
||||||
|
clusters = cluster_by_threshold(texts, threshold=0.7)
|
||||||
|
multi = [c for c in clusters if len(c) >= 2]
|
||||||
|
assert multi, "expected the two APM memories to cluster together"
|
||||||
|
# Unrelated p06 memory should NOT be in that cluster
|
||||||
|
apm_ids = {mems[i].id for i in multi[0]}
|
||||||
|
assert len(apm_ids) == 2
|
||||||
|
all_ids = {m.id for m in mems}
|
||||||
|
assert apm_ids.issubset(all_ids)
|
||||||
|
|
||||||
|
|
||||||
|
# --- create_merge_candidate idempotency ---
|
||||||
|
|
||||||
|
|
||||||
|
def test_create_merge_candidate_inserts_row(tmp_data_dir):
|
||||||
|
init_db()
|
||||||
|
m1 = create_memory("knowledge", "APM uses NX for DXF conversion")
|
||||||
|
m2 = create_memory("knowledge", "APM uses NX for DXF-to-STL")
|
||||||
|
|
||||||
|
cid = create_merge_candidate(
|
||||||
|
memory_ids=[m1.id, m2.id],
|
||||||
|
similarity=0.92,
|
||||||
|
proposed_content="APM uses NX for DXF→STL conversion",
|
||||||
|
proposed_memory_type="knowledge",
|
||||||
|
proposed_project="",
|
||||||
|
proposed_tags=["apm", "nx"],
|
||||||
|
proposed_confidence=0.6,
|
||||||
|
reason="near-paraphrase",
|
||||||
|
)
|
||||||
|
assert cid is not None
|
||||||
|
|
||||||
|
pending = get_merge_candidates(status="pending")
|
||||||
|
assert len(pending) == 1
|
||||||
|
assert pending[0]["id"] == cid
|
||||||
|
assert pending[0]["similarity"] == pytest.approx(0.92)
|
||||||
|
assert len(pending[0]["sources"]) == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_create_merge_candidate_idempotent(tmp_data_dir):
|
||||||
|
init_db()
|
||||||
|
m1 = create_memory("knowledge", "Fact A")
|
||||||
|
m2 = create_memory("knowledge", "Fact A slightly reworded")
|
||||||
|
|
||||||
|
first = create_merge_candidate(
|
||||||
|
memory_ids=[m1.id, m2.id],
|
||||||
|
similarity=0.9,
|
||||||
|
proposed_content="merged",
|
||||||
|
proposed_memory_type="knowledge",
|
||||||
|
proposed_project="",
|
||||||
|
)
|
||||||
|
# Same id set, different order → dedupe skips
|
||||||
|
second = create_merge_candidate(
|
||||||
|
memory_ids=[m2.id, m1.id],
|
||||||
|
similarity=0.9,
|
||||||
|
proposed_content="merged (again)",
|
||||||
|
proposed_memory_type="knowledge",
|
||||||
|
proposed_project="",
|
||||||
|
)
|
||||||
|
assert first is not None
|
||||||
|
assert second is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_create_merge_candidate_requires_two_ids(tmp_data_dir):
|
||||||
|
init_db()
|
||||||
|
m1 = create_memory("knowledge", "lonely")
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
create_merge_candidate(
|
||||||
|
memory_ids=[m1.id],
|
||||||
|
similarity=1.0,
|
||||||
|
proposed_content="x",
|
||||||
|
proposed_memory_type="knowledge",
|
||||||
|
proposed_project="",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# --- merge_memories end-to-end ---
|
||||||
|
|
||||||
|
|
||||||
|
def test_merge_memories_happy_path(tmp_data_dir):
|
||||||
|
init_db()
|
||||||
|
m1 = create_memory(
|
||||||
|
"knowledge", "APM uses NX for DXF conversion",
|
||||||
|
project="apm", confidence=0.6, domain_tags=["apm", "nx"],
|
||||||
|
)
|
||||||
|
m2 = create_memory(
|
||||||
|
"knowledge", "APM does DXF to STL via NX bridge",
|
||||||
|
project="apm", confidence=0.8, domain_tags=["apm", "bridge"],
|
||||||
|
)
|
||||||
|
# Bump reference counts so sum is meaningful
|
||||||
|
with get_connection() as conn:
|
||||||
|
conn.execute("UPDATE memories SET reference_count = 3 WHERE id = ?", (m1.id,))
|
||||||
|
conn.execute("UPDATE memories SET reference_count = 5 WHERE id = ?", (m2.id,))
|
||||||
|
|
||||||
|
cid = create_merge_candidate(
|
||||||
|
memory_ids=[m1.id, m2.id],
|
||||||
|
similarity=0.92,
|
||||||
|
proposed_content="APM uses NX bridge for DXF→STL conversion",
|
||||||
|
proposed_memory_type="knowledge",
|
||||||
|
proposed_project="apm",
|
||||||
|
proposed_tags=["apm", "nx", "bridge"],
|
||||||
|
proposed_confidence=0.7,
|
||||||
|
reason="duplicates",
|
||||||
|
)
|
||||||
|
new_id = merge_memories(candidate_id=cid, actor="human-triage")
|
||||||
|
assert new_id is not None
|
||||||
|
|
||||||
|
# Sources superseded
|
||||||
|
with get_connection() as conn:
|
||||||
|
s1 = conn.execute("SELECT status FROM memories WHERE id = ?", (m1.id,)).fetchone()
|
||||||
|
s2 = conn.execute("SELECT status FROM memories WHERE id = ?", (m2.id,)).fetchone()
|
||||||
|
merged = conn.execute(
|
||||||
|
"SELECT content, status, confidence, reference_count, project "
|
||||||
|
"FROM memories WHERE id = ?", (new_id,)
|
||||||
|
).fetchone()
|
||||||
|
cand = conn.execute(
|
||||||
|
"SELECT status, result_memory_id FROM memory_merge_candidates WHERE id = ?",
|
||||||
|
(cid,),
|
||||||
|
).fetchone()
|
||||||
|
assert s1["status"] == "superseded"
|
||||||
|
assert s2["status"] == "superseded"
|
||||||
|
assert merged["status"] == "active"
|
||||||
|
assert merged["project"] == "apm"
|
||||||
|
# confidence = max of sources (0.8), not the proposed 0.7 (proposed is hint;
|
||||||
|
# merge_memories picks max of actual source confidences — verify).
|
||||||
|
assert merged["confidence"] == pytest.approx(0.8)
|
||||||
|
# reference_count = sum (3 + 5 = 8)
|
||||||
|
assert int(merged["reference_count"]) == 8
|
||||||
|
assert cand["status"] == "approved"
|
||||||
|
assert cand["result_memory_id"] == new_id
|
||||||
|
|
||||||
|
|
||||||
|
def test_merge_memories_content_override(tmp_data_dir):
|
||||||
|
init_db()
|
||||||
|
m1 = create_memory("knowledge", "draft A", project="p05-interferometer")
|
||||||
|
m2 = create_memory("knowledge", "draft B", project="p05-interferometer")
|
||||||
|
|
||||||
|
cid = create_merge_candidate(
|
||||||
|
memory_ids=[m1.id, m2.id],
|
||||||
|
similarity=0.9,
|
||||||
|
proposed_content="AI draft",
|
||||||
|
proposed_memory_type="knowledge",
|
||||||
|
proposed_project="p05-interferometer",
|
||||||
|
)
|
||||||
|
new_id = merge_memories(
|
||||||
|
candidate_id=cid,
|
||||||
|
actor="human-triage",
|
||||||
|
override_content="human-edited final text",
|
||||||
|
override_tags=["optics", "custom"],
|
||||||
|
)
|
||||||
|
assert new_id is not None
|
||||||
|
with get_connection() as conn:
|
||||||
|
row = conn.execute(
|
||||||
|
"SELECT content, domain_tags FROM memories WHERE id = ?", (new_id,)
|
||||||
|
).fetchone()
|
||||||
|
assert row["content"] == "human-edited final text"
|
||||||
|
# domain_tags JSON should contain the override
|
||||||
|
assert "optics" in row["domain_tags"]
|
||||||
|
assert "custom" in row["domain_tags"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_merge_memories_writes_audit(tmp_data_dir):
|
||||||
|
init_db()
|
||||||
|
m1 = create_memory("knowledge", "alpha")
|
||||||
|
m2 = create_memory("knowledge", "alpha variant")
|
||||||
|
cid = create_merge_candidate(
|
||||||
|
memory_ids=[m1.id, m2.id], similarity=0.9,
|
||||||
|
proposed_content="alpha merged",
|
||||||
|
proposed_memory_type="knowledge", proposed_project="",
|
||||||
|
)
|
||||||
|
new_id = merge_memories(candidate_id=cid)
|
||||||
|
assert new_id
|
||||||
|
|
||||||
|
audit_new = get_memory_audit(new_id)
|
||||||
|
actions_new = {a["action"] for a in audit_new}
|
||||||
|
assert "created_via_merge" in actions_new
|
||||||
|
|
||||||
|
audit_m1 = get_memory_audit(m1.id)
|
||||||
|
actions_m1 = {a["action"] for a in audit_m1}
|
||||||
|
assert "superseded" in actions_m1
|
||||||
|
|
||||||
|
|
||||||
|
def test_merge_memories_aborts_if_source_not_active(tmp_data_dir):
|
||||||
|
init_db()
|
||||||
|
m1 = create_memory("knowledge", "one")
|
||||||
|
m2 = create_memory("knowledge", "two")
|
||||||
|
cid = create_merge_candidate(
|
||||||
|
memory_ids=[m1.id, m2.id], similarity=0.9,
|
||||||
|
proposed_content="merged",
|
||||||
|
proposed_memory_type="knowledge", proposed_project="",
|
||||||
|
)
|
||||||
|
# Tamper: supersede one source before the merge runs
|
||||||
|
with get_connection() as conn:
|
||||||
|
conn.execute("UPDATE memories SET status = 'superseded' WHERE id = ?", (m1.id,))
|
||||||
|
result = merge_memories(candidate_id=cid)
|
||||||
|
assert result is None
|
||||||
|
|
||||||
|
# Candidate still pending
|
||||||
|
pending = get_merge_candidates(status="pending")
|
||||||
|
assert any(c["id"] == cid for c in pending)
|
||||||
|
|
||||||
|
|
||||||
|
def test_merge_memories_rejects_already_resolved(tmp_data_dir):
|
||||||
|
init_db()
|
||||||
|
m1 = create_memory("knowledge", "x")
|
||||||
|
m2 = create_memory("knowledge", "y")
|
||||||
|
cid = create_merge_candidate(
|
||||||
|
memory_ids=[m1.id, m2.id], similarity=0.9,
|
||||||
|
proposed_content="xy",
|
||||||
|
proposed_memory_type="knowledge", proposed_project="",
|
||||||
|
)
|
||||||
|
first = merge_memories(candidate_id=cid)
|
||||||
|
assert first is not None
|
||||||
|
# second call — already approved, should return None
|
||||||
|
second = merge_memories(candidate_id=cid)
|
||||||
|
assert second is None
|
||||||
|
|
||||||
|
|
||||||
|
# --- reject_merge_candidate ---
|
||||||
|
|
||||||
|
|
||||||
|
def test_reject_merge_candidate_leaves_sources_untouched(tmp_data_dir):
|
||||||
|
init_db()
|
||||||
|
m1 = create_memory("knowledge", "a")
|
||||||
|
m2 = create_memory("knowledge", "b")
|
||||||
|
cid = create_merge_candidate(
|
||||||
|
memory_ids=[m1.id, m2.id], similarity=0.9,
|
||||||
|
proposed_content="a+b",
|
||||||
|
proposed_memory_type="knowledge", proposed_project="",
|
||||||
|
)
|
||||||
|
ok = reject_merge_candidate(cid, actor="human-triage", note="false positive")
|
||||||
|
assert ok
|
||||||
|
|
||||||
|
# Sources still active
|
||||||
|
with get_connection() as conn:
|
||||||
|
s1 = conn.execute("SELECT status FROM memories WHERE id = ?", (m1.id,)).fetchone()
|
||||||
|
s2 = conn.execute("SELECT status FROM memories WHERE id = ?", (m2.id,)).fetchone()
|
||||||
|
cand = conn.execute(
|
||||||
|
"SELECT status FROM memory_merge_candidates WHERE id = ?", (cid,)
|
||||||
|
).fetchone()
|
||||||
|
assert s1["status"] == "active"
|
||||||
|
assert s2["status"] == "active"
|
||||||
|
assert cand["status"] == "rejected"
|
||||||
|
|
||||||
|
|
||||||
|
def test_reject_merge_candidate_idempotent(tmp_data_dir):
|
||||||
|
init_db()
|
||||||
|
m1 = create_memory("knowledge", "p")
|
||||||
|
m2 = create_memory("knowledge", "q")
|
||||||
|
cid = create_merge_candidate(
|
||||||
|
memory_ids=[m1.id, m2.id], similarity=0.9,
|
||||||
|
proposed_content="pq",
|
||||||
|
proposed_memory_type="knowledge", proposed_project="",
|
||||||
|
)
|
||||||
|
assert reject_merge_candidate(cid) is True
|
||||||
|
# second reject — already rejected, returns False
|
||||||
|
assert reject_merge_candidate(cid) is False
|
||||||
|
|
||||||
|
|
||||||
|
# --- Schema sanity ---
|
||||||
|
|
||||||
|
|
||||||
|
def test_merge_candidates_table_exists(tmp_data_dir):
|
||||||
|
init_db()
|
||||||
|
with get_connection() as conn:
|
||||||
|
cols = [r["name"] for r in conn.execute("PRAGMA table_info(memory_merge_candidates)").fetchall()]
|
||||||
|
expected = {"id", "status", "memory_ids", "similarity", "proposed_content",
|
||||||
|
"proposed_memory_type", "proposed_project", "proposed_tags",
|
||||||
|
"proposed_confidence", "reason", "created_at", "resolved_at",
|
||||||
|
"resolved_by", "result_memory_id"}
|
||||||
|
assert expected.issubset(set(cols))
|
||||||
160
tests/test_patch_entity.py
Normal file
160
tests/test_patch_entity.py
Normal file
@@ -0,0 +1,160 @@
|
|||||||
|
"""PATCH /entities/{id} — edit mutable fields without cloning (sprint P1)."""
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from fastapi.testclient import TestClient
|
||||||
|
|
||||||
|
from atocore.engineering.service import (
|
||||||
|
create_entity,
|
||||||
|
get_entity,
|
||||||
|
init_engineering_schema,
|
||||||
|
update_entity,
|
||||||
|
)
|
||||||
|
from atocore.main import app
|
||||||
|
from atocore.models.database import init_db
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def env(tmp_data_dir, tmp_path, monkeypatch):
|
||||||
|
registry_path = tmp_path / "test-registry.json"
|
||||||
|
registry_path.write_text('{"projects": []}', encoding="utf-8")
|
||||||
|
monkeypatch.setenv("ATOCORE_PROJECT_REGISTRY_PATH", str(registry_path))
|
||||||
|
from atocore import config
|
||||||
|
config.settings = config.Settings()
|
||||||
|
init_db()
|
||||||
|
init_engineering_schema()
|
||||||
|
yield tmp_data_dir
|
||||||
|
|
||||||
|
|
||||||
|
def test_update_entity_description(env):
|
||||||
|
e = create_entity(entity_type="component", name="t", description="old desc")
|
||||||
|
updated = update_entity(e.id, description="new desc")
|
||||||
|
assert updated.description == "new desc"
|
||||||
|
assert get_entity(e.id).description == "new desc"
|
||||||
|
|
||||||
|
|
||||||
|
def test_update_entity_properties_merge(env):
|
||||||
|
e = create_entity(
|
||||||
|
entity_type="component",
|
||||||
|
name="t2",
|
||||||
|
properties={"color": "red", "kg": 5},
|
||||||
|
)
|
||||||
|
updated = update_entity(
|
||||||
|
e.id, properties_patch={"color": "blue", "material": "invar"},
|
||||||
|
)
|
||||||
|
assert updated.properties == {"color": "blue", "kg": 5, "material": "invar"}
|
||||||
|
|
||||||
|
|
||||||
|
def test_update_entity_properties_null_deletes_key(env):
|
||||||
|
e = create_entity(
|
||||||
|
entity_type="component",
|
||||||
|
name="t3",
|
||||||
|
properties={"color": "red", "kg": 5},
|
||||||
|
)
|
||||||
|
updated = update_entity(e.id, properties_patch={"color": None})
|
||||||
|
assert "color" not in updated.properties
|
||||||
|
assert updated.properties.get("kg") == 5
|
||||||
|
|
||||||
|
|
||||||
|
def test_update_entity_confidence_bounds(env):
|
||||||
|
e = create_entity(entity_type="component", name="t4", confidence=0.5)
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
update_entity(e.id, confidence=1.5)
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
update_entity(e.id, confidence=-0.1)
|
||||||
|
|
||||||
|
|
||||||
|
def test_update_entity_source_refs_append_dedup(env):
|
||||||
|
e = create_entity(
|
||||||
|
entity_type="component",
|
||||||
|
name="t5",
|
||||||
|
source_refs=["session:a", "session:b"],
|
||||||
|
)
|
||||||
|
updated = update_entity(
|
||||||
|
e.id, append_source_refs=["session:b", "session:c"],
|
||||||
|
)
|
||||||
|
assert updated.source_refs == ["session:a", "session:b", "session:c"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_update_entity_returns_none_for_unknown(env):
|
||||||
|
assert update_entity("nonexistent", description="x") is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_api_patch_happy_path(env):
|
||||||
|
e = create_entity(
|
||||||
|
entity_type="component",
|
||||||
|
name="tower",
|
||||||
|
description="old",
|
||||||
|
properties={"material": "steel"},
|
||||||
|
confidence=0.6,
|
||||||
|
)
|
||||||
|
client = TestClient(app)
|
||||||
|
r = client.patch(
|
||||||
|
f"/entities/{e.id}",
|
||||||
|
json={
|
||||||
|
"description": "three-stage tower",
|
||||||
|
"properties": {"material": "invar", "height_mm": 1200},
|
||||||
|
"confidence": 0.9,
|
||||||
|
"source_refs": ["session:s1"],
|
||||||
|
"note": "from voice session",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
assert r.status_code == 200, r.text
|
||||||
|
body = r.json()
|
||||||
|
assert body["description"] == "three-stage tower"
|
||||||
|
assert body["properties"]["material"] == "invar"
|
||||||
|
assert body["properties"]["height_mm"] == 1200
|
||||||
|
assert body["confidence"] == 0.9
|
||||||
|
assert "session:s1" in body["source_refs"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_api_patch_omitted_fields_unchanged(env):
|
||||||
|
e = create_entity(
|
||||||
|
entity_type="component",
|
||||||
|
name="keep-desc",
|
||||||
|
description="keep me",
|
||||||
|
)
|
||||||
|
client = TestClient(app)
|
||||||
|
r = client.patch(
|
||||||
|
f"/entities/{e.id}",
|
||||||
|
json={"confidence": 0.7},
|
||||||
|
)
|
||||||
|
assert r.status_code == 200
|
||||||
|
assert r.json()["description"] == "keep me"
|
||||||
|
|
||||||
|
|
||||||
|
def test_api_patch_404_on_missing(env):
|
||||||
|
client = TestClient(app)
|
||||||
|
r = client.patch("/entities/does-not-exist", json={"description": "x"})
|
||||||
|
assert r.status_code == 404
|
||||||
|
|
||||||
|
|
||||||
|
def test_api_patch_rejects_bad_confidence(env):
|
||||||
|
e = create_entity(entity_type="component", name="bad-conf")
|
||||||
|
client = TestClient(app)
|
||||||
|
r = client.patch(f"/entities/{e.id}", json={"confidence": 2.0})
|
||||||
|
assert r.status_code == 400
|
||||||
|
|
||||||
|
|
||||||
|
def test_api_patch_aliased_under_v1(env):
|
||||||
|
e = create_entity(entity_type="component", name="v1-patch")
|
||||||
|
client = TestClient(app)
|
||||||
|
r = client.patch(
|
||||||
|
f"/v1/entities/{e.id}",
|
||||||
|
json={"description": "via v1"},
|
||||||
|
)
|
||||||
|
assert r.status_code == 200
|
||||||
|
assert get_entity(e.id).description == "via v1"
|
||||||
|
|
||||||
|
|
||||||
|
def test_api_patch_audit_row_written(env):
|
||||||
|
from atocore.engineering.service import get_entity_audit
|
||||||
|
|
||||||
|
e = create_entity(entity_type="component", name="audit-check")
|
||||||
|
client = TestClient(app)
|
||||||
|
client.patch(
|
||||||
|
f"/entities/{e.id}",
|
||||||
|
json={"description": "new", "note": "manual edit"},
|
||||||
|
)
|
||||||
|
audit = get_entity_audit(e.id)
|
||||||
|
actions = [a["action"] for a in audit]
|
||||||
|
assert "updated" in actions
|
||||||
148
tests/test_phase6_living_taxonomy.py
Normal file
148
tests/test_phase6_living_taxonomy.py
Normal file
@@ -0,0 +1,148 @@
|
|||||||
|
"""Phase 6 tests — Living Taxonomy: detector + transient-to-durable extension."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from datetime import datetime, timedelta, timezone
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from atocore.memory.service import (
|
||||||
|
create_memory,
|
||||||
|
extend_reinforced_valid_until,
|
||||||
|
)
|
||||||
|
from atocore.models.database import get_connection, init_db
|
||||||
|
|
||||||
|
|
||||||
|
def _set_memory_fields(mem_id, reference_count=None, valid_until=None):
|
||||||
|
"""Helper to force memory state for tests."""
|
||||||
|
with get_connection() as conn:
|
||||||
|
fields, params = [], []
|
||||||
|
if reference_count is not None:
|
||||||
|
fields.append("reference_count = ?")
|
||||||
|
params.append(reference_count)
|
||||||
|
if valid_until is not None:
|
||||||
|
fields.append("valid_until = ?")
|
||||||
|
params.append(valid_until)
|
||||||
|
params.append(mem_id)
|
||||||
|
conn.execute(
|
||||||
|
f"UPDATE memories SET {', '.join(fields)} WHERE id = ?",
|
||||||
|
params,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# --- Transient-to-durable extension (C.3) ---
|
||||||
|
|
||||||
|
|
||||||
|
def test_extend_extends_imminent_valid_until(tmp_data_dir):
|
||||||
|
init_db()
|
||||||
|
mem = create_memory("knowledge", "Reinforced content for extension")
|
||||||
|
soon = (datetime.now(timezone.utc) + timedelta(days=7)).strftime("%Y-%m-%d")
|
||||||
|
_set_memory_fields(mem.id, reference_count=6, valid_until=soon)
|
||||||
|
|
||||||
|
result = extend_reinforced_valid_until()
|
||||||
|
assert len(result) == 1
|
||||||
|
assert result[0]["memory_id"] == mem.id
|
||||||
|
assert result[0]["action"] == "extended"
|
||||||
|
# New expiry should be ~90 days out
|
||||||
|
new_date = datetime.strptime(result[0]["new_valid_until"], "%Y-%m-%d")
|
||||||
|
days_out = (new_date - datetime.now(timezone.utc).replace(tzinfo=None)).days
|
||||||
|
assert 85 <= days_out <= 92 # ~90 days, some slop for test timing
|
||||||
|
|
||||||
|
|
||||||
|
def test_extend_makes_permanent_at_high_reference_count(tmp_data_dir):
|
||||||
|
init_db()
|
||||||
|
mem = create_memory("knowledge", "Heavy-referenced content")
|
||||||
|
soon = (datetime.now(timezone.utc) + timedelta(days=7)).strftime("%Y-%m-%d")
|
||||||
|
_set_memory_fields(mem.id, reference_count=15, valid_until=soon)
|
||||||
|
|
||||||
|
result = extend_reinforced_valid_until()
|
||||||
|
assert len(result) == 1
|
||||||
|
assert result[0]["action"] == "made_permanent"
|
||||||
|
assert result[0]["new_valid_until"] is None
|
||||||
|
|
||||||
|
# Verify the DB reflects the cleared expiry
|
||||||
|
with get_connection() as conn:
|
||||||
|
row = conn.execute(
|
||||||
|
"SELECT valid_until FROM memories WHERE id = ?", (mem.id,)
|
||||||
|
).fetchone()
|
||||||
|
assert row["valid_until"] is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_extend_skips_not_expiring_soon(tmp_data_dir):
|
||||||
|
init_db()
|
||||||
|
mem = create_memory("knowledge", "Far-future expiry")
|
||||||
|
far = (datetime.now(timezone.utc) + timedelta(days=365)).strftime("%Y-%m-%d")
|
||||||
|
_set_memory_fields(mem.id, reference_count=6, valid_until=far)
|
||||||
|
|
||||||
|
result = extend_reinforced_valid_until(imminent_expiry_days=30)
|
||||||
|
assert result == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_extend_skips_low_reference_count(tmp_data_dir):
|
||||||
|
init_db()
|
||||||
|
mem = create_memory("knowledge", "Not reinforced enough")
|
||||||
|
soon = (datetime.now(timezone.utc) + timedelta(days=7)).strftime("%Y-%m-%d")
|
||||||
|
_set_memory_fields(mem.id, reference_count=2, valid_until=soon)
|
||||||
|
|
||||||
|
result = extend_reinforced_valid_until(min_reference_count=5)
|
||||||
|
assert result == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_extend_skips_permanent_memory(tmp_data_dir):
|
||||||
|
"""Memory with no valid_until is already permanent — shouldn't touch."""
|
||||||
|
init_db()
|
||||||
|
mem = create_memory("knowledge", "Already permanent")
|
||||||
|
_set_memory_fields(mem.id, reference_count=20)
|
||||||
|
# no valid_until
|
||||||
|
|
||||||
|
result = extend_reinforced_valid_until()
|
||||||
|
assert result == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_extend_writes_audit_row(tmp_data_dir):
|
||||||
|
init_db()
|
||||||
|
mem = create_memory("knowledge", "Audited extension")
|
||||||
|
soon = (datetime.now(timezone.utc) + timedelta(days=7)).strftime("%Y-%m-%d")
|
||||||
|
_set_memory_fields(mem.id, reference_count=6, valid_until=soon)
|
||||||
|
|
||||||
|
extend_reinforced_valid_until()
|
||||||
|
|
||||||
|
from atocore.memory.service import get_memory_audit
|
||||||
|
audit = get_memory_audit(mem.id)
|
||||||
|
actions = [a["action"] for a in audit]
|
||||||
|
assert "valid_until_extended" in actions
|
||||||
|
entry = next(a for a in audit if a["action"] == "valid_until_extended")
|
||||||
|
assert entry["actor"] == "transient-to-durable"
|
||||||
|
|
||||||
|
|
||||||
|
# --- Emerging detector (smoke tests — detector runs against live DB state
|
||||||
|
# so we test the shape of results rather than full integration here) ---
|
||||||
|
|
||||||
|
|
||||||
|
def test_detector_imports_cleanly():
|
||||||
|
"""Detector module must import without errors (it's called from nightly cron)."""
|
||||||
|
import importlib.util
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
# Load the detector script as a module
|
||||||
|
script = Path(__file__).resolve().parent.parent / "scripts" / "detect_emerging.py"
|
||||||
|
assert script.exists()
|
||||||
|
spec = importlib.util.spec_from_file_location("detect_emerging", script)
|
||||||
|
mod = importlib.util.module_from_spec(spec)
|
||||||
|
# Don't actually run main() — just verify it parses and defines expected names
|
||||||
|
spec.loader.exec_module(mod)
|
||||||
|
assert hasattr(mod, "main")
|
||||||
|
assert hasattr(mod, "PROJECT_MIN_MEMORIES")
|
||||||
|
assert hasattr(mod, "PROJECT_ALERT_THRESHOLD")
|
||||||
|
|
||||||
|
|
||||||
|
def test_detector_handles_empty_db(tmp_data_dir):
|
||||||
|
"""Detector should handle zero memories without crashing."""
|
||||||
|
init_db()
|
||||||
|
# Don't create any memories. Just verify the queries work via the service layer.
|
||||||
|
from atocore.memory.service import get_memories
|
||||||
|
active = get_memories(active_only=True, limit=500)
|
||||||
|
candidates = get_memories(status="candidate", limit=500)
|
||||||
|
assert active == []
|
||||||
|
assert candidates == []
|
||||||
296
tests/test_tag_canon.py
Normal file
296
tests/test_tag_canon.py
Normal file
@@ -0,0 +1,296 @@
|
|||||||
|
"""Phase 7C — tag canonicalization tests.
|
||||||
|
|
||||||
|
Covers:
|
||||||
|
- prompt parser (fences, prose, empty)
|
||||||
|
- normalizer (identity, protected tokens, empty)
|
||||||
|
- get_tag_distribution counts across active memories
|
||||||
|
- apply_tag_alias rewrites + dedupes + audits
|
||||||
|
- create / approve / reject lifecycle
|
||||||
|
- idempotency (dup proposals skipped)
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from atocore.memory._tag_canon_prompt import (
|
||||||
|
PROTECTED_PROJECT_TOKENS,
|
||||||
|
build_user_message,
|
||||||
|
normalize_alias_item,
|
||||||
|
parse_canon_output,
|
||||||
|
)
|
||||||
|
from atocore.memory.service import (
|
||||||
|
apply_tag_alias,
|
||||||
|
approve_tag_alias,
|
||||||
|
create_memory,
|
||||||
|
create_tag_alias_proposal,
|
||||||
|
get_memory_audit,
|
||||||
|
get_tag_alias_proposals,
|
||||||
|
get_tag_distribution,
|
||||||
|
reject_tag_alias,
|
||||||
|
)
|
||||||
|
from atocore.models.database import get_connection, init_db
|
||||||
|
|
||||||
|
|
||||||
|
# --- Prompt parser ---
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_canon_output_handles_fences():
|
||||||
|
raw = "```json\n{\"aliases\": [{\"alias\": \"fw\", \"canonical\": \"firmware\", \"confidence\": 0.9}]}\n```"
|
||||||
|
items = parse_canon_output(raw)
|
||||||
|
assert len(items) == 1
|
||||||
|
assert items[0]["alias"] == "fw"
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_canon_output_handles_prose_prefix():
|
||||||
|
raw = "Here you go:\n{\"aliases\": [{\"alias\": \"ml\", \"canonical\": \"machine-learning\", \"confidence\": 0.9}]}"
|
||||||
|
items = parse_canon_output(raw)
|
||||||
|
assert len(items) == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_canon_output_empty_list():
|
||||||
|
assert parse_canon_output("{\"aliases\": []}") == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_canon_output_malformed():
|
||||||
|
assert parse_canon_output("not json at all") == []
|
||||||
|
assert parse_canon_output("") == []
|
||||||
|
|
||||||
|
|
||||||
|
# --- Normalizer ---
|
||||||
|
|
||||||
|
|
||||||
|
def test_normalize_alias_strips_and_lowercases():
|
||||||
|
n = normalize_alias_item({"alias": " FW ", "canonical": "Firmware", "confidence": 0.95, "reason": "abbrev"})
|
||||||
|
assert n == {"alias": "fw", "canonical": "firmware", "confidence": 0.95, "reason": "abbrev"}
|
||||||
|
|
||||||
|
|
||||||
|
def test_normalize_rejects_identity():
|
||||||
|
assert normalize_alias_item({"alias": "foo", "canonical": "foo", "confidence": 0.9}) is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_normalize_rejects_empty():
|
||||||
|
assert normalize_alias_item({"alias": "", "canonical": "foo", "confidence": 0.9}) is None
|
||||||
|
assert normalize_alias_item({"alias": "foo", "canonical": "", "confidence": 0.9}) is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_normalize_protects_project_tokens():
|
||||||
|
# Project ids must not be canonicalized — they're their own namespace
|
||||||
|
assert "p04" in PROTECTED_PROJECT_TOKENS
|
||||||
|
assert normalize_alias_item({"alias": "p04", "canonical": "p04-gigabit", "confidence": 1.0}) is None
|
||||||
|
assert normalize_alias_item({"alias": "p04-gigabit", "canonical": "p04", "confidence": 1.0}) is None
|
||||||
|
assert normalize_alias_item({"alias": "apm", "canonical": "part-manager", "confidence": 1.0}) is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_normalize_clamps_confidence():
|
||||||
|
hi = normalize_alias_item({"alias": "a", "canonical": "b", "confidence": 2.5})
|
||||||
|
assert hi["confidence"] == 1.0
|
||||||
|
lo = normalize_alias_item({"alias": "a", "canonical": "b", "confidence": -0.5})
|
||||||
|
assert lo["confidence"] == 0.0
|
||||||
|
|
||||||
|
|
||||||
|
def test_normalize_handles_non_numeric_confidence():
|
||||||
|
n = normalize_alias_item({"alias": "a", "canonical": "b", "confidence": "not a number"})
|
||||||
|
assert n is not None and n["confidence"] == 0.0
|
||||||
|
|
||||||
|
|
||||||
|
# --- build_user_message ---
|
||||||
|
|
||||||
|
|
||||||
|
def test_build_user_message_includes_top_tags():
|
||||||
|
dist = {"firmware": 23, "fw": 5, "optics": 18, "optical": 2}
|
||||||
|
msg = build_user_message(dist)
|
||||||
|
assert "firmware: 23" in msg
|
||||||
|
assert "optics: 18" in msg
|
||||||
|
assert "aliases" in msg.lower() or "JSON" in msg
|
||||||
|
|
||||||
|
|
||||||
|
def test_build_user_message_empty():
|
||||||
|
msg = build_user_message({})
|
||||||
|
assert "Empty" in msg or "empty" in msg
|
||||||
|
|
||||||
|
|
||||||
|
# --- get_tag_distribution ---
|
||||||
|
|
||||||
|
|
||||||
|
def test_tag_distribution_counts_active_only(tmp_data_dir):
|
||||||
|
init_db()
|
||||||
|
create_memory("knowledge", "a", domain_tags=["firmware", "p06"])
|
||||||
|
create_memory("knowledge", "b", domain_tags=["firmware"])
|
||||||
|
create_memory("knowledge", "c", domain_tags=["optics"])
|
||||||
|
|
||||||
|
# Add an invalid memory — should NOT be counted
|
||||||
|
m_invalid = create_memory("knowledge", "d", domain_tags=["firmware", "ignored"])
|
||||||
|
with get_connection() as conn:
|
||||||
|
conn.execute("UPDATE memories SET status = 'invalid' WHERE id = ?", (m_invalid.id,))
|
||||||
|
|
||||||
|
dist = get_tag_distribution()
|
||||||
|
assert dist.get("firmware") == 2 # two active memories
|
||||||
|
assert dist.get("optics") == 1
|
||||||
|
assert dist.get("p06") == 1
|
||||||
|
assert "ignored" not in dist
|
||||||
|
|
||||||
|
|
||||||
|
def test_tag_distribution_min_count_filter(tmp_data_dir):
|
||||||
|
init_db()
|
||||||
|
create_memory("knowledge", "a", domain_tags=["firmware"])
|
||||||
|
create_memory("knowledge", "b", domain_tags=["firmware"])
|
||||||
|
create_memory("knowledge", "c", domain_tags=["once"])
|
||||||
|
|
||||||
|
dist = get_tag_distribution(min_count=2)
|
||||||
|
assert "firmware" in dist
|
||||||
|
assert "once" not in dist
|
||||||
|
|
||||||
|
|
||||||
|
# --- apply_tag_alias ---
|
||||||
|
|
||||||
|
|
||||||
|
def test_apply_tag_alias_rewrites_across_memories(tmp_data_dir):
|
||||||
|
init_db()
|
||||||
|
m1 = create_memory("knowledge", "a", domain_tags=["fw", "p06"])
|
||||||
|
m2 = create_memory("knowledge", "b", domain_tags=["fw"])
|
||||||
|
m3 = create_memory("knowledge", "c", domain_tags=["optics"]) # untouched
|
||||||
|
|
||||||
|
result = apply_tag_alias("fw", "firmware")
|
||||||
|
assert result["memories_touched"] == 2
|
||||||
|
|
||||||
|
import json as _json
|
||||||
|
with get_connection() as conn:
|
||||||
|
r1 = conn.execute("SELECT domain_tags FROM memories WHERE id = ?", (m1.id,)).fetchone()
|
||||||
|
r2 = conn.execute("SELECT domain_tags FROM memories WHERE id = ?", (m2.id,)).fetchone()
|
||||||
|
r3 = conn.execute("SELECT domain_tags FROM memories WHERE id = ?", (m3.id,)).fetchone()
|
||||||
|
assert "firmware" in _json.loads(r1["domain_tags"])
|
||||||
|
assert "fw" not in _json.loads(r1["domain_tags"])
|
||||||
|
assert "firmware" in _json.loads(r2["domain_tags"])
|
||||||
|
assert _json.loads(r3["domain_tags"]) == ["optics"] # untouched
|
||||||
|
|
||||||
|
|
||||||
|
def test_apply_tag_alias_dedupes_when_both_present(tmp_data_dir):
|
||||||
|
"""Memory has both fw AND firmware → rewrite collapses to just firmware."""
|
||||||
|
init_db()
|
||||||
|
m = create_memory("knowledge", "dual-tagged", domain_tags=["fw", "firmware", "p06"])
|
||||||
|
|
||||||
|
result = apply_tag_alias("fw", "firmware")
|
||||||
|
assert result["memories_touched"] == 1
|
||||||
|
|
||||||
|
import json as _json
|
||||||
|
with get_connection() as conn:
|
||||||
|
r = conn.execute("SELECT domain_tags FROM memories WHERE id = ?", (m.id,)).fetchone()
|
||||||
|
tags = _json.loads(r["domain_tags"])
|
||||||
|
assert tags.count("firmware") == 1
|
||||||
|
assert "fw" not in tags
|
||||||
|
assert "p06" in tags
|
||||||
|
|
||||||
|
|
||||||
|
def test_apply_tag_alias_skips_memories_without_alias(tmp_data_dir):
|
||||||
|
init_db()
|
||||||
|
m = create_memory("knowledge", "no match", domain_tags=["optics", "p04"])
|
||||||
|
result = apply_tag_alias("fw", "firmware")
|
||||||
|
assert result["memories_touched"] == 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_apply_tag_alias_writes_audit(tmp_data_dir):
|
||||||
|
init_db()
|
||||||
|
m = create_memory("knowledge", "audited", domain_tags=["fw"])
|
||||||
|
apply_tag_alias("fw", "firmware", actor="auto-tag-canon")
|
||||||
|
|
||||||
|
audit = get_memory_audit(m.id)
|
||||||
|
actions = [a["action"] for a in audit]
|
||||||
|
assert "tag_canonicalized" in actions
|
||||||
|
entry = next(a for a in audit if a["action"] == "tag_canonicalized")
|
||||||
|
assert entry["actor"] == "auto-tag-canon"
|
||||||
|
assert "fw → firmware" in entry["note"]
|
||||||
|
assert "fw" in entry["before"]["domain_tags"]
|
||||||
|
assert "firmware" in entry["after"]["domain_tags"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_apply_tag_alias_rejects_identity(tmp_data_dir):
|
||||||
|
init_db()
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
apply_tag_alias("foo", "foo")
|
||||||
|
|
||||||
|
|
||||||
|
def test_apply_tag_alias_rejects_empty(tmp_data_dir):
|
||||||
|
init_db()
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
apply_tag_alias("", "firmware")
|
||||||
|
|
||||||
|
|
||||||
|
# --- Proposal lifecycle ---
|
||||||
|
|
||||||
|
|
||||||
|
def test_create_proposal_inserts_pending(tmp_data_dir):
|
||||||
|
init_db()
|
||||||
|
pid = create_tag_alias_proposal("fw", "firmware", confidence=0.65,
|
||||||
|
alias_count=5, canonical_count=23,
|
||||||
|
reason="standard abbreviation")
|
||||||
|
assert pid is not None
|
||||||
|
|
||||||
|
rows = get_tag_alias_proposals(status="pending")
|
||||||
|
assert len(rows) == 1
|
||||||
|
assert rows[0]["alias"] == "fw"
|
||||||
|
assert rows[0]["confidence"] == pytest.approx(0.65)
|
||||||
|
|
||||||
|
|
||||||
|
def test_create_proposal_idempotent(tmp_data_dir):
|
||||||
|
init_db()
|
||||||
|
first = create_tag_alias_proposal("fw", "firmware", confidence=0.6)
|
||||||
|
second = create_tag_alias_proposal("fw", "firmware", confidence=0.7)
|
||||||
|
assert first is not None
|
||||||
|
assert second is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_approve_applies_rewrite(tmp_data_dir):
|
||||||
|
init_db()
|
||||||
|
m = create_memory("knowledge", "x", domain_tags=["fw"])
|
||||||
|
pid = create_tag_alias_proposal("fw", "firmware", confidence=0.7)
|
||||||
|
result = approve_tag_alias(pid, actor="human-triage")
|
||||||
|
assert result is not None
|
||||||
|
assert result["memories_touched"] == 1
|
||||||
|
|
||||||
|
# Proposal now approved with applied_to_memories recorded
|
||||||
|
rows = get_tag_alias_proposals(status="approved")
|
||||||
|
assert len(rows) == 1
|
||||||
|
assert rows[0]["applied_to_memories"] == 1
|
||||||
|
|
||||||
|
# Memory actually rewritten
|
||||||
|
import json as _json
|
||||||
|
with get_connection() as conn:
|
||||||
|
r = conn.execute("SELECT domain_tags FROM memories WHERE id = ?", (m.id,)).fetchone()
|
||||||
|
assert "firmware" in _json.loads(r["domain_tags"])
|
||||||
|
|
||||||
|
|
||||||
|
def test_approve_already_resolved_returns_none(tmp_data_dir):
|
||||||
|
init_db()
|
||||||
|
pid = create_tag_alias_proposal("a", "b", confidence=0.6)
|
||||||
|
approve_tag_alias(pid)
|
||||||
|
assert approve_tag_alias(pid) is None # second approve — no-op
|
||||||
|
|
||||||
|
|
||||||
|
def test_reject_leaves_memories_untouched(tmp_data_dir):
|
||||||
|
init_db()
|
||||||
|
m = create_memory("knowledge", "x", domain_tags=["fw"])
|
||||||
|
pid = create_tag_alias_proposal("fw", "firmware", confidence=0.6)
|
||||||
|
assert reject_tag_alias(pid)
|
||||||
|
|
||||||
|
rows = get_tag_alias_proposals(status="rejected")
|
||||||
|
assert len(rows) == 1
|
||||||
|
|
||||||
|
# Memory still has the original tag
|
||||||
|
import json as _json
|
||||||
|
with get_connection() as conn:
|
||||||
|
r = conn.execute("SELECT domain_tags FROM memories WHERE id = ?", (m.id,)).fetchone()
|
||||||
|
assert "fw" in _json.loads(r["domain_tags"])
|
||||||
|
|
||||||
|
|
||||||
|
# --- Schema sanity ---
|
||||||
|
|
||||||
|
|
||||||
|
def test_tag_aliases_table_exists(tmp_data_dir):
|
||||||
|
init_db()
|
||||||
|
with get_connection() as conn:
|
||||||
|
cols = [r["name"] for r in conn.execute("PRAGMA table_info(tag_aliases)").fetchall()]
|
||||||
|
expected = {"id", "alias", "canonical", "status", "confidence",
|
||||||
|
"alias_count", "canonical_count", "reason",
|
||||||
|
"applied_to_memories", "created_at", "resolved_at", "resolved_by"}
|
||||||
|
assert expected.issubset(set(cols))
|
||||||
398
tests/test_v1_0_write_invariants.py
Normal file
398
tests/test_v1_0_write_invariants.py
Normal file
@@ -0,0 +1,398 @@
|
|||||||
|
"""V1-0 write-time invariant tests.
|
||||||
|
|
||||||
|
Covers the Engineering V1 completion plan Phase V1-0 acceptance:
|
||||||
|
- F-1 shared-header fields: extractor_version + canonical_home + hand_authored
|
||||||
|
land in the entities table with working defaults
|
||||||
|
- F-8 provenance enforcement: create_entity raises without source_refs
|
||||||
|
unless hand_authored=True
|
||||||
|
- F-5 synchronous conflict-detection hook on any active-entity write
|
||||||
|
(create_entity with status="active" + the pre-existing promote_entity
|
||||||
|
path); fail-open per conflict-model.md:256
|
||||||
|
- Q-3 "flag, never block": a conflict never 4xx-blocks the write
|
||||||
|
- Q-4 partial trust: get_entities scope_only filters candidates out
|
||||||
|
|
||||||
|
Plan: docs/plans/engineering-v1-completion-plan.md
|
||||||
|
Spec: docs/architecture/engineering-v1-acceptance.md
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from atocore.engineering.service import (
|
||||||
|
EXTRACTOR_VERSION,
|
||||||
|
create_entity,
|
||||||
|
create_relationship,
|
||||||
|
get_entities,
|
||||||
|
get_entity,
|
||||||
|
init_engineering_schema,
|
||||||
|
promote_entity,
|
||||||
|
supersede_entity,
|
||||||
|
)
|
||||||
|
from atocore.models.database import get_connection, init_db
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- F-1: shared-header fields ----------
|
||||||
|
|
||||||
|
|
||||||
|
def test_entity_row_has_shared_header_fields(tmp_data_dir):
|
||||||
|
init_db()
|
||||||
|
init_engineering_schema()
|
||||||
|
with get_connection() as conn:
|
||||||
|
cols = {row["name"] for row in conn.execute("PRAGMA table_info(entities)").fetchall()}
|
||||||
|
assert "extractor_version" in cols
|
||||||
|
assert "canonical_home" in cols
|
||||||
|
assert "hand_authored" in cols
|
||||||
|
|
||||||
|
|
||||||
|
def test_created_entity_has_default_extractor_version_and_canonical_home(tmp_data_dir):
|
||||||
|
init_db()
|
||||||
|
init_engineering_schema()
|
||||||
|
e = create_entity(
|
||||||
|
entity_type="component",
|
||||||
|
name="Pivot Pin",
|
||||||
|
project="p04-gigabit",
|
||||||
|
source_refs=["test:fixture"],
|
||||||
|
)
|
||||||
|
assert e.extractor_version == EXTRACTOR_VERSION
|
||||||
|
assert e.canonical_home == "entity"
|
||||||
|
assert e.hand_authored is False
|
||||||
|
|
||||||
|
# round-trip through get_entity to confirm the row mapper returns
|
||||||
|
# the same values (not just the return-by-construct path)
|
||||||
|
got = get_entity(e.id)
|
||||||
|
assert got is not None
|
||||||
|
assert got.extractor_version == EXTRACTOR_VERSION
|
||||||
|
assert got.canonical_home == "entity"
|
||||||
|
assert got.hand_authored is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_explicit_extractor_version_is_persisted(tmp_data_dir):
|
||||||
|
init_db()
|
||||||
|
init_engineering_schema()
|
||||||
|
e = create_entity(
|
||||||
|
entity_type="decision",
|
||||||
|
name="Pick GF-PTFE pads",
|
||||||
|
project="p04-gigabit",
|
||||||
|
source_refs=["interaction:abc"],
|
||||||
|
extractor_version="custom-v2.3",
|
||||||
|
)
|
||||||
|
got = get_entity(e.id)
|
||||||
|
assert got.extractor_version == "custom-v2.3"
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- F-8: provenance enforcement ----------
|
||||||
|
|
||||||
|
|
||||||
|
def test_create_entity_without_provenance_raises(tmp_data_dir):
|
||||||
|
init_db()
|
||||||
|
init_engineering_schema()
|
||||||
|
with pytest.raises(ValueError, match="source_refs required"):
|
||||||
|
create_entity(
|
||||||
|
entity_type="component",
|
||||||
|
name="No Provenance",
|
||||||
|
project="p04-gigabit",
|
||||||
|
hand_authored=False, # explicit — bypasses the test-conftest auto-flag
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_create_entity_with_hand_authored_needs_no_source_refs(tmp_data_dir):
|
||||||
|
init_db()
|
||||||
|
init_engineering_schema()
|
||||||
|
e = create_entity(
|
||||||
|
entity_type="component",
|
||||||
|
name="Human Entry",
|
||||||
|
project="p04-gigabit",
|
||||||
|
hand_authored=True,
|
||||||
|
)
|
||||||
|
assert e.hand_authored is True
|
||||||
|
got = get_entity(e.id)
|
||||||
|
assert got.hand_authored is True
|
||||||
|
# source_refs stays empty — the hand_authored flag IS the provenance
|
||||||
|
assert got.source_refs == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_create_entity_with_empty_source_refs_list_is_treated_as_missing(tmp_data_dir):
|
||||||
|
init_db()
|
||||||
|
init_engineering_schema()
|
||||||
|
with pytest.raises(ValueError, match="source_refs required"):
|
||||||
|
create_entity(
|
||||||
|
entity_type="component",
|
||||||
|
name="Empty Refs",
|
||||||
|
project="p04-gigabit",
|
||||||
|
source_refs=[],
|
||||||
|
hand_authored=False,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_promote_rejects_legacy_candidate_without_provenance(tmp_data_dir):
|
||||||
|
"""Regression (Codex V1-0 probe): candidate rows can exist in the DB
|
||||||
|
from before V1-0 enforcement (or from paths that bypass create_entity).
|
||||||
|
promote_entity must re-check the invariant and refuse to flip a
|
||||||
|
no-provenance candidate to active. Without this check, the active
|
||||||
|
store can leak F-8 violations in from legacy data."""
|
||||||
|
init_db()
|
||||||
|
init_engineering_schema()
|
||||||
|
|
||||||
|
# Simulate a pre-V1-0 candidate by inserting directly into the table,
|
||||||
|
# bypassing the service-layer invariant. Real legacy rows look exactly
|
||||||
|
# like this: empty source_refs, hand_authored=0.
|
||||||
|
import uuid as _uuid
|
||||||
|
entity_id = str(_uuid.uuid4())
|
||||||
|
with get_connection() as conn:
|
||||||
|
conn.execute(
|
||||||
|
"INSERT INTO entities (id, entity_type, name, project, "
|
||||||
|
"description, properties, status, confidence, source_refs, "
|
||||||
|
"extractor_version, canonical_home, hand_authored, "
|
||||||
|
"created_at, updated_at) "
|
||||||
|
"VALUES (?, 'component', 'Legacy Orphan', 'p04-gigabit', "
|
||||||
|
"'', '{}', 'candidate', 1.0, '[]', '', 'entity', 0, "
|
||||||
|
"CURRENT_TIMESTAMP, CURRENT_TIMESTAMP)",
|
||||||
|
(entity_id,),
|
||||||
|
)
|
||||||
|
|
||||||
|
with pytest.raises(ValueError, match="source_refs required"):
|
||||||
|
promote_entity(entity_id)
|
||||||
|
|
||||||
|
# And the row stays a candidate — no half-transition.
|
||||||
|
got = get_entity(entity_id)
|
||||||
|
assert got is not None
|
||||||
|
assert got.status == "candidate"
|
||||||
|
|
||||||
|
|
||||||
|
def test_api_promote_returns_400_on_legacy_no_provenance(tmp_data_dir):
|
||||||
|
"""R14 (Codex, 2026-04-22): the HTTP promote route must translate
|
||||||
|
the V1-0 ValueError for no-provenance candidates into 400, not 500.
|
||||||
|
Previously the route didn't catch ValueError so legacy bad
|
||||||
|
candidates surfaced as a server error."""
|
||||||
|
init_db()
|
||||||
|
init_engineering_schema()
|
||||||
|
|
||||||
|
import uuid as _uuid
|
||||||
|
from fastapi.testclient import TestClient
|
||||||
|
from atocore.main import app
|
||||||
|
|
||||||
|
entity_id = str(_uuid.uuid4())
|
||||||
|
with get_connection() as conn:
|
||||||
|
conn.execute(
|
||||||
|
"INSERT INTO entities (id, entity_type, name, project, "
|
||||||
|
"description, properties, status, confidence, source_refs, "
|
||||||
|
"extractor_version, canonical_home, hand_authored, "
|
||||||
|
"created_at, updated_at) "
|
||||||
|
"VALUES (?, 'component', 'Legacy HTTP', 'p04-gigabit', "
|
||||||
|
"'', '{}', 'candidate', 1.0, '[]', '', 'entity', 0, "
|
||||||
|
"CURRENT_TIMESTAMP, CURRENT_TIMESTAMP)",
|
||||||
|
(entity_id,),
|
||||||
|
)
|
||||||
|
|
||||||
|
client = TestClient(app)
|
||||||
|
r = client.post(f"/entities/{entity_id}/promote")
|
||||||
|
assert r.status_code == 400
|
||||||
|
assert "source_refs required" in r.json().get("detail", "")
|
||||||
|
|
||||||
|
# Row still candidate — the 400 didn't half-transition.
|
||||||
|
got = get_entity(entity_id)
|
||||||
|
assert got is not None
|
||||||
|
assert got.status == "candidate"
|
||||||
|
|
||||||
|
|
||||||
|
def test_promote_accepts_candidate_flagged_hand_authored(tmp_data_dir):
|
||||||
|
"""The other side of the promote re-check: hand_authored=1 with
|
||||||
|
empty source_refs still lets promote succeed, matching
|
||||||
|
create_entity's symmetry."""
|
||||||
|
init_db()
|
||||||
|
init_engineering_schema()
|
||||||
|
|
||||||
|
import uuid as _uuid
|
||||||
|
entity_id = str(_uuid.uuid4())
|
||||||
|
with get_connection() as conn:
|
||||||
|
conn.execute(
|
||||||
|
"INSERT INTO entities (id, entity_type, name, project, "
|
||||||
|
"description, properties, status, confidence, source_refs, "
|
||||||
|
"extractor_version, canonical_home, hand_authored, "
|
||||||
|
"created_at, updated_at) "
|
||||||
|
"VALUES (?, 'component', 'Hand Authored Candidate', "
|
||||||
|
"'p04-gigabit', '', '{}', 'candidate', 1.0, '[]', '', "
|
||||||
|
"'entity', 1, CURRENT_TIMESTAMP, CURRENT_TIMESTAMP)",
|
||||||
|
(entity_id,),
|
||||||
|
)
|
||||||
|
|
||||||
|
assert promote_entity(entity_id) is True
|
||||||
|
assert get_entity(entity_id).status == "active"
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- F-5: synchronous conflict-detection hook ----------
|
||||||
|
|
||||||
|
|
||||||
|
def test_active_create_runs_conflict_detection_hook(tmp_data_dir, monkeypatch):
|
||||||
|
"""status=active writes trigger detect_conflicts_for_entity."""
|
||||||
|
init_db()
|
||||||
|
init_engineering_schema()
|
||||||
|
|
||||||
|
called_with: list[str] = []
|
||||||
|
|
||||||
|
def _fake_detect(entity_id: str):
|
||||||
|
called_with.append(entity_id)
|
||||||
|
return []
|
||||||
|
|
||||||
|
import atocore.engineering.conflicts as conflicts_mod
|
||||||
|
monkeypatch.setattr(conflicts_mod, "detect_conflicts_for_entity", _fake_detect)
|
||||||
|
|
||||||
|
e = create_entity(
|
||||||
|
entity_type="component",
|
||||||
|
name="Active With Hook",
|
||||||
|
project="p04-gigabit",
|
||||||
|
source_refs=["test:hook"],
|
||||||
|
status="active",
|
||||||
|
)
|
||||||
|
|
||||||
|
assert called_with == [e.id]
|
||||||
|
|
||||||
|
|
||||||
|
def test_supersede_runs_conflict_detection_on_new_active(tmp_data_dir, monkeypatch):
|
||||||
|
"""Regression (Codex V1-0 probe): per plan's 'every active-entity
|
||||||
|
write path', supersede_entity must trigger synchronous conflict
|
||||||
|
detection. The subject is the `superseded_by` entity — the one
|
||||||
|
whose graph state just changed because a new `supersedes` edge was
|
||||||
|
rooted at it."""
|
||||||
|
init_db()
|
||||||
|
init_engineering_schema()
|
||||||
|
|
||||||
|
old = create_entity(
|
||||||
|
entity_type="component",
|
||||||
|
name="Old Pad",
|
||||||
|
project="p04-gigabit",
|
||||||
|
source_refs=["test:old"],
|
||||||
|
status="active",
|
||||||
|
)
|
||||||
|
new = create_entity(
|
||||||
|
entity_type="component",
|
||||||
|
name="New Pad",
|
||||||
|
project="p04-gigabit",
|
||||||
|
source_refs=["test:new"],
|
||||||
|
status="active",
|
||||||
|
)
|
||||||
|
|
||||||
|
called_with: list[str] = []
|
||||||
|
|
||||||
|
def _fake_detect(entity_id: str):
|
||||||
|
called_with.append(entity_id)
|
||||||
|
return []
|
||||||
|
|
||||||
|
import atocore.engineering.conflicts as conflicts_mod
|
||||||
|
monkeypatch.setattr(conflicts_mod, "detect_conflicts_for_entity", _fake_detect)
|
||||||
|
|
||||||
|
assert supersede_entity(old.id, superseded_by=new.id) is True
|
||||||
|
|
||||||
|
# The detector fires on the `superseded_by` entity — the one whose
|
||||||
|
# edges just grew a new `supersedes` relationship.
|
||||||
|
assert new.id in called_with
|
||||||
|
|
||||||
|
|
||||||
|
def test_supersede_hook_fails_open(tmp_data_dir, monkeypatch):
|
||||||
|
"""Supersede must survive a broken detector per Q-3 flag-never-block."""
|
||||||
|
init_db()
|
||||||
|
init_engineering_schema()
|
||||||
|
|
||||||
|
old = create_entity(
|
||||||
|
entity_type="component", name="Old2", project="p04-gigabit",
|
||||||
|
source_refs=["test:old"], status="active",
|
||||||
|
)
|
||||||
|
new = create_entity(
|
||||||
|
entity_type="component", name="New2", project="p04-gigabit",
|
||||||
|
source_refs=["test:new"], status="active",
|
||||||
|
)
|
||||||
|
|
||||||
|
def _boom(entity_id: str):
|
||||||
|
raise RuntimeError("synthetic detector failure")
|
||||||
|
|
||||||
|
import atocore.engineering.conflicts as conflicts_mod
|
||||||
|
monkeypatch.setattr(conflicts_mod, "detect_conflicts_for_entity", _boom)
|
||||||
|
|
||||||
|
# The supersede still succeeds despite the detector blowing up.
|
||||||
|
assert supersede_entity(old.id, superseded_by=new.id) is True
|
||||||
|
assert get_entity(old.id).status == "superseded"
|
||||||
|
|
||||||
|
|
||||||
|
def test_candidate_create_does_not_run_conflict_hook(tmp_data_dir, monkeypatch):
|
||||||
|
"""status=candidate writes do NOT trigger detection — the hook is
|
||||||
|
for active rows only, per V1-0 scope. Candidates are checked at
|
||||||
|
promote time."""
|
||||||
|
init_db()
|
||||||
|
init_engineering_schema()
|
||||||
|
|
||||||
|
called: list[str] = []
|
||||||
|
|
||||||
|
def _fake_detect(entity_id: str):
|
||||||
|
called.append(entity_id)
|
||||||
|
return []
|
||||||
|
|
||||||
|
import atocore.engineering.conflicts as conflicts_mod
|
||||||
|
monkeypatch.setattr(conflicts_mod, "detect_conflicts_for_entity", _fake_detect)
|
||||||
|
|
||||||
|
create_entity(
|
||||||
|
entity_type="component",
|
||||||
|
name="Candidate No Hook",
|
||||||
|
project="p04-gigabit",
|
||||||
|
source_refs=["test:cand"],
|
||||||
|
status="candidate",
|
||||||
|
)
|
||||||
|
|
||||||
|
assert called == []
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- Q-3: flag, never block ----------
|
||||||
|
|
||||||
|
|
||||||
|
def test_conflict_detector_failure_does_not_block_write(tmp_data_dir, monkeypatch):
|
||||||
|
"""Per conflict-model.md:256: detection errors must not fail the
|
||||||
|
write. The entity is still created; only a warning is logged."""
|
||||||
|
init_db()
|
||||||
|
init_engineering_schema()
|
||||||
|
|
||||||
|
def _boom(entity_id: str):
|
||||||
|
raise RuntimeError("synthetic detector failure")
|
||||||
|
|
||||||
|
import atocore.engineering.conflicts as conflicts_mod
|
||||||
|
monkeypatch.setattr(conflicts_mod, "detect_conflicts_for_entity", _boom)
|
||||||
|
|
||||||
|
# The write still succeeds — no exception propagates.
|
||||||
|
e = create_entity(
|
||||||
|
entity_type="component",
|
||||||
|
name="Hook Fails Open",
|
||||||
|
project="p04-gigabit",
|
||||||
|
source_refs=["test:failopen"],
|
||||||
|
status="active",
|
||||||
|
)
|
||||||
|
assert get_entity(e.id) is not None
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- Q-4 (partial): trust-hierarchy — scope_only filters candidates ----------
|
||||||
|
|
||||||
|
|
||||||
|
def test_scope_only_active_does_not_return_candidates(tmp_data_dir):
|
||||||
|
"""V1-0 partial Q-4: active-scoped listing never returns candidates.
|
||||||
|
Full trust-hierarchy coverage (no-auto-project-state, etc.) ships in
|
||||||
|
V1-E per plan."""
|
||||||
|
init_db()
|
||||||
|
init_engineering_schema()
|
||||||
|
|
||||||
|
active = create_entity(
|
||||||
|
entity_type="component",
|
||||||
|
name="Active Alpha",
|
||||||
|
project="p04-gigabit",
|
||||||
|
source_refs=["test:alpha"],
|
||||||
|
status="active",
|
||||||
|
)
|
||||||
|
candidate = create_entity(
|
||||||
|
entity_type="component",
|
||||||
|
name="Candidate Beta",
|
||||||
|
project="p04-gigabit",
|
||||||
|
source_refs=["test:beta"],
|
||||||
|
status="candidate",
|
||||||
|
)
|
||||||
|
|
||||||
|
listed = get_entities(project="p04-gigabit", status="active", scope_only=True)
|
||||||
|
ids = {e.id for e in listed}
|
||||||
|
assert active.id in ids
|
||||||
|
assert candidate.id not in ids
|
||||||
61
tests/test_v1_aliases.py
Normal file
61
tests/test_v1_aliases.py
Normal file
@@ -0,0 +1,61 @@
|
|||||||
|
"""Tests for /v1 API aliases — stable contract for external clients."""
|
||||||
|
|
||||||
|
from fastapi.testclient import TestClient
|
||||||
|
|
||||||
|
from atocore.main import app
|
||||||
|
|
||||||
|
|
||||||
|
def test_v1_health_alias_matches_unversioned():
|
||||||
|
client = TestClient(app)
|
||||||
|
unversioned = client.get("/health")
|
||||||
|
versioned = client.get("/v1/health")
|
||||||
|
assert unversioned.status_code == 200
|
||||||
|
assert versioned.status_code == 200
|
||||||
|
assert versioned.json()["build_sha"] == unversioned.json()["build_sha"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_v1_projects_alias_returns_same_shape():
|
||||||
|
client = TestClient(app)
|
||||||
|
unversioned = client.get("/projects")
|
||||||
|
versioned = client.get("/v1/projects")
|
||||||
|
assert unversioned.status_code == 200
|
||||||
|
assert versioned.status_code == 200
|
||||||
|
assert versioned.json() == unversioned.json()
|
||||||
|
|
||||||
|
|
||||||
|
def test_v1_entities_get_alias_reachable(tmp_data_dir):
|
||||||
|
from atocore.engineering.service import init_engineering_schema
|
||||||
|
|
||||||
|
init_engineering_schema()
|
||||||
|
client = TestClient(app)
|
||||||
|
response = client.get("/v1/entities")
|
||||||
|
assert response.status_code == 200
|
||||||
|
body = response.json()
|
||||||
|
assert "entities" in body
|
||||||
|
|
||||||
|
|
||||||
|
def test_v1_paths_appear_in_openapi():
|
||||||
|
client = TestClient(app)
|
||||||
|
spec = client.get("/openapi.json").json()
|
||||||
|
paths = spec["paths"]
|
||||||
|
expected = [
|
||||||
|
"/v1/entities",
|
||||||
|
"/v1/relationships",
|
||||||
|
"/v1/ingest",
|
||||||
|
"/v1/context/build",
|
||||||
|
"/v1/projects",
|
||||||
|
"/v1/memory",
|
||||||
|
"/v1/projects/{project_name}/mirror",
|
||||||
|
"/v1/health",
|
||||||
|
]
|
||||||
|
for path in expected:
|
||||||
|
assert path in paths, f"{path} missing from OpenAPI spec"
|
||||||
|
|
||||||
|
|
||||||
|
def test_unversioned_paths_still_present_in_openapi():
|
||||||
|
"""Regression: /v1 aliases must not displace the original paths."""
|
||||||
|
client = TestClient(app)
|
||||||
|
spec = client.get("/openapi.json").json()
|
||||||
|
paths = spec["paths"]
|
||||||
|
for path in ("/entities", "/projects", "/health", "/context/build"):
|
||||||
|
assert path in paths, f"unversioned {path} missing — aliases must coexist"
|
||||||
163
tests/test_wiki_pages.py
Normal file
163
tests/test_wiki_pages.py
Normal file
@@ -0,0 +1,163 @@
|
|||||||
|
"""Tests for the new wiki pages shipped in the UI refresh:
|
||||||
|
- /wiki/capture (7I follow-up)
|
||||||
|
- /wiki/memories/{id} (7E)
|
||||||
|
- /wiki/domains/{tag} (7F)
|
||||||
|
- /wiki/activity (activity feed)
|
||||||
|
- home refresh (topnav + activity snippet)
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from atocore.engineering.wiki import (
|
||||||
|
render_activity,
|
||||||
|
render_capture,
|
||||||
|
render_domain,
|
||||||
|
render_homepage,
|
||||||
|
render_memory_detail,
|
||||||
|
)
|
||||||
|
from atocore.engineering.service import init_engineering_schema
|
||||||
|
from atocore.memory.service import create_memory
|
||||||
|
from atocore.models.database import init_db
|
||||||
|
|
||||||
|
|
||||||
|
def _init_all():
|
||||||
|
"""Wiki pages read from both the memory and engineering schemas, so
|
||||||
|
tests need both initialized (the engineering schema is a separate
|
||||||
|
init_engineering_schema() call)."""
|
||||||
|
init_db()
|
||||||
|
init_engineering_schema()
|
||||||
|
|
||||||
|
|
||||||
|
def test_capture_page_renders_as_fallback(tmp_data_dir):
|
||||||
|
_init_all()
|
||||||
|
html = render_capture()
|
||||||
|
# Page is reachable but now labeled as a fallback, not promoted
|
||||||
|
assert "fallback only" in html
|
||||||
|
assert "sanctioned capture surfaces are Claude Code" in html
|
||||||
|
# Form inputs still exist for emergency use
|
||||||
|
assert "cap-prompt" in html
|
||||||
|
assert "cap-response" in html
|
||||||
|
|
||||||
|
|
||||||
|
def test_capture_not_in_topnav(tmp_data_dir):
|
||||||
|
"""The paste form should NOT appear in topnav — it's not the sanctioned path."""
|
||||||
|
_init_all()
|
||||||
|
html = render_homepage()
|
||||||
|
assert "/wiki/capture" not in html
|
||||||
|
assert "📥 Capture" not in html
|
||||||
|
|
||||||
|
|
||||||
|
def test_memory_detail_renders(tmp_data_dir):
|
||||||
|
_init_all()
|
||||||
|
m = create_memory(
|
||||||
|
"knowledge", "APM uses NX bridge for DXF → STL",
|
||||||
|
project="apm", confidence=0.7, domain_tags=["apm", "nx", "cad"],
|
||||||
|
)
|
||||||
|
html = render_memory_detail(m.id)
|
||||||
|
assert html is not None
|
||||||
|
assert "APM uses NX" in html
|
||||||
|
assert "Audit trail" in html
|
||||||
|
# Tag links go to domain pages
|
||||||
|
assert '/wiki/domains/apm' in html
|
||||||
|
assert '/wiki/domains/nx' in html
|
||||||
|
# Project link present
|
||||||
|
assert '/wiki/projects/apm' in html
|
||||||
|
|
||||||
|
|
||||||
|
def test_memory_detail_404(tmp_data_dir):
|
||||||
|
_init_all()
|
||||||
|
assert render_memory_detail("nonexistent-id") is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_domain_page_lists_memories(tmp_data_dir):
|
||||||
|
_init_all()
|
||||||
|
create_memory("knowledge", "optics fact 1", project="p04-gigabit",
|
||||||
|
domain_tags=["optics"])
|
||||||
|
create_memory("knowledge", "optics fact 2", project="p05-interferometer",
|
||||||
|
domain_tags=["optics", "metrology"])
|
||||||
|
create_memory("knowledge", "other", project="p06-polisher",
|
||||||
|
domain_tags=["firmware"])
|
||||||
|
|
||||||
|
html = render_domain("optics")
|
||||||
|
assert "Domain: <code>optics</code>" in html
|
||||||
|
assert "p04-gigabit" in html
|
||||||
|
assert "p05-interferometer" in html
|
||||||
|
assert "optics fact 1" in html
|
||||||
|
assert "optics fact 2" in html
|
||||||
|
# Unrelated memory should NOT appear
|
||||||
|
assert "other" not in html or "firmware" not in html
|
||||||
|
|
||||||
|
|
||||||
|
def test_domain_page_empty(tmp_data_dir):
|
||||||
|
_init_all()
|
||||||
|
html = render_domain("definitely-not-a-tag")
|
||||||
|
assert "No memories currently carry" in html
|
||||||
|
|
||||||
|
|
||||||
|
def test_domain_page_normalizes_tag(tmp_data_dir):
|
||||||
|
_init_all()
|
||||||
|
create_memory("knowledge", "x", domain_tags=["firmware"])
|
||||||
|
# Case-insensitive
|
||||||
|
assert "firmware" in render_domain("FIRMWARE")
|
||||||
|
# Whitespace tolerant
|
||||||
|
assert "firmware" in render_domain(" firmware ")
|
||||||
|
|
||||||
|
|
||||||
|
def test_activity_feed_renders(tmp_data_dir):
|
||||||
|
_init_all()
|
||||||
|
m = create_memory("knowledge", "activity test")
|
||||||
|
html = render_activity()
|
||||||
|
assert "Activity Feed" in html
|
||||||
|
# The newly-created memory should appear as a "created" event
|
||||||
|
assert "created" in html
|
||||||
|
# Short timestamp format
|
||||||
|
assert m.id[:8] in html
|
||||||
|
|
||||||
|
|
||||||
|
def test_activity_feed_groups_by_action_and_actor(tmp_data_dir):
|
||||||
|
_init_all()
|
||||||
|
for i in range(3):
|
||||||
|
create_memory("knowledge", f"m{i}", actor="test-actor")
|
||||||
|
|
||||||
|
html = render_activity()
|
||||||
|
# Summary row should show "created: 3" or similar
|
||||||
|
assert "created" in html
|
||||||
|
assert "test-actor" in html
|
||||||
|
|
||||||
|
|
||||||
|
def test_homepage_has_topnav_and_activity(tmp_data_dir):
|
||||||
|
_init_all()
|
||||||
|
create_memory("knowledge", "homepage test")
|
||||||
|
html = render_homepage()
|
||||||
|
# Topnav with expected items (Capture removed — it's not sanctioned capture)
|
||||||
|
assert "🏠 Home" in html
|
||||||
|
assert "📡 Activity" in html
|
||||||
|
assert "/wiki/activity" in html
|
||||||
|
assert "/wiki/capture" not in html
|
||||||
|
# Activity snippet
|
||||||
|
assert "What the brain is doing" in html
|
||||||
|
|
||||||
|
|
||||||
|
def test_memory_detail_shows_superseded_sources(tmp_data_dir):
|
||||||
|
"""After a merge, sources go to status=superseded. Detail page should
|
||||||
|
still render them."""
|
||||||
|
from atocore.memory.service import (
|
||||||
|
create_merge_candidate, merge_memories,
|
||||||
|
)
|
||||||
|
_init_all()
|
||||||
|
m1 = create_memory("knowledge", "alpha variant 1", project="test")
|
||||||
|
m2 = create_memory("knowledge", "alpha variant 2", project="test")
|
||||||
|
cid = create_merge_candidate(
|
||||||
|
memory_ids=[m1.id, m2.id], similarity=0.9,
|
||||||
|
proposed_content="alpha merged",
|
||||||
|
proposed_memory_type="knowledge", proposed_project="test",
|
||||||
|
)
|
||||||
|
merge_memories(cid, actor="auto-dedup-tier1")
|
||||||
|
|
||||||
|
# Source detail page should render and show the superseded status
|
||||||
|
html1 = render_memory_detail(m1.id)
|
||||||
|
assert html1 is not None
|
||||||
|
assert "superseded" in html1
|
||||||
|
assert "auto-dedup-tier1" in html1 # audit trail shows who merged
|
||||||
132
tests/test_wikilinks.py
Normal file
132
tests/test_wikilinks.py
Normal file
@@ -0,0 +1,132 @@
|
|||||||
|
"""Issue B — wikilinks with redlinks + cross-project resolution."""
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from fastapi.testclient import TestClient
|
||||||
|
|
||||||
|
from atocore.engineering.service import (
|
||||||
|
create_entity,
|
||||||
|
init_engineering_schema,
|
||||||
|
)
|
||||||
|
from atocore.engineering.wiki import (
|
||||||
|
_resolve_wikilink,
|
||||||
|
_wikilink_transform,
|
||||||
|
render_entity,
|
||||||
|
render_new_entity_form,
|
||||||
|
render_project,
|
||||||
|
)
|
||||||
|
from atocore.main import app
|
||||||
|
from atocore.models.database import init_db
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def env(tmp_data_dir, tmp_path, monkeypatch):
|
||||||
|
registry_path = tmp_path / "test-registry.json"
|
||||||
|
registry_path.write_text('{"projects": []}', encoding="utf-8")
|
||||||
|
monkeypatch.setenv("ATOCORE_PROJECT_REGISTRY_PATH", str(registry_path))
|
||||||
|
from atocore import config
|
||||||
|
config.settings = config.Settings()
|
||||||
|
init_db()
|
||||||
|
init_engineering_schema()
|
||||||
|
yield tmp_data_dir
|
||||||
|
|
||||||
|
|
||||||
|
def test_resolve_wikilink_same_project_is_live(env):
|
||||||
|
tower = create_entity(entity_type="component", name="Tower", project="p05")
|
||||||
|
href, cls, _ = _resolve_wikilink("Tower", current_project="p05")
|
||||||
|
assert href == f"/wiki/entities/{tower.id}"
|
||||||
|
assert cls == "wikilink"
|
||||||
|
|
||||||
|
|
||||||
|
def test_resolve_wikilink_missing_is_redlink(env):
|
||||||
|
href, cls, suffix = _resolve_wikilink("DoesNotExist", current_project="p05")
|
||||||
|
assert "/wiki/new" in href
|
||||||
|
assert "name=DoesNotExist" in href
|
||||||
|
assert cls == "redlink"
|
||||||
|
|
||||||
|
|
||||||
|
def test_resolve_wikilink_cross_project_indicator(env):
|
||||||
|
other = create_entity(entity_type="material", name="Invar", project="p06")
|
||||||
|
href, cls, suffix = _resolve_wikilink("Invar", current_project="p05")
|
||||||
|
assert href == f"/wiki/entities/{other.id}"
|
||||||
|
assert "wikilink-cross" in cls
|
||||||
|
assert "in p06" in suffix
|
||||||
|
|
||||||
|
|
||||||
|
def test_resolve_wikilink_case_insensitive(env):
|
||||||
|
tower = create_entity(entity_type="component", name="Tower", project="p05")
|
||||||
|
href, cls, _ = _resolve_wikilink("tower", current_project="p05")
|
||||||
|
assert href == f"/wiki/entities/{tower.id}"
|
||||||
|
assert cls == "wikilink"
|
||||||
|
|
||||||
|
|
||||||
|
def test_transform_replaces_brackets_with_anchor(env):
|
||||||
|
create_entity(entity_type="component", name="Base Plate", project="p05")
|
||||||
|
out = _wikilink_transform("See [[Base Plate]] for details.", current_project="p05")
|
||||||
|
assert '<a href="/wiki/entities/' in out
|
||||||
|
assert 'class="wikilink"' in out
|
||||||
|
assert "[[Base Plate]]" not in out
|
||||||
|
|
||||||
|
|
||||||
|
def test_transform_redlink_for_missing(env):
|
||||||
|
out = _wikilink_transform("Mentions [[Ghost]] nowhere.", current_project="p05")
|
||||||
|
assert 'class="redlink"' in out
|
||||||
|
assert "/wiki/new?name=Ghost" in out
|
||||||
|
|
||||||
|
|
||||||
|
def test_transform_alias_syntax(env):
|
||||||
|
tower = create_entity(entity_type="component", name="Tower", project="p05")
|
||||||
|
out = _wikilink_transform("The [[Tower|big tower]] is tall.", current_project="p05")
|
||||||
|
assert f'href="/wiki/entities/{tower.id}"' in out
|
||||||
|
assert ">big tower<" in out
|
||||||
|
|
||||||
|
|
||||||
|
def test_render_entity_description_has_redlink(env):
|
||||||
|
a = create_entity(
|
||||||
|
entity_type="component",
|
||||||
|
name="EntityA",
|
||||||
|
project="p05",
|
||||||
|
description="This depends on [[MissingPart]] which does not exist.",
|
||||||
|
)
|
||||||
|
html = render_entity(a.id)
|
||||||
|
assert 'class="redlink"' in html
|
||||||
|
assert "/wiki/new?name=MissingPart" in html
|
||||||
|
|
||||||
|
|
||||||
|
def test_regression_redlink_becomes_live_once_target_created(env):
|
||||||
|
a = create_entity(
|
||||||
|
entity_type="component",
|
||||||
|
name="EntityA",
|
||||||
|
project="p05",
|
||||||
|
description="Connected to [[EntityB]].",
|
||||||
|
)
|
||||||
|
# Pre-create: redlink.
|
||||||
|
html_before = render_entity(a.id)
|
||||||
|
assert 'class="redlink"' in html_before
|
||||||
|
|
||||||
|
b = create_entity(entity_type="component", name="EntityB", project="p05")
|
||||||
|
html_after = render_entity(a.id)
|
||||||
|
assert 'class="redlink"' not in html_after
|
||||||
|
assert f"/wiki/entities/{b.id}" in html_after
|
||||||
|
|
||||||
|
|
||||||
|
def test_new_entity_form_prefills_name():
|
||||||
|
html = render_new_entity_form(name="FreshEntity", project="p05")
|
||||||
|
assert 'value="FreshEntity"' in html
|
||||||
|
assert 'value="p05"' in html
|
||||||
|
assert "entity_type" in html
|
||||||
|
assert 'method="post"' not in html # JS-driven
|
||||||
|
|
||||||
|
|
||||||
|
def test_wiki_new_route_renders(env):
|
||||||
|
client = TestClient(app)
|
||||||
|
r = client.get("/wiki/new?name=NewThing&project=p05")
|
||||||
|
assert r.status_code == 200
|
||||||
|
assert "NewThing" in r.text
|
||||||
|
assert "Create entity" in r.text
|
||||||
|
|
||||||
|
|
||||||
|
def test_wiki_new_url_escapes_special_chars(env):
|
||||||
|
# "steel (likely)" is the kind of awkward name AKC produces
|
||||||
|
href, cls, _ = _resolve_wikilink("steel (likely)", current_project="p05")
|
||||||
|
assert cls == "redlink"
|
||||||
|
assert "name=steel%20%28likely%29" in href
|
||||||
Reference in New Issue
Block a user