Compare commits
6 Commits
akc-wiki-h
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
| 4ca81e9b36 | |||
| 22a37a7241 | |||
| 2712c5d2d0 | |||
| 9ab5b3c9d8 | |||
| 44724c81ab | |||
| ce3a87857e |
@@ -6,10 +6,10 @@
|
|||||||
|
|
||||||
## Orientation
|
## Orientation
|
||||||
|
|
||||||
- **live_sha** (Dalidou `/health` build_sha): `775960c` (verified 2026-04-16 via /health, build_time 2026-04-16T17:59:30Z)
|
- **live_sha** (Dalidou `/health` build_sha): `2712c5d` (verified 2026-04-22 via Codex after V1-0 deploy; status=ok)
|
||||||
- **last_updated**: 2026-04-18 by Claude (Phase 7A — Memory Consolidation "sleep cycle" V1 on branch, not yet deployed)
|
- **last_updated**: 2026-04-22 by Claude (V1-0 write-time invariants landed + deployed + prod backfill complete)
|
||||||
- **main_tip**: `999788b`
|
- **main_tip**: `2712c5d`
|
||||||
- **test_count**: 533 (prior 521 + 12 new wikilink/redlink tests)
|
- **test_count**: 547 (prior 533 + 10 V1-0 invariant tests + 4 Codex-review regression tests)
|
||||||
- **harness**: `17/18 PASS` on live Dalidou (p04-constraints expects "Zerodur" — retrieval content gap, not regression)
|
- **harness**: `17/18 PASS` on live Dalidou (p04-constraints expects "Zerodur" — retrieval content gap, not regression)
|
||||||
- **vectors**: 33,253
|
- **vectors**: 33,253
|
||||||
- **active_memories**: 84 (31 project, 23 knowledge, 10 episodic, 8 adaptation, 7 preference, 5 identity)
|
- **active_memories**: 84 (31 project, 23 knowledge, 10 episodic, 8 adaptation, 7 preference, 5 identity)
|
||||||
@@ -23,6 +23,8 @@
|
|||||||
- **capture_clients**: claude-code (Stop hook + cwd project inference), openclaw (before_agent_start + llm_output plugin, verified live)
|
- **capture_clients**: claude-code (Stop hook + cwd project inference), openclaw (before_agent_start + llm_output plugin, verified live)
|
||||||
- **wiki**: http://dalidou:8100/wiki (browse), /wiki/projects/{id}, /wiki/entities/{id}, /wiki/search
|
- **wiki**: http://dalidou:8100/wiki (browse), /wiki/projects/{id}, /wiki/entities/{id}, /wiki/search
|
||||||
- **dashboard**: http://dalidou:8100/admin/dashboard (now shows pipeline health, interaction totals by client, all registered projects)
|
- **dashboard**: http://dalidou:8100/admin/dashboard (now shows pipeline health, interaction totals by client, all registered projects)
|
||||||
|
- **active_track**: Engineering V1 Completion (started 2026-04-22). V1-0 landed (`2712c5d`), V1-A gated on soak (~2026-04-26) + density (100+ active memories, currently 84). Plan: `docs/plans/engineering-v1-completion-plan.md`. Resume map: `docs/plans/v1-resume-state.md`.
|
||||||
|
- **open_branches**: `claude/r14-promote-400` at `3888db9` — P2 route fix (ValueError → 400), pending Codex review
|
||||||
|
|
||||||
## Active Plan
|
## Active Plan
|
||||||
|
|
||||||
@@ -143,9 +145,15 @@ One branch `codex/extractor-eval-loop` for Day 1-5, a second `codex/retrieval-ha
|
|||||||
| R11 | Codex | P2 | src/atocore/api/routes.py:773-845 | `POST /admin/extract-batch` still accepts `mode="llm"` inside the container and returns a successful 0-candidate result instead of surfacing that host-only LLM extraction is unavailable from this runtime. That is a misleading API contract for operators. | fixed | Claude | 2026-04-12 | (pending) |
|
| R11 | Codex | P2 | src/atocore/api/routes.py:773-845 | `POST /admin/extract-batch` still accepts `mode="llm"` inside the container and returns a successful 0-candidate result instead of surfacing that host-only LLM extraction is unavailable from this runtime. That is a misleading API contract for operators. | fixed | Claude | 2026-04-12 | (pending) |
|
||||||
| R12 | Codex | P2 | scripts/batch_llm_extract_live.py:39-190 | The host-side extractor duplicates the LLM system prompt and JSON parsing logic from `src/atocore/memory/extractor_llm.py`. It works today, but this is now a prompt/parser drift risk across the container and host implementations. | fixed | Claude | 2026-04-12 | (pending) |
|
| R12 | Codex | P2 | scripts/batch_llm_extract_live.py:39-190 | The host-side extractor duplicates the LLM system prompt and JSON parsing logic from `src/atocore/memory/extractor_llm.py`. It works today, but this is now a prompt/parser drift risk across the container and host implementations. | fixed | Claude | 2026-04-12 | (pending) |
|
||||||
| R13 | Codex | P2 | DEV-LEDGER.md:12 | The new `286 passing` test-count claim is not reproducibly auditable from the current audit environments: neither Dalidou nor the clean worktree has `pytest` available. The claim may be true in Claude's dev shell, but it remains unverified in this audit. | fixed | Claude | 2026-04-12 | (pending) |
|
| R13 | Codex | P2 | DEV-LEDGER.md:12 | The new `286 passing` test-count claim is not reproducibly auditable from the current audit environments: neither Dalidou nor the clean worktree has `pytest` available. The claim may be true in Claude's dev shell, but it remains unverified in this audit. | fixed | Claude | 2026-04-12 | (pending) |
|
||||||
|
| R14 | Codex | P2 | src/atocore/api/routes.py (POST /entities/{id}/promote) | The HTTP `POST /entities/{id}/promote` route does not translate the new service-layer `ValueError("source_refs required: cannot promote a candidate with no provenance...")` into a 400. A legacy no-provenance candidate promoted through the API currently surfaces as a 500. Does not block V1-0 acceptance; tidy in a follow-up. | open | Claude | 2026-04-22 | |
|
||||||
|
|
||||||
## Recent Decisions
|
## Recent Decisions
|
||||||
|
|
||||||
|
- **2026-04-22** **V1-0 done: approved, merged, deployed, prod backfilled.** Codex pulled `f16cd52`, re-ran the two original probes (both pass), re-ran the three targeted regression suites (all pass). Squash-merged to main as `2712c5d`. Dalidou deployed via canonical deploy script; `/health` reports build_sha=`2712c5d2d03cb2a6af38b559664afd1c4cd0e050`, status=ok. Validated backup snapshot taken at `/srv/storage/atocore/backups/snapshots/20260422T190624Z` before backfill. Prod backfill: `--dry-run` found 31 active/superseded entities with no provenance; list reviewed and sane; live run updated 31 rows via the default `hand_authored=1` flag path; follow-up dry-run returned 0 rows remaining. Residual logged as R14 (P2): `POST /entities/{id}/promote` HTTP route doesn't translate the new service-layer `ValueError` into a 400 — legacy bad candidate promotes via the API return 500 instead. Does not block V1-0 acceptance. V1-0 closed. Next: V1-A (Q-001 subsystem-scoped variant + Q-6 integration). V1-A holds until the soak window ends ~2026-04-26 and the 100-memory density target is hit. *Approved + landed by:* Codex. *Ratified by:* Antoine.
|
||||||
|
- **2026-04-22** **Engineering V1 Completion Plan — Codex sign-off (third round)**. Codex's third-round audit closed the remaining five open questions with concrete resolutions, patched inline in `docs/plans/engineering-v1-completion-plan.md`: (1) F-7 row rewritten with ground truth — schema + preserve-original + test coverage already exist (`graduated_to_entity_id` at `database.py:143-146`, `graduated` status in memory service, promote hook at `service.py:354-356,389-451`, tests at `test_engineering_v1_phase5.py:67-90`); **real gaps** are missing direct `POST /memory/{id}/graduate` route and spec's `knowledge→Fact` mismatch (no `fact` entity type exists; reconcile to `parameter` or similar); V1-E 2 → **3–4 days**; (2) Q-5 determinism reframed — don't stabilize the call to `datetime.now()`, inject regenerated timestamp + checksum as renderer inputs, remove DB iteration ordering dependencies; V1-D scope updated; (3) `project` vs `project_id` — doc note only, no rename, resolved; (4) total estimate 16.5–17.5 → **17.5–19.5 focused days** with calendar buffer on top; (5) "Minions" must not be canonized in D-3 release notes — neutral wording ("queued background processing / async workers") only. **Agreement reached**: Claude + Codex + Antoine aligned. V1-0 is ready to start once the current pipeline soak window ends (~2026-04-26) and the 100-memory density target is hit. *Patched by:* Codex. *Signed off by:* Codex ("with those edits, I'd sign off on the five questions"). *Accepted by:* Antoine. *Executor (V1-0 onwards):* Claude.
|
||||||
|
- **2026-04-22** **Engineering V1 Completion Plan revised per Codex second-round file-level audit** — three findings folded in, all with exact file:line refs from Codex: (1) F-1 downgraded from ✅ to 🟡 — `extractor_version` and `canonical_home` missing from `Entity` dataclass and `entities` table per `engineering-v1-acceptance.md:45`; V1-0 scope now adds both fields via additive migration + doc note that `project` IS `project_id` per "fields equivalent to" spec wording; (2) F-2 replaced with ground-truth per-query status: 9 of 20 v1-required queries done (Q-004/Q-005/Q-006/Q-008/Q-009/Q-011/Q-013/Q-016/Q-017), 1 partial (Q-001 needs subsystem-scoped variant), 10 missing (Q-002/003/007/010/012/014/018/019/020); V1-A scope shrank to Q-001 shape fix + Q-6 integration (pillar queries already implemented); V1-C closes the 8 remaining new queries + Q-020 deferred to V1-D; (3) F-5 reframed — generic `conflicts` + `conflict_members` schema already present at `database.py:190`, no migration needed; divergence is detector body (per-type dispatch needs generalization) + routes (`/admin/conflicts/*` needs `/conflicts/*` alias). Total revised to 16.5–17.5 days, ~60 tests. Plan: `docs/plans/engineering-v1-completion-plan.md` at commit `ce3a878` (Codex pulled clean). Three of Codex's eight open questions now answered; remaining: F-7 graduation depth, mirror determinism, `project` rename question, velocity calibration, minions naming. *Proposed by:* Claude. *Reviewed by:* Codex (two rounds).
|
||||||
|
- **2026-04-22** **Engineering V1 Completion Plan revised per Codex first-round review** — original six-phase order (queries → ingest → mirror → graduation → provenance → ops) rejected by Codex as backward: provenance-at-write (F-8) and conflict-detection hooks (F-5 minimal) must precede any phase that writes active entities. Revised to seven phases: V1-0 write-time invariants (F-8 + F-5 hooks + F-1 audit) as hard prerequisite, V1-A minimum query slice proving the model, V1-B ingest, V1-C full query catalog, V1-D mirror, V1-E graduation, V1-F full F-5 spec + ops + docs. Also softened "parallel with Now list" — real collision points listed explicitly; schedule shifted ~4 weeks to reflect that V1-0 cannot start during pipeline soak. Withdrew the "50–70% built" global framing in favor of the per-criterion gap table. Workspace sync note added: Codex's Playground workspace can't see the plan file; canonical dev tree is Windows `C:\Users\antoi\ATOCore`. Plan: `docs/plans/engineering-v1-completion-plan.md`. Awaiting Codex file-level audit once workspace syncs. *Proposed by:* Claude. *First-round review by:* Codex.
|
||||||
|
- **2026-04-22** gbrain-inspired "Phase 8 Minions + typed edges" plan **rejected as packaged** — wrong sequencing (leapfrogged `master-plan-status.md` Now list), wrong predicate set (6 vs V1's 17), wrong canonical boundary (edges-on-wikilinks instead of typed entities+relationships per `memory-vs-entities.md`). Mechanic (durable jobs + typed graph) deferred to V1 home. Record: `docs/decisions/2026-04-22-gbrain-plan-rejection.md`. *Proposed by:* Claude. *Reviewed/rejected by:* Codex. *Ratified by:* Antoine.
|
||||||
- **2026-04-12** Day 4 gate cleared: LLM-assisted extraction via `claude -p` (OAuth, no API key) is the path forward. Rule extractor stays as default for structural cues. *Proposed by:* Claude. *Ratified by:* Antoine.
|
- **2026-04-12** Day 4 gate cleared: LLM-assisted extraction via `claude -p` (OAuth, no API key) is the path forward. Rule extractor stays as default for structural cues. *Proposed by:* Claude. *Ratified by:* Antoine.
|
||||||
- **2026-04-12** First live triage: 16 promoted, 35 rejected from 51 LLM-extracted candidates. 31% accept rate. Active memory count 20->36. *Executed by:* Claude. *Ratified by:* Antoine.
|
- **2026-04-12** First live triage: 16 promoted, 35 rejected from 51 LLM-extracted candidates. 31% accept rate. Active memory count 20->36. *Executed by:* Claude. *Ratified by:* Antoine.
|
||||||
- **2026-04-12** No API keys allowed in AtoCore — LLM-assisted features use OAuth via `claude -p` or equivalent CLI-authenticated paths. *Proposed by:* Antoine.
|
- **2026-04-12** No API keys allowed in AtoCore — LLM-assisted features use OAuth via `claude -p` or equivalent CLI-authenticated paths. *Proposed by:* Antoine.
|
||||||
@@ -160,6 +168,22 @@ One branch `codex/extractor-eval-loop` for Day 1-5, a second `codex/retrieval-ha
|
|||||||
|
|
||||||
## Session Log
|
## Session Log
|
||||||
|
|
||||||
|
- **2026-04-22 Codex + Antoine (V1-0 closed)** Codex approved `f16cd52` after re-running both original probes (legacy-candidate promote + supersede hook — both correct) and the three targeted regression suites (`test_v1_0_write_invariants.py`, `test_engineering_v1_phase5.py`, `test_inbox_crossproject.py` — all pass). Squash-merged to main as `2712c5d` ("feat(engineering): enforce V1-0 write invariants"). Deployed to Dalidou via the canonical deploy script; `/health` build_sha=`2712c5d2d03cb2a6af38b559664afd1c4cd0e050` status=ok. Validated backup snapshot at `/srv/storage/atocore/backups/snapshots/20260422T190624Z` taken BEFORE prod backfill. Prod backfill of `scripts/v1_0_backfill_provenance.py` against live DB: dry-run found 31 active/superseded entities with no provenance, list reviewed and looked sane; live run with default `hand_authored=1` flag path updated 31 rows; follow-up dry-run returned 0 rows remaining → no lingering F-8 violations in prod. Codex logged one residual P2 (R14): HTTP `POST /entities/{id}/promote` route doesn't translate the new service-layer `ValueError` into 400 — legacy bad candidate promoted through the API surfaces as 500. Not blocking. V1-0 closed. **Gates for V1-A**: soak window ends ~2026-04-26; 100-active-memory density target (currently 84 active + the ~31 newly flagged ones — need to check how those count in density math). V1-A holds until both gates clear.
|
||||||
|
|
||||||
|
- **2026-04-22 Claude (V1-0 patches per Codex review)** Codex audit of commit `cbf9e03` surfaced two P1 gaps + one P2 scope concern, all verified with code-level probes. **P1 #1**: `promote_entity` didn't re-check the F-8 invariant — a legacy candidate with empty `source_refs` and `hand_authored=0` could still promote to active, violating the plan's "invariant at both `create_entity` and `promote_entity`". Fixed: `promote_entity` at `service.py:365-379` now raises `ValueError("source_refs required: cannot promote a candidate with no provenance...")` before flipping status. Stays symmetric with the create-side error. **P1 #2**: `supersede_entity` was missing the F-5 hook the plan requires on every active-entity write path. The `supersedes` relationship rooted at the `superseded_by` entity can create a conflict the detector should catch. Fixed at `service.py:581-591`: calls `detect_conflicts_for_entity(superseded_by)` with fail-open per Q-3. **P2**: backfill script's `--invalidate-instead` flag queried both active AND superseded rows; invalidating already-superseded rows would collapse history. Fixed at `scripts/v1_0_backfill_provenance.py:52-63`: `--invalidate-instead` now scopes to `status='active'` only (default flag-hand_authored mode stays broad as it's additive/non-destructive). Help text tightened to make the destructive posture explicit. **Four new regression tests** in `test_v1_0_write_invariants.py`: (1) `test_promote_rejects_legacy_candidate_without_provenance` — directly inserts a legacy candidate and confirms promote raises + row stays candidate; (2) `test_promote_accepts_candidate_flagged_hand_authored` — symmetry check; (3) `test_supersede_runs_conflict_detection_on_new_active` — monkeypatches detector, confirms hook fires on `superseded_by`; (4) `test_supersede_hook_fails_open` — Q-3 check for supersede path. **Test count**: 543 → 547 (+4 regression). Full suite `547 passed in 81.07s`. Next: commit patches on branch, push, Codex re-review.
|
||||||
|
|
||||||
|
- **2026-04-22 Claude (V1-0 landed on branch)** First V1 completion phase done on branch `claude/v1-0-write-invariants`. **F-1 schema remediation**: added `extractor_version`, `canonical_home`, `hand_authored` columns to `entities` via idempotent ALTERs in both `_apply_migrations` (`database.py:148-170`) and `init_engineering_schema` (`service.py:95-139`). CREATE TABLE also updated so fresh DBs get the columns natively. New `_table_exists` helper at `database.py:378`. `Entity` dataclass gains the three fields with sensible defaults. `EXTRACTOR_VERSION = "v1.0.0"` module constant at top of `service.py`. `_row_to_entity` tolerates rows without the new columns so tests predating V1-0 still pass. **F-8 provenance enforcement**: `create_entity` raises `ValueError("source_refs required: ...")` when called without non-empty source_refs AND without `hand_authored=True`. New kwargs `hand_authored: bool = False` and `extractor_version: str | None = None` threaded through `service.create_entity`, the `EntityCreateRequest` Pydantic model, the API route, and the wiki `/wiki/new` form body (form writes `hand_authored: true` since human entries are hand-authored by definition). **F-5 hook on active create**: `create_entity(status="active")` now calls `detect_conflicts_for_entity` with fail-open per `conflict-model.md:256` (errors log warning, write still succeeds). The promote path's existing hook at `service.py:400-404` was kept as-is. **Doc note** added to `engineering-ontology-v1.md` recording that `project` IS the `project_id` per "fields equivalent to" wording. **Backfill script** at `scripts/v1_0_backfill_provenance.py` — idempotent, defaults to flagging no-provenance active entities as `hand_authored=1`, supports `--dry-run` and `--invalidate-instead`. **Tests**: 10 new in `tests/test_v1_0_write_invariants.py` covering F-1 fields, F-8 raise path, F-8 hand_authored bypass, F-5 active-create hook, F-5 candidate-no-hook, Q-3 fail-open on detector error, Q-4 partial (scope_only=active excludes candidates). **Test fixes**: three pre-existing tests adapted — `test_requirement_name_conflict_detected` + `test_conflict_resolution_dismiss_leaves_entities_alone` now read from `list_open_conflicts` because the V1-0 hook records the conflict at create-time (detector dedup returns [] on re-run); `test_api_post_entity_with_null_project_stores_global` sends `hand_authored: true` since the fixture has no source_refs. **conftest.py monkeypatch**: wraps `create_entity` so tests missing both source_refs and hand_authored default to `hand_authored=True` (reasonable since tests author their own fixture data). Production paths (API route, wiki form, graduation scripts) all pass explicit values and are unaffected by the monkeypatch. **Test count**: 533 → 543 (+10), full suite `543 passed in 77.86s`. **Not yet**: commit + push + Codex review + deploy. **Branch**: `claude/v1-0-write-invariants`.
|
||||||
|
|
||||||
|
- **2026-04-22 Codex (late night)** Third-round audit closed the remaining five open questions. Patched `docs/plans/engineering-v1-completion-plan.md` inline (no commit by Codex). **F-7 finding (P1):** graduation stack is partially built — `_graduation_prompt.py`, `scripts/graduate_memories.py`, `database.py:143-146` (`graduated_to_entity_id`), memory `graduated` status, promote-preserves-original at `service.py:354-356,389-451`, tests at `test_engineering_v1_phase5.py:67-90` all exist. Real gaps: no direct `POST /memory/{id}/graduate` route at `routes.py:756`; spec's `knowledge→Fact` doesn't match ontology (`service.py:16` has no `fact` type — reconcile to `parameter` or similar). V1-E estimate 2 → 3–4 days. **Q-5 finding (P2):** "stabilize timestamp" insufficient — renderer reads wall-clock in `_footer()` at `mirror.py:320`; fix is inject regenerated timestamp + checksum as renderer inputs + sort DB iteration + remove dict ordering deps. V1-D scope patched. **Remaining three (P3):** `project` stays as doc-note equivalence (no rename); total estimate 17.5–19.5 focused days; release notes must NOT canonize "Minions" — neutral "queued background processing / async workers" only. **Sign-off:** "with those edits, I'd sign off on the five questions. The only non-architectural uncertainty left in the plan is scheduling discipline against the current Now list; that does not block V1-0 once the soak window and memory-density gate clear." **Status:** Claude + Codex agreed. Plan frozen pending Antoine final accept and gate clearance. Claude to commit Codex's patches + push.
|
||||||
|
|
||||||
|
- **2026-04-22 Claude (late night)** Codex second-round review did the full file-level audit and came back with three P1/P2 findings, all with exact file:line refs. Verified each against current code before revising. (1) **F-1 not clean**: `Entity` dataclass at `service.py:67` and `entities` table schema are missing the `extractor_version` and `canonical_home` shared-header fields required by `engineering-v1-acceptance.md:45`; `project` field is the project identifier but not named `project_id` as spec writes (spec wording "fields equivalent to" allows the naming, but needs explicit doc note). V1-0 scope now includes adding both missing fields via additive `_apply_migrations` pattern. (2) **F-2 needed exact statuses, not guesses**: per-function audit gave ground truth — 9 of 20 v1-required queries done, 1 partial (Q-001 returns project-wide tree not subsystem-scoped expand=contains per `engineering-query-catalog.md:71`), 10 missing. V1-A scope shrank to Q-001 shape fix + Q-6 integration (most pillar queries already implemented); V1-C closes the 8 net-new queries + Q-020 to V1-D. (3) **F-5 misframed**: the generic `conflicts` + `conflict_members` schema is ALREADY spec-compliant at `database.py:190`; divergence is detector body at `conflicts.py:36` (per-type dispatch needs generalization) + route path (`/admin/conflicts/*` needs `/conflicts/*` alias). V1-F no longer includes a schema migration; detector generalization + route alignment only. Totals revised to 16.5–17.5 days, ~60 tests (down from 12–17 / 65 because V1-A and V1-F scopes both shrank after audit). Three of the eight open questions resolved. Remaining open: F-7 graduation depth, mirror determinism, `project` naming, velocity calibration, minions-as-V2 naming. No code changes this session — plan + ledger only. Next: commit + push revised plan, then await Antoine+Codex joint sign-off before V1-0 starts.
|
||||||
|
|
||||||
|
- **2026-04-22 Claude (night)** Codex first-round review of the V1 Completion Plan summary came back with four findings. Three substantive, one workspace-sync: (1) "50–70% built" too loose — replaced with per-criterion table, global framing withdrawn; (2) phase order backward — provenance-at-write (F-8) and conflict hooks (F-5 minimal) depend-upon by every later phase but were in V1-E; new V1-0 prerequisite phase inserted to establish write-time invariants, and V1-A shrunk to a minimum query slice (four pillars Q-001/Q-005/Q-006/Q-017 + Q-6 integration) rather than full catalog closure; (3) "parallel with Now list / disjoint surfaces" too strong — real collisions listed explicitly (V1-0 provenance + memory extractor write path, V1-E graduation + memory module, V1-F conflicts migration + memory promote); schedule shifted ~4 weeks, V1-0 cannot start during pipeline soak; (4) Codex's Playground workspace can't see the plan file or the `src/atocore/engineering/` code — added a Workspace note to the plan directing per-file audit at the Windows canonical dev tree (`C:\Users\antoi\ATOCore`) and noting the three visible file paths (`docs/plans/engineering-v1-completion-plan.md`, `docs/decisions/2026-04-22-gbrain-plan-rejection.md`, `DEV-LEDGER.md`). Revised plan estimate: 12–17 days across 7 phases (up from 11–14 / 6), ~65 tests added (up from ~50). V1-0 is a hard prerequisite; no later phase starts until it lands. Pending Antoine decision on workspace sync (commit+push vs paste-to-Codex) so Codex can do the file-level audit. No code changes this session.
|
||||||
|
|
||||||
|
- **2026-04-22 Claude (late eve)** After the rejection, read the four core V1 architecture docs end-to-end (`engineering-ontology-v1.md`, `engineering-query-catalog.md`, `memory-vs-entities.md`, `engineering-v1-acceptance.md`) plus the four supporting docs (`promotion-rules.md`, `conflict-model.md`, `human-mirror-rules.md`, `tool-handoff-boundaries.md`). Cross-referenced against current code in `src/atocore/engineering/`. **Key finding:** V1 is already 50–70% built — entity types (16, superset of V1's 12), all 18 V1 relationship types, 4-state lifecycle, CRUD + supersede + invalidate + PATCH, queries module with most killer-correctness queries (orphan_requirements, risky_decisions, unsupported_claims, impact_analysis, evidence_chain), conflicts module scaffolded, mirror scaffolded, graduation endpoint scaffolded. Recent commits e147ab2/b94f9df/081c058/069d155/b1a3dd0 are all V1 entity-layer work. Drafted `docs/plans/engineering-v1-completion-plan.md` reframing the work as **V1 completion, not V1 start**. Six sequential phases V1-A through V1-F, estimated 11–14 days, ~50 new tests (533 → ~580). Phases run in parallel with the Now list (pipeline soak + density + multi-model triage + p04-constraints) because surfaces are disjoint. Plan explicitly defers the minions/queue mechanic per acceptance-doc negative list. Pending Codex audit of the plan itself — especially the F-2 query gap list (Claude didn't read each query function end-to-end), F-5 conflicts schema divergence (per-type detectors vs spec's generic slot-keyed shape), and F-7 graduation depth. No code changes this session.
|
||||||
|
|
||||||
|
- **2026-04-22 Claude (eve)** gbrain review session. Antoine surfaced https://github.com/garrytan/gbrain for compare/contrast. Claude drafted a "Phase 8 Minions + typed edges" plan pairing a durable job queue with a 6-predicate edge upgrade over wikilinks. Codex reviewed and rejected as packaged: (1) sequencing leapfrogged the `master-plan-status.md` Now list (pipeline soak → 100+ memories → multi-model triage → p04-constraints fix); (2) 6 predicates vs V1's 17 across Structural/Intent/Validation/Provenance families — would have been schema debt on day one per `engineering-ontology-v1.md:112-137`; (3) "edges over wikilinks" bypassed the V1 canonical entity + promotion contract in `memory-vs-entities.md`. Claude verified each high finding against the cited files and concurred. The underlying mechanic (durable background jobs + typed relationship graph) is still a valid future direction, but its correct home is the Engineering V1 sprint under **Next** in `master-plan-status.md:179`, not a leapfrog phase. Decision record: `docs/decisions/2026-04-22-gbrain-plan-rejection.md`. No code changes this session. Next (pending Antoine ack): read the four V1 architecture docs end-to-end, then draft an Engineering V1 foundation plan that follows the existing contract, not a new external reference. Phase 8 (OpenClaw) name remains untouched — Claude's misuse of "Phase 8" in the rejected plan was a naming collision, not a renaming.
|
||||||
|
|
||||||
- **2026-04-22 Claude (pm)** Issue B (wiki redlinks) landed — last remaining P2 from Antoine's sprint plan. `_wikilink_transform(text, current_project)` in `src/atocore/engineering/wiki.py` replaces `[[Name]]` / `[[Name|Display]]` tokens (pre-markdown) with HTML anchors. Resolution order: same-project exact-name match → live `wikilink`; other-project match → live link with `(in project X)` scope indicator (`wikilink-cross`); no match → `redlink` pointing at `/wiki/new?name=<quoted>&project=<current>`. New route `GET /wiki/new` renders a pre-filled "create this entity" form that POSTs to `/v1/entities` via a minimal inline fetch() and redirects to the new entity's wiki page on success. Transform applied in `render_project` (over the mirror markdown) and `render_entity` (over the description body). CSS: dashed-underline accent for live wikilinks, red italic + dashed for redlinks. 12 new tests including the regression from the spec (entity A references `[[EntityB]]` → initial render has `class="redlink"`; after EntityB is created, re-render no longer has redlink and includes `/wiki/entities/{b.id}`). Tests 521 → 533. All 6 acceptance criteria from the sprint plan ("daily-usable") now green: retract/supersede, edit without cloning, cross-project has a home, visual evidence, wiki readable, AKC can capture reliably.
|
- **2026-04-22 Claude (pm)** Issue B (wiki redlinks) landed — last remaining P2 from Antoine's sprint plan. `_wikilink_transform(text, current_project)` in `src/atocore/engineering/wiki.py` replaces `[[Name]]` / `[[Name|Display]]` tokens (pre-markdown) with HTML anchors. Resolution order: same-project exact-name match → live `wikilink`; other-project match → live link with `(in project X)` scope indicator (`wikilink-cross`); no match → `redlink` pointing at `/wiki/new?name=<quoted>&project=<current>`. New route `GET /wiki/new` renders a pre-filled "create this entity" form that POSTs to `/v1/entities` via a minimal inline fetch() and redirects to the new entity's wiki page on success. Transform applied in `render_project` (over the mirror markdown) and `render_entity` (over the description body). CSS: dashed-underline accent for live wikilinks, red italic + dashed for redlinks. 12 new tests including the regression from the spec (entity A references `[[EntityB]]` → initial render has `class="redlink"`; after EntityB is created, re-render no longer has redlink and includes `/wiki/entities/{b.id}`). Tests 521 → 533. All 6 acceptance criteria from the sprint plan ("daily-usable") now green: retract/supersede, edit without cloning, cross-project has a home, visual evidence, wiki readable, AKC can capture reliably.
|
||||||
|
|
||||||
- **2026-04-22 Claude** PATCH `/entities/{id}` + Issue D (/v1/engineering/* aliases) landed. New `update_entity()` in `src/atocore/engineering/service.py` supports partial updates to description (replace), properties (shallow merge — `null` value deletes a key), confidence (0..1, 400 on bounds violation), source_refs (append + dedup). Writes an `updated` audit row with full before/after snapshots. Forbidden via this path: entity_type / project / name / status — those require supersede+create or the dedicated status endpoints, by design. New route `PATCH /entities/{id}` aliased under `/v1`. Issue D: all 10 `/engineering/*` query paths (decisions, systems, components/{id}/requirements, changes, gaps + sub-paths, impact, evidence) added to the `/v1` allowlist. 12 new PATCH tests (merge, null-delete, confidence bounds, source_refs dedup, 404, audit row, v1 alias). Tests 509 → 521. Next: commit + deploy, then Issue B (wiki redlinks) as the last remaining P2 per Antoine's sprint order.
|
- **2026-04-22 Claude** PATCH `/entities/{id}` + Issue D (/v1/engineering/* aliases) landed. New `update_entity()` in `src/atocore/engineering/service.py` supports partial updates to description (replace), properties (shallow merge — `null` value deletes a key), confidence (0..1, 400 on bounds violation), source_refs (append + dedup). Writes an `updated` audit row with full before/after snapshots. Forbidden via this path: entity_type / project / name / status — those require supersede+create or the dedicated status endpoints, by design. New route `PATCH /entities/{id}` aliased under `/v1`. Issue D: all 10 `/engineering/*` query paths (decisions, systems, components/{id}/requirements, changes, gaps + sub-paths, impact, evidence) added to the `/v1` allowlist. 12 new PATCH tests (merge, null-delete, confidence bounds, source_refs dedup, 404, audit row, v1 alias). Tests 509 → 521. Next: commit + deploy, then Issue B (wiki redlinks) as the last remaining P2 per Antoine's sprint order.
|
||||||
|
|||||||
@@ -159,6 +159,17 @@ Every major object should support fields equivalent to:
|
|||||||
- `created_at`
|
- `created_at`
|
||||||
- `updated_at`
|
- `updated_at`
|
||||||
- `notes` (optional)
|
- `notes` (optional)
|
||||||
|
- `extractor_version` (V1-0)
|
||||||
|
- `canonical_home` (V1-0)
|
||||||
|
|
||||||
|
**Naming note (V1-0, 2026-04-22).** The AtoCore `entities` table and
|
||||||
|
`Entity` dataclass name the project-identifier field `project`, not
|
||||||
|
`project_id`. This doc's "fields equivalent to" wording allows that
|
||||||
|
naming flexibility — the `project` field on entity rows IS the
|
||||||
|
`project_id` per spec. No storage rename is planned; downstream readers
|
||||||
|
should treat `entity.project` as the project identifier. This was
|
||||||
|
resolved in Codex's third-round audit of the V1 Completion Plan (see
|
||||||
|
`docs/plans/engineering-v1-completion-plan.md`).
|
||||||
|
|
||||||
## Suggested Status Lifecycle
|
## Suggested Status Lifecycle
|
||||||
|
|
||||||
|
|||||||
@@ -1,6 +1,31 @@
|
|||||||
# AtoCore — Current State (2026-04-19)
|
# AtoCore — Current State (2026-04-22)
|
||||||
|
|
||||||
Live deploy: `877b97e` · Dalidou health: ok · Harness: 17/18.
|
Live deploy: `2712c5d` · Dalidou health: ok · Harness: 17/18 · Tests: 547 passing.
|
||||||
|
|
||||||
|
## V1-0 landed 2026-04-22
|
||||||
|
|
||||||
|
Engineering V1 completion track has started. **V1-0 write-time invariants**
|
||||||
|
merged and deployed: F-1 shared-header fields (`extractor_version`,
|
||||||
|
`canonical_home`, `hand_authored`) added to `entities`, F-8 provenance
|
||||||
|
enforcement at both `create_entity` and `promote_entity`, F-5 synchronous
|
||||||
|
conflict-detection hook on every active-entity write path (create, promote,
|
||||||
|
supersede) with Q-3 fail-open. Prod backfill ran cleanly — 31 legacy
|
||||||
|
active/superseded entities flagged `hand_authored=1`, follow-up dry-run
|
||||||
|
returned 0 remaining rows. Test count 533 → 547 (+14).
|
||||||
|
|
||||||
|
R14 (P2, non-blocking): `POST /entities/{id}/promote` route fix translates
|
||||||
|
the new `ValueError` into 400. Branch `claude/r14-promote-400` pending
|
||||||
|
Codex review + squash-merge.
|
||||||
|
|
||||||
|
**Next in the V1 track:** V1-A (minimal query slice + Q-6 killer-correctness
|
||||||
|
integration). Gated on pipeline soak (~2026-04-26) + 100+ active memory
|
||||||
|
density target. See `docs/plans/engineering-v1-completion-plan.md` for
|
||||||
|
the full 7-phase roadmap and `docs/plans/v1-resume-state.md` for the
|
||||||
|
"you are here" map.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Snapshot from previous update (2026-04-19)
|
||||||
|
|
||||||
## The numbers
|
## The numbers
|
||||||
|
|
||||||
|
|||||||
113
docs/decisions/2026-04-22-gbrain-plan-rejection.md
Normal file
113
docs/decisions/2026-04-22-gbrain-plan-rejection.md
Normal file
@@ -0,0 +1,113 @@
|
|||||||
|
# Decision record: gbrain-inspired "Phase 8 Minions + typed edges" plan rejected
|
||||||
|
|
||||||
|
**Date:** 2026-04-22
|
||||||
|
**Author of plan:** Claude
|
||||||
|
**Reviewer:** Codex
|
||||||
|
**Ratified by:** Antoine
|
||||||
|
**Status:** Rejected as packaged. Underlying mechanic (durable background jobs + typed relationships) deferred to its correct home.
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
Antoine surfaced https://github.com/garrytan/gbrain and asked for a compare/contrast and a
|
||||||
|
plan to improve AtoCore. Claude proposed a "Phase 8" plan pairing:
|
||||||
|
|
||||||
|
1. A Minion-style durable job queue replacing the nightly cron pipeline
|
||||||
|
2. A typed-edge upgrade over existing wikilinks, with a six-predicate set
|
||||||
|
(`mentions`, `decided_by`, `supersedes`, `evidences`, `part_of`, `blocks`)
|
||||||
|
|
||||||
|
Codex reviewed and rejected the plan as packaged. This record captures what went wrong,
|
||||||
|
what was right, and where the ideas should actually land.
|
||||||
|
|
||||||
|
## What Codex flagged (verified against repo)
|
||||||
|
|
||||||
|
### High — wrong sequencing
|
||||||
|
|
||||||
|
`docs/master-plan-status.md` defines the **Now** list:
|
||||||
|
|
||||||
|
1. Observe the enhanced pipeline for a week
|
||||||
|
2. Knowledge density — batch-extract over all 234 interactions, target 100+ memories
|
||||||
|
3. Multi-model triage (Phase 11 entry)
|
||||||
|
4. Fix p04-constraints harness failure
|
||||||
|
|
||||||
|
Engineering V1 appears under **Next** (line 179) as
|
||||||
|
"Engineering V1 implementation sprint — once knowledge density is sufficient and the
|
||||||
|
pipeline feels boring and dependable."
|
||||||
|
|
||||||
|
Claude's plan jumped over all four **Now** items. That was the primary sequencing error.
|
||||||
|
|
||||||
|
### High — wrong predicate set
|
||||||
|
|
||||||
|
`docs/architecture/engineering-ontology-v1.md` already defines a 17-predicate V1
|
||||||
|
ontology across four families:
|
||||||
|
|
||||||
|
- **Structural:** `CONTAINS`, `PART_OF`, `INTERFACES_WITH`
|
||||||
|
- **Intent / logic:** `SATISFIES`, `CONSTRAINED_BY`, `BASED_ON_ASSUMPTION`,
|
||||||
|
`AFFECTED_BY_DECISION`, `SUPERSEDES`
|
||||||
|
- **Validation:** `ANALYZED_BY`, `VALIDATED_BY`, `SUPPORTS`, `CONFLICTS_WITH`,
|
||||||
|
`DEPENDS_ON`
|
||||||
|
- **Artifact / provenance:** `DESCRIBED_BY`, `UPDATED_BY_SESSION`, `EVIDENCED_BY`,
|
||||||
|
`SUMMARIZED_IN`
|
||||||
|
|
||||||
|
Claude's six-predicate set was a gbrain-shaped subset that could not express the V1
|
||||||
|
example statements at lines 141–147 of that doc. Shipping it first would have been
|
||||||
|
schema debt on day one.
|
||||||
|
|
||||||
|
### High — wrong canonical boundary
|
||||||
|
|
||||||
|
`docs/architecture/memory-vs-entities.md` and
|
||||||
|
`docs/architecture/engineering-v1-acceptance.md` establish that V1 is **typed
|
||||||
|
entities plus typed relationships**, with one canonical home per concept, a shared
|
||||||
|
candidate-review / promotion flow, provenance, conflict handling, and mirror
|
||||||
|
generation. Claude's "typed edges on top of wikilinks" framing bypassed the canonical
|
||||||
|
entity contract — it would have produced labelled links over notes without the
|
||||||
|
promotion / canonicalization machinery V1 actually requires.
|
||||||
|
|
||||||
|
### Medium — overstated problem
|
||||||
|
|
||||||
|
Claude described the nightly pipeline as a "monolithic bash script" that needed to be
|
||||||
|
replaced. The actual runtime is API-driven (`src/atocore/api/routes.py:516`,
|
||||||
|
`src/atocore/interactions/service.py:55`), SQLite is already in WAL with a busy
|
||||||
|
timeout (`src/atocore/models/database.py:151`), and the reflection loop is explicit
|
||||||
|
capture / reinforce / extract. The queue argument overstated the current shape.
|
||||||
|
|
||||||
|
## What was right
|
||||||
|
|
||||||
|
- gbrain is genuine validation of the general pattern: **durable background jobs +
|
||||||
|
typed relationship graph compound value**. The gbrain v0.12.0 graph release and
|
||||||
|
Minions benchmark (both 2026-04-18) are evidence, not just inspiration.
|
||||||
|
- Async-ification of extraction with retries, per-job visibility, and SLOs remains a
|
||||||
|
real future win for AtoCore — but **additively, behind flags, after V1**, not as a
|
||||||
|
replacement for the current explicit endpoints.
|
||||||
|
|
||||||
|
## What we will do instead
|
||||||
|
|
||||||
|
1. **Keep to the `master-plan-status.md` Now list.** No leapfrog. Observe the
|
||||||
|
pipeline (including the confidence-decay Step F4 first real run), land knowledge
|
||||||
|
density via full-backlog batch extract, progress multi-model triage, fix
|
||||||
|
p04-constraints.
|
||||||
|
2. **When Engineering V1 is ready to start** (criterion: pipeline feels boring and
|
||||||
|
dependable, knowledge density ≥ 100 active memories), write a V1 foundation plan
|
||||||
|
that follows `engineering-ontology-v1.md`, `engineering-query-catalog.md`,
|
||||||
|
`memory-vs-entities.md`, and `engineering-v1-acceptance.md` — entities +
|
||||||
|
relationships + memory-to-entity bridge + mirror / query surfaces, in that order.
|
||||||
|
3. **Async workerization is optional and later.** Only after V1 is working, and only
|
||||||
|
if observed contention or latency warrants it. Jobs stay in the primary SQLite
|
||||||
|
(WAL already in place). No separate DB unless contention is measured.
|
||||||
|
|
||||||
|
## Lesson for future plans
|
||||||
|
|
||||||
|
A plan built from a **new external reference** (gbrain) without reading the
|
||||||
|
repository's own architecture docs will mis-specify predicates, boundaries, and
|
||||||
|
sequencing — even when the underlying mechanic is valid. Read the four V1
|
||||||
|
architecture docs end-to-end before proposing schema work.
|
||||||
|
|
||||||
|
## References
|
||||||
|
|
||||||
|
- https://github.com/garrytan/gbrain
|
||||||
|
- `docs/master-plan-status.md` (Now / Next / Later)
|
||||||
|
- `docs/architecture/engineering-ontology-v1.md`
|
||||||
|
- `docs/architecture/engineering-query-catalog.md`
|
||||||
|
- `docs/architecture/memory-vs-entities.md`
|
||||||
|
- `docs/architecture/engineering-v1-acceptance.md`
|
||||||
|
- `docs/architecture/llm-client-integration.md`
|
||||||
|
- `docs/architecture/human-mirror-rules.md`
|
||||||
@@ -168,16 +168,40 @@ These are the current practical priorities.
|
|||||||
"Zerodur" for p04 constraint queries. Investigate if it's a missing
|
"Zerodur" for p04 constraint queries. Investigate if it's a missing
|
||||||
memory or retrieval ranking issue.
|
memory or retrieval ranking issue.
|
||||||
|
|
||||||
|
## Active — Engineering V1 Completion Track (started 2026-04-22)
|
||||||
|
|
||||||
|
The Engineering V1 sprint moved from **Next** to **Active** on 2026-04-22.
|
||||||
|
The discovery from the gbrain review was that V1 entity infrastructure
|
||||||
|
had been built incrementally already; the sprint is a **completion** plan
|
||||||
|
against `engineering-v1-acceptance.md`, not a greenfield build. Full plan:
|
||||||
|
`docs/plans/engineering-v1-completion-plan.md`. "You are here" single-page
|
||||||
|
map: `docs/plans/v1-resume-state.md`.
|
||||||
|
|
||||||
|
Seven phases, ~17.5–19.5 focused days, runs in parallel with the Now list
|
||||||
|
where surfaces are disjoint, pauses when they collide.
|
||||||
|
|
||||||
|
| Phase | Scope | Status |
|
||||||
|
|---|---|---|
|
||||||
|
| V1-0 | Write-time invariants: F-1 header fields + F-8 provenance enforcement + F-5 hook on every active-entity write + Q-3 flag-never-block | ✅ done 2026-04-22 (`2712c5d`) |
|
||||||
|
| V1-A | Minimum query slice: Q-001 subsystem-scoped variant + Q-6 killer-correctness integration test on p05-interferometer | 🟡 gated — starts when soak (~2026-04-26) + density (100+ active memories) gates clear |
|
||||||
|
| V1-B | KB-CAD + KB-FEM ingest (`POST /ingest/kb-cad/export`, `POST /ingest/kb-fem/export`) + D-2 schema docs | pending V1-A |
|
||||||
|
| V1-C | Close the remaining 8 queries (Q-002/003/007/010/012/014/018/019; Q-020 to V1-D) | pending V1-B |
|
||||||
|
| V1-D | Full mirror surface (3 spec routes + regenerate + determinism + disputed + curated markers) + Q-5 golden file | pending V1-C |
|
||||||
|
| V1-E | Memory→entity graduation end-to-end + remaining Q-4 trust tests | pending V1-D (note: collides with memory extractor; pauses for multi-model triage work) |
|
||||||
|
| V1-F | F-5 detector generalization + route alias + O-1/O-2/O-3 operational + D-1/D-3/D-4 docs | finish line |
|
||||||
|
|
||||||
|
R14 (P2, non-blocking): `POST /entities/{id}/promote` route returns 500
|
||||||
|
on the new V1-0 `ValueError` instead of 400. Fix on branch
|
||||||
|
`claude/r14-promote-400`, pending Codex review.
|
||||||
|
|
||||||
## Next
|
## Next
|
||||||
|
|
||||||
These are the next major layers after the current stabilization pass.
|
These are the next major layers after V1 and the current stabilization pass.
|
||||||
|
|
||||||
1. Phase 6 AtoDrive — clarify Google Drive as a trusted operational
|
1. Phase 6 AtoDrive — clarify Google Drive as a trusted operational
|
||||||
source and ingest from it
|
source and ingest from it
|
||||||
2. Phase 13 Hardening — Chroma backup policy, monitoring, alerting,
|
2. Phase 13 Hardening — Chroma backup policy, monitoring, alerting,
|
||||||
failure visibility beyond log files
|
failure visibility beyond log files
|
||||||
3. Engineering V1 implementation sprint — once knowledge density is
|
|
||||||
sufficient and the pipeline feels boring and dependable
|
|
||||||
|
|
||||||
## Later
|
## Later
|
||||||
|
|
||||||
|
|||||||
617
docs/plans/engineering-v1-completion-plan.md
Normal file
617
docs/plans/engineering-v1-completion-plan.md
Normal file
@@ -0,0 +1,617 @@
|
|||||||
|
# Engineering V1 Completion Plan
|
||||||
|
|
||||||
|
**Date:** 2026-04-22
|
||||||
|
**Author:** Claude (after reading the four V1 architecture docs + promotion-rules,
|
||||||
|
conflict-model, human-mirror-rules, tool-handoff-boundaries end-to-end)
|
||||||
|
**Status:** Draft, pending Codex review
|
||||||
|
**Replaces:** the rejected "Phase 8 Minions + typed edges" plan (see
|
||||||
|
`docs/decisions/2026-04-22-gbrain-plan-rejection.md`)
|
||||||
|
|
||||||
|
## Position
|
||||||
|
|
||||||
|
This is **not** a plan to start Engineering V1. It is a plan to **finish** V1.
|
||||||
|
|
||||||
|
**Against what criterion?** Each F/Q/O/D item in `engineering-v1-acceptance.md`
|
||||||
|
gets scored individually in the Gap audit table below with exact code/test/doc
|
||||||
|
references. No global percentage. The headline framing from the first draft
|
||||||
|
("50–70% built") is withdrawn — it's either done per-criterion or it isn't.
|
||||||
|
|
||||||
|
The relevant observation is narrower: the entity schema, the full
|
||||||
|
relationship type set, the 4-state lifecycle, basic CRUD and most of the
|
||||||
|
killer-correctness query functions are already implemented in
|
||||||
|
`src/atocore/engineering/*.py` in the Windows working tree at
|
||||||
|
`C:\Users\antoi\ATOCore` (the canonical dev workspace, per
|
||||||
|
`CLAUDE.md`). The recent commits e147ab2, b94f9df, 081c058, 069d155, b1a3dd0
|
||||||
|
are V1 entity-layer work. **Codex auditors working in a different
|
||||||
|
workspace / branch should sync from the canonical dev tree before
|
||||||
|
per-file review** — see the "Workspace note" at the end of this doc.
|
||||||
|
|
||||||
|
The question this plan answers: given the current code state, in what
|
||||||
|
order should the remaining V1 acceptance criteria be closed so that
|
||||||
|
every phase builds on invariants the earlier phases already enforced?
|
||||||
|
|
||||||
|
## Corrected sequencing principle (post-Codex review 2026-04-22)
|
||||||
|
|
||||||
|
The first draft ordered phases F-1 → F-2 → F-3 → F-4 → F-5 → F-6 → F-7 → F-8
|
||||||
|
following the acceptance doc's suggested reading order. Codex rejected
|
||||||
|
that ordering. The correct dependency order, which this revision adopts, is:
|
||||||
|
|
||||||
|
1. **Write-time invariants come first.** Every later phase creates active
|
||||||
|
entities. Provenance-at-write (F-8) and synchronous conflict-detection
|
||||||
|
hooks (F-5 minimal) must be enforced **before** any phase that writes
|
||||||
|
entities at scale (ingest, graduation, or broad query coverage that
|
||||||
|
depends on the model being populated).
|
||||||
|
2. **Query closure sits on top of the schema + invariants**, not ahead of
|
||||||
|
them. A minimum query slice that proves the model is fine early. The
|
||||||
|
full catalog closure waits until after the write paths are invariant-safe.
|
||||||
|
3. **Mirror is a derived consumer** of the entity layer, not a midstream
|
||||||
|
milestone. It comes after the entity layer produces enforced, correct data.
|
||||||
|
4. **Graduation and full conflict-spec compliance** are finishing work that
|
||||||
|
depend on everything above being stable.
|
||||||
|
|
||||||
|
The acceptance criteria are unchanged. Only the order of closing them changes.
|
||||||
|
|
||||||
|
## How this plan respects the rejected-plan lessons
|
||||||
|
|
||||||
|
- **No new predicates.** The V1 ontology in `engineering-ontology-v1.md:112-137`
|
||||||
|
already defines 18 relationship types; `service.py:38-62` already implements
|
||||||
|
them. Nothing added, nothing reshaped.
|
||||||
|
- **No new canonical boundary.** Typed entities + typed relationships with
|
||||||
|
promotion-based candidate flow per `memory-vs-entities.md`. Not
|
||||||
|
edges-over-wikilinks.
|
||||||
|
- **No leapfrog of `master-plan-status.md` Now list.** This plan is **in
|
||||||
|
parallel** with (not ahead of) the Now items because V1 entity work is
|
||||||
|
already happening alongside them. The sequencing section below is explicit.
|
||||||
|
- **Queue/worker infrastructure is explicitly out of scope.** The "flag it for
|
||||||
|
later" note at the end of this doc is the only mention, per
|
||||||
|
`engineering-v1-acceptance.md:378` negative list.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Gap audit against `engineering-v1-acceptance.md`
|
||||||
|
|
||||||
|
Each criterion marked: ✅ done / 🟡 partial / ❌ missing. "Partial" means the
|
||||||
|
capability exists but does not yet match spec shape or coverage.
|
||||||
|
|
||||||
|
### Functional (F-1 through F-8)
|
||||||
|
|
||||||
|
| ID | Criterion | Status | Evidence |
|
||||||
|
|----|-----------|--------|----------|
|
||||||
|
| F-1 | 12 V1 entity types, 4 relationship families, shared header fields, 4-state lifecycle | 🟡 partial (per Codex 2026-04-22 audit) | `service.py:16-36` has 16 types (superset of V1's 12), `service.py:38-62` has 18 relationship types, `service.py:64` statuses, `Entity` dataclass at line 67. **Gaps vs `engineering-v1-acceptance.md:45`**: `extractor_version` missing from dataclass and `entities` table; `canonical_home` missing from dataclass and table; `project` field is the project identifier but not named `project_id` as spec uses — spec says "fields equivalent to" so naming flexibility is allowed but needs an explicit doc note. Remediation lands in V1-0 |
|
||||||
|
| F-2 | All v1-required Q-001 through Q-020 implemented, with provenance where required | 🟡 partial (per Codex 2026-04-22 per-function audit) | **Ground truth from per-function read of `queries.py` + `routes.py:2092+`:** Q-001 partial (`system_map()` returns project-wide tree, not the catalog's subsystem-scoped `GET /entities/Subsystem/<id>?expand=contains` shape per `engineering-query-catalog.md:71`); Q-002 missing; Q-003 missing; Q-004 done (covered by `system_map()`); Q-005 done (`requirements_for()`); Q-006 done (`orphan_requirements()`); Q-007 missing; Q-008 done (`decisions_affecting()`); Q-009 done (`risky_decisions()`); Q-010 missing; Q-011 done (`unsupported_claims()`); Q-012 missing; Q-013 done (`recent_changes()`); Q-014 missing; Q-016 done (`impact_analysis()`); Q-017 done (`evidence_chain()`); Q-018 missing; Q-019 missing; Q-020 missing (mirror route in spec shape). **Net: 9 of 20 v1-required queries done, 1 partial (Q-001), 10 missing.** Q-015 is v1-stretch, out of scope |
|
||||||
|
| F-3 | `POST /ingest/kb-cad/export` and `POST /ingest/kb-fem/export` | ❌ missing | No `/ingest/kb-cad` or `/ingest/kb-fem` route in `api/routes.py`. No schema doc under `docs/architecture/` |
|
||||||
|
| F-4 | Candidate review queue end-to-end (list/promote/reject/edit) | 🟡 partial for entities | Memory side shipped in Phase 9 Commit C. Entity side has `promote_entity`, `supersede_entity`, `invalidate_active_entity` but reject path and editable-before-promote may not match spec shape. Need to verify `GET /entities?status=candidate` returns spec shape |
|
||||||
|
| F-5 | Conflict detector fires synchronously; `POST /conflicts/{id}/resolve` + dismiss | 🟡 partial (per Codex 2026-04-22 audit — schema present, detector+routes divergent) | **Schema is already spec-shaped**: `database.py:190` defines the generic `conflicts` + `conflict_members` tables per `conflict-model.md`; `conflicts.py:154` persists through them. **Divergences are in detection and API, not schema**: (1) `conflicts.py:36` dispatches per-type detectors only (`_check_component_conflicts`, `_check_requirement_conflicts`) — needs generalization to slot-key-driven detection; (2) routes live at `/admin/conflicts/*`, spec says `/conflicts/*` — needs alias + deprecation. **No schema migration needed** |
|
||||||
|
| F-6 | Mirror: `/mirror/{project}/overview`, `/decisions`, `/subsystems/{id}`, `/regenerate`; files under `/srv/storage/atocore/data/mirror/`; disputed + curated markers; deterministic output | 🟡 partial | `mirror.py` has `generate_project_overview` with header/state/system/decisions/requirements/materials/vendors/memories/footer sections. API at `/projects/{project_name}/mirror` and `.html`. **Gaps**: no separate `/mirror/{project}/decisions` or `/mirror/{project}/subsystems/{id}` routes, no `POST /regenerate` endpoint, no debounced-async-on-write, no daily refresh, no `⚠ disputed` markers wired to conflicts, no `(curated)` override annotations verified, no golden-file test for determinism |
|
||||||
|
| F-7 | Memory→entity graduation: `POST /memory/{id}/graduate` + `graduated` status + forward pointer + original preserved | 🟡 partial (per Codex 2026-04-22 third-round audit) | `_graduation_prompt.py` exists; `scripts/graduate_memories.py` creates entity candidates from active memories; `database.py:143-146` adds `graduated_to_entity_id`; `memory.service` already has a `graduated` status; `service.py:354-356,389-451` preserves the original memory and marks it `graduated` with a forward pointer on entity promote; `tests/test_engineering_v1_phase5.py:67-90` covers that flow. **Gaps vs spec**: no direct `POST /memory/{id}/graduate` route yet (current surface is batch/admin-driven via `/admin/graduation/request`); no explicit acceptance tests yet for `adaptation→decision` and `project→requirement`; spec wording `knowledge→Fact` does not match the current ontology (there is no `fact` entity type in `service.py` / `_graduation_prompt.py`) and should be reconciled to an actual V1 type such as `parameter` or another ontology-defined entity. |
|
||||||
|
| F-8 | Every active entity has `source_refs`; Q-017 returns ≥1 row for every active entity | 🟡 partial | `Entity.source_refs` field exists; Q-017 (`evidence_chain`) exists. **Gap**: is provenance enforced at write time (not NULL), or just encouraged? Per spec it must be mandatory |
|
||||||
|
|
||||||
|
### Quality (Q-1 through Q-6)
|
||||||
|
|
||||||
|
| ID | Criterion | Status | Evidence |
|
||||||
|
|----|-----------|--------|----------|
|
||||||
|
| Q-1 | All pre-V1 tests still pass | ✅ presumed | 533 tests passing per DEV-LEDGER line 12 |
|
||||||
|
| Q-2 | Each F criterion has happy-path + error-path test, <10s each, <30s total | 🟡 partial | 16 + 15 + 15 + 12 = 58 tests in engineering/queries/v1-phase5/patch files. Need to verify coverage of each F criterion one-for-one |
|
||||||
|
| Q-3 | Conflict invariants enforced by tests (contradictory imports produce conflict, can't promote both, flag-never-block) | 🟡 partial | Tests likely exist in `test_engineering_v1_phase5.py` — verify explicit coverage of the three invariants |
|
||||||
|
| Q-4 | Trust hierarchy enforced by tests (candidates never in context, active-only reinforcement, no auto-project-state writes) | 🟡 partial | Phase 9 Commit B covered the memory side; verify entity side has equivalent tests |
|
||||||
|
| Q-5 | Mirror has golden-file test, deterministic output | ❌ missing | No golden file seen; mirror output reads wall-clock time inside `_footer()` (`mirror.py:320-327`). Determinism should come from injecting the regenerated timestamp/checksum as inputs to the renderer and pinning them in the golden-file test, not from calling `datetime.now()` inside render code |
|
||||||
|
| Q-6 | Killer correctness queries pass against seeded real-ish data (5 seed cases per Q-006/Q-009/Q-011) | ❌ likely missing | No fixture file named for this seen. The three queries exist but there's no evidence of the single integration test described in Q-6 |
|
||||||
|
|
||||||
|
### Operational (O-1 through O-5)
|
||||||
|
|
||||||
|
| ID | Criterion | Status | Evidence |
|
||||||
|
|----|-----------|--------|----------|
|
||||||
|
| O-1 | Schema migration additive, idempotent, tested against fresh + prod-copy DB | 🟡 presumed | `_apply_migrations` pattern is in use per CLAUDE.md sessions; tables exist. Need one confirmation run against a Dalidou backup copy |
|
||||||
|
| O-2 | Backup includes new tables; full restore drill passes; post-restore Q-001 works | ❌ not done | No evidence a restore drill has been run on V1 entity state. `docs/backup-restore-procedure.md` exists but drill is an explicit V1 prerequisite |
|
||||||
|
| O-3 | Performance bounds: write <100ms p99, query <500ms p99 at 1000 entities, mirror <5s per project | 🟡 unmeasured | 35 entities in system — bounds unmeasured at scale. Spec says "sanity-checked, not benchmarked", so this is a one-off manual check |
|
||||||
|
| O-4 | No new manual ops burden | 🟡 | Mirror regen auto-triggers not wired yet (see F-6 gap) — they must be wired for O-4 to pass |
|
||||||
|
| O-5 | Phase 9 reflection loop unchanged for identity/preference/episodic | ✅ presumed | `memory-vs-entities.md` says these three types don't interact with engineering layer. No recent change to memory extractor for these types |
|
||||||
|
|
||||||
|
### Documentation (D-1 through D-4)
|
||||||
|
|
||||||
|
| ID | Criterion | Status | Evidence |
|
||||||
|
|----|-----------|--------|----------|
|
||||||
|
| D-1 | 12 per-entity-type spec docs under `docs/architecture/entities/` | ❌ missing | No `docs/architecture/entities/` folder |
|
||||||
|
| D-2 | `kb-cad-export-schema.md` + `kb-fem-export-schema.md` | ❌ missing | No such files in `docs/architecture/` |
|
||||||
|
| D-3 | `docs/v1-release-notes.md` | ❌ missing | Not written yet (appropriately — it's written when V1 is done) |
|
||||||
|
| D-4 | `master-plan-status.md` + `current-state.md` updated with V1 completion | ❌ not yet | `master-plan-status.md:179` still has V1 under **Next** |
|
||||||
|
|
||||||
|
### Summary (revised per Codex 2026-04-22 per-file audit)
|
||||||
|
|
||||||
|
- **Functional:** 0/8 ✅, 7/8 🟡 partial (F-1 downgraded from ✅ — two header fields missing; F-2 through F-7 partial), 1/8 ❌ missing (F-3 ingest endpoints) → the entity layer shape is real but not yet spec-clean; write-time invariants come first, then everything builds on stable invariants
|
||||||
|
- **F-2 detail:** 9 of 20 v1-required queries done, 1 partial (Q-001 needs subsystem-scoped variant), 10 missing
|
||||||
|
- **F-5 detail:** generic `conflicts` + `conflict_members` schema already present (no migration needed); detector body + routes diverge from spec
|
||||||
|
- **Quality:** 1/6 ✅, 3/6 🟡 partial, 2/6 ❌ missing → golden file + killer-correctness integration test are the two clear gaps
|
||||||
|
- **Operational:** 0/5 ✅ (none fully verified), 3/5 🟡, 1/5 ❌ → backup drill is the one hard blocker here
|
||||||
|
- **Documentation:** 0/4 ✅, 4/4 ❌ → all 4 docs need writing
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Proposed completion order (revised post-Codex review)
|
||||||
|
|
||||||
|
Seven phases instead of six. The new V1-0 establishes the write-time
|
||||||
|
invariants (provenance enforcement F-8 + synchronous conflict hooks F-5
|
||||||
|
minimal) that every later phase depends on. V1-A becomes a **minimal query
|
||||||
|
slice** that proves the model on one project, not a full catalog closure.
|
||||||
|
Full query catalog closure moves to V1-C. Full F-5 spec compliance (the
|
||||||
|
generic `conflicts`/`conflict_members` slot-key schema) stays in V1-F
|
||||||
|
because that's the final shape, but the *minimal hooks* that fire
|
||||||
|
synchronously on writes land in V1-0.
|
||||||
|
|
||||||
|
Skipped by construction: F-1 core schema (already implemented) and O-5
|
||||||
|
(identity/preference/episodic don't touch the engineering layer).
|
||||||
|
|
||||||
|
### Phase V1-0: Write-time invariants (F-8 + F-5 minimal + F-1 audit)
|
||||||
|
|
||||||
|
**Scope:**
|
||||||
|
- **F-1 remediation (Codex audit 2026-04-22 already completed).** Add
|
||||||
|
the two missing shared-header fields to the `Entity` dataclass
|
||||||
|
(`service.py:67`) and the `entities` table schema:
|
||||||
|
- `extractor_version TEXT` — semver-ish string carrying the extractor
|
||||||
|
module version per `promotion-rules.md:268`. Backfill existing rows
|
||||||
|
with `"0.0.0"` or `NULL` flagged as unknown. Every future
|
||||||
|
write carries the current `EXTRACTOR_VERSION` constant.
|
||||||
|
- `canonical_home TEXT` — which layer is canonical for this concept.
|
||||||
|
For entities, value is always `"entity"`. For future graduation
|
||||||
|
records it may be `"memory"` (frozen pointer). Backfill active
|
||||||
|
rows with `"entity"`.
|
||||||
|
- Additive migration via the existing `_apply_migrations` pattern,
|
||||||
|
idempotent, safe on replay.
|
||||||
|
- Add doc note in `engineering-ontology-v1.md` clarifying that the
|
||||||
|
`project` field IS the `project_id` per spec — "fields equivalent
|
||||||
|
to" wording in the spec allows this, but make it explicit so
|
||||||
|
future readers don't trip on the naming.
|
||||||
|
- **F-8 provenance enforcement.** Add a NOT-NULL invariant at
|
||||||
|
`create_entity` and `promote_entity` that `source_refs` is non-empty
|
||||||
|
OR an explicit `hand_authored=True` flag is set (per
|
||||||
|
`promotion-rules.md:253`). Backfill any existing active entities that
|
||||||
|
fail the invariant — either attach provenance, flag as hand-authored,
|
||||||
|
or invalidate. Every future active entity has provenance by schema,
|
||||||
|
not by discipline.
|
||||||
|
- **F-5 minimal hooks.** Wire synchronous conflict detection into every
|
||||||
|
active-entity write path (`create_entity` with status=active,
|
||||||
|
`promote_entity`, `supersede_entity`). The *detector* can stay in its
|
||||||
|
current per-type form (`_check_component_conflicts`,
|
||||||
|
`_check_requirement_conflicts`); the *hook* must fire on every write.
|
||||||
|
Full generic slot-keyed schema lands in V1-F; the hook shape must be
|
||||||
|
generic enough that V1-F is a detector-body swap, not an API refactor.
|
||||||
|
- **Q-3 "flag never block" test.** The hook must return conflict-id in
|
||||||
|
the response body but never 4xx-block the write. One test per write
|
||||||
|
path demonstrating this.
|
||||||
|
- **Q-4 trust-hierarchy test for candidates.** One test: entity
|
||||||
|
candidates never appear in `/context/build` output. (Full trust tests
|
||||||
|
land in V1-E; this is the one that V1-0 can cover without graduation
|
||||||
|
being ready.)
|
||||||
|
|
||||||
|
**Acceptance:** F-1 ✅ (after `extractor_version` + `canonical_home`
|
||||||
|
land + doc note on `project` naming), F-8 ✅, F-5 hooks ✅, Q-3 ✅,
|
||||||
|
partial Q-4 ✅.
|
||||||
|
|
||||||
|
**Estimated size:** 3 days (two small schema additions + invariant
|
||||||
|
patches + hook wiring + tests; no audit overhead — Codex already did
|
||||||
|
that part).
|
||||||
|
|
||||||
|
**Tests added:** ~10.
|
||||||
|
|
||||||
|
**Why first:** every later phase writes entities. Without F-8 + F-5
|
||||||
|
hooks, V1-A through V1-F can leak invalid state into the store that
|
||||||
|
must then be cleaned up.
|
||||||
|
|
||||||
|
### Phase V1-A: Minimal query slice that proves the model (partial F-2 + Q-6)
|
||||||
|
|
||||||
|
**Scope:**
|
||||||
|
- Pick the **four pillar queries**: Q-001 (subsystem contents),
|
||||||
|
Q-005 (component satisfies requirements), Q-006 (orphan requirements —
|
||||||
|
killer correctness), Q-017 (evidence chain). These exercise structural +
|
||||||
|
intent + killer-correctness + provenance.
|
||||||
|
- **Q-001 needs a shape fix**: Codex's audit confirms the existing
|
||||||
|
`system_map()` returns a project-wide tree, not the spec's
|
||||||
|
subsystem-scoped `GET /entities/Subsystem/<id>?expand=contains`.
|
||||||
|
Add a subsystem-scoped variant (the existing project-wide route stays
|
||||||
|
for Q-004). This is the only shape fix in V1-A; larger query additions
|
||||||
|
move to V1-C.
|
||||||
|
- Q-005, Q-006, Q-017 are already implemented per Codex audit. V1-A
|
||||||
|
verifies them against seeded data; no code changes expected.
|
||||||
|
- Seed p05-interferometer with Q-6 integration data (one satisfying
|
||||||
|
Component + one orphan Requirement + one Decision on flagged
|
||||||
|
Assumption + one supported ValidationClaim + one unsupported
|
||||||
|
ValidationClaim).
|
||||||
|
- All three killer-correctness queries (Q-006, Q-009, Q-011) are
|
||||||
|
**already implemented** per Codex audit. V1-A runs them as a single
|
||||||
|
integration test against the seed data.
|
||||||
|
|
||||||
|
**Acceptance:** Q-001 subsystem-scoped variant + Q-6 integration test.
|
||||||
|
Partial F-2 (remaining 10 missing + 1 partial queries land in V1-C).
|
||||||
|
|
||||||
|
**Estimated size:** 1.5 days (scope shrunk — most pillar queries already
|
||||||
|
work per Codex audit; only Q-001 shape fix + seed data + integration
|
||||||
|
test required).
|
||||||
|
|
||||||
|
**Tests added:** ~4.
|
||||||
|
|
||||||
|
**Why second:** proves the entity layer shape works end-to-end on real
|
||||||
|
data before we start bolting ingest, graduation, or mirror onto it. If
|
||||||
|
the four pillar queries don't work, stopping here is cheap.
|
||||||
|
|
||||||
|
### Phase V1-B: KB-CAD / KB-FEM ingest (F-3) + D-2 schema docs
|
||||||
|
|
||||||
|
**Scope:**
|
||||||
|
- Write `docs/architecture/kb-cad-export-schema.md` and
|
||||||
|
`kb-fem-export-schema.md` (matches D-2).
|
||||||
|
- Implement `POST /ingest/kb-cad/export` and `POST /ingest/kb-fem/export`
|
||||||
|
per `tool-handoff-boundaries.md` sketches. Validator + entity-candidate
|
||||||
|
producer + provenance population **using the F-8 invariant from V1-0**.
|
||||||
|
- Hand-craft one real KB-CAD export for p05-interferometer and
|
||||||
|
round-trip it: export → candidate queue → reviewer promotes → queryable
|
||||||
|
via V1-A's four pillar queries.
|
||||||
|
- Tests: valid export → candidates created; invalid export → 400;
|
||||||
|
duplicate re-export → no duplicate candidates; re-export with changed
|
||||||
|
value → new candidate + conflict row (exercises V1-0's F-5 hook on a
|
||||||
|
real workload).
|
||||||
|
|
||||||
|
**Acceptance:** F-3 ✅, D-2 ✅.
|
||||||
|
|
||||||
|
**Estimated size:** 2 days.
|
||||||
|
|
||||||
|
**Tests added:** ~8.
|
||||||
|
|
||||||
|
**Why third:** ingest is the first real stress test of the V1-0
|
||||||
|
invariants. A re-import that creates a conflict must trigger the V1-0
|
||||||
|
hook; if it doesn't, V1-0 is incomplete and we catch it before going
|
||||||
|
further.
|
||||||
|
|
||||||
|
### Phase V1-C: Close the rest of the query catalog (remaining F-2)
|
||||||
|
|
||||||
|
**Scope:** close the 10 missing queries per Codex's audit. Already-done
|
||||||
|
queries (Q-004/Q-005/Q-006/Q-008/Q-009/Q-011/Q-013/Q-016/Q-017) are
|
||||||
|
verified but not rewritten.
|
||||||
|
- Q-002 (component → parents, inverse of CONTAINS)
|
||||||
|
- Q-003 (subsystem interfaces, Interface as simple string label)
|
||||||
|
- Q-007 (component → constraints via CONSTRAINED_BY)
|
||||||
|
- Q-010 (ValidationClaim → supporting results + AnalysisModel trace)
|
||||||
|
- Q-012 (conflicting results on same claim — exercises V1-0's F-5 hook)
|
||||||
|
- Q-014 (decision log ordered + superseded chain)
|
||||||
|
- Q-018 (`include=superseded` for supersession chains)
|
||||||
|
- Q-019 (Material → components, derived from Component.material field
|
||||||
|
per `engineering-query-catalog.md:266`, no edge needed)
|
||||||
|
- Q-020 (project overview mirror route) — deferred to V1-D where the
|
||||||
|
mirror lands in full.
|
||||||
|
|
||||||
|
**Acceptance:** F-2 ✅ (all 19 of 20 v1-required queries; Q-020 in V1-D).
|
||||||
|
|
||||||
|
**Estimated size:** 2 days (eight new query functions + routes +
|
||||||
|
per-query happy-path tests).
|
||||||
|
|
||||||
|
**Tests added:** ~12.
|
||||||
|
|
||||||
|
**Why fourth:** with the model proven (V1-A) and ingest exercising the
|
||||||
|
write invariants (V1-B), filling in the remaining queries is mechanical.
|
||||||
|
They all sit on top of the same entity store and V1-0 invariants.
|
||||||
|
|
||||||
|
### Phase V1-D: Full Mirror surface (F-6) + determinism golden file (Q-5) + Q-020
|
||||||
|
|
||||||
|
**Scope:**
|
||||||
|
- Split the single `/projects/{project_name}/mirror` route into the three
|
||||||
|
spec routes: `/mirror/{project}/overview` (= Q-020),
|
||||||
|
`/mirror/{project}/decisions`, `/mirror/{project}/subsystems/{subsystem}`.
|
||||||
|
- Add `POST /mirror/{project}/regenerate`.
|
||||||
|
- Move generated files to `/srv/storage/atocore/data/mirror/{project}/`.
|
||||||
|
- **Deterministic output:** inject regenerated timestamp + checksum as
|
||||||
|
renderer inputs (pinned by golden tests), sort every iteration, and
|
||||||
|
remove `dict` / database ordering dependencies. The renderer should
|
||||||
|
not call wall-clock time directly.
|
||||||
|
- `⚠ disputed` markers inline wherever an open conflict touches a
|
||||||
|
rendered field (uses V1-0's F-5 hook output).
|
||||||
|
- `(curated)` annotations where project_state overrides entity state.
|
||||||
|
- Regeneration triggers: synchronous on regenerate, debounced async on
|
||||||
|
entity write (30s window), daily scheduled refresh via existing
|
||||||
|
nightly cron (one new cron line, not a new queue).
|
||||||
|
- `mirror_regeneration_failures` table.
|
||||||
|
- Golden-file test: fixture project state → render → bytes equal.
|
||||||
|
|
||||||
|
**Acceptance:** F-6 ✅, Q-5 ✅, Q-020 ✅, O-4 moves toward ✅.
|
||||||
|
|
||||||
|
**Estimated size:** 3–4 days.
|
||||||
|
|
||||||
|
**Tests added:** ~15.
|
||||||
|
|
||||||
|
**Why fifth:** mirror is a derived consumer. It cannot be correct
|
||||||
|
before the entity store + queries + conflict hooks are correct. It
|
||||||
|
lands after everything it depends on is stable.
|
||||||
|
|
||||||
|
### Phase V1-E: Memory→entity graduation end-to-end (F-7) + remaining Q-4
|
||||||
|
|
||||||
|
**Scope:**
|
||||||
|
- Verify and close F-7 spec gaps:
|
||||||
|
- Add the missing direct `POST /memory/{id}/graduate` route, reusing the
|
||||||
|
same prompt/parser as the batch graduation path.
|
||||||
|
- Keep `/admin/graduation/request` as the bulk lane; direct route is the
|
||||||
|
per-memory acceptance surface.
|
||||||
|
- Preserve current behavior where promote marks source memories
|
||||||
|
`status="graduated"` and sets `graduated_to_entity_id`.
|
||||||
|
- Flow tested for `adaptation` → Decision and `project` → Requirement.
|
||||||
|
- Reconcile the spec's `knowledge` → Fact wording with the actual V1
|
||||||
|
ontology (no `fact` entity type exists today). Prefer doc alignment to
|
||||||
|
an existing typed entity such as `parameter`, rather than adding a vague
|
||||||
|
catch-all `Fact` type late in V1.
|
||||||
|
- Schema is mostly already in place: `graduated` status exists in memory
|
||||||
|
service, `graduated_to_entity_id` column + index exist, and promote
|
||||||
|
preserves the original memory. Remaining work is route surface,
|
||||||
|
ontology/spec reconciliation, and targeted end-to-end tests.
|
||||||
|
- **Q-4 full trust-hierarchy tests**: no auto-write to project_state
|
||||||
|
from any promote path; active-only reinforcement for entities; etc.
|
||||||
|
(The entity-candidates-excluded-from-context test shipped in V1-0.)
|
||||||
|
|
||||||
|
**Acceptance:** F-7 ✅, Q-4 ✅.
|
||||||
|
|
||||||
|
**Estimated size:** 3–4 days.
|
||||||
|
|
||||||
|
**Tests added:** ~8.
|
||||||
|
|
||||||
|
**Why sixth:** graduation touches memory-layer semantics (adds a
|
||||||
|
`graduated` status, flows memory→entity, requires memory-module changes).
|
||||||
|
Doing it after the entity layer is fully invariant-safe + query-complete
|
||||||
|
+ mirror-derived means the memory side only has to deal with one shape:
|
||||||
|
a stable, tested entity layer.
|
||||||
|
|
||||||
|
### Phase V1-F: Full F-5 spec compliance + O-1/O-2/O-3 + D-1/D-3/D-4
|
||||||
|
|
||||||
|
**Scope:**
|
||||||
|
- **F-5 full spec compliance** (Codex 2026-04-22 audit already confirmed
|
||||||
|
the gap shape — schema is spec-compliant, divergence is in detector +
|
||||||
|
routes only).
|
||||||
|
- **Detector generalization.** Replace the per-type dispatch at
|
||||||
|
`conflicts.py:36` (`_check_component_conflicts`,
|
||||||
|
`_check_requirement_conflicts`) with a slot-key-driven generic
|
||||||
|
detector that reads the per-entity-type conflict slot from a
|
||||||
|
registry and queries the already-generic `conflicts` +
|
||||||
|
`conflict_members` tables. The V1-0 hook shape was chosen to make
|
||||||
|
this a detector-body swap, not an API change.
|
||||||
|
- **Route alignment.** Add `/conflicts/*` routes as the canonical
|
||||||
|
surface per `conflict-model.md:187`. Keep `/admin/conflicts/*` as
|
||||||
|
aliases for one release, deprecate in D-3 release notes, remove
|
||||||
|
in V1.1.
|
||||||
|
- **No schema migration needed** (the tables at `database.py:190`
|
||||||
|
already match the spec).
|
||||||
|
- **O-1:** Run the full migration against a Dalidou backup copy.
|
||||||
|
Confirm additive, idempotent, safe to run twice.
|
||||||
|
- **O-2:** Run a full restore drill on the test project per
|
||||||
|
`docs/backup-restore-procedure.md`. Post-restore, Q-001 returns
|
||||||
|
correct shape. `POST /admin/backup` snapshot includes the new tables.
|
||||||
|
- **O-3:** Manual sanity-check of the three performance bounds.
|
||||||
|
- **D-1:** Write 12 short spec docs under `docs/architecture/entities/`
|
||||||
|
(one per V1 entity type).
|
||||||
|
- **D-3:** Write `docs/v1-release-notes.md`.
|
||||||
|
- **D-4:** Update `master-plan-status.md` and `current-state.md` —
|
||||||
|
move engineering V1 from **Next** to **What Is Real Today**.
|
||||||
|
|
||||||
|
**Acceptance:** F-5 ✅, O-1 ✅, O-2 ✅, O-3 ✅, D-1 ✅, D-3 ✅, D-4 ✅ →
|
||||||
|
**V1 is done.**
|
||||||
|
|
||||||
|
**Estimated size:** 3 days (F-5 migration if needed is the main unknown;
|
||||||
|
D-1 entity docs at ~30 min each ≈ 6 hours; verification is fast).
|
||||||
|
|
||||||
|
**Tests added:** ~6 (F-5 spec-shape tests; verification adds no automated
|
||||||
|
tests).
|
||||||
|
|
||||||
|
### Total (revised after Codex 2026-04-22 audit)
|
||||||
|
|
||||||
|
- Phase budgets: V1-0 (3) + V1-A (1.5) + V1-B (2) + V1-C (2) + V1-D (3-4)
|
||||||
|
+ V1-E (3-4) + V1-F (3) ≈ **17.5–19.5 days of focused work**. This is a
|
||||||
|
realistic engineering-effort estimate, but a single-operator calendar
|
||||||
|
plan should still carry context-switch / soak / review buffer on top.
|
||||||
|
- Adds roughly **60 tests** (533 → ~593).
|
||||||
|
- Branch strategy: one branch per phase (V1-0 → V1-F), each squash-merged
|
||||||
|
to main after Codex review. Phases sequential because each builds on
|
||||||
|
the previous. **V1-0 is a hard prerequisite for all later phases** —
|
||||||
|
nothing starts until V1-0 lands.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Sequencing with the `master-plan-status.md` Now list
|
||||||
|
|
||||||
|
The **Now** list from master-plan-status.md:159-169 is:
|
||||||
|
|
||||||
|
1. Observe the enhanced pipeline (1 week soak — first F4 confidence decay
|
||||||
|
run was 2026-04-19 per Trusted State, so soak window ends ~2026-04-26)
|
||||||
|
2. Knowledge density — batch extract over 234 interactions, target 100+
|
||||||
|
active memories (currently 84)
|
||||||
|
3. Multi-model triage (Phase 11 entry)
|
||||||
|
4. Fix p04-constraints harness failure
|
||||||
|
|
||||||
|
**Principle (revised per Codex review):** V1 work and the Now list are
|
||||||
|
**less disjoint than the first draft claimed**. Real collision points:
|
||||||
|
|
||||||
|
| V1 phase | Collides with Now list at |
|
||||||
|
|---|---|
|
||||||
|
| V1-0 provenance enforcement | memory extractor write path if it shares helper functions; context assembly for the Q-4 partial trust test |
|
||||||
|
| V1-0 F-5 hooks | any write path that creates active rows (limited collision; entity writes are separate from memory writes) |
|
||||||
|
| V1-B KB-CAD/FEM ingest | none on the Now list, but adds an ingest surface that becomes operational burden (ties to O-4 "no new manual ops") |
|
||||||
|
| V1-D mirror regen triggers | scheduling / ops behavior that intersects with "boring and dependable" gate — mirror regen failures become an observable that the pipeline soak must accommodate |
|
||||||
|
| V1-E graduation | memory module (new `graduated` status, memory→entity flow); direct collision with memory extractor + triage |
|
||||||
|
| V1-F F-5 migration | conflicts.py touches the write path shared with memory promotion |
|
||||||
|
|
||||||
|
**Recommended schedule (revised):**
|
||||||
|
|
||||||
|
- **This week (2026-04-22 to 2026-04-26):** Pipeline soak continues.
|
||||||
|
Density batch-extract continues. V1 work **waits** — V1-0 would start
|
||||||
|
touching write paths, which is explicitly something we should not do
|
||||||
|
during a soak window. Density target (100+ active memories) and the
|
||||||
|
pipeline soak complete first.
|
||||||
|
- **Week of 2026-04-27:** If soak is clean and density reached, V1-0
|
||||||
|
starts. V1-0 is a hard prerequisite and cannot be skipped or parallelized.
|
||||||
|
- **Weeks of 2026-05-04 and 2026-05-11:** V1-A through V1-D in order.
|
||||||
|
Multi-model triage work (Now list item 3) continues in parallel only
|
||||||
|
if its touch-surface is triage-path-only (memory side). Any memory
|
||||||
|
extractor change pauses V1-E.
|
||||||
|
- **Week of 2026-05-18 approx:** V1-E (graduation). **This phase must
|
||||||
|
not run in parallel with memory extractor changes** — it directly
|
||||||
|
modifies memory module semantics. Multi-model triage should be settled
|
||||||
|
before V1-E starts.
|
||||||
|
- **Week of 2026-05-25:** V1-F.
|
||||||
|
- **End date target:** ~2026-06-01, four weeks later than the first
|
||||||
|
draft's 2026-05-18 soft target. The shift is deliberate — the first
|
||||||
|
draft's "parallel / disjoint" claim understated the real collisions.
|
||||||
|
|
||||||
|
**Pause points (explicit):**
|
||||||
|
|
||||||
|
- Any Now-list item that regresses the pipeline → V1 pauses immediately.
|
||||||
|
- Memory extractor changes in flight → V1-E pauses until they land and
|
||||||
|
soak.
|
||||||
|
- p04-constraints fix requires retrieval ranking changes → V1 does not
|
||||||
|
pause (retrieval is genuinely disjoint from entities).
|
||||||
|
- Multi-model triage work touching the entity extractor path (if one
|
||||||
|
gets prototyped) → V1-0 pauses until the triage decision settles.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Test project
|
||||||
|
|
||||||
|
Per `engineering-v1-acceptance.md:379`, the recommended test bed is
|
||||||
|
**p05-interferometer** — "the optical/structural domain has the cleanest
|
||||||
|
entity model". I agree. Every F-2, F-3, F-6 criterion asserts against this
|
||||||
|
project.
|
||||||
|
|
||||||
|
p06-polisher is the backup test bed if p05 turns out to have data gaps
|
||||||
|
(polisher suite is actively worked and has more content).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## What V1 completion does NOT include
|
||||||
|
|
||||||
|
Per the negative list in `engineering-v1-acceptance.md:351-373`, all of the
|
||||||
|
following are **explicitly out of scope** for this plan:
|
||||||
|
|
||||||
|
- LLM extractor for entities (rule-based is V1)
|
||||||
|
- Auto-promotion of candidates (human-only in V1)
|
||||||
|
- Write-back to KB-CAD / KB-FEM
|
||||||
|
- Multi-user auth
|
||||||
|
- Real-time UI (API + Mirror markdown only)
|
||||||
|
- Cross-project rollups
|
||||||
|
- Time-travel queries (Q-015 stays stretch)
|
||||||
|
- Nightly conflict sweep (synchronous only)
|
||||||
|
- Incremental Chroma snapshots
|
||||||
|
- Retention cleanup script
|
||||||
|
- Backup encryption
|
||||||
|
- Off-Dalidou backup target (already exists at clawdbot per ledger, but
|
||||||
|
not a V1 criterion)
|
||||||
|
- **Async job queue / minions pattern** (the rejected plan's centerpiece —
|
||||||
|
explicitly deferred to post-V1 per the negative list)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Open questions for Codex (post-second-round revision)
|
||||||
|
|
||||||
|
Three of the original eight questions (F-1 field audit, F-2 per-query
|
||||||
|
audit, F-5 schema divergence) were answered by Codex's 2026-04-22 audit
|
||||||
|
and folded into the plan. One open question remains; the rest are now
|
||||||
|
resolved in-plan:
|
||||||
|
|
||||||
|
1. **Parallel schedule vs Now list.** The first-round review correctly
|
||||||
|
softened this from "fully parallel" to "less disjoint than claimed".
|
||||||
|
Is the revised collision table + pause-points section enough, or
|
||||||
|
should specific Now-list items gate specific V1 phases more strictly?
|
||||||
|
|
||||||
|
2. **F-7 graduation gap depth.** Resolved by Codex audit. The schema and
|
||||||
|
preserve-original-memory hook are already in place, so V1-E is not a
|
||||||
|
greenfield build. But the direct `/memory/{id}/graduate` route and the
|
||||||
|
ontology/spec mismatch around `knowledge` → `Fact` are still open, so
|
||||||
|
V1-E is closer to **3–4 days** than 2.
|
||||||
|
|
||||||
|
3. **Mirror determinism — where does `now` go?** Resolved. Keep the
|
||||||
|
regenerated timestamp in the rendered output if desired, but pass it
|
||||||
|
into the renderer as an input value. Golden-file tests pin that input;
|
||||||
|
render code must not read the clock directly.
|
||||||
|
|
||||||
|
4. **`project` field naming.** Resolved. Keep the existing `project`
|
||||||
|
field; add the explicit doc note that it is the project identifier for
|
||||||
|
V1 acceptance purposes. No storage rename needed.
|
||||||
|
|
||||||
|
5. **Velocity calibration.** Resolved. **17.5–19.5 focused days** is a
|
||||||
|
fair engineering-effort estimate after the F-7 audit. For an actual
|
||||||
|
operator schedule, keep additional buffer for context switching, soak,
|
||||||
|
and review rounds.
|
||||||
|
|
||||||
|
6. **Minions/queue as V2 item in D-3.** Resolved. Do not name the
|
||||||
|
rejected "Minions" plan in V1 release notes. If D-3 includes a future
|
||||||
|
work section, refer to it neutrally as "queued background processing /
|
||||||
|
async workers" rather than canonizing a V2 codename before V2 is
|
||||||
|
designed.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Risks
|
||||||
|
|
||||||
|
| Risk | Mitigation |
|
||||||
|
|---|---|
|
||||||
|
| V1 work slows the Now list | V1 pauses on any Now-list blocker. Codex veto on any V1 PR that touches memory extractor, retrieval ranking, or triage paths |
|
||||||
|
| F-5 detector generalization harder than estimated | Codex audit confirmed schema is already spec-compliant; only detector body + routes need work. If detector generalization still slips, keep per-type detectors and document as a V1.1 cleanup (detection correctness is unaffected, only code organization) |
|
||||||
|
| Mirror determinism regresses existing mirror output | Keep `/projects/{project_name}/mirror` alias returning the current shape; new `/mirror/{project}/overview` is the spec-compliant one. Deprecate old in V1 release notes |
|
||||||
|
| Golden file churn as templates evolve | Standard workflow: updating a golden file is a normal part of template work, documented in V1-C commit message |
|
||||||
|
| Backup drill on Dalidou is disruptive | Run against a clone of the Dalidou DB at a safe hour; no production drill required for V1 acceptance |
|
||||||
|
| p05-interferometer data gaps | Fall back to p06-polisher per this plan's test-project section |
|
||||||
|
| Scope creep during V1-A query audit | Any query that isn't in the v1-required set (Q-021 onward) is out of scope, period |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## What this plan is **for**
|
||||||
|
|
||||||
|
1. A checklist Claude can follow to close V1.
|
||||||
|
2. A review target for Codex — every phase has explicit acceptance
|
||||||
|
criteria tied to the acceptance doc.
|
||||||
|
3. A communication artifact for Antoine — "here's what's left, here's why,
|
||||||
|
here's the order, here's the risk."
|
||||||
|
|
||||||
|
## What this plan is **not**
|
||||||
|
|
||||||
|
1. Not a commitment to start tomorrow. Pipeline soak + density density
|
||||||
|
come first in parallel; V1-A can start this week only because it's
|
||||||
|
zero-risk additive work.
|
||||||
|
2. Not a rewrite. Every phase builds on existing code.
|
||||||
|
3. Not an ontology debate. The ontology is fixed in
|
||||||
|
`engineering-ontology-v1.md`. Any desire to change it is V2 material.
|
||||||
|
|
||||||
|
## Workspace note (for Codex audit)
|
||||||
|
|
||||||
|
Codex's first-round review (2026-04-22) flagged that
|
||||||
|
`docs/plans/engineering-v1-completion-plan.md` and `DEV-LEDGER.md` were
|
||||||
|
**not visible** in the Playground workspace they were running against,
|
||||||
|
and that `src/atocore/engineering/` appeared empty there.
|
||||||
|
|
||||||
|
The canonical dev workspace for AtoCore is the Windows path
|
||||||
|
`C:\Users\antoi\ATOCore` (per `CLAUDE.md`). The engineering layer code
|
||||||
|
(`src/atocore/engineering/service.py`, `queries.py`, `conflicts.py`,
|
||||||
|
`mirror.py`, `_graduation_prompt.py`, `wiki.py`, `triage_ui.py`) exists
|
||||||
|
there and is what the recent commits (e147ab2, b94f9df, 081c058, 069d155,
|
||||||
|
b1a3dd0) touched. The Windows working tree is what this plan was written
|
||||||
|
against.
|
||||||
|
|
||||||
|
Before the file-level audit:
|
||||||
|
|
||||||
|
1. Confirm which branch / SHA Codex is reviewing. The Windows working
|
||||||
|
tree has uncommitted changes to this plan + DEV-LEDGER as of
|
||||||
|
2026-04-22; commit will be made only after Antoine approves sync.
|
||||||
|
2. If Codex is reviewing `ATOCore-clean` or a Playground snapshot, that
|
||||||
|
tree may lag the canonical dev tree. Sync or re-clone from the
|
||||||
|
Windows working tree / current `origin/main` before per-file audit.
|
||||||
|
3. The three visible-to-Codex file paths for this plan are:
|
||||||
|
- `docs/plans/engineering-v1-completion-plan.md` (this file)
|
||||||
|
- `docs/decisions/2026-04-22-gbrain-plan-rejection.md` (prior decision)
|
||||||
|
- `DEV-LEDGER.md` (Recent Decisions + Session Log entries 2026-04-22)
|
||||||
|
|
||||||
|
## References
|
||||||
|
|
||||||
|
- `docs/architecture/engineering-ontology-v1.md`
|
||||||
|
- `docs/architecture/engineering-query-catalog.md`
|
||||||
|
- `docs/architecture/memory-vs-entities.md`
|
||||||
|
- `docs/architecture/engineering-v1-acceptance.md`
|
||||||
|
- `docs/architecture/promotion-rules.md`
|
||||||
|
- `docs/architecture/conflict-model.md`
|
||||||
|
- `docs/architecture/human-mirror-rules.md`
|
||||||
|
- `docs/architecture/tool-handoff-boundaries.md`
|
||||||
|
- `docs/master-plan-status.md` (Now/Next/Later list)
|
||||||
|
- `docs/decisions/2026-04-22-gbrain-plan-rejection.md` (the rejected plan)
|
||||||
|
- `src/atocore/engineering/service.py` (current V1 entity service)
|
||||||
|
- `src/atocore/engineering/queries.py` (current V1 query implementations)
|
||||||
|
- `src/atocore/engineering/conflicts.py` (current conflicts module)
|
||||||
|
- `src/atocore/engineering/mirror.py` (current mirror module)
|
||||||
161
docs/plans/v1-resume-state.md
Normal file
161
docs/plans/v1-resume-state.md
Normal file
@@ -0,0 +1,161 @@
|
|||||||
|
# V1 Completion — Resume State
|
||||||
|
|
||||||
|
**Last updated:** 2026-04-22 (after V1-0 landed + R14 branch pushed)
|
||||||
|
**Purpose:** single-page "you are here" so any future session can pick up
|
||||||
|
the V1 completion sprint without re-reading the full plan history.
|
||||||
|
|
||||||
|
## State of play
|
||||||
|
|
||||||
|
- **V1-0 is DONE.** Merged to main as `2712c5d`, deployed to Dalidou,
|
||||||
|
prod backfill ran cleanly (31 legacy entities flagged
|
||||||
|
`hand_authored=1`, zero violations remaining).
|
||||||
|
- **R14 is on a branch.** `claude/r14-promote-400` at `3888db9` —
|
||||||
|
HTTP promote route returns 400 instead of 500 on V1-0 `ValueError`.
|
||||||
|
Pending Codex review + squash-merge. Non-blocking for V1-A.
|
||||||
|
- **V1-A is next but GATED.** Doesn't start until both gates clear.
|
||||||
|
|
||||||
|
## Start-gates for V1-A
|
||||||
|
|
||||||
|
| Gate | Condition | Status as of 2026-04-22 |
|
||||||
|
|---|---|---|
|
||||||
|
| Soak | Four clean nightly cycles since F4 confidence-decay first real run 2026-04-19 | Day 3 of 4 — expected clear around **2026-04-26** |
|
||||||
|
| Density | 100+ active memories | 84 active as of last ledger update — need +16. Lever: `scripts/batch_llm_extract_live.py` against 234-interaction backlog |
|
||||||
|
|
||||||
|
**When both are green, start V1-A.** If only one is green, hold.
|
||||||
|
|
||||||
|
## Pre-flight checklist when resuming
|
||||||
|
|
||||||
|
Before opening the V1-A branch, run through this in order:
|
||||||
|
|
||||||
|
1. `git checkout main && git pull` — make sure you're at the tip
|
||||||
|
2. Check `DEV-LEDGER.md` **Orientation** for current `live_sha`, `test_count`, `active_memories`
|
||||||
|
3. Check `/health` on Dalidou returns the same `build_sha` as Orientation
|
||||||
|
4. Check the dashboard for pipeline health: http://dalidou:8100/admin/dashboard
|
||||||
|
5. Confirm R14 branch status — either merged or explicitly deferred
|
||||||
|
6. Re-read the two core plan docs:
|
||||||
|
- `docs/plans/engineering-v1-completion-plan.md` — the full 7-phase plan
|
||||||
|
- `docs/architecture/engineering-v1-acceptance.md` — the acceptance contract
|
||||||
|
7. Skim the relevant spec docs for the phase you're about to start:
|
||||||
|
- V1-A: `engineering-query-catalog.md` (Q-001 + Q-006/Q-009/Q-011 killer queries)
|
||||||
|
- V1-B: `tool-handoff-boundaries.md` (KB-CAD/KB-FEM export shapes)
|
||||||
|
- V1-C: `engineering-query-catalog.md` (all remaining v1-required queries)
|
||||||
|
- V1-D: `human-mirror-rules.md` (mirror spec end-to-end)
|
||||||
|
- V1-E: `memory-vs-entities.md` (graduation flow)
|
||||||
|
- V1-F: `conflict-model.md` (generic slot-key detector)
|
||||||
|
|
||||||
|
## What V1-A looks like when started
|
||||||
|
|
||||||
|
**Branch:** `claude/v1-a-pillar-queries`
|
||||||
|
|
||||||
|
**Scope (~1.5 days):**
|
||||||
|
- **Q-001 shape fix.** Add a subsystem-scoped variant of `system_map()`
|
||||||
|
matching `GET /entities/Subsystem/<id>?expand=contains` per
|
||||||
|
`engineering-query-catalog.md:71`. The project-wide version stays
|
||||||
|
(it serves Q-004).
|
||||||
|
- **Q-6 integration test.** Seed p05-interferometer with five cases:
|
||||||
|
1 satisfying Component, 1 orphan Requirement, 1 Decision on flagged
|
||||||
|
Assumption, 1 supported ValidationClaim, 1 unsupported ValidationClaim.
|
||||||
|
One test asserting Q-006 / Q-009 / Q-011 return exactly the expected
|
||||||
|
members.
|
||||||
|
- The four "pillar" queries (Q-001, Q-005, Q-006, Q-017) already work
|
||||||
|
per Codex's 2026-04-22 audit. V1-A does NOT re-implement them —
|
||||||
|
V1-A verifies them on seeded data.
|
||||||
|
|
||||||
|
**Acceptance:** Q-001 subsystem-scoped variant + Q-6 integration test both
|
||||||
|
green. F-2 moves from 🟡 partial to slightly-less-partial.
|
||||||
|
|
||||||
|
**Estimated tests added:** ~4 (not ~12 — V1-A scope shrank after Codex
|
||||||
|
confirmed most queries already work).
|
||||||
|
|
||||||
|
## Map of the remaining phases
|
||||||
|
|
||||||
|
```
|
||||||
|
V1-0 ✅ write-time invariants landed 2026-04-22 (2712c5d)
|
||||||
|
↓
|
||||||
|
V1-A 🟡 minimum query slice gated on soak + density (~1.5d when started)
|
||||||
|
↓
|
||||||
|
V1-B KB-CAD/KB-FEM ingest + D-2 ~2d
|
||||||
|
↓
|
||||||
|
V1-C close 8 remaining queries ~2d
|
||||||
|
↓
|
||||||
|
V1-D full mirror + determinism ~3-4d (biggest phase)
|
||||||
|
↓
|
||||||
|
V1-E graduation + trust tests ~3-4d (pauses for multi-model triage)
|
||||||
|
↓
|
||||||
|
V1-F F-5 generalization + ops + docs ~3d — V1 done
|
||||||
|
```
|
||||||
|
|
||||||
|
## Parallel work that can run WITHOUT touching V1
|
||||||
|
|
||||||
|
These are genuinely disjoint surfaces; pick any of them during the gate
|
||||||
|
pause or as scheduling allows:
|
||||||
|
|
||||||
|
- **Density batch-extract** — *required* to unblock V1-A. Not optional.
|
||||||
|
- **p04-constraints harness fix** — retrieval-ranking change, fully
|
||||||
|
disjoint from entities. Safe to do anywhere in the V1 track.
|
||||||
|
- **Multi-model triage (Phase 11 entry)** — memory-side work, disjoint
|
||||||
|
from V1-A/B/C/D. **Pause before V1-E starts** because V1-E touches
|
||||||
|
memory module semantics.
|
||||||
|
|
||||||
|
## What NOT to do
|
||||||
|
|
||||||
|
- Don't start V1-A until both gates are green.
|
||||||
|
- Don't touch the memory extractor write path while V1-E is open.
|
||||||
|
- Don't name the rejected "Minions" plan in any doc — neutral wording
|
||||||
|
only ("queued background processing / async workers") per Codex
|
||||||
|
sign-off.
|
||||||
|
- Don't rename the `project` field to `project_id` — Codex + Antoine
|
||||||
|
agreed it stays as `project`, with a doc note in
|
||||||
|
`engineering-ontology-v1.md` that this IS the project_id per spec.
|
||||||
|
|
||||||
|
## Open review findings
|
||||||
|
|
||||||
|
| id | severity | summary | status |
|
||||||
|
|---|---|---|---|
|
||||||
|
| R14 | P2 | `POST /entities/{id}/promote` returns 500 on V1-0 `ValueError` instead of 400 | fixed on branch `claude/r14-promote-400`, pending Codex review |
|
||||||
|
|
||||||
|
Closed V1-0 findings: P1 "promote path allows provenance-less legacy
|
||||||
|
candidates" (service.py:365-379), P1 "supersede path missing F-5 hook"
|
||||||
|
(service.py:581-591), P2 "`--invalidate-instead` backfill too broad"
|
||||||
|
(v1_0_backfill_provenance.py:52-63). All three patched and approved in
|
||||||
|
the squash-merge to `2712c5d`.
|
||||||
|
|
||||||
|
## How agreement between Claude + Codex has worked so far
|
||||||
|
|
||||||
|
Three review rounds before V1-0 started + three during implementation:
|
||||||
|
|
||||||
|
1. **Rejection round.** Claude drafted a gbrain-inspired "Phase 8
|
||||||
|
Minions + typed edges" plan; Codex rejected as wrong-packaging.
|
||||||
|
Record: `docs/decisions/2026-04-22-gbrain-plan-rejection.md`.
|
||||||
|
2. **Completion-plan rewrite.** Claude rewrote against
|
||||||
|
`engineering-v1-acceptance.md`. Codex first-round review fixed the
|
||||||
|
phase order (provenance-first).
|
||||||
|
3. **Per-file audit.** Codex's second-round audit found F-1 / F-2 /
|
||||||
|
F-5 gaps, all folded in.
|
||||||
|
4. **Sign-off round.** Codex's third-round review resolved the five
|
||||||
|
remaining open questions inline and signed off: *"with those edits,
|
||||||
|
I'd sign off on the five questions."*
|
||||||
|
5. **V1-0 review.** Codex found two P1 gaps (promote re-check missing,
|
||||||
|
supersede hook missing) + one P2 (backfill scope too broad). All
|
||||||
|
three patched. Codex re-ran probes + regression suites, approved,
|
||||||
|
squash-merged.
|
||||||
|
6. **V1-0 deploy + prod backfill.** Codex deployed + ran backfill,
|
||||||
|
logged R14 as P2 residual.
|
||||||
|
|
||||||
|
Protocol has been: Claude writes, Codex audits, human Antoine ratifies.
|
||||||
|
Continue this for V1-A onward.
|
||||||
|
|
||||||
|
## References
|
||||||
|
|
||||||
|
- `docs/plans/engineering-v1-completion-plan.md` — full 7-phase plan
|
||||||
|
- `docs/decisions/2026-04-22-gbrain-plan-rejection.md` — prior rejection
|
||||||
|
- `docs/architecture/engineering-ontology-v1.md` — V1 ontology (18 predicates)
|
||||||
|
- `docs/architecture/engineering-query-catalog.md` — Q-001 through Q-020 spec
|
||||||
|
- `docs/architecture/engineering-v1-acceptance.md` — F/Q/O/D acceptance table
|
||||||
|
- `docs/architecture/promotion-rules.md` — candidate → active flow
|
||||||
|
- `docs/architecture/conflict-model.md` — F-5 spec
|
||||||
|
- `docs/architecture/human-mirror-rules.md` — V1-D spec
|
||||||
|
- `docs/architecture/memory-vs-entities.md` — V1-E spec
|
||||||
|
- `docs/architecture/tool-handoff-boundaries.md` — V1-B KB-CAD/KB-FEM
|
||||||
|
- `docs/master-plan-status.md` — Now / Active / Next / Later
|
||||||
|
- `DEV-LEDGER.md` — Orientation + Open Review Findings + Session Log
|
||||||
167
scripts/v1_0_backfill_provenance.py
Normal file
167
scripts/v1_0_backfill_provenance.py
Normal file
@@ -0,0 +1,167 @@
|
|||||||
|
"""V1-0 one-time backfill: flag existing active entities that have no
|
||||||
|
provenance (empty source_refs) as hand_authored=1 so they stop failing
|
||||||
|
the F-8 invariant.
|
||||||
|
|
||||||
|
Runs against the live AtoCore DB. Idempotent: a second run after the
|
||||||
|
first touches nothing because the flagged rows already have
|
||||||
|
hand_authored=1.
|
||||||
|
|
||||||
|
Per the Engineering V1 Completion Plan (V1-0 scope), the three options
|
||||||
|
for an existing active entity without provenance are:
|
||||||
|
|
||||||
|
1. Attach provenance — impossible without human review, not automatable
|
||||||
|
2. Flag hand-authored — safe, additive, this script's default
|
||||||
|
3. Invalidate — destructive, requires operator sign-off
|
||||||
|
|
||||||
|
This script picks option (2) by default. Add --dry-run to see what
|
||||||
|
would change without writing. Add --invalidate-instead to pick option
|
||||||
|
(3) for all rows (not recommended for first run).
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
python scripts/v1_0_backfill_provenance.py --base-url http://dalidou:8100 --dry-run
|
||||||
|
python scripts/v1_0_backfill_provenance.py --base-url http://dalidou:8100
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import sqlite3
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
|
||||||
|
def run(db_path: Path, dry_run: bool, invalidate_instead: bool) -> int:
|
||||||
|
if not db_path.exists():
|
||||||
|
print(f"ERROR: db not found: {db_path}", file=sys.stderr)
|
||||||
|
return 2
|
||||||
|
|
||||||
|
conn = sqlite3.connect(str(db_path))
|
||||||
|
conn.row_factory = sqlite3.Row
|
||||||
|
|
||||||
|
# Verify the V1-0 migration ran: if hand_authored column is missing
|
||||||
|
# the operator hasn't deployed V1-0 yet, and running this script
|
||||||
|
# would crash. Fail loud rather than attempt the ALTER here.
|
||||||
|
cols = {r["name"] for r in conn.execute("PRAGMA table_info(entities)").fetchall()}
|
||||||
|
if "hand_authored" not in cols:
|
||||||
|
print(
|
||||||
|
"ERROR: entities table lacks the hand_authored column. "
|
||||||
|
"Deploy V1-0 migrations first (init_db + init_engineering_schema).",
|
||||||
|
file=sys.stderr,
|
||||||
|
)
|
||||||
|
return 2
|
||||||
|
|
||||||
|
# Scope differs by mode:
|
||||||
|
# - Default (flag hand_authored=1): safe/additive, applies to active
|
||||||
|
# AND superseded rows so the historical trail is consistent.
|
||||||
|
# - --invalidate-instead: destructive — scope to ACTIVE rows only.
|
||||||
|
# Invalidating already-superseded history would collapse the audit
|
||||||
|
# trail, which the plan's remediation scope never intended
|
||||||
|
# (V1-0 talks about existing active no-provenance entities).
|
||||||
|
if invalidate_instead:
|
||||||
|
scope_sql = "status = 'active'"
|
||||||
|
else:
|
||||||
|
scope_sql = "status IN ('active', 'superseded')"
|
||||||
|
rows = conn.execute(
|
||||||
|
f"SELECT id, entity_type, name, project, status, source_refs, hand_authored "
|
||||||
|
f"FROM entities WHERE {scope_sql} AND hand_authored = 0"
|
||||||
|
).fetchall()
|
||||||
|
|
||||||
|
needs_fix = []
|
||||||
|
for row in rows:
|
||||||
|
refs_raw = row["source_refs"] or "[]"
|
||||||
|
try:
|
||||||
|
refs = json.loads(refs_raw)
|
||||||
|
except Exception:
|
||||||
|
refs = []
|
||||||
|
if not refs:
|
||||||
|
needs_fix.append(row)
|
||||||
|
|
||||||
|
print(f"found {len(needs_fix)} active/superseded entities with no provenance")
|
||||||
|
for row in needs_fix:
|
||||||
|
print(
|
||||||
|
f" - {row['id'][:8]} [{row['entity_type']}] "
|
||||||
|
f"{row['name']!r} project={row['project']!r} status={row['status']}"
|
||||||
|
)
|
||||||
|
|
||||||
|
if dry_run:
|
||||||
|
print("--dry-run: no changes written")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
if not needs_fix:
|
||||||
|
print("nothing to do")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
action = "invalidate" if invalidate_instead else "flag hand_authored=1"
|
||||||
|
print(f"applying: {action}")
|
||||||
|
|
||||||
|
cur = conn.cursor()
|
||||||
|
for row in needs_fix:
|
||||||
|
if invalidate_instead:
|
||||||
|
cur.execute(
|
||||||
|
"UPDATE entities SET status = 'invalid', "
|
||||||
|
"updated_at = CURRENT_TIMESTAMP WHERE id = ?",
|
||||||
|
(row["id"],),
|
||||||
|
)
|
||||||
|
cur.execute(
|
||||||
|
"INSERT INTO memory_audit "
|
||||||
|
"(id, memory_id, action, actor, before_json, after_json, note, entity_kind) "
|
||||||
|
"VALUES (?, ?, 'invalidated', 'v1_0_backfill', ?, ?, ?, 'entity')",
|
||||||
|
(
|
||||||
|
f"v10bf-{row['id'][:8]}-inv",
|
||||||
|
row["id"],
|
||||||
|
json.dumps({"status": row["status"]}),
|
||||||
|
json.dumps({"status": "invalid"}),
|
||||||
|
"V1-0 backfill: invalidated, no provenance",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
cur.execute(
|
||||||
|
"UPDATE entities SET hand_authored = 1, "
|
||||||
|
"updated_at = CURRENT_TIMESTAMP WHERE id = ?",
|
||||||
|
(row["id"],),
|
||||||
|
)
|
||||||
|
cur.execute(
|
||||||
|
"INSERT INTO memory_audit "
|
||||||
|
"(id, memory_id, action, actor, before_json, after_json, note, entity_kind) "
|
||||||
|
"VALUES (?, ?, 'hand_authored_flagged', 'v1_0_backfill', ?, ?, ?, 'entity')",
|
||||||
|
(
|
||||||
|
f"v10bf-{row['id'][:8]}-ha",
|
||||||
|
row["id"],
|
||||||
|
json.dumps({"hand_authored": False}),
|
||||||
|
json.dumps({"hand_authored": True}),
|
||||||
|
"V1-0 backfill: flagged hand_authored since source_refs empty",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
conn.commit()
|
||||||
|
print(f"done: updated {len(needs_fix)} entities")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
parser = argparse.ArgumentParser(description=__doc__)
|
||||||
|
parser.add_argument(
|
||||||
|
"--db",
|
||||||
|
type=Path,
|
||||||
|
default=Path("data/db/atocore.db"),
|
||||||
|
help="Path to the SQLite database (default: data/db/atocore.db)",
|
||||||
|
)
|
||||||
|
parser.add_argument("--dry-run", action="store_true", help="Report only; no writes")
|
||||||
|
parser.add_argument(
|
||||||
|
"--invalidate-instead",
|
||||||
|
action="store_true",
|
||||||
|
help=(
|
||||||
|
"DESTRUCTIVE. Invalidate active rows with no provenance instead "
|
||||||
|
"of flagging them hand_authored. Scoped to status='active' only "
|
||||||
|
"(superseded rows are left alone to preserve audit history). "
|
||||||
|
"Not recommended for first run — start with --dry-run, then "
|
||||||
|
"the default hand_authored flag path."
|
||||||
|
),
|
||||||
|
)
|
||||||
|
args = parser.parse_args()
|
||||||
|
return run(args.db, args.dry_run, args.invalidate_instead)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
@@ -1457,6 +1457,11 @@ class EntityCreateRequest(BaseModel):
|
|||||||
status: str = "active"
|
status: str = "active"
|
||||||
confidence: float = 1.0
|
confidence: float = 1.0
|
||||||
source_refs: list[str] | None = None
|
source_refs: list[str] | None = None
|
||||||
|
# V1-0 provenance enforcement (F-8). Clients must either pass
|
||||||
|
# non-empty source_refs or set hand_authored=true. The service layer
|
||||||
|
# raises ValueError otherwise, surfaced here as 400.
|
||||||
|
hand_authored: bool = False
|
||||||
|
extractor_version: str | None = None
|
||||||
|
|
||||||
|
|
||||||
class EntityPromoteRequest(BaseModel):
|
class EntityPromoteRequest(BaseModel):
|
||||||
@@ -1486,6 +1491,8 @@ def api_create_entity(req: EntityCreateRequest) -> dict:
|
|||||||
confidence=req.confidence,
|
confidence=req.confidence,
|
||||||
source_refs=req.source_refs,
|
source_refs=req.source_refs,
|
||||||
actor="api-http",
|
actor="api-http",
|
||||||
|
hand_authored=req.hand_authored,
|
||||||
|
extractor_version=req.extractor_version,
|
||||||
)
|
)
|
||||||
except ValueError as e:
|
except ValueError as e:
|
||||||
raise HTTPException(status_code=400, detail=str(e))
|
raise HTTPException(status_code=400, detail=str(e))
|
||||||
|
|||||||
@@ -63,6 +63,12 @@ RELATIONSHIP_TYPES = [
|
|||||||
|
|
||||||
ENTITY_STATUSES = ["candidate", "active", "superseded", "invalid"]
|
ENTITY_STATUSES = ["candidate", "active", "superseded", "invalid"]
|
||||||
|
|
||||||
|
# V1-0: extractor version this module writes into new entity rows.
|
||||||
|
# Per promotion-rules.md:268, every candidate must record the version of
|
||||||
|
# the extractor that produced it so later re-evaluation is auditable.
|
||||||
|
# Bump this when extraction logic materially changes.
|
||||||
|
EXTRACTOR_VERSION = "v1.0.0"
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
class Entity:
|
class Entity:
|
||||||
@@ -77,6 +83,10 @@ class Entity:
|
|||||||
source_refs: list[str] = field(default_factory=list)
|
source_refs: list[str] = field(default_factory=list)
|
||||||
created_at: str = ""
|
created_at: str = ""
|
||||||
updated_at: str = ""
|
updated_at: str = ""
|
||||||
|
# V1-0 shared-header fields per engineering-v1-acceptance.md:45.
|
||||||
|
extractor_version: str = ""
|
||||||
|
canonical_home: str = "entity"
|
||||||
|
hand_authored: bool = False
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
@@ -103,10 +113,25 @@ def init_engineering_schema() -> None:
|
|||||||
status TEXT NOT NULL DEFAULT 'active',
|
status TEXT NOT NULL DEFAULT 'active',
|
||||||
confidence REAL NOT NULL DEFAULT 1.0,
|
confidence REAL NOT NULL DEFAULT 1.0,
|
||||||
source_refs TEXT NOT NULL DEFAULT '[]',
|
source_refs TEXT NOT NULL DEFAULT '[]',
|
||||||
|
extractor_version TEXT NOT NULL DEFAULT '',
|
||||||
|
canonical_home TEXT NOT NULL DEFAULT 'entity',
|
||||||
|
hand_authored INTEGER NOT NULL DEFAULT 0,
|
||||||
created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||||
updated_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP
|
updated_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP
|
||||||
)
|
)
|
||||||
""")
|
""")
|
||||||
|
# V1-0 (Engineering V1 completion): the three shared-header fields
|
||||||
|
# per engineering-v1-acceptance.md:45. Idempotent ALTERs for
|
||||||
|
# databases created before V1-0 land these columns without a full
|
||||||
|
# migration. Fresh DBs get them via the CREATE TABLE above; the
|
||||||
|
# ALTERs below are a no-op there.
|
||||||
|
from atocore.models.database import _column_exists # late import; avoids cycle
|
||||||
|
if not _column_exists(conn, "entities", "extractor_version"):
|
||||||
|
conn.execute("ALTER TABLE entities ADD COLUMN extractor_version TEXT DEFAULT ''")
|
||||||
|
if not _column_exists(conn, "entities", "canonical_home"):
|
||||||
|
conn.execute("ALTER TABLE entities ADD COLUMN canonical_home TEXT DEFAULT 'entity'")
|
||||||
|
if not _column_exists(conn, "entities", "hand_authored"):
|
||||||
|
conn.execute("ALTER TABLE entities ADD COLUMN hand_authored INTEGER DEFAULT 0")
|
||||||
conn.execute("""
|
conn.execute("""
|
||||||
CREATE TABLE IF NOT EXISTS relationships (
|
CREATE TABLE IF NOT EXISTS relationships (
|
||||||
id TEXT PRIMARY KEY,
|
id TEXT PRIMARY KEY,
|
||||||
@@ -149,6 +174,8 @@ def create_entity(
|
|||||||
confidence: float = 1.0,
|
confidence: float = 1.0,
|
||||||
source_refs: list[str] | None = None,
|
source_refs: list[str] | None = None,
|
||||||
actor: str = "api",
|
actor: str = "api",
|
||||||
|
hand_authored: bool = False,
|
||||||
|
extractor_version: str | None = None,
|
||||||
) -> Entity:
|
) -> Entity:
|
||||||
if entity_type not in ENTITY_TYPES:
|
if entity_type not in ENTITY_TYPES:
|
||||||
raise ValueError(f"Invalid entity type: {entity_type}. Must be one of {ENTITY_TYPES}")
|
raise ValueError(f"Invalid entity type: {entity_type}. Must be one of {ENTITY_TYPES}")
|
||||||
@@ -157,6 +184,21 @@ def create_entity(
|
|||||||
if not name or not name.strip():
|
if not name or not name.strip():
|
||||||
raise ValueError("Entity name must be non-empty")
|
raise ValueError("Entity name must be non-empty")
|
||||||
|
|
||||||
|
refs = list(source_refs) if source_refs else []
|
||||||
|
|
||||||
|
# V1-0 (F-8 provenance enforcement, engineering-v1-acceptance.md:147):
|
||||||
|
# every new entity row must carry non-empty source_refs OR be explicitly
|
||||||
|
# flagged hand_authored. This is the non-negotiable invariant every
|
||||||
|
# later V1 phase depends on — without it, active entities can escape
|
||||||
|
# into the graph with no traceable origin. Raises at the write seam so
|
||||||
|
# the bug is impossible to introduce silently.
|
||||||
|
if not refs and not hand_authored:
|
||||||
|
raise ValueError(
|
||||||
|
"source_refs required: every entity must carry provenance "
|
||||||
|
"(source_chunk_id / source_interaction_id / kb_cad_export_id / ...) "
|
||||||
|
"or set hand_authored=True to explicitly flag a direct human write"
|
||||||
|
)
|
||||||
|
|
||||||
# Phase 5: enforce project canonicalization contract at the write seam.
|
# Phase 5: enforce project canonicalization contract at the write seam.
|
||||||
# Aliases like "p04" become "p04-gigabit" so downstream reads stay
|
# Aliases like "p04" become "p04-gigabit" so downstream reads stay
|
||||||
# consistent with the registry.
|
# consistent with the registry.
|
||||||
@@ -165,18 +207,22 @@ def create_entity(
|
|||||||
entity_id = str(uuid.uuid4())
|
entity_id = str(uuid.uuid4())
|
||||||
now = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S")
|
now = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S")
|
||||||
props = properties or {}
|
props = properties or {}
|
||||||
refs = source_refs or []
|
ev = extractor_version if extractor_version is not None else EXTRACTOR_VERSION
|
||||||
|
|
||||||
with get_connection() as conn:
|
with get_connection() as conn:
|
||||||
conn.execute(
|
conn.execute(
|
||||||
"""INSERT INTO entities
|
"""INSERT INTO entities
|
||||||
(id, entity_type, name, project, description, properties,
|
(id, entity_type, name, project, description, properties,
|
||||||
status, confidence, source_refs, created_at, updated_at)
|
status, confidence, source_refs,
|
||||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
|
extractor_version, canonical_home, hand_authored,
|
||||||
|
created_at, updated_at)
|
||||||
|
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
|
||||||
(
|
(
|
||||||
entity_id, entity_type, name.strip(), project,
|
entity_id, entity_type, name.strip(), project,
|
||||||
description, json.dumps(props), status, confidence,
|
description, json.dumps(props), status, confidence,
|
||||||
json.dumps(refs), now, now,
|
json.dumps(refs),
|
||||||
|
ev, "entity", 1 if hand_authored else 0,
|
||||||
|
now, now,
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -194,14 +240,31 @@ def create_entity(
|
|||||||
"project": project,
|
"project": project,
|
||||||
"status": status,
|
"status": status,
|
||||||
"confidence": confidence,
|
"confidence": confidence,
|
||||||
|
"hand_authored": hand_authored,
|
||||||
|
"extractor_version": ev,
|
||||||
},
|
},
|
||||||
)
|
)
|
||||||
|
|
||||||
|
# V1-0 (F-5 hook, engineering-v1-acceptance.md:99): synchronous
|
||||||
|
# conflict detection on any active-entity write. The promote path
|
||||||
|
# already had this hook (see promote_entity below); V1-0 adds it to
|
||||||
|
# direct-active creates so every active row — however it got that
|
||||||
|
# way — is checked. Fail-open per "flag, never block" rule in
|
||||||
|
# conflict-model.md:256: detector errors log but never fail the write.
|
||||||
|
if status == "active":
|
||||||
|
try:
|
||||||
|
from atocore.engineering.conflicts import detect_conflicts_for_entity
|
||||||
|
detect_conflicts_for_entity(entity_id)
|
||||||
|
except Exception as e:
|
||||||
|
log.warning("conflict_detection_failed", entity_id=entity_id, error=str(e))
|
||||||
|
|
||||||
return Entity(
|
return Entity(
|
||||||
id=entity_id, entity_type=entity_type, name=name.strip(),
|
id=entity_id, entity_type=entity_type, name=name.strip(),
|
||||||
project=project, description=description, properties=props,
|
project=project, description=description, properties=props,
|
||||||
status=status, confidence=confidence, source_refs=refs,
|
status=status, confidence=confidence, source_refs=refs,
|
||||||
created_at=now, updated_at=now,
|
created_at=now, updated_at=now,
|
||||||
|
extractor_version=ev, canonical_home="entity",
|
||||||
|
hand_authored=hand_authored,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -361,6 +424,20 @@ def promote_entity(
|
|||||||
if entity is None or entity.status != "candidate":
|
if entity is None or entity.status != "candidate":
|
||||||
return False
|
return False
|
||||||
|
|
||||||
|
# V1-0 (F-8 provenance re-check at promote). The invariant must hold at
|
||||||
|
# BOTH create_entity AND promote_entity per the plan, because candidate
|
||||||
|
# rows can exist in the DB from before V1-0 (no enforcement at their
|
||||||
|
# create time) or can be inserted by code paths that bypass the service
|
||||||
|
# layer. Block any candidate with empty source_refs that is NOT flagged
|
||||||
|
# hand_authored from ever becoming active. Same error shape as the
|
||||||
|
# create-side check for symmetry.
|
||||||
|
if not (entity.source_refs or []) and not entity.hand_authored:
|
||||||
|
raise ValueError(
|
||||||
|
"source_refs required: cannot promote a candidate with no "
|
||||||
|
"provenance. Attach source_refs via PATCH /entities/{id}, "
|
||||||
|
"or flag hand_authored=true before promoting."
|
||||||
|
)
|
||||||
|
|
||||||
if target_project is not None:
|
if target_project is not None:
|
||||||
new_project = (
|
new_project = (
|
||||||
resolve_project_name(target_project) if target_project else ""
|
resolve_project_name(target_project) if target_project else ""
|
||||||
@@ -503,6 +580,22 @@ def supersede_entity(
|
|||||||
superseded_by=superseded_by,
|
superseded_by=superseded_by,
|
||||||
error=str(e),
|
error=str(e),
|
||||||
)
|
)
|
||||||
|
|
||||||
|
# V1-0 (F-5 hook on supersede, per plan's "every active-entity
|
||||||
|
# write path"). Supersede demotes `entity_id` AND adds a
|
||||||
|
# `supersedes` relationship rooted at the already-active
|
||||||
|
# `superseded_by`. That new edge can create a conflict the
|
||||||
|
# detector should catch synchronously. Fail-open per
|
||||||
|
# conflict-model.md:256.
|
||||||
|
try:
|
||||||
|
from atocore.engineering.conflicts import detect_conflicts_for_entity
|
||||||
|
detect_conflicts_for_entity(superseded_by)
|
||||||
|
except Exception as e:
|
||||||
|
log.warning(
|
||||||
|
"conflict_detection_failed",
|
||||||
|
entity_id=superseded_by,
|
||||||
|
error=str(e),
|
||||||
|
)
|
||||||
return True
|
return True
|
||||||
|
|
||||||
|
|
||||||
@@ -774,6 +867,15 @@ def get_entity_with_context(entity_id: str) -> dict | None:
|
|||||||
|
|
||||||
|
|
||||||
def _row_to_entity(row) -> Entity:
|
def _row_to_entity(row) -> Entity:
|
||||||
|
# V1-0 shared-header fields are optional on read — rows that predate
|
||||||
|
# V1-0 migration have NULL / missing values, so defaults kick in and
|
||||||
|
# older tests that build Entity() without the new fields keep passing.
|
||||||
|
# `row.keys()` lets us tolerate SQLite rows that lack the columns
|
||||||
|
# entirely (pre-migration sqlite3.Row).
|
||||||
|
keys = set(row.keys())
|
||||||
|
extractor_version = (row["extractor_version"] or "") if "extractor_version" in keys else ""
|
||||||
|
canonical_home = (row["canonical_home"] or "entity") if "canonical_home" in keys else "entity"
|
||||||
|
hand_authored = bool(row["hand_authored"]) if "hand_authored" in keys and row["hand_authored"] is not None else False
|
||||||
return Entity(
|
return Entity(
|
||||||
id=row["id"],
|
id=row["id"],
|
||||||
entity_type=row["entity_type"],
|
entity_type=row["entity_type"],
|
||||||
@@ -786,6 +888,9 @@ def _row_to_entity(row) -> Entity:
|
|||||||
source_refs=json.loads(row["source_refs"] or "[]"),
|
source_refs=json.loads(row["source_refs"] or "[]"),
|
||||||
created_at=row["created_at"] or "",
|
created_at=row["created_at"] or "",
|
||||||
updated_at=row["updated_at"] or "",
|
updated_at=row["updated_at"] or "",
|
||||||
|
extractor_version=extractor_version,
|
||||||
|
canonical_home=canonical_home,
|
||||||
|
hand_authored=hand_authored,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -240,30 +240,10 @@ def render_homepage() -> str:
|
|||||||
|
|
||||||
# Quick stats
|
# Quick stats
|
||||||
all_entities = get_entities(limit=500)
|
all_entities = get_entities(limit=500)
|
||||||
all_memories_raw = get_memories(active_only=True, limit=500)
|
all_memories = get_memories(active_only=True, limit=500)
|
||||||
# Partition real knowledge from ambient provenance so counts are honest.
|
|
||||||
# Each memory lands in exactly one bucket (low-signal takes priority).
|
|
||||||
all_memories: list = []
|
|
||||||
akc_session_count = 0
|
|
||||||
low_signal_count = 0
|
|
||||||
for _m in all_memories_raw:
|
|
||||||
if _is_low_signal_memory(_m):
|
|
||||||
low_signal_count += 1
|
|
||||||
elif _is_akc_session_memory(_m):
|
|
||||||
akc_session_count += 1
|
|
||||||
else:
|
|
||||||
all_memories.append(_m)
|
|
||||||
pending = get_memories(status="candidate", limit=500)
|
pending = get_memories(status="candidate", limit=500)
|
||||||
lines.append('<h2>System</h2>')
|
lines.append('<h2>System</h2>')
|
||||||
lines.append(
|
lines.append(f'<p>{len(all_entities)} entities · {len(all_memories)} active memories · {len(projects)} projects</p>')
|
||||||
f'<p>{len(all_entities)} entities · {len(all_memories)} memories · '
|
|
||||||
f'{len(projects)} projects'
|
|
||||||
+ (f' · <span style="color:#888;">{akc_session_count} AKC session snapshots'
|
|
||||||
+ (f", {low_signal_count} low-signal hidden" if low_signal_count else "")
|
|
||||||
+ '</span>'
|
|
||||||
if akc_session_count or low_signal_count else '')
|
|
||||||
+ '</p>'
|
|
||||||
)
|
|
||||||
|
|
||||||
# Triage queue prompt — surfaced prominently if non-empty
|
# Triage queue prompt — surfaced prominently if non-empty
|
||||||
if pending:
|
if pending:
|
||||||
@@ -306,44 +286,6 @@ import re as _re
|
|||||||
_WIKILINK_PATTERN = _re.compile(r"\[\[([^\[\]|]+?)(?:\|([^\[\]]+?))?\]\]")
|
_WIKILINK_PATTERN = _re.compile(r"\[\[([^\[\]|]+?)(?:\|([^\[\]]+?))?\]\]")
|
||||||
|
|
||||||
|
|
||||||
# ----------------------------------------------------------------- signal/noise
|
|
||||||
# Memories with these patterns are low-signal ambient artefacts — they
|
|
||||||
# inflate lists on the homepage and domain pages without being informative.
|
|
||||||
# They remain in the DB (for provenance / audit) but are filtered from
|
|
||||||
# default browsing surfaces. Pass `include_low_signal=True` on a page query
|
|
||||||
# param to surface them.
|
|
||||||
_LOW_SIGNAL_CONTENT_PATTERNS = (
|
|
||||||
"(no transcript)", # silent-mic AKC sessions
|
|
||||||
"synthetic AKC integration", # E2E test pollution
|
|
||||||
"AKC-E2E-", # E2E test prefix in content
|
|
||||||
"AKC-IMG-TEST-", # image-upload test prefix
|
|
||||||
"IMG integration test — synthetic", # E2E narrative header
|
|
||||||
)
|
|
||||||
|
|
||||||
# AKC voice-session ambient memories follow this pattern — they're
|
|
||||||
# provenance records, not knowledge. Collapse them behind a link on domain
|
|
||||||
# pages instead of rendering each inline.
|
|
||||||
_AKC_SESSION_HEADER = "AKC voice session "
|
|
||||||
|
|
||||||
|
|
||||||
def _is_low_signal_memory(mem) -> bool:
|
|
||||||
"""True for memories whose content is known ambient/test pollution."""
|
|
||||||
content = (getattr(mem, "content", "") or "")
|
|
||||||
if not content:
|
|
||||||
return True
|
|
||||||
return any(p in content for p in _LOW_SIGNAL_CONTENT_PATTERNS)
|
|
||||||
|
|
||||||
|
|
||||||
def _is_akc_session_memory(mem) -> bool:
|
|
||||||
"""True for AKC voice-session ambient snapshots (have value as provenance,
|
|
||||||
but shouldn't clutter topical listings)."""
|
|
||||||
content = (getattr(mem, "content", "") or "")
|
|
||||||
tags = getattr(mem, "domain_tags", None) or []
|
|
||||||
if any(t in ("session", "akc") for t in tags) and "voice" in tags:
|
|
||||||
return True
|
|
||||||
return content.startswith(_AKC_SESSION_HEADER)
|
|
||||||
|
|
||||||
|
|
||||||
def _resolve_wikilink(target: str, current_project: str | None) -> tuple[str, str, str]:
|
def _resolve_wikilink(target: str, current_project: str | None) -> tuple[str, str, str]:
|
||||||
"""Resolve a ``[[Name]]`` target to ``(href, css_class, extra_suffix)``.
|
"""Resolve a ``[[Name]]`` target to ``(href, css_class, extra_suffix)``.
|
||||||
|
|
||||||
@@ -449,6 +391,8 @@ def render_new_entity_form(name: str = "", project: str = "") -> str:
|
|||||||
entity_type: fd.get('entity_type'),
|
entity_type: fd.get('entity_type'),
|
||||||
project: fd.get('project') || '',
|
project: fd.get('project') || '',
|
||||||
description: fd.get('description') || '',
|
description: fd.get('description') || '',
|
||||||
|
// V1-0: human writes via the wiki form are hand_authored by definition.
|
||||||
|
hand_authored: true,
|
||||||
};
|
};
|
||||||
try {
|
try {
|
||||||
const r = await fetch('/v1/entities', {
|
const r = await fetch('/v1/entities', {
|
||||||
@@ -932,22 +876,8 @@ def render_domain(tag: str) -> str:
|
|||||||
breadcrumbs=[("Wiki", "/wiki"), ("Domains", "")])
|
breadcrumbs=[("Wiki", "/wiki"), ("Domains", "")])
|
||||||
|
|
||||||
all_mems = get_memories(active_only=True, limit=500)
|
all_mems = get_memories(active_only=True, limit=500)
|
||||||
matching_all = [m for m in all_mems
|
matching = [m for m in all_mems
|
||||||
if any((t or "").lower() == tag for t in (m.domain_tags or []))]
|
if any((t or "").lower() == tag for t in (m.domain_tags or []))]
|
||||||
|
|
||||||
# Partition: low-signal test pollution is hidden entirely, ambient AKC
|
|
||||||
# session memories are collapsed (shown as a count + link to
|
|
||||||
# /wiki/activity). Priority: low-signal > session > real.
|
|
||||||
matching: list = []
|
|
||||||
akc_sessions: list = []
|
|
||||||
hidden_low_signal = 0
|
|
||||||
for m in matching_all:
|
|
||||||
if _is_low_signal_memory(m):
|
|
||||||
hidden_low_signal += 1
|
|
||||||
elif _is_akc_session_memory(m):
|
|
||||||
akc_sessions.append(m)
|
|
||||||
else:
|
|
||||||
matching.append(m)
|
|
||||||
|
|
||||||
# Group by project
|
# Group by project
|
||||||
by_project: dict[str, list] = {}
|
by_project: dict[str, list] = {}
|
||||||
@@ -956,18 +886,6 @@ def render_domain(tag: str) -> str:
|
|||||||
|
|
||||||
lines = [f'<h1>Domain: <code>{tag}</code></h1>']
|
lines = [f'<h1>Domain: <code>{tag}</code></h1>']
|
||||||
lines.append(f'<p class="meta">{len(matching)} active memories across {len(by_project)} projects</p>')
|
lines.append(f'<p class="meta">{len(matching)} active memories across {len(by_project)} projects</p>')
|
||||||
if akc_sessions or hidden_low_signal:
|
|
||||||
noise_bits = []
|
|
||||||
if akc_sessions:
|
|
||||||
noise_bits.append(
|
|
||||||
f'<a href="/wiki/activity">{len(akc_sessions)} AKC voice session snapshots</a>'
|
|
||||||
)
|
|
||||||
if hidden_low_signal:
|
|
||||||
noise_bits.append(f"{hidden_low_signal} low-signal memories hidden")
|
|
||||||
lines.append(
|
|
||||||
f'<p class="meta" style="font-size:0.85rem; color:#888;">'
|
|
||||||
f'Ambient provenance not listed: {" · ".join(noise_bits)}.</p>'
|
|
||||||
)
|
|
||||||
|
|
||||||
if not matching:
|
if not matching:
|
||||||
lines.append(
|
lines.append(
|
||||||
|
|||||||
@@ -146,6 +146,28 @@ def _apply_migrations(conn: sqlite3.Connection) -> None:
|
|||||||
"CREATE INDEX IF NOT EXISTS idx_memories_graduated ON memories(graduated_to_entity_id)"
|
"CREATE INDEX IF NOT EXISTS idx_memories_graduated ON memories(graduated_to_entity_id)"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
# V1-0 (Engineering V1 completion): shared header fields per
|
||||||
|
# engineering-v1-acceptance.md:45. Three columns on `entities`:
|
||||||
|
# - extractor_version: which extractor produced this row. Lets old
|
||||||
|
# candidates be re-evaluated with a newer extractor per
|
||||||
|
# promotion-rules.md:268.
|
||||||
|
# - canonical_home: which layer holds the canonical record. Always
|
||||||
|
# "entity" for rows written via create_entity; reserved for future
|
||||||
|
# cross-layer bookkeeping.
|
||||||
|
# - hand_authored: 1 when the row was created directly by a human
|
||||||
|
# without source provenance. Enforced by the write path so every
|
||||||
|
# non-hand-authored row must carry non-empty source_refs (F-8).
|
||||||
|
# The entities table itself is created by init_engineering_schema
|
||||||
|
# (see engineering/service.py); these ALTERs cover existing DBs
|
||||||
|
# where the original CREATE TABLE predates V1-0.
|
||||||
|
if _table_exists(conn, "entities"):
|
||||||
|
if not _column_exists(conn, "entities", "extractor_version"):
|
||||||
|
conn.execute("ALTER TABLE entities ADD COLUMN extractor_version TEXT DEFAULT ''")
|
||||||
|
if not _column_exists(conn, "entities", "canonical_home"):
|
||||||
|
conn.execute("ALTER TABLE entities ADD COLUMN canonical_home TEXT DEFAULT 'entity'")
|
||||||
|
if not _column_exists(conn, "entities", "hand_authored"):
|
||||||
|
conn.execute("ALTER TABLE entities ADD COLUMN hand_authored INTEGER DEFAULT 0")
|
||||||
|
|
||||||
# Phase 4 (Robustness V1): append-only audit log for memory mutations.
|
# Phase 4 (Robustness V1): append-only audit log for memory mutations.
|
||||||
# Every create/update/promote/reject/supersede/invalidate/reinforce/expire/
|
# Every create/update/promote/reject/supersede/invalidate/reinforce/expire/
|
||||||
# auto_promote writes one row here. before/after are JSON snapshots of the
|
# auto_promote writes one row here. before/after are JSON snapshots of the
|
||||||
@@ -352,6 +374,14 @@ def _column_exists(conn: sqlite3.Connection, table: str, column: str) -> bool:
|
|||||||
return any(row["name"] == column for row in rows)
|
return any(row["name"] == column for row in rows)
|
||||||
|
|
||||||
|
|
||||||
|
def _table_exists(conn: sqlite3.Connection, table: str) -> bool:
|
||||||
|
row = conn.execute(
|
||||||
|
"SELECT name FROM sqlite_master WHERE type='table' AND name=?",
|
||||||
|
(table,),
|
||||||
|
).fetchone()
|
||||||
|
return row is not None
|
||||||
|
|
||||||
|
|
||||||
@contextmanager
|
@contextmanager
|
||||||
def get_connection() -> Generator[sqlite3.Connection, None, None]:
|
def get_connection() -> Generator[sqlite3.Connection, None, None]:
|
||||||
"""Get a database connection with row factory."""
|
"""Get a database connection with row factory."""
|
||||||
|
|||||||
@@ -16,6 +16,36 @@ os.environ["ATOCORE_DATA_DIR"] = _default_test_dir
|
|||||||
os.environ["ATOCORE_DEBUG"] = "true"
|
os.environ["ATOCORE_DEBUG"] = "true"
|
||||||
|
|
||||||
|
|
||||||
|
# V1-0: every entity created in a test is "hand authored" by the test
|
||||||
|
# author — fixture data, not extracted content. Rather than rewrite 100+
|
||||||
|
# existing test call sites, wrap create_entity so that tests which don't
|
||||||
|
# provide source_refs get hand_authored=True automatically. Tests that
|
||||||
|
# explicitly pass source_refs or hand_authored are unaffected. This keeps
|
||||||
|
# the F-8 invariant enforced in production (the API, the wiki form, and
|
||||||
|
# graduation scripts all go through the unwrapped function) while leaving
|
||||||
|
# the existing test corpus intact.
|
||||||
|
def _patch_create_entity_for_tests():
|
||||||
|
from atocore.engineering import service as _svc
|
||||||
|
|
||||||
|
_original = _svc.create_entity
|
||||||
|
|
||||||
|
def _create_entity_test(*args, **kwargs):
|
||||||
|
# Only auto-flag when hand_authored isn't explicitly specified.
|
||||||
|
# Tests that want to exercise the F-8 raise path pass
|
||||||
|
# hand_authored=False explicitly and should hit the error.
|
||||||
|
if (
|
||||||
|
not kwargs.get("source_refs")
|
||||||
|
and "hand_authored" not in kwargs
|
||||||
|
):
|
||||||
|
kwargs["hand_authored"] = True
|
||||||
|
return _original(*args, **kwargs)
|
||||||
|
|
||||||
|
_svc.create_entity = _create_entity_test
|
||||||
|
|
||||||
|
|
||||||
|
_patch_create_entity_for_tests()
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
@pytest.fixture
|
||||||
def tmp_data_dir(tmp_path):
|
def tmp_data_dir(tmp_path):
|
||||||
"""Provide a temporary data directory for tests."""
|
"""Provide a temporary data directory for tests."""
|
||||||
|
|||||||
@@ -143,8 +143,11 @@ def test_requirement_name_conflict_detected(tmp_data_dir):
|
|||||||
r2 = create_entity("requirement", "Surface figure < 25nm",
|
r2 = create_entity("requirement", "Surface figure < 25nm",
|
||||||
project="p-test", description="Different interpretation")
|
project="p-test", description="Different interpretation")
|
||||||
|
|
||||||
detected = detect_conflicts_for_entity(r2.id)
|
# V1-0 synchronous hook: the conflict is already detected at r2's
|
||||||
assert len(detected) == 1
|
# create-time, so a redundant detect call returns [] due to
|
||||||
|
# _record_conflict dedup. Assert on list_open_conflicts instead —
|
||||||
|
# that's what the intent of this test really tests: duplicate
|
||||||
|
# active requirements surface as an open conflict.
|
||||||
conflicts = list_open_conflicts(project="p-test")
|
conflicts = list_open_conflicts(project="p-test")
|
||||||
assert any(c["slot_kind"] == "requirement.name" for c in conflicts)
|
assert any(c["slot_kind"] == "requirement.name" for c in conflicts)
|
||||||
|
|
||||||
@@ -191,8 +194,12 @@ def test_conflict_resolution_dismiss_leaves_entities_alone(tmp_data_dir):
|
|||||||
description="first meaning")
|
description="first meaning")
|
||||||
r2 = create_entity("requirement", "Dup req", project="p-test",
|
r2 = create_entity("requirement", "Dup req", project="p-test",
|
||||||
description="second meaning")
|
description="second meaning")
|
||||||
detected = detect_conflicts_for_entity(r2.id)
|
# V1-0 synchronous hook already recorded the conflict at r2's
|
||||||
conflict_id = detected[0]
|
# create-time. Look it up via list_open_conflicts rather than
|
||||||
|
# calling the detector again (which returns [] due to dedup).
|
||||||
|
open_list = list_open_conflicts(project="p-test")
|
||||||
|
assert open_list, "expected conflict recorded by create-time hook"
|
||||||
|
conflict_id = open_list[0]["id"]
|
||||||
|
|
||||||
assert resolve_conflict(conflict_id, "dismiss")
|
assert resolve_conflict(conflict_id, "dismiss")
|
||||||
# Both still active — dismiss just clears the conflict marker
|
# Both still active — dismiss just clears the conflict marker
|
||||||
|
|||||||
@@ -132,6 +132,7 @@ def test_api_post_entity_with_null_project_stores_global(seeded_db):
|
|||||||
"entity_type": "material",
|
"entity_type": "material",
|
||||||
"name": "Titanium",
|
"name": "Titanium",
|
||||||
"project": None,
|
"project": None,
|
||||||
|
"hand_authored": True, # V1-0 F-8: test fixture, no source_refs
|
||||||
})
|
})
|
||||||
assert r.status_code == 200
|
assert r.status_code == 200
|
||||||
|
|
||||||
|
|||||||
362
tests/test_v1_0_write_invariants.py
Normal file
362
tests/test_v1_0_write_invariants.py
Normal file
@@ -0,0 +1,362 @@
|
|||||||
|
"""V1-0 write-time invariant tests.
|
||||||
|
|
||||||
|
Covers the Engineering V1 completion plan Phase V1-0 acceptance:
|
||||||
|
- F-1 shared-header fields: extractor_version + canonical_home + hand_authored
|
||||||
|
land in the entities table with working defaults
|
||||||
|
- F-8 provenance enforcement: create_entity raises without source_refs
|
||||||
|
unless hand_authored=True
|
||||||
|
- F-5 synchronous conflict-detection hook on any active-entity write
|
||||||
|
(create_entity with status="active" + the pre-existing promote_entity
|
||||||
|
path); fail-open per conflict-model.md:256
|
||||||
|
- Q-3 "flag, never block": a conflict never 4xx-blocks the write
|
||||||
|
- Q-4 partial trust: get_entities scope_only filters candidates out
|
||||||
|
|
||||||
|
Plan: docs/plans/engineering-v1-completion-plan.md
|
||||||
|
Spec: docs/architecture/engineering-v1-acceptance.md
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from atocore.engineering.service import (
|
||||||
|
EXTRACTOR_VERSION,
|
||||||
|
create_entity,
|
||||||
|
create_relationship,
|
||||||
|
get_entities,
|
||||||
|
get_entity,
|
||||||
|
init_engineering_schema,
|
||||||
|
promote_entity,
|
||||||
|
supersede_entity,
|
||||||
|
)
|
||||||
|
from atocore.models.database import get_connection, init_db
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- F-1: shared-header fields ----------
|
||||||
|
|
||||||
|
|
||||||
|
def test_entity_row_has_shared_header_fields(tmp_data_dir):
|
||||||
|
init_db()
|
||||||
|
init_engineering_schema()
|
||||||
|
with get_connection() as conn:
|
||||||
|
cols = {row["name"] for row in conn.execute("PRAGMA table_info(entities)").fetchall()}
|
||||||
|
assert "extractor_version" in cols
|
||||||
|
assert "canonical_home" in cols
|
||||||
|
assert "hand_authored" in cols
|
||||||
|
|
||||||
|
|
||||||
|
def test_created_entity_has_default_extractor_version_and_canonical_home(tmp_data_dir):
|
||||||
|
init_db()
|
||||||
|
init_engineering_schema()
|
||||||
|
e = create_entity(
|
||||||
|
entity_type="component",
|
||||||
|
name="Pivot Pin",
|
||||||
|
project="p04-gigabit",
|
||||||
|
source_refs=["test:fixture"],
|
||||||
|
)
|
||||||
|
assert e.extractor_version == EXTRACTOR_VERSION
|
||||||
|
assert e.canonical_home == "entity"
|
||||||
|
assert e.hand_authored is False
|
||||||
|
|
||||||
|
# round-trip through get_entity to confirm the row mapper returns
|
||||||
|
# the same values (not just the return-by-construct path)
|
||||||
|
got = get_entity(e.id)
|
||||||
|
assert got is not None
|
||||||
|
assert got.extractor_version == EXTRACTOR_VERSION
|
||||||
|
assert got.canonical_home == "entity"
|
||||||
|
assert got.hand_authored is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_explicit_extractor_version_is_persisted(tmp_data_dir):
|
||||||
|
init_db()
|
||||||
|
init_engineering_schema()
|
||||||
|
e = create_entity(
|
||||||
|
entity_type="decision",
|
||||||
|
name="Pick GF-PTFE pads",
|
||||||
|
project="p04-gigabit",
|
||||||
|
source_refs=["interaction:abc"],
|
||||||
|
extractor_version="custom-v2.3",
|
||||||
|
)
|
||||||
|
got = get_entity(e.id)
|
||||||
|
assert got.extractor_version == "custom-v2.3"
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- F-8: provenance enforcement ----------
|
||||||
|
|
||||||
|
|
||||||
|
def test_create_entity_without_provenance_raises(tmp_data_dir):
|
||||||
|
init_db()
|
||||||
|
init_engineering_schema()
|
||||||
|
with pytest.raises(ValueError, match="source_refs required"):
|
||||||
|
create_entity(
|
||||||
|
entity_type="component",
|
||||||
|
name="No Provenance",
|
||||||
|
project="p04-gigabit",
|
||||||
|
hand_authored=False, # explicit — bypasses the test-conftest auto-flag
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_create_entity_with_hand_authored_needs_no_source_refs(tmp_data_dir):
|
||||||
|
init_db()
|
||||||
|
init_engineering_schema()
|
||||||
|
e = create_entity(
|
||||||
|
entity_type="component",
|
||||||
|
name="Human Entry",
|
||||||
|
project="p04-gigabit",
|
||||||
|
hand_authored=True,
|
||||||
|
)
|
||||||
|
assert e.hand_authored is True
|
||||||
|
got = get_entity(e.id)
|
||||||
|
assert got.hand_authored is True
|
||||||
|
# source_refs stays empty — the hand_authored flag IS the provenance
|
||||||
|
assert got.source_refs == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_create_entity_with_empty_source_refs_list_is_treated_as_missing(tmp_data_dir):
|
||||||
|
init_db()
|
||||||
|
init_engineering_schema()
|
||||||
|
with pytest.raises(ValueError, match="source_refs required"):
|
||||||
|
create_entity(
|
||||||
|
entity_type="component",
|
||||||
|
name="Empty Refs",
|
||||||
|
project="p04-gigabit",
|
||||||
|
source_refs=[],
|
||||||
|
hand_authored=False,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_promote_rejects_legacy_candidate_without_provenance(tmp_data_dir):
|
||||||
|
"""Regression (Codex V1-0 probe): candidate rows can exist in the DB
|
||||||
|
from before V1-0 enforcement (or from paths that bypass create_entity).
|
||||||
|
promote_entity must re-check the invariant and refuse to flip a
|
||||||
|
no-provenance candidate to active. Without this check, the active
|
||||||
|
store can leak F-8 violations in from legacy data."""
|
||||||
|
init_db()
|
||||||
|
init_engineering_schema()
|
||||||
|
|
||||||
|
# Simulate a pre-V1-0 candidate by inserting directly into the table,
|
||||||
|
# bypassing the service-layer invariant. Real legacy rows look exactly
|
||||||
|
# like this: empty source_refs, hand_authored=0.
|
||||||
|
import uuid as _uuid
|
||||||
|
entity_id = str(_uuid.uuid4())
|
||||||
|
with get_connection() as conn:
|
||||||
|
conn.execute(
|
||||||
|
"INSERT INTO entities (id, entity_type, name, project, "
|
||||||
|
"description, properties, status, confidence, source_refs, "
|
||||||
|
"extractor_version, canonical_home, hand_authored, "
|
||||||
|
"created_at, updated_at) "
|
||||||
|
"VALUES (?, 'component', 'Legacy Orphan', 'p04-gigabit', "
|
||||||
|
"'', '{}', 'candidate', 1.0, '[]', '', 'entity', 0, "
|
||||||
|
"CURRENT_TIMESTAMP, CURRENT_TIMESTAMP)",
|
||||||
|
(entity_id,),
|
||||||
|
)
|
||||||
|
|
||||||
|
with pytest.raises(ValueError, match="source_refs required"):
|
||||||
|
promote_entity(entity_id)
|
||||||
|
|
||||||
|
# And the row stays a candidate — no half-transition.
|
||||||
|
got = get_entity(entity_id)
|
||||||
|
assert got is not None
|
||||||
|
assert got.status == "candidate"
|
||||||
|
|
||||||
|
|
||||||
|
def test_promote_accepts_candidate_flagged_hand_authored(tmp_data_dir):
|
||||||
|
"""The other side of the promote re-check: hand_authored=1 with
|
||||||
|
empty source_refs still lets promote succeed, matching
|
||||||
|
create_entity's symmetry."""
|
||||||
|
init_db()
|
||||||
|
init_engineering_schema()
|
||||||
|
|
||||||
|
import uuid as _uuid
|
||||||
|
entity_id = str(_uuid.uuid4())
|
||||||
|
with get_connection() as conn:
|
||||||
|
conn.execute(
|
||||||
|
"INSERT INTO entities (id, entity_type, name, project, "
|
||||||
|
"description, properties, status, confidence, source_refs, "
|
||||||
|
"extractor_version, canonical_home, hand_authored, "
|
||||||
|
"created_at, updated_at) "
|
||||||
|
"VALUES (?, 'component', 'Hand Authored Candidate', "
|
||||||
|
"'p04-gigabit', '', '{}', 'candidate', 1.0, '[]', '', "
|
||||||
|
"'entity', 1, CURRENT_TIMESTAMP, CURRENT_TIMESTAMP)",
|
||||||
|
(entity_id,),
|
||||||
|
)
|
||||||
|
|
||||||
|
assert promote_entity(entity_id) is True
|
||||||
|
assert get_entity(entity_id).status == "active"
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- F-5: synchronous conflict-detection hook ----------
|
||||||
|
|
||||||
|
|
||||||
|
def test_active_create_runs_conflict_detection_hook(tmp_data_dir, monkeypatch):
|
||||||
|
"""status=active writes trigger detect_conflicts_for_entity."""
|
||||||
|
init_db()
|
||||||
|
init_engineering_schema()
|
||||||
|
|
||||||
|
called_with: list[str] = []
|
||||||
|
|
||||||
|
def _fake_detect(entity_id: str):
|
||||||
|
called_with.append(entity_id)
|
||||||
|
return []
|
||||||
|
|
||||||
|
import atocore.engineering.conflicts as conflicts_mod
|
||||||
|
monkeypatch.setattr(conflicts_mod, "detect_conflicts_for_entity", _fake_detect)
|
||||||
|
|
||||||
|
e = create_entity(
|
||||||
|
entity_type="component",
|
||||||
|
name="Active With Hook",
|
||||||
|
project="p04-gigabit",
|
||||||
|
source_refs=["test:hook"],
|
||||||
|
status="active",
|
||||||
|
)
|
||||||
|
|
||||||
|
assert called_with == [e.id]
|
||||||
|
|
||||||
|
|
||||||
|
def test_supersede_runs_conflict_detection_on_new_active(tmp_data_dir, monkeypatch):
|
||||||
|
"""Regression (Codex V1-0 probe): per plan's 'every active-entity
|
||||||
|
write path', supersede_entity must trigger synchronous conflict
|
||||||
|
detection. The subject is the `superseded_by` entity — the one
|
||||||
|
whose graph state just changed because a new `supersedes` edge was
|
||||||
|
rooted at it."""
|
||||||
|
init_db()
|
||||||
|
init_engineering_schema()
|
||||||
|
|
||||||
|
old = create_entity(
|
||||||
|
entity_type="component",
|
||||||
|
name="Old Pad",
|
||||||
|
project="p04-gigabit",
|
||||||
|
source_refs=["test:old"],
|
||||||
|
status="active",
|
||||||
|
)
|
||||||
|
new = create_entity(
|
||||||
|
entity_type="component",
|
||||||
|
name="New Pad",
|
||||||
|
project="p04-gigabit",
|
||||||
|
source_refs=["test:new"],
|
||||||
|
status="active",
|
||||||
|
)
|
||||||
|
|
||||||
|
called_with: list[str] = []
|
||||||
|
|
||||||
|
def _fake_detect(entity_id: str):
|
||||||
|
called_with.append(entity_id)
|
||||||
|
return []
|
||||||
|
|
||||||
|
import atocore.engineering.conflicts as conflicts_mod
|
||||||
|
monkeypatch.setattr(conflicts_mod, "detect_conflicts_for_entity", _fake_detect)
|
||||||
|
|
||||||
|
assert supersede_entity(old.id, superseded_by=new.id) is True
|
||||||
|
|
||||||
|
# The detector fires on the `superseded_by` entity — the one whose
|
||||||
|
# edges just grew a new `supersedes` relationship.
|
||||||
|
assert new.id in called_with
|
||||||
|
|
||||||
|
|
||||||
|
def test_supersede_hook_fails_open(tmp_data_dir, monkeypatch):
|
||||||
|
"""Supersede must survive a broken detector per Q-3 flag-never-block."""
|
||||||
|
init_db()
|
||||||
|
init_engineering_schema()
|
||||||
|
|
||||||
|
old = create_entity(
|
||||||
|
entity_type="component", name="Old2", project="p04-gigabit",
|
||||||
|
source_refs=["test:old"], status="active",
|
||||||
|
)
|
||||||
|
new = create_entity(
|
||||||
|
entity_type="component", name="New2", project="p04-gigabit",
|
||||||
|
source_refs=["test:new"], status="active",
|
||||||
|
)
|
||||||
|
|
||||||
|
def _boom(entity_id: str):
|
||||||
|
raise RuntimeError("synthetic detector failure")
|
||||||
|
|
||||||
|
import atocore.engineering.conflicts as conflicts_mod
|
||||||
|
monkeypatch.setattr(conflicts_mod, "detect_conflicts_for_entity", _boom)
|
||||||
|
|
||||||
|
# The supersede still succeeds despite the detector blowing up.
|
||||||
|
assert supersede_entity(old.id, superseded_by=new.id) is True
|
||||||
|
assert get_entity(old.id).status == "superseded"
|
||||||
|
|
||||||
|
|
||||||
|
def test_candidate_create_does_not_run_conflict_hook(tmp_data_dir, monkeypatch):
|
||||||
|
"""status=candidate writes do NOT trigger detection — the hook is
|
||||||
|
for active rows only, per V1-0 scope. Candidates are checked at
|
||||||
|
promote time."""
|
||||||
|
init_db()
|
||||||
|
init_engineering_schema()
|
||||||
|
|
||||||
|
called: list[str] = []
|
||||||
|
|
||||||
|
def _fake_detect(entity_id: str):
|
||||||
|
called.append(entity_id)
|
||||||
|
return []
|
||||||
|
|
||||||
|
import atocore.engineering.conflicts as conflicts_mod
|
||||||
|
monkeypatch.setattr(conflicts_mod, "detect_conflicts_for_entity", _fake_detect)
|
||||||
|
|
||||||
|
create_entity(
|
||||||
|
entity_type="component",
|
||||||
|
name="Candidate No Hook",
|
||||||
|
project="p04-gigabit",
|
||||||
|
source_refs=["test:cand"],
|
||||||
|
status="candidate",
|
||||||
|
)
|
||||||
|
|
||||||
|
assert called == []
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- Q-3: flag, never block ----------
|
||||||
|
|
||||||
|
|
||||||
|
def test_conflict_detector_failure_does_not_block_write(tmp_data_dir, monkeypatch):
|
||||||
|
"""Per conflict-model.md:256: detection errors must not fail the
|
||||||
|
write. The entity is still created; only a warning is logged."""
|
||||||
|
init_db()
|
||||||
|
init_engineering_schema()
|
||||||
|
|
||||||
|
def _boom(entity_id: str):
|
||||||
|
raise RuntimeError("synthetic detector failure")
|
||||||
|
|
||||||
|
import atocore.engineering.conflicts as conflicts_mod
|
||||||
|
monkeypatch.setattr(conflicts_mod, "detect_conflicts_for_entity", _boom)
|
||||||
|
|
||||||
|
# The write still succeeds — no exception propagates.
|
||||||
|
e = create_entity(
|
||||||
|
entity_type="component",
|
||||||
|
name="Hook Fails Open",
|
||||||
|
project="p04-gigabit",
|
||||||
|
source_refs=["test:failopen"],
|
||||||
|
status="active",
|
||||||
|
)
|
||||||
|
assert get_entity(e.id) is not None
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- Q-4 (partial): trust-hierarchy — scope_only filters candidates ----------
|
||||||
|
|
||||||
|
|
||||||
|
def test_scope_only_active_does_not_return_candidates(tmp_data_dir):
|
||||||
|
"""V1-0 partial Q-4: active-scoped listing never returns candidates.
|
||||||
|
Full trust-hierarchy coverage (no-auto-project-state, etc.) ships in
|
||||||
|
V1-E per plan."""
|
||||||
|
init_db()
|
||||||
|
init_engineering_schema()
|
||||||
|
|
||||||
|
active = create_entity(
|
||||||
|
entity_type="component",
|
||||||
|
name="Active Alpha",
|
||||||
|
project="p04-gigabit",
|
||||||
|
source_refs=["test:alpha"],
|
||||||
|
status="active",
|
||||||
|
)
|
||||||
|
candidate = create_entity(
|
||||||
|
entity_type="component",
|
||||||
|
name="Candidate Beta",
|
||||||
|
project="p04-gigabit",
|
||||||
|
source_refs=["test:beta"],
|
||||||
|
status="candidate",
|
||||||
|
)
|
||||||
|
|
||||||
|
listed = get_entities(project="p04-gigabit", status="active", scope_only=True)
|
||||||
|
ids = {e.id for e in listed}
|
||||||
|
assert active.id in ids
|
||||||
|
assert candidate.id not in ids
|
||||||
@@ -161,103 +161,3 @@ def test_memory_detail_shows_superseded_sources(tmp_data_dir):
|
|||||||
assert html1 is not None
|
assert html1 is not None
|
||||||
assert "superseded" in html1
|
assert "superseded" in html1
|
||||||
assert "auto-dedup-tier1" in html1 # audit trail shows who merged
|
assert "auto-dedup-tier1" in html1 # audit trail shows who merged
|
||||||
|
|
||||||
|
|
||||||
# -------------------------------------------------- low-signal wiki filters
|
|
||||||
# Ambient AKC session memories and test pollution shouldn't dominate domain
|
|
||||||
# pages / homepage counts. These tests lock the partitioning behaviour.
|
|
||||||
|
|
||||||
def test_domain_page_hides_empty_transcript_sessions(tmp_data_dir):
|
|
||||||
"""Silent-mic AKC sessions (content has '(no transcript)') are ambient
|
|
||||||
noise — they go into the hidden count, not the main list."""
|
|
||||||
_init_all()
|
|
||||||
# One real knowledge memory with tag "optics"
|
|
||||||
create_memory(
|
|
||||||
"knowledge",
|
|
||||||
"CGH null corrector supports F/1.2 asphere testing",
|
|
||||||
project="p05", confidence=0.9, domain_tags=["optics", "cgh"],
|
|
||||||
)
|
|
||||||
# One silent AKC session with the same tag — should NOT appear
|
|
||||||
create_memory(
|
|
||||||
"episodic",
|
|
||||||
"AKC voice session abc (gen-002)\nDuration: 60s, 2 captures\n"
|
|
||||||
"\n## Transcript\n(no transcript)\n",
|
|
||||||
project="p05", confidence=0.7,
|
|
||||||
domain_tags=["optics", "session", "akc", "voice"],
|
|
||||||
)
|
|
||||||
html = render_domain("optics")
|
|
||||||
assert "CGH null corrector" in html
|
|
||||||
# The hidden-count banner should be present
|
|
||||||
assert "low-signal" in html or "Ambient provenance" in html
|
|
||||||
# And the empty-transcript content itself is not rendered inline
|
|
||||||
assert "(no transcript)" not in html
|
|
||||||
|
|
||||||
|
|
||||||
def test_domain_page_collapses_akc_session_snapshots(tmp_data_dir):
|
|
||||||
"""AKC voice-session memories are provenance records — count them as
|
|
||||||
a single collapsed link, don't inline every one."""
|
|
||||||
_init_all()
|
|
||||||
for i in range(5):
|
|
||||||
create_memory(
|
|
||||||
"episodic",
|
|
||||||
f"AKC voice session session-{i} (gen-00{i})\nDuration: 120s, 3 captures\n"
|
|
||||||
f"\n## Transcript\nReal transcript number {i}",
|
|
||||||
project="p05", confidence=0.7,
|
|
||||||
domain_tags=["optics", "session", "akc", "voice"],
|
|
||||||
)
|
|
||||||
html = render_domain("optics")
|
|
||||||
# Inline count should mention AKC session snapshots
|
|
||||||
assert "AKC voice session snapshots" in html
|
|
||||||
# None of the session transcripts should be pasted inline on the domain
|
|
||||||
# page (they're provenance, linked via /wiki/activity)
|
|
||||||
assert "Real transcript number 0" not in html
|
|
||||||
|
|
||||||
|
|
||||||
def test_homepage_stats_exclude_ambient_memory(tmp_data_dir):
|
|
||||||
"""Homepage system-stats line shows real memory count, pushes ambient
|
|
||||||
counts into a dimmed sub-segment."""
|
|
||||||
_init_all()
|
|
||||||
# 2 real memories + 3 ambient sessions + 1 silent junk
|
|
||||||
create_memory("knowledge", "Real fact 1", project="p05", confidence=0.8)
|
|
||||||
create_memory("knowledge", "Real fact 2", project="p05", confidence=0.8)
|
|
||||||
for i in range(3):
|
|
||||||
create_memory(
|
|
||||||
"episodic",
|
|
||||||
f"AKC voice session s{i} (gen-00{i})\nReal transcript x",
|
|
||||||
project="p05", confidence=0.7,
|
|
||||||
domain_tags=["session", "akc", "voice"],
|
|
||||||
)
|
|
||||||
create_memory(
|
|
||||||
"episodic",
|
|
||||||
"AKC voice session silent (gen-099)\nDuration: 30s, 0 captures\n"
|
|
||||||
"\n## Transcript\n(no transcript)\n",
|
|
||||||
project="p05", confidence=0.7,
|
|
||||||
domain_tags=["session", "akc", "voice"],
|
|
||||||
)
|
|
||||||
html = render_homepage()
|
|
||||||
assert "3 AKC session snapshots" in html
|
|
||||||
assert "low-signal hidden" in html
|
|
||||||
# Main count reflects only real knowledge
|
|
||||||
assert "2 memories" in html
|
|
||||||
|
|
||||||
|
|
||||||
def test_low_signal_predicate_catches_known_patterns():
|
|
||||||
from atocore.engineering.wiki import _is_low_signal_memory, _is_akc_session_memory
|
|
||||||
from dataclasses import dataclass
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class M:
|
|
||||||
content: str = ""
|
|
||||||
domain_tags: list = None
|
|
||||||
|
|
||||||
# Explicit empty-transcript — low signal
|
|
||||||
assert _is_low_signal_memory(M(content="AKC voice session x\n## Transcript\n(no transcript)\n"))
|
|
||||||
# E2E test pollution — low signal
|
|
||||||
assert _is_low_signal_memory(M(content="IMG integration test — synthetic session"))
|
|
||||||
assert _is_low_signal_memory(M(content="synthetic AKC integration session"))
|
|
||||||
# Real knowledge — NOT low signal
|
|
||||||
assert not _is_low_signal_memory(M(content="The CGH is mounted to the fold mirror via…"))
|
|
||||||
# AKC session tag predicate
|
|
||||||
assert _is_akc_session_memory(M(content="anything", domain_tags=["session", "akc", "voice"]))
|
|
||||||
assert _is_akc_session_memory(M(content="AKC voice session abc"))
|
|
||||||
assert not _is_akc_session_memory(M(content="Real fact", domain_tags=["optics"]))
|
|
||||||
|
|||||||
Reference in New Issue
Block a user