3 Commits

Author SHA1 Message Date
d6ce6128cf docs(arch): human-mirror-rules + engineering-v1-acceptance, sprint complete
Session 4 of the four-session plan. Final two engineering planning
docs, plus master-plan-status.md updated to reflect that the
engineering layer planning sprint is now complete.

docs/architecture/human-mirror-rules.md
---------------------------------------
The Layer 3 derived markdown view spec:

- The non-negotiable rule: the Mirror is read-only from the
  human's perspective; edits go to the canonical home and the
  Mirror picks them up on regeneration
- 3 V1 template families: Project Overview, Decision Log,
  Subsystem Detail
- Explicit V1 exclusions: per-component pages, per-decision
  pages, cross-project rollups, time-series pages, diff pages,
  conflict queue render, per-memory pages
- Mirror files live in /srv/storage/atocore/data/mirror/ NOT in
  the source vault (sources stay read-only per the operating
  model)
- 3 regeneration triggers: explicit POST, debounced async on
  entity write, daily scheduled refresh
- "Do not edit" header banner with checksum so unchanged inputs
  skip work
- Conflicts and project_state overrides surface inline so the
  trust hierarchy is visible in the human reading experience
- Templates checked in under templates/mirror/, edited via PR
- Deterministic output is a V1 requirement so future Mirror
  diffing works without rework
- Open questions for V1: debounce window, scheduler integration,
  template testing approach, directory listing endpoint, empty
  state rendering

docs/architecture/engineering-v1-acceptance.md
----------------------------------------------
The measurable done definition:

- Single-sentence definition: V1 is done when every v1-required
  query in engineering-query-catalog.md returns a correct result
  for one chosen test project, the Human Mirror renders a
  coherent overview, and a real KB-CAD or KB-FEM export round-
  trips through ingest -> review queue -> active entity without
  violating any conflict or trust invariant
- 23 acceptance criteria across 4 categories:
  * Functional (8): entity store, all 20 v1-required queries,
    tool ingest endpoints, candidate review queue, conflict
    detection, Human Mirror, memory-to-entity graduation,
    complete provenance chain
  * Quality (6): existing tests pass, V1 has its own coverage,
    conflict invariants enforced, trust hierarchy enforced,
    Mirror reproducible via golden file, killer correctness
    queries pass against representative data
  * Operational (5): safe migration, backup/restore drill,
    performance bounds, no new manual ops burden, Phase 9 not
    regressed
  * Documentation (4): per-entity-type spec docs, KB schema docs,
    V1 release notes, master-plan-status updated
- Explicit negative list of things V1 does NOT need to do:
  no LLM extractor, no auto-promotion, no write-back, no
  multi-user, no real-time UI, no cross-project rollups,
  no time-travel, no nightly conflict sweep, no incremental
  Chroma, no retention cleanup, no encryption, no off-Dalidou
  backup target
- Recommended implementation order: F-1 -> F-8 in sequence,
  with the graduation flow (F-7) saved for last as the most
  cross-cutting change
- Anticipated friction points called out in advance:
  graduation cross-cuts memory module, Mirror determinism trap,
  conflict detector subtle correctness, provenance backfill
  for graduated entities

master-plan-status.md updated
-----------------------------
- Engineering Layer Planning Sprint section now marked complete
  with all 8 architecture docs listed
- Note that the next concrete step is the V1 implementation
  sprint following engineering-v1-acceptance.md as its checklist

Pure doc work. No code, no schema, no behavior changes.

After this commit, the engineering planning sprint is fully done
(8/8 docs) and Phase 9 is fully complete (Commits A/B/C all
shipped, validated, and pushed). AtoCore is ready for either
the engineering V1 implementation sprint OR a pause for real-
world Phase 9 usage, depending on which the user prefers next.
2026-04-07 06:55:43 -04:00
368adf2ebc docs(arch): tool-handoff-boundaries + representation-authority
Session 3 of the four-session plan. Two more engineering planning
docs that lock in the most contentious architectural decisions
before V1 implementation begins.

docs/architecture/tool-handoff-boundaries.md
--------------------------------------------
Locks in the V1 read/write relationship with external tools:

- AtoCore is a one-way mirror in V1. External tools push,
  AtoCore reads, AtoCore never writes back.
- Per-tool stance table covering KB-CAD, KB-FEM, NX, PKM, Gitea
  repos, OpenClaw, AtoDrive, PLM/vendor systems
- Two new ingest endpoints proposed for V1:
  POST /ingest/kb-cad/export and POST /ingest/kb-fem/export
- Sketch JSON shapes for both exports (intentionally minimal,
  to be refined in dedicated schema docs during implementation)
- Drift handling: KB-CAD changes a value -> creates an entity
  candidate -> existing active becomes a conflict member ->
  human resolves via the conflict model
- Hard-line invariants V1 will not cross: no write to external
  tools, no live polling, no silent merging, no schema fan-out,
  no external-tool-specific logic in entity types
- Why not bidirectional: schema drift, conflict semantics, trust
  hierarchy, velocity, reversibility
- V2+ deferred items: selective write-back annotations, light
  polling, direct NX integration, cost/vendor/PLM connections
- Open questions for the implementation sprint: schema location,
  who runs the exporter, full-vs-incremental, exporter auth

docs/architecture/representation-authority.md
---------------------------------------------
The canonical-home matrix that says where each kind of fact
actually lives:

- Six representation layers identified: PKM, KB project,
  Gitea repos, AtoCore memories, AtoCore entities, AtoCore
  project_state
- The hard rule: every fact kind has exactly one canonical
  home; other layers may hold derived copies but never disagree
- Comprehensive matrix covering 22 fact kinds (CAD geometry,
  CAD-side structure, FEM mesh, FEM results, code, repo docs,
  PKM prose, identity, preference, episodic, decision,
  requirement, constraint, validation claim, material,
  parameter, project status, ADRs, runbooks, backup metadata,
  interactions)
- Cross-layer supremacy rule: project_state > tool-of-origin >
  entities > active memories > source chunks
- Three worked examples showing how the rules apply:
  * "what material does the lateral support pad use?" (KB-CAD
    canonical, project_state override possible)
  * "did we decide to merge the bind mounts?" (Gitea + memory
    both canonical for different aspects)
  * "what's p05's current next focus?" (project_state always
    wins for current state queries)
- Concrete consequences for V1 implementation: Material and
  Parameter are mostly KB-CAD shadows; Decisions / Requirements /
  Constraints / ValidationClaims are AtoCore-canonical; PKM is
  never authoritative; project_state is the override layer;
  the conflict model is the enforcement mechanism
- Out of scope for V1: facts about other people, vendor/cost
  facts, time-bounded facts, cross-project shared facts
- Open questions for V1: how the reviewer sees canonical home
  in the UI, whether entities need an explicit canonical_home
  field, how project_state overrides surface in query results

This is pure doc work. No code, no schema, no behavior changes.
After this commit the engineering planning sprint is 6 of 8 docs
done — only human-mirror-rules and engineering-v1-acceptance
remain.
2026-04-07 06:50:56 -04:00
a637017900 slash command for daily AtoCore use + backup-restore procedure
Session 2 of the four-session plan. Lands two operational pieces:
the Claude Code slash command that makes AtoCore reachable from
inside any Claude Code session, and the full backup/restore
procedure doc that turns the backup endpoint code into a real
operational drill.

Slash command (.claude/commands/atocore-context.md)
---------------------------------------------------
- Project-level slash command following the standard frontmatter
  format (description + argument-hint)
- Parses the user prompt and an optional trailing project id, with
  case-insensitive matching against the registered project ids
  (atocore, p04-gigabit, p05-interferometer, p06-polisher and
  their aliases)
- Calls POST /context/build on the live AtoCore service, defaulting
  to http://dalidou:8100 (overridable via ATOCORE_API_BASE env var)
- Renders the formatted context pack inline so the user can see
  exactly what AtoCore would feed an LLM, plus a stats banner and a
  per-chunk source list
- Includes graceful failure handling for network errors, 4xx, 5xx,
  and the empty-result case
- Defines a future capture path that POSTs to /interactions for the
  Phase 9 reflection loop. The current command leaves capture as
  manual / opt-in pending a clean post-turn hook design

.gitignore changes
------------------
- Replaced wholesale .claude/ ignore with .claude/* + exceptions
  for .claude/commands/ so project slash commands can be tracked
- Other .claude/* paths (worktrees, settings, local state) remain
  ignored

Backup-restore procedure (docs/backup-restore-procedure.md)
-----------------------------------------------------------
- Defines what gets backed up (SQLite + registry always, Chroma
  optional under ingestion lock) and what doesn't (sources, code,
  logs, cache, tmp)
- Documents the snapshot directory layout and the timestamp format
- Three trigger paths in priority order:
  - via POST /admin/backup with {include_chroma: true|false}
  - via the standalone src/atocore/ops/backup.py module
  - via cold filesystem copy with brief downtime as last resort
- Listing and validation procedure with the /admin/backup and
  /admin/backup/{stamp}/validate endpoints
- Full step-by-step restore procedure with mandatory pre-flight
  safety snapshot, ownership/permission requirements, and the
  post-restore verification checks
- Rollback path using the pre-restore safety copy
- Retention policy (last 7 daily / 4 weekly / 6 monthly) and
  explicit acknowledgment that the cleanup job is not yet
  implemented
- Drill schedule: quarterly full restore drill, post-migration
  drill, post-incident validation
- Common failure mode table with diagnoses
- Quickstart cheat sheet at the end for daily reference
- Open follow-ups: cleanup script, off-Dalidou target,
  encryption, automatic post-backup validation, incremental
  Chroma snapshots

The procedure has not yet been exercised against the live Dalidou
instance — that is the next step the user runs themselves once
the slash command is in place.
2026-04-07 06:46:50 -04:00
8 changed files with 1929 additions and 9 deletions

View File

@@ -0,0 +1,123 @@
---
description: Pull a context pack from the live AtoCore service for the current prompt
argument-hint: <prompt text> [project-id]
---
You are about to enrich a user prompt with context from the live AtoCore
service. This is the daily-use entry point for AtoCore from inside Claude
Code.
## Step 1 — parse the arguments
The user invoked `/atocore-context` with the following arguments:
```
$ARGUMENTS
```
Treat the **entire argument string** as the prompt text by default. If the
last whitespace-separated token looks like a registered project id (matches
one of `atocore`, `p04-gigabit`, `p04`, `p05-interferometer`, `p05`,
`p06-polisher`, `p06`, or any case-insensitive variant), treat it as the
project hint and use the rest as the prompt text. Otherwise, leave the
project hint empty.
## Step 2 — call the AtoCore /context/build endpoint
Use the Bash tool to call AtoCore. The default endpoint is the live
Dalidou instance. Read `ATOCORE_API_BASE` from the environment if set,
otherwise default to `http://dalidou:3000` (the gitea host) — wait,
no, AtoCore lives on a different port. Default to `http://dalidou:8100`
which is the AtoCore service port from `pyproject.toml` and `config.py`.
Build the JSON body with `jq -n` so quoting is safe. Run something like:
```bash
ATOCORE_API_BASE="${ATOCORE_API_BASE:-http://dalidou:8100}"
PROMPT_TEXT='<the prompt text from step 1>'
PROJECT_HINT='<the project hint or empty string>'
if [ -n "$PROJECT_HINT" ]; then
BODY=$(jq -n --arg p "$PROMPT_TEXT" --arg proj "$PROJECT_HINT" \
'{prompt:$p, project:$proj}')
else
BODY=$(jq -n --arg p "$PROMPT_TEXT" '{prompt:$p}')
fi
curl -fsS -X POST "$ATOCORE_API_BASE/context/build" \
-H "Content-Type: application/json" \
-d "$BODY"
```
If `jq` is not available on the host, fall back to a Python one-liner:
```bash
python -c "import json,sys; print(json.dumps({'prompt': sys.argv[1], 'project': sys.argv[2]} if sys.argv[2] else {'prompt': sys.argv[1]}))" "$PROMPT_TEXT" "$PROJECT_HINT"
```
## Step 3 — present the context pack to the user
The response is JSON with at least these fields:
`formatted_context`, `chunks_used`, `total_chars`, `budget`,
`budget_remaining`, `duration_ms`, and a `chunks` array.
Print the response in a readable summary:
1. Print a one-line stats banner: `chunks=N, chars=X/budget, duration=Yms`
2. Print the `formatted_context` block verbatim inside a fenced text
code block so the user can read what AtoCore would feed an LLM
3. Print the `chunks` array as a small bulleted list with `source_file`,
`heading_path`, and `score` per chunk
If the response is empty (`chunks_used=0`, no project state, no
memories), tell the user explicitly: "AtoCore returned no context for
this prompt — either the corpus does not have relevant information or
the project hint is wrong. Try `/atocore-context <prompt> <project-id>`."
If the curl call fails:
- Network error → tell the user the AtoCore service may be down at
`$ATOCORE_API_BASE` and suggest checking `curl $ATOCORE_API_BASE/health`
- 4xx → print the error body verbatim, the API error message is usually
enough
- 5xx → print the error body and suggest checking the service logs
## Step 4 — capture the interaction (optional, opt-in)
If the user has previously asked the assistant to capture interactions
into AtoCore (or if the slash command was invoked with the trailing
literal `--capture` token), also POST the captured exchange to
`/interactions` so the Phase 9 reflection loop sees it. Skip this step
silently otherwise. The capture body is:
```json
{
"prompt": "<user prompt>",
"response": "",
"response_summary": "",
"project": "<project hint or empty>",
"client": "claude-code-slash",
"session_id": "<a stable id for this Claude Code session>",
"memories_used": ["<from chunks array if available>"],
"chunks_used": ["<chunk_id from chunks array>"],
"context_pack": {"chunks_used": <N>, "total_chars": <X>}
}
```
Note that the response field stays empty here — the LLM hasn't actually
answered yet at the moment the slash command runs. A separate post-turn
hook (not part of this command) would update the same interaction with
the response, OR a follow-up `/atocore-record-response <interaction-id>`
command would do it. For now, leave that as future work.
## Notes for the assistant
- DO NOT invent project ids that aren't in the registry. If the user
passed something that doesn't match, treat it as part of the prompt.
- DO NOT silently fall back to a different endpoint. If `ATOCORE_API_BASE`
is wrong, surface the network error and let the user fix the env var.
- DO NOT hide the formatted context pack from the user. The whole point
of this command is to show what AtoCore would feed an LLM, so the user
can decide if it's relevant.
- The output goes into the user's working context as background — they
may follow up with their actual question, and the AtoCore context pack
acts as informal injected knowledge.

4
.gitignore vendored
View File

@@ -10,4 +10,6 @@ htmlcov/
.coverage
venv/
.venv/
.claude/
.claude/*
!.claude/commands/
!.claude/commands/**

View File

@@ -0,0 +1,434 @@
# Engineering Layer V1 Acceptance Criteria
## Why this document exists
The engineering layer planning sprint produced 7 architecture
docs. None of them on their own says "you're done with V1, ship
it". This document does. It translates the planning into
measurable, falsifiable acceptance criteria so the implementation
sprint can know unambiguously when V1 is complete.
The acceptance criteria are organized into four categories:
1. **Functional** — what the system must be able to do
2. **Quality** — how well it must do it
3. **Operational** — what running it must look like
4. **Documentation** — what must be written down
V1 is "done" only when **every criterion in this document is met
against at least one of the three active projects** (`p04-gigabit`,
`p05-interferometer`, `p06-polisher`). The choice of which
project is the test bed is up to the implementer, but the same
project must satisfy all functional criteria.
## The single-sentence definition
> AtoCore Engineering Layer V1 is done when, against one chosen
> active project, every v1-required query in
> `engineering-query-catalog.md` returns a correct result, the
> Human Mirror renders a coherent project overview, and a real
> KB-CAD or KB-FEM export round-trips through the ingest →
> review queue → active entity flow without violating any
> conflict or trust invariant.
Everything below is the operational form of that sentence.
## Category 1 — Functional acceptance
### F-1: Entity store implemented per the V1 ontology
- The 12 V1 entity types from `engineering-ontology-v1.md` exist
in the database with the schema described there
- The 4 relationship families (Structural, Intent, Validation,
Provenance) are implemented as edges with the relationship
types listed in the catalog
- Every entity has the shared header fields:
`id, type, name, project_id, status, confidence, source_refs,
created_at, updated_at, extractor_version, canonical_home`
- The status lifecycle matches the memory layer:
`candidate → active → superseded | invalid`
### F-2: All v1-required queries return correct results
For the chosen test project, every query Q-001 through Q-020 in
`engineering-query-catalog.md` must:
- be implemented as an API endpoint with the shape specified in
the catalog
- return the expected result shape against real data
- include the provenance chain when the catalog requires it
- handle the empty case (no matches) gracefully — empty array,
not 500
The "killer correctness queries" — Q-006 (orphan requirements),
Q-009 (decisions on flagged assumptions), Q-011 (unsupported
validation claims) — are non-negotiable. If any of those three
returns wrong results, V1 is not done.
### F-3: Tool ingest endpoints are live
Both endpoints from `tool-handoff-boundaries.md` are implemented:
- `POST /ingest/kb-cad/export` accepts the documented JSON
shape, validates it, and produces entity candidates
- `POST /ingest/kb-fem/export` ditto
- Both refuse exports with invalid schemas (4xx with a clear
error)
- Both return a summary of created/dropped/failed counts
- Both never auto-promote anything; everything lands as
`status="candidate"`
- Both carry source identifiers (exporter name, exporter version,
source artifact id) into the candidate's provenance fields
A real KB-CAD export — even a hand-crafted one if the actual
exporter doesn't exist yet — must round-trip through the endpoint
and produce reviewable candidates for the test project.
### F-4: Candidate review queue works end to end
Per `promotion-rules.md`:
- `GET /entities?status=candidate` lists the queue
- `POST /entities/{id}/promote` moves candidate → active
- `POST /entities/{id}/reject` moves candidate → invalid
- The same shapes work for memories (already shipped in Phase 9 C)
- The reviewer can edit a candidate's content via
`PUT /entities/{id}` before promoting
- Every promote/reject is logged with timestamp and reason
### F-5: Conflict detection fires
Per `conflict-model.md`:
- The synchronous detector runs at every active write
(create, promote, project_state set, KB import)
- A test must demonstrate that pushing a contradictory KB-CAD
export creates a `conflicts` row with both members linked
- The reviewer can resolve the conflict via
`POST /conflicts/{id}/resolve` with one of the supported
actions (supersede_others, no_action, dismiss)
- Resolution updates the underlying entities according to the
chosen action
### F-6: Human Mirror renders for the test project
Per `human-mirror-rules.md`:
- `GET /mirror/{project}/overview` returns rendered markdown
- `GET /mirror/{project}/decisions` returns rendered markdown
- `GET /mirror/{project}/subsystems/{subsystem}` returns
rendered markdown for at least one subsystem
- `POST /mirror/{project}/regenerate` triggers regeneration on
demand
- Generated files appear under `/srv/storage/atocore/data/mirror/`
with the "do not edit" header banner
- Disputed markers appear inline when conflicts exist
- Project-state overrides display with the `(curated)` annotation
- Output is deterministic (the same inputs produce the same
bytes, suitable for diffing)
### F-7: Memory-to-entity graduation works for at least one type
Per `memory-vs-entities.md`:
- `POST /memory/{id}/graduate` exists
- Graduating a memory of type `adaptation` produces a Decision
entity candidate with the memory's content as a starting point
- The original memory row stays at `status="graduated"` (a new
status added by the engineering layer migration)
- The graduated memory has a forward pointer to the entity
candidate's id
- Promoting the entity candidate does NOT delete the original
memory
- The same graduation flow works for `project` → Requirement
and `knowledge` → Fact entity types (test the path; doesn't
have to be exhaustive)
### F-8: Provenance chain is complete
For every active entity in the test project, the following must
be true:
- It links back to at least one source via `source_refs` (which
is one or more of: source_chunk_id, source_interaction_id,
source_artifact_id from KB import)
- The provenance chain can be walked from the entity to the
underlying raw text (source_chunks) or external artifact
- Q-017 (the evidence query) returns at least one row for every
active entity
If any active entity has no provenance, it's a bug — provenance
is mandatory at write time per the promotion rules.
## Category 2 — Quality acceptance
### Q-1: All existing tests still pass
The full pre-V1 test suite (currently 160 tests) must still
pass. The V1 implementation may add new tests but cannot regress
any existing test.
### Q-2: V1 has its own test coverage
For each of F-1 through F-8 above, at least one automated test
exists that:
- exercises the happy path
- covers at least one error path
- runs in CI in under 10 seconds (no real network, no real LLM)
The full V1 test suite should be under 30 seconds total runtime
to keep the development loop fast.
### Q-3: Conflict invariants are enforced by tests
Specific tests must demonstrate:
- Two contradictory KB exports produce a conflict (not silent
overwrite)
- A reviewer can't accidentally promote both members of an open
conflict to active without resolving the conflict first
- The "flag, never block" rule holds — writes still succeed
even when they create a conflict
### Q-4: Trust hierarchy is enforced by tests
Specific tests must demonstrate:
- Entity candidates can never appear in context packs
- Reinforcement only touches active memories (already covered
by Phase 9 Commit B tests, but the same property must hold
for entities once they exist)
- Nothing automatically writes to project_state ever
- Candidates can never satisfy Q-005 (only active entities count)
### Q-5: The Human Mirror is reproducible
A golden-file test exists for at least one Mirror page. Updating
the golden file is a normal part of template work (single
command, well-documented). The test fails if the renderer
produces different bytes for the same input, catching
non-determinism.
### Q-6: Killer correctness queries pass against real-ish data
The test bed for Q-006, Q-009, Q-011 is not synthetic. The
implementation must seed the test project with at least:
- One Requirement that has a satisfying Component (Q-006 should
not flag it)
- One Requirement with no satisfying Component (Q-006 must flag it)
- One Decision based on an Assumption flagged as `needs_review`
(Q-009 must flag the Decision)
- One ValidationClaim with at least one supporting Result
(Q-011 should not flag it)
- One ValidationClaim with no supporting Result (Q-011 must flag it)
These five seed cases run as a single integration test that
exercises the killer correctness queries against actual
representative data.
## Category 3 — Operational acceptance
### O-1: Migration is safe and reversible
The V1 schema migration (adding the `entities`, `relationships`,
`conflicts`, `conflict_members` tables, plus `mirror_regeneration_failures`)
must:
- run cleanly against a production-shape database
- be implemented via the same `_apply_migrations` pattern as
Phase 9 (additive only, idempotent, safe to run twice)
- be tested by spinning up a fresh DB AND running against a
copy of the live Dalidou DB taken from a backup
### O-2: Backup and restore still work
The backup endpoint must include the new tables. A restore drill
on the test project must:
- successfully back up the V1 entity state via
`POST /admin/backup`
- successfully validate the snapshot
- successfully restore from the snapshot per
`docs/backup-restore-procedure.md`
- pass post-restore verification including a Q-001 query against
the test project
The drill must be performed once before V1 is declared done.
### O-3: Performance bounds
These are starting bounds; tune later if real usage shows
problems:
- Single-entity write (`POST /entities/...`): under 100ms p99
on the production Dalidou hardware
- Single Q-001 / Q-005 / Q-008 query: under 500ms p99 against
a project with up to 1000 entities
- Mirror regeneration of one project overview: under 5 seconds
for a project with up to 1000 entities
- Conflict detector at write time: adds no more than 50ms p99
to a write that doesn't actually produce a conflict
These bounds are not tested by automated benchmarks in V1 (that
would be over-engineering). They are sanity-checked by the
developer running the operations against the test project.
### O-4: No new manual ops burden
V1 should not introduce any new "you have to remember to run X
every day" requirement. Specifically:
- Mirror regeneration is automatic (debounced async + daily
refresh), no manual cron entry needed
- Conflict detection is automatic at write time, no manual sweep
needed in V1 (the nightly sweep is V2)
- Backup retention cleanup is **still** an open follow-up from
the operational baseline; V1 does not block on it
### O-5: No regressions in Phase 9 reflection loop
The capture, reinforcement, and extraction loop from Phase 9
A/B/C must continue to work end to end with the engineering
layer in place. Specifically:
- Memories whose types are NOT in the engineering layer
(identity, preference, episodic) keep working exactly as
before
- Memories whose types ARE in the engineering layer (project,
knowledge, adaptation) can still be created hand or by
extraction; the deprecation rule from `memory-vs-entities.md`
("no new writes after V1 ships") is implemented as a
configurable warning, not a hard block, so existing
workflows aren't disrupted
## Category 4 — Documentation acceptance
### D-1: Per-entity-type spec docs
Each of the 12 V1 entity types has a short spec doc under
`docs/architecture/entities/` covering:
- the entity's purpose
- its required and optional fields
- its lifecycle quirks (if any beyond the standard
candidate/active/superseded/invalid)
- which queries it appears in (cross-reference to the catalog)
- which relationship types reference it
These docs can be terse — a page each, mostly bullet lists.
Their purpose is to make the entity model legible to a future
maintainer, not to be reference manuals.
### D-2: KB-CAD and KB-FEM export schema docs
`docs/architecture/kb-cad-export-schema.md` and
`docs/architecture/kb-fem-export-schema.md` are written and
match the implemented validators.
### D-3: V1 release notes
A `docs/v1-release-notes.md` summarizes:
- What V1 added (entities, relationships, conflicts, mirror,
ingest endpoints)
- What V1 deferred (auto-promotion, BOM/cost/manufacturing
entities, NX direct integration, cross-project rollups)
- The migration story for existing memories (graduation flow)
- Known limitations and the V2 roadmap pointers
### D-4: master-plan-status.md and current-state.md updated
Both top-level status docs reflect V1's completion:
- Phase 6 (AtoDrive) and the engineering layer are explicitly
marked as separate tracks
- The engineering planning sprint section is marked complete
- Phase 9 stays at "baseline complete" (V1 doesn't change Phase 9)
- The engineering layer V1 is added as its own line item
## What V1 explicitly does NOT need to do
To prevent scope creep, here is the negative list. None of the
following are V1 acceptance criteria:
- **No LLM extractor.** The Phase 9 C rule-based extractor is
the entity extractor for V1 too, just with new rules added for
entity types.
- **No auto-promotion of candidates.** Per `promotion-rules.md`.
- **No write-back to KB-CAD or KB-FEM.** Per
`tool-handoff-boundaries.md`.
- **No multi-user / per-reviewer auth.** Single-user assumed.
- **No real-time UI.** API + Mirror markdown is the V1 surface.
A web UI is V2+.
- **No cross-project rollups.** Per `human-mirror-rules.md`.
- **No time-travel queries** (Q-015 stays v1-stretch).
- **No nightly conflict sweep.** Synchronous detection only in V1.
- **No incremental Chroma snapshots.** The current full-copy
approach in `backup-restore-procedure.md` is fine for V1.
- **No retention cleanup script.** Still an open follow-up.
- **No backup encryption.** Still an open follow-up.
- **No off-Dalidou backup target.** Still an open follow-up.
## How to use this document during implementation
When the implementation sprint begins:
1. Read this doc once, top to bottom
2. Pick the test project (probably p05-interferometer because
the optical/structural domain has the cleanest entity model)
3. For each section, write the test or the implementation, in
roughly the order: F-1 → F-2 → F-3 → F-4 → F-5 → F-6 → F-7 → F-8
4. Each acceptance criterion's test should be written **before
or alongside** the implementation, not after
5. Run the full test suite at every commit
6. When every box is checked, write D-3 (release notes), update
D-4 (status docs), and call V1 done
The implementation sprint should not touch anything outside the
scope listed here. If a desire arises to add something not in
this doc, that's a V2 conversation, not a V1 expansion.
## Anticipated friction points
These are the things I expect will be hard during implementation:
1. **The graduation flow (F-7)** is the most cross-cutting
change because it touches the existing memory module.
Worth doing it last so the memory module is stable for
all the V1 entity work first.
2. **The Mirror's deterministic-output requirement (Q-5)** will
bite if the implementer iterates over Python dicts without
sorting. Plan to use `sorted()` literally everywhere.
3. **Conflict detection (F-5)** has subtle correctness traps:
the slot key extraction must be stable, the dedup-of-existing-conflicts
logic must be right, and the synchronous detector must not
slow writes meaningfully (Q-3 / O-3 cover this, but watch).
4. **Provenance backfill** for entities that come from the
existing memory layer via graduation (F-7) is the trickiest
part: the original memory may not have had a strict
`source_chunk_id`, in which case the graduated entity also
doesn't have one. The implementation needs an "orphan
provenance" allowance for graduated entities, with a
warning surfaced in the Mirror.
These aren't blockers, just the parts of the V1 spec I'd
attack with extra care.
## TL;DR
- Engineering V1 is done when every box in this doc is checked
against one chosen active project
- Functional: 8 criteria covering entities, queries, ingest,
review queue, conflicts, mirror, graduation, provenance
- Quality: 6 criteria covering tests, golden files, killer
correctness, trust enforcement
- Operational: 5 criteria covering migration safety, backup
drill, performance bounds, no new manual ops, Phase 9 not
regressed
- Documentation: 4 criteria covering entity specs, KB schema
docs, release notes, top-level status updates
- Negative list: a clear set of things V1 deliberately does
NOT need to do, to prevent scope creep
- The implementation sprint follows this doc as a checklist

View File

@@ -0,0 +1,384 @@
# Human Mirror Rules (Layer 3 → derived markdown views)
## Why this document exists
The engineering layer V1 stores facts as typed entities and
relationships in a SQL database. That representation is excellent
for queries, conflict detection, and automated reasoning, but
it's terrible for the human reading experience. People want to
read prose, not crawl JSON.
The Human Mirror is the layer that turns the typed entity store
into human-readable markdown pages. It's strictly a derived view —
nothing in the Human Mirror is canonical, every page is regenerated
from current entity state on demand.
This document defines:
- what the Human Mirror generates
- when it regenerates
- how the human edits things they see in the Mirror
- how the canonical-vs-derived rule is enforced (so editing the
derived markdown can't silently corrupt the entity store)
## The non-negotiable rule
> **The Human Mirror is read-only from the human's perspective.**
>
> If the human wants to change a fact they see in the Mirror, they
> change it in the canonical home (per `representation-authority.md`),
> NOT in the Mirror page. The next regeneration picks up the change.
This rule is what makes the whole derived-view approach safe. If
the human is allowed to edit Mirror pages directly, the
canonical-vs-derived split breaks and the Mirror becomes a second
source of truth that disagrees with the entity store.
The technical enforcement is that every Mirror page carries a
header banner that says "this file is generated from AtoCore
entity state, do not edit", and the file is regenerated from the
entity store on every change to its underlying entities. Manual
edits will be silently overwritten on the next regeneration.
## What the Mirror generates in V1
Three template families, each producing one or more pages per
project:
### 1. Project Overview
One page per registered project. Renders:
- Project header (id, aliases, description)
- Subsystem tree (from Q-001 / Q-004 in the query catalog)
- Active Decisions affecting this project (Q-008, ordered by date)
- Open Requirements with coverage status (Q-005, Q-006)
- Open ValidationClaims with support status (Q-010, Q-011)
- Currently flagged conflicts (from the conflict model)
- Recent changes (Q-013) — last 14 days
This is the most important Mirror page. It's the page someone
opens when they want to know "what's the state of this project
right now". It deliberately mirrors what `current-state.md` does
for AtoCore itself but generated entirely from typed state.
### 2. Decision Log
One page per project. Renders:
- All active Decisions in chronological order (newest first)
- Each Decision shows: id, what was decided, when, the affected
Subsystem/Component, the supporting evidence (Q-014, Q-017)
- Superseded Decisions appear as collapsed "history" entries
with a forward link to whatever superseded them
- Conflicting Decisions get a "⚠ disputed" marker
This is the human-readable form of the engineering query catalog's
Q-014 query.
### 3. Subsystem Detail
One page per Subsystem (so a few per project). Renders:
- Subsystem header
- Components contained in this subsystem (Q-001)
- Interfaces this subsystem has (Q-003)
- Constraints applying to it (Q-007)
- Decisions affecting it (Q-008)
- Validation status: which Requirements are satisfied,
which are open (Q-005, Q-006)
- Change history within this subsystem (Q-013 scoped)
Subsystem detail pages are what someone reads when they're
working on a specific part of the system and want everything
relevant in one place.
## What the Mirror does NOT generate in V1
Intentionally excluded so the V1 implementation stays scoped:
- **Per-component detail pages.** Components are listed in
Subsystem pages but don't get their own pages. Reduces page
count from hundreds to dozens.
- **Per-Decision detail pages.** Decisions appear inline in
Project Overview and Decision Log; their full text plus
evidence chain is shown there, not on a separate page.
- **Cross-project rollup pages.** No "all projects at a glance"
page in V1. Each project is its own report.
- **Time-series / historical pages.** The Mirror is always
"current state". History is accessible via Decision Log and
superseded chains, but no "what was true on date X" page exists
in V1 (Q-015 is v1-stretch in the query catalog for the same
reason).
- **Diff pages between two timestamps.** Same reasoning.
- **Render of the conflict queue itself.** Conflicts appear
inline in the relevant Mirror pages with the "⚠ disputed"
marker and a link to `/conflicts/{id}`, but there's no
Mirror page that lists all conflicts. Use `GET /conflicts`.
- **Per-memory pages.** Memories are not engineering entities;
they appear in context packs and the review queue, not in the
Human Mirror.
## Where Mirror pages live
Two options were considered. The chosen V1 path is option B:
**Option A — write Mirror pages back into the source vault.**
Generate `/srv/storage/atocore/sources/vault/mirror/p05/overview.md`
so the human reads them in their normal Obsidian / markdown
viewer. **Rejected** because writing into the source vault
violates the "sources are read-only" rule from
`tool-handoff-boundaries.md` and the operating model.
**Option B (chosen) — write Mirror pages into a dedicated AtoCore
output dir, served via the API.** Generate under
`/srv/storage/atocore/data/mirror/p05/overview.md`. The human
reads them via:
- the API endpoints `GET /mirror/{project}/overview`,
`GET /mirror/{project}/decisions`,
`GET /mirror/{project}/subsystems/{subsystem}` (all return
rendered markdown as text/markdown)
- a future "Mirror viewer" in the Claude Code slash command
`/atocore-mirror <project>` that fetches the rendered markdown
and displays it inline
- direct file access on Dalidou for power users:
`cat /srv/storage/atocore/data/mirror/p05/overview.md`
The dedicated dir keeps the Mirror clearly separated from the
canonical sources and makes regeneration safe (it's just a
directory wipe + write).
## When the Mirror regenerates
Three triggers, in order from cheapest to most expensive:
### 1. On explicit human request
```
POST /mirror/{project}/regenerate
```
Returns the timestamp of the regeneration and the list of files
written. This is the path the human takes when they've just
curated something into project_state and want to see the Mirror
reflect it immediately.
### 2. On entity write (debounced, async, per project)
When any entity in a project changes status (candidate → active,
active → superseded), a regeneration of that project's Mirror is
queued. The queue is debounced — multiple writes within a 30-second
window only trigger one regeneration. This keeps the Mirror
"close to current" without generating a Mirror update on every
single API call.
The implementation is a simple dict of "next regeneration time"
per project, checked by a background task. No cron, no message
queue, no Celery. Just a `dict[str, datetime]` and a thread.
### 3. On scheduled refresh (daily)
Once per day at a quiet hour, every project's Mirror regenerates
unconditionally. This catches any state drift from manual
project_state edits that bypassed the entity write hooks, and
provides a baseline guarantee that the Mirror is at most 24
hours stale.
The schedule runs from the same machinery as the future backup
retention job, so we get one cron-equivalent system to maintain
instead of two.
## What if regeneration fails
The Mirror has to be resilient. If regeneration fails for a
project (e.g. a query catalog query crashes, a template rendering
error), the existing Mirror files are **not** deleted. The
existing files stay in place (showing the last successful state)
and a regeneration error is recorded in:
- the API response if the trigger was explicit
- a log entry at warning level for the async path
- a `mirror_regeneration_failures` table for the daily refresh
This means the human can always read the Mirror, even if the
last 5 minutes of changes haven't made it in yet. Stale is
better than blank.
## How the human curates "around" the Mirror
The Mirror reflects the current entity state. If the human
doesn't like what they see, the right edits go into one of:
| What you want to change | Where you change it |
|---|---|
| A Decision's text | `PUT /entities/Decision/{id}` (or `PUT /memory/{id}` if it's still memory-layer) |
| A Decision's status (active → superseded) | `POST /entities/Decision/{id}/supersede` (V1 entity API) |
| Whether a Component "satisfies" a Requirement | edit the relationship directly via the entity API (V1) |
| The current trusted next focus shown on the Project Overview | `POST /project/state` with `category=status, key=next_focus` |
| A typo in a generated heading or label | edit the **template**, not the rendered file. Templates live in `templates/mirror/` (V1 implementation) |
| Source of a fact ("this came from KB-CAD on day X") | not editable by hand — it's automatically populated from provenance |
The rule is consistent: edit the canonical home, regenerate (or
let the auto-trigger fire), see the change reflected in the
Mirror.
## Templates
The Mirror uses Jinja2-style templates checked into the repo
under `templates/mirror/`. Each template is a markdown file with
placeholders that the renderer fills from query catalog results.
Template list for V1:
- `templates/mirror/project-overview.md.j2`
- `templates/mirror/decision-log.md.j2`
- `templates/mirror/subsystem-detail.md.j2`
Editing a template is a code change, reviewed via normal git PRs.
The templates are deliberately small and readable so the human
can tweak the output format without touching renderer code.
The renderer is a thin module:
```python
# src/atocore/mirror/renderer.py (V1, not yet implemented)
def render_project_overview(project: str) -> str:
"""Generate the project overview markdown for one project."""
facts = collect_project_overview_facts(project)
template = load_template("project-overview.md.j2")
return template.render(**facts)
```
## The "do not edit" header
Every generated Mirror file starts with a fixed banner:
```markdown
<!--
This file is generated by AtoCore from current entity state.
DO NOT EDIT — manual changes will be silently overwritten on
the next regeneration.
Edit the canonical home instead. See:
https://docs.atocore.../representation-authority.md
Regenerated: 2026-04-07T12:34:56Z
Source entities: <commit-like checksum of input data>
-->
```
The checksum at the end lets the renderer skip work when nothing
relevant has changed since the last regeneration. If the inputs
match the previous run's checksum, the existing file is left
untouched.
## Conflicts in the Mirror
Per the conflict model, any open conflict on a fact that appears
in the Mirror gets a visible disputed marker:
```markdown
- Lateral support material: **GF-PTFE** ⚠ disputed
- The KB-CAD import on 2026-04-07 reported PEEK; conflict #c-039.
```
The disputed marker is a hyperlink (in renderer terms; the markdown
output is a relative link) to the conflict detail page in the API
or to the conflict id for direct lookup. The reviewer follows the
link, resolves the conflict via `POST /conflicts/{id}/resolve`,
and on the next regeneration the marker disappears.
## Project-state overrides in the Mirror
When a Mirror page would show a value derived from entities, but
project_state has an override on the same key, **the Mirror shows
the project_state value** with a small annotation noting the
override:
```markdown
- Next focus: **Wave 2 trusted-operational ingestion** (curated)
```
The `(curated)` annotation tells the reader "this is from the
trusted-state Layer 3, not from extracted entities". This makes
the trust hierarchy visible in the human reading experience.
## The "Mirror diff" workflow (post-V1, but designed for)
A common workflow after V1 ships will be:
1. Reviewer has curated some new entities
2. They want to see "what changed in the Mirror as a result"
3. They want to share that diff with someone else as evidence
To support this, the Mirror generator writes its output
deterministically (sorted iteration, stable timestamp formatting)
so a `git diff` between two regenerated states is meaningful.
V1 doesn't add an explicit "diff between two Mirror snapshots"
endpoint — that's deferred. But the deterministic-output
property is a V1 requirement so future diffing works without
re-renderer-design work.
## What the Mirror enables
With the Mirror in place:
- **OpenClaw can read project state in human form.** The
read-only AtoCore helper skill on the T420 already calls
`/context/build`; in V1 it gains the option to call
`/mirror/{project}/overview` to get a fully-rendered markdown
page instead of just retrieved chunks. This is much faster
than crawling individual entities for general questions.
- **The human gets a daily-readable artifact.** Every morning,
Antoine can `cat /srv/storage/atocore/data/mirror/p05/overview.md`
and see the current state of p05 in his preferred reading
format. No API calls, no JSON parsing.
- **Cross-collaborator sharing.** If you ever want to send
someone a project overview without giving them AtoCore access,
the Mirror file is a self-contained markdown document they can
read in any markdown viewer.
- **Claude Code integration.** A future
`/atocore-mirror <project>` slash command renders the Mirror
inline, complementing the existing `/atocore-context` command
with a human-readable view of "what does AtoCore think about
this project right now".
## Open questions for V1 implementation
1. **What's the regeneration debounce window?** 30 seconds is the
starting value but should be tuned with real usage.
2. **Does the daily refresh need a separate trigger mechanism, or
is it just a long-period entry in the same in-process scheduler
that handles the debounced async refreshes?** Probably the
latter — keep it simple.
3. **How are templates tested?** Likely a small set of fixture
project states + golden output files, with a single test that
asserts `render(fixture) == golden`. Updating golden files is
a normal part of template work.
4. **Are Mirror pages discoverable via a directory listing
endpoint?** `GET /mirror/{project}` returns the list of
available pages for that project. Probably yes; cheap to add.
5. **How does the Mirror handle a project that has zero entities
yet?** Render an empty-state page that says "no curated facts
yet — add some via /memory or /entities/Decision". Better than
a blank file.
## TL;DR
- The Human Mirror generates 3 template families per project
(Overview, Decision Log, Subsystem Detail) from current entity
state
- It's strictly read-only from the human's perspective; edits go
to the canonical home and the Mirror picks them up on
regeneration
- Three regeneration triggers: explicit POST, debounced
async-on-write, daily scheduled refresh
- Mirror files live in `/srv/storage/atocore/data/mirror/`
(NOT in the source vault — sources stay read-only)
- Conflicts and project_state overrides are visible inline in
the rendered markdown so the trust hierarchy shows through
- Templates are checked into the repo and edited via PR; the
rendered files are derived and never canonical
- Deterministic output is a V1 requirement so future diffing
works without rework

View File

@@ -0,0 +1,273 @@
# Representation Authority (canonical home matrix)
## Why this document exists
The same fact about an engineering project can show up in many
places: a markdown note in the PKM, a structured field in KB-CAD,
a commit message in a Gitea repo, an active memory in AtoCore, an
entity in the engineering layer, a row in trusted project state.
**Without an explicit rule about which representation is
authoritative for which kind of fact, the system will accumulate
contradictions and the human will lose trust in all of them.**
This document is the canonical-home matrix. Every kind of fact
that AtoCore handles has exactly one authoritative representation,
and every other place that holds a copy of that fact is, by
definition, a derived view that may be stale.
## The representations in scope
Six places where facts can live in this ecosystem:
| Layer | What it is | Who edits it | How it's structured |
|---|---|---|---|
| **PKM** | Antoine's Obsidian-style markdown vault under `/srv/storage/atocore/sources/vault/` | Antoine, by hand | unstructured markdown with optional frontmatter |
| **KB project** | the engineering Knowledge Base (KB-CAD / KB-FEM repos and any companion docs) | Antoine, semi-structured | per-tool typed records |
| **Gitea repos** | source code repos under `dalidou:3000/Antoine/*` (Fullum-Interferometer, polisher-sim, ATOCore itself, ...) | Antoine via git commits | code, READMEs, repo-specific markdown |
| **AtoCore memories** | rows in the `memories` table | hand-authored or extracted from interactions | typed (identity / preference / project / episodic / knowledge / adaptation) |
| **AtoCore entities** | rows in the `entities` table (V1, not yet built) | imported from KB exports or extracted from interactions | typed entities + relationships per the V1 ontology |
| **AtoCore project state** | rows in the `project_state` table (Layer 3, trusted) | hand-curated only, never automatic | category + key + value |
## The canonical home rule
> For each kind of fact, exactly one of the six representations is
> the authoritative source. The other five may hold derived
> copies, but they are not allowed to disagree with the
> authoritative one. When they disagree, the disagreement is a
> conflict and surfaces via the conflict model.
The matrix below assigns the authoritative representation per fact
kind. It is the practical answer to the question "where does this
fact actually live?" for daily decisions.
## The canonical-home matrix
| Fact kind | Canonical home | Why | How it gets into AtoCore |
|---|---|---|---|
| **CAD geometry** (the actual model) | NX (or successor CAD tool) | the only place that can render and validate it | not in AtoCore at all in V1 |
| **CAD-side structure** (subsystem tree, component list, materials, parameters) | KB-CAD | KB-CAD is the structured wrapper around NX | KB-CAD export → `/ingest/kb-cad/export` → entities |
| **FEM mesh & solver settings** | KB-FEM (wrapping the FEM tool) | only the solver representation can run | not in AtoCore at all in V1 |
| **FEM results & validation outcomes** | KB-FEM | KB-FEM owns the outcome records | KB-FEM export → `/ingest/kb-fem/export` → entities |
| **Source code** | Gitea repos | repos are version-controlled and reviewable | indirectly via repo markdown ingestion (Phase 1) |
| **Repo-level documentation** (READMEs, design docs in the repo) | Gitea repos | lives next to the code it documents | ingested as source chunks; never hand-edited in AtoCore |
| **Project-level prose notes** (decisions in long-form, journal-style entries, working notes) | PKM | the place Antoine actually writes when thinking | ingested as source chunks; the extractor proposes candidates from these for the review queue |
| **Identity** ("the user is a mechanical engineer running AtoCore") | AtoCore memories (`identity` type) | nowhere else holds personal identity | hand-authored via `POST /memory` or extracted from interactions |
| **Preference** ("prefers small reviewable diffs", "uses SI units") | AtoCore memories (`preference` type) | nowhere else holds personal preferences | hand-authored or extracted |
| **Episodic** ("on April 6 we debugged the EXDEV bug") | AtoCore memories (`episodic` type) | nowhere else has time-bound personal recall | extracted from captured interactions |
| **Decision** (a structured engineering decision) | AtoCore **entities** (Decision) once the engineering layer ships; AtoCore memories (`adaptation`) until then | needs structured supersession, audit trail, and link to affected components | extracted from PKM or interactions; promoted via review queue |
| **Requirement** | AtoCore **entities** (Requirement) | needs structured satisfaction tracking | extracted from PKM, KB-CAD, or interactions |
| **Constraint** | AtoCore **entities** (Constraint) | needs structured link to the entity it constrains | extracted from PKM, KB-CAD, or interactions |
| **Validation claim** | AtoCore **entities** (ValidationClaim) | needs structured link to supporting Result | extracted from KB-FEM exports or interactions |
| **Material** | KB-CAD if the material is on a real component; AtoCore entity (Material) if it's a project-wide material decision not yet attached to geometry | structured properties live in KB-CAD's material database | KB-CAD export, or hand-authored as a Material entity |
| **Parameter** | KB-CAD or KB-FEM depending on whether it's a geometry or solver parameter; AtoCore entity (Parameter) if it's a higher-level project parameter not in either tool | structured numeric values with units live in their tool of origin | KB export, or hand-authored |
| **Project status / current focus / next milestone** | AtoCore **project_state** (Layer 3) | the trust hierarchy says trusted state is the highest authority for "what is the current state of the project" | hand-curated via `POST /project/state` |
| **Architectural decision records (ADRs)** | depends on form: long-form ADR markdown lives in the repo; the structured fact about which ADR was selected lives in the AtoCore Decision entity | both representations are useful for different audiences | repo ingestion provides the prose; the entity is created by extraction or hand-authored |
| **Operational runbooks** | repo (next to the code they describe) | lives with the system it operates | not promoted into AtoCore entities — runbooks are reference material, not facts |
| **Backup metadata** (snapshot timestamps, integrity status) | the backup-metadata.json files under `/srv/storage/atocore/backups/` | each snapshot is its own self-describing record | not in AtoCore's database; queried via the `/admin/backup` endpoints |
| **Conversation history with AtoCore (interactions)** | AtoCore `interactions` table | nowhere else has the prompt + context pack + response triple | written by capture (Phase 9 Commit A) |
## The supremacy rule for cross-layer facts
When the same fact has copies in multiple representations and they
disagree, the trust hierarchy applies in this order:
1. **AtoCore project_state** (Layer 3) is highest authority for any
"current state of the project" question. This is why it requires
manual curation and never gets touched by automatic processes.
2. **The tool-of-origin canonical home** is highest authority for
facts that are tool-managed: KB-CAD wins over AtoCore entities
for CAD-side structure facts; KB-FEM wins for FEM result facts.
3. **AtoCore entities** are highest authority for facts that are
AtoCore-managed: Decisions, Requirements, Constraints,
ValidationClaims (when the supporting Results are still loose).
4. **Active AtoCore memories** are highest authority for personal
facts (identity, preference, episodic).
5. **Source chunks (PKM, repos, ingested docs)** are lowest
authority — they are the raw substrate from which higher layers
are extracted, but they may be stale, contradictory among
themselves, or out of date.
This is the same hierarchy enforced by `conflict-model.md`. This
document just makes it explicit per fact kind.
## Examples
### Example 1 — "what material does the lateral support pad use?"
Possible representations:
- KB-CAD has the field `component.lateral-support-pad.material = "GF-PTFE"`
- A PKM note from last month says "considering PEEK for the
lateral support, GF-PTFE was the previous choice"
- An AtoCore Material entity says `GF-PTFE`
- An AtoCore project_state entry says `p05 / decision /
lateral_support_material = GF-PTFE`
Which one wins for the question "what's the current material"?
- **project_state wins** if the query is "what is the current
trusted answer for p05's lateral support material" (Layer 3)
- **KB-CAD wins** if project_state has not been curated for this
field yet, because KB-CAD is the canonical home for CAD-side
structure
- **The Material entity** is a derived view from KB-CAD; if it
disagrees with KB-CAD, the entity is wrong and a conflict is
surfaced
- **The PKM note** is historical context, not authoritative for
"current"
### Example 2 — "did we decide to merge the bind mounts?"
Possible representations:
- A working session interaction is captured in the `interactions`
table with the response containing `## Decision: merge the two
bind mounts into one`
- The Phase 9 Commit C extractor produced a candidate adaptation
memory from that decision
- A reviewer promoted the candidate to active
- The AtoCore source repo has the actual code change in commit
`d0ff8b5` and the docker-compose.yml is in its post-merge form
Which one wins for "is this decision real and current"?
- **The Gitea repo** wins for "is this decision implemented" —
the docker-compose.yml is the canonical home for the actual
bind mount configuration
- **The active adaptation memory** wins for "did we decide this"
— that's exactly what the Commit C lifecycle is for
- **The interaction record** is the audit trail — it's
authoritative for "when did this conversation happen and what
did the LLM say", but not for "is this decision current"
- **The source chunks** from PKM are not relevant here because no
PKM note about this decision exists yet (and that's fine —
decisions don't have to live in PKM if they live in the repo
and the AtoCore memory)
### Example 3 — "what's p05's current next focus?"
Possible representations:
- The PKM has a `current-status.md` note updated last week
- AtoCore project_state has `p05 / status / next_focus = "wave 2 ingestion"`
- A captured interaction from yesterday discussed the next focus
at length
Which one wins?
- **project_state wins**, full stop. The trust hierarchy says
Layer 3 is canonical for current state. This is exactly the
reason project_state exists.
- The PKM note is historical context.
- The interaction is conversation history.
- If project_state and the PKM disagree, the human updates one or
the other to bring them in line — usually by re-curating
project_state if the conversation revealed a real change.
## What this means for the engineering layer V1 implementation
Several concrete consequences fall out of the matrix:
1. **The Material and Parameter entity types are mostly KB-CAD
shadows in V1.** They exist in AtoCore so other entities
(Decisions, Requirements) can reference them with structured
links, but their authoritative values come from KB-CAD imports.
If KB-CAD doesn't know about a material, the AtoCore entity is
the canonical home only because nothing else is.
2. **Decisions / Requirements / Constraints / ValidationClaims
are AtoCore-canonical.** These don't have a natural home in
KB-CAD or KB-FEM. They live in AtoCore as first-class entities
with full lifecycle and supersession.
3. **The PKM is never authoritative.** It is the substrate for
extraction. The reviewer promotes things out of it; they don't
point at PKM notes as the "current truth".
4. **project_state is the override layer.** Whenever the human
wants to declare "the current truth is X regardless of what
the entities and memories and KB exports say", they curate
into project_state. Layer 3 is intentionally small and
intentionally manual.
5. **The conflict model is the enforcement mechanism.** When two
representations disagree on a fact whose canonical home rule
should pick a winner, the conflict surfaces via the
`/conflicts` endpoint and the reviewer resolves it. The
matrix in this document tells the reviewer who is supposed
to win in each scenario; they're not making the decision blind.
## What the matrix does NOT define
1. **Facts about people other than the user.** No "team member"
entity, no per-collaborator preferences. AtoCore is
single-user in V1.
2. **Facts about AtoCore itself as a project.** Those are project
memories and project_state entries under `project=atocore`,
same lifecycle as any other project's facts.
3. **Vendor / supplier / cost facts.** Out of V1 scope.
4. **Time-bounded facts** (a value that was true between two
dates and may not be true now). The current matrix treats all
active facts as currently-true and uses supersession to
represent change. Temporal facts are a V2 concern.
5. **Cross-project shared facts** (a Material that is reused across
p04, p05, and p06). Currently each project has its own copy.
Cross-project deduplication is also a V2 concern.
## The "single canonical home" invariant in practice
The hard rule that every fact has exactly one canonical home is
the load-bearing invariant of this matrix. To enforce it
operationally:
- **Extraction never duplicates.** When the extractor scans an
interaction or a source chunk and proposes a candidate, the
candidate is dropped if it duplicates an already-active record
in the canonical home (the existing extractor implementation
already does this for memories; the entity extractor will
follow the same pattern).
- **Imports never duplicate.** When KB-CAD pushes the same
Component twice with the same value, the second push is
recognized as identical and updates the `last_imported_at`
timestamp without creating a new entity.
- **Imports surface drift as conflict.** When KB-CAD pushes the
same Component with a different value, that's a conflict per
the conflict model — never a silent overwrite.
- **Hand-curation into project_state always wins.** A
project_state entry can disagree with an entity or a KB
export; the project_state entry is correct by fiat (Layer 3
trust), and the reviewer is responsible for bringing the lower
layers in line if appropriate.
## Open questions for V1 implementation
1. **How does the reviewer see the canonical home for a fact in
the UI?** Probably by including the fact's authoritative
layer in the entity / memory detail view: "this Material is
currently mirrored from KB-CAD; the canonical home is KB-CAD".
2. **Who owns running the KB-CAD / KB-FEM exporter?** The
`tool-handoff-boundaries.md` doc lists this as an open
question; same answer applies here.
3. **Do we need an explicit `canonical_home` field on entity
rows?** A field that records "this entity is canonical here"
vs "this entity is a mirror of <external system>". Probably
yes; deferred to the entity schema spec.
4. **How are project_state overrides surfaced in the engineering
layer query results?** When a query (e.g. Q-001 "what does
this subsystem contain?") would return entity rows, the result
should also flag any project_state entries that contradict the
entities — letting the reviewer see the override at query
time, not just in the conflict queue.
## TL;DR
- Six representation layers: PKM, KB project, repos, AtoCore
memories, AtoCore entities, AtoCore project_state
- Every fact kind has exactly one canonical home
- The trust hierarchy resolves cross-layer conflicts:
project_state > tool-of-origin (KB-CAD/KB-FEM) > entities >
active memories > source chunks
- Decisions / Requirements / Constraints / ValidationClaims are
AtoCore-canonical (no other system has a natural home for them)
- Materials / Parameters / CAD-side structure are KB-CAD-canonical
- FEM results / validation outcomes are KB-FEM-canonical
- project_state is the human override layer, top of the
hierarchy, manually curated only
- Conflicts surface via `/conflicts` and the reviewer applies the
matrix to pick a winner

View File

@@ -0,0 +1,339 @@
# Tool Hand-off Boundaries (KB-CAD / KB-FEM and friends)
## Why this document exists
The engineering layer V1 will accumulate typed entities about
projects, subsystems, components, materials, requirements,
constraints, decisions, parameters, analysis models, results, and
validation claims. Many of those concepts also live in real
external tools — CAD systems, FEM solvers, BOM managers, PLM
databases, vendor portals.
The first big design decision before writing any entity-layer code
is: **what is AtoCore's read/write relationship with each of those
external tools?**
The wrong answer in either direction is expensive:
- Too read-only: AtoCore becomes a stale shadow of the tools and
loses the trust battle the moment a value drifts.
- Too bidirectional: AtoCore takes on responsibilities it can't
reliably honor (live sync, conflict resolution against external
schemas, write-back validation), and the project never ships.
This document picks a position for V1.
## The position
> **AtoCore is a one-way mirror in V1.** External tools push
> structured exports into AtoCore. AtoCore never pushes back.
That position has three corollaries:
1. **External tools remain the source of truth for everything they
already manage.** A CAD model is canonical for geometry; a FEM
project is canonical for meshes and solver settings; KB-CAD is
canonical for whatever KB-CAD already calls canonical.
2. **AtoCore is the source of truth for the *AtoCore-shaped*
record** of those facts: the Decision that selected the geometry,
the Requirement the geometry satisfies, the ValidationClaim the
FEM result supports. AtoCore does not duplicate the external
tool's primary representation; it stores the structured *facts
about* it.
3. **The boundary is enforced by absence.** No write endpoint in
AtoCore ever generates a `.prt`, a `.fem`, an export to a PLM
schema, or a vendor purchase order. If we find ourselves wanting
to add such an endpoint in V1, we should stop and reconsider
the V1 scope.
## Why one-way and not bidirectional
Bidirectional sync between independent systems is one of the
hardest problems in engineering software. The honest reasons we
are not attempting it in V1:
1. **Schema drift.** External tools evolve their schemas
independently. A bidirectional sync would have to track every
schema version of every external tool we touch. That is a
permanent maintenance tax.
2. **Conflict semantics.** When AtoCore and an external tool
disagree on the same field, "who wins" is a per-tool, per-field
decision. There is no general rule. Bidirectional sync would
require us to specify that decision exhaustively.
3. **Trust hierarchy.** AtoCore's whole point is the trust
hierarchy: trusted project state > entities > memories. If we
let entities push values back into the external tools, we
silently elevate AtoCore's confidence to "high enough to write
to a CAD model", which it almost never deserves.
4. **Velocity.** A bidirectional engineering layer is a
multi-year project. A one-way mirror is a months project. The
value-to-effort ratio favors one-way for V1 by an enormous
margin.
5. **Reversibility.** We can always add bidirectional sync later
on a per-tool basis once V1 has shown itself to be useful. We
cannot easily walk back a half-finished bidirectional sync that
has already corrupted data in someone's CAD model.
## Per-tool stance for V1
| External tool | V1 stance | What AtoCore reads in | What AtoCore writes back |
|---|---|---|---|
| **KB-CAD** (Antoine's CAD knowledge base) | one-way mirror | structured exports of subsystems, components, materials, parameters via a documented JSON or CSV shape | nothing |
| **KB-FEM** (Antoine's FEM knowledge base) | one-way mirror | structured exports of analysis models, results, validation claims | nothing |
| **NX / Siemens NX** (the CAD tool itself) | not connected in V1 | nothing direct — only what KB-CAD exports about NX projects | nothing |
| **PKM (Obsidian / markdown vault)** | already connected via the ingestion pipeline (Phase 1) | full markdown/text corpus per the ingestion-waves doc | nothing |
| **Gitea repos** | already connected via the ingestion pipeline | repo markdown/text per project | nothing |
| **OpenClaw** (the LLM agent) | already connected via the read-only helper skill on the T420 | nothing — OpenClaw reads from AtoCore | nothing — OpenClaw does not write into AtoCore |
| **AtoDrive** (operational truth layer, future) | future: bidirectional with AtoDrive itself, but AtoDrive is internal to AtoCore so this isn't an external tool boundary | n/a in V1 | n/a in V1 |
| **PLM / vendor portals / cost systems** | not in V1 scope | nothing | nothing |
## What "one-way mirror" actually looks like in code
AtoCore exposes an ingestion endpoint per external tool that
accepts a structured export and turns it into entity candidates.
The endpoint is read-side from AtoCore's perspective (it reads
from a file or HTTP body), even though the external tool is the
one initiating the call.
Proposed V1 ingestion endpoints:
```
POST /ingest/kb-cad/export body: KB-CAD export JSON
POST /ingest/kb-fem/export body: KB-FEM export JSON
```
Each endpoint:
1. Validates the export against the documented schema
2. Maps each export record to an entity candidate (status="candidate")
3. Carries the export's source identifier into the candidate's
provenance fields (source_artifact_id, exporter_version, etc.)
4. Returns a summary: how many candidates were created, how many
were dropped as duplicates, how many failed schema validation
5. Does NOT auto-promote anything
The KB-CAD and KB-FEM teams (which is to say, future-you) own the
exporter scripts that produce these JSON bodies. Those scripts
live in the KB-CAD / KB-FEM repos respectively, not in AtoCore.
## The export schemas (sketch, not final)
These are starting shapes, intentionally minimal. The schemas
will be refined in `kb-cad-export-schema.md` and
`kb-fem-export-schema.md` once the V1 ontology lands.
### KB-CAD export shape (starting sketch)
```json
{
"exporter": "kb-cad",
"exporter_version": "1.0.0",
"exported_at": "2026-04-07T12:00:00Z",
"project": "p05-interferometer",
"subsystems": [
{
"id": "subsystem.optical-frame",
"name": "Optical frame",
"parent": null,
"components": [
{
"id": "component.lateral-support-pad",
"name": "Lateral support pad",
"material": "GF-PTFE",
"parameters": {
"thickness_mm": 3.0,
"preload_n": 12.0
},
"source_artifact": "kb-cad://p05/subsystems/optical-frame#lateral-support"
}
]
}
]
}
```
### KB-FEM export shape (starting sketch)
```json
{
"exporter": "kb-fem",
"exporter_version": "1.0.0",
"exported_at": "2026-04-07T12:00:00Z",
"project": "p05-interferometer",
"analysis_models": [
{
"id": "model.optical-frame-modal",
"name": "Optical frame modal analysis v3",
"subsystem": "subsystem.optical-frame",
"results": [
{
"id": "result.first-mode-frequency",
"name": "First-mode frequency",
"value": 187.4,
"unit": "Hz",
"supports_validation_claim": "claim.frame-rigidity-min-150hz",
"source_artifact": "kb-fem://p05/models/optical-frame-modal#first-mode"
}
]
}
]
}
```
These shapes will evolve. The point of including them now is to
make the one-way mirror concrete: it is a small, well-defined
JSON shape, not "AtoCore reaches into KB-CAD's database".
## What AtoCore is allowed to do with the imported records
After ingestion, the imported records become entity candidates
in AtoCore's own table. From that point forward they follow the
exact same lifecycle as any other candidate:
- they sit at status="candidate" until a human reviews them
- the reviewer promotes them to status="active" or rejects them
- the active entities are queryable via the engineering query
catalog (Q-001 through Q-020)
- the active entities can be referenced from Decisions, Requirements,
ValidationClaims, etc. via the V1 relationship types
The imported records are never automatically pushed into trusted
project state, never modified in place after import (they are
superseded by re-imports, not edited), and never written back to
the external tool.
## What happens when KB-CAD changes a value AtoCore already has
This is the canonical "drift" scenario. The flow:
1. KB-CAD exports a fresh JSON. Component `component.lateral-support-pad`
now has `material: "PEEK"` instead of `material: "GF-PTFE"`.
2. AtoCore's ingestion endpoint sees the same `id` and a different
value.
3. The ingestion endpoint creates a new entity candidate with the
new value, **does NOT delete or modify the existing active
entity**, and creates a `conflicts` row linking the two members
(per the conflict model doc).
4. The reviewer sees an open conflict on the next visit to
`/conflicts`.
5. The reviewer either:
- **promotes the new value** (the active is superseded, the
candidate becomes the new active, the audit trail keeps both)
- **rejects the new value** (the candidate is invalidated, the
active stays — useful when the export was wrong)
- **dismisses the conflict** (declares them not actually about
the same thing, both stay active)
The reviewer never touches KB-CAD from AtoCore. If the resolution
implies a change in KB-CAD itself, the reviewer makes that change
in KB-CAD, then re-exports.
## What about NX directly?
NX (Siemens NX) is the underlying CAD tool that KB-CAD wraps.
**NX is not connected to AtoCore in V1.** Any facts about NX
projects flow through KB-CAD as the structured intermediate. This
gives us:
- **One schema to maintain.** AtoCore only has to understand the
KB-CAD export shape, not the NX API.
- **One ownership boundary.** KB-CAD owns the question of "what's
in NX". AtoCore owns the question of "what's in the typed
knowledge base".
- **Future flexibility.** When NX is replaced or upgraded, only
KB-CAD has to adapt; AtoCore doesn't notice.
The same logic applies to FEM solvers (Nastran, Abaqus, ANSYS):
KB-FEM is the structured intermediate, AtoCore never talks to the
solver directly.
## The hard-line invariants
These are the things V1 will not do, regardless of how convenient
they might seem:
1. **No write to external tools.** No POST/PUT/PATCH to any
external API, no file generation that gets written into a
CAD/FEM project tree, no email/chat sends.
2. **No live polling.** AtoCore does not poll KB-CAD or KB-FEM on
a schedule. Imports are explicit pushes from the external tool
into AtoCore's ingestion endpoint.
3. **No silent merging.** Every value drift surfaces as a
conflict for the reviewer (per the conflict model doc).
4. **No schema fan-out.** AtoCore does not store every field that
KB-CAD knows about. Only fields that map to one of the V1
entity types make it into AtoCore. Everything else is dropped
at the import boundary.
5. **No external-tool-specific logic in entity types.** A
`Component` in AtoCore is the same shape regardless of whether
it came from KB-CAD, KB-FEM, the PKM, or a hand-curated
project state entry. The source is recorded in provenance,
not in the entity shape.
## What this enables
With the one-way mirror locked in, V1 implementation can focus on:
- The entity table and its lifecycle
- The two `/ingest/kb-cad/export` and `/ingest/kb-fem/export`
endpoints with their JSON validators
- The candidate review queue extension (already designed in
`promotion-rules.md`)
- The conflict model (already designed in `conflict-model.md`)
- The query catalog implementation (already designed in
`engineering-query-catalog.md`)
None of those are unbounded. Each is a finite, well-defined
implementation task. The one-way mirror is the choice that makes
V1 finishable.
## What V2 might consider (deferred)
After V1 has been live and demonstrably useful for a quarter or
two, the questions that become reasonable to revisit:
1. **Selective write-back to KB-CAD for low-risk fields.** For
example, AtoCore could push back a "Decision id linked to this
component" annotation that KB-CAD then displays without it
being canonical there. Read-only annotations from AtoCore's
perspective, advisory metadata from KB-CAD's perspective.
2. **Live polling for very small payloads.** A daily poll of
"what subsystem ids exist in KB-CAD now" so AtoCore can flag
subsystems that disappeared from KB-CAD without an explicit
AtoCore invalidation.
3. **Direct NX integration** if the KB-CAD layer becomes a
bottleneck — but only if the friction is real, not theoretical.
4. **Cost / vendor / PLM connections** for projects where the
procurement cycle is part of the active engineering work.
None of these are V1 work and they are listed only so the V1
design intentionally leaves room for them later.
## Open questions for the V1 implementation sprint
1. **Where do the export schemas live?** Probably in
`docs/architecture/kb-cad-export-schema.md` and
`docs/architecture/kb-fem-export-schema.md`, drafted during
the implementation sprint.
2. **Who runs the exporter?** A scheduled job on the KB-CAD /
KB-FEM hosts, triggered by the human after a meaningful
change, or both?
3. **Is the export incremental or full?** Full is simpler but
more expensive. Incremental needs delta semantics. V1 starts
with full and revisits when full becomes too slow.
4. **How is the exporter authenticated to AtoCore?** Probably
the existing PAT model (one PAT per exporter, scoped to
`write:engineering-import` once that scope exists). Worth a
quick auth design pass before the endpoints exist.
## TL;DR
- AtoCore is a one-way mirror in V1: external tools push,
AtoCore reads, AtoCore never writes back
- Two import endpoints for V1: KB-CAD and KB-FEM, each with a
documented JSON export shape
- Drift surfaces as conflicts in the existing conflict model
- No NX, no FEM solvers, no PLM, no vendor portals, no
cost/BOM systems in V1
- Bidirectional sync is reserved for V2+ on a per-tool basis,
only after V1 demonstrates value

View File

@@ -0,0 +1,360 @@
# AtoCore Backup and Restore Procedure
## Scope
This document defines the operational procedure for backing up and
restoring AtoCore's machine state on the Dalidou deployment. It is
the practical companion to `docs/backup-strategy.md` (which defines
the strategy) and `src/atocore/ops/backup.py` (which implements the
mechanics).
The intent is that this procedure can be followed by anyone with
SSH access to Dalidou and the AtoCore admin endpoints.
## What gets backed up
A `create_runtime_backup` snapshot contains, in order of importance:
| Artifact | Source path on Dalidou | Backup destination | Always included |
|---|---|---|---|
| SQLite database | `/srv/storage/atocore/data/db/atocore.db` | `<backup_root>/db/atocore.db` | yes |
| Project registry JSON | `/srv/storage/atocore/config/project-registry.json` | `<backup_root>/config/project-registry.json` | yes (if file exists) |
| Backup metadata | (generated) | `<backup_root>/backup-metadata.json` | yes |
| Chroma vector store | `/srv/storage/atocore/data/chroma/` | `<backup_root>/chroma/` | only when `include_chroma=true` |
The SQLite snapshot uses the online `conn.backup()` API and is safe
to take while the database is in use. The Chroma snapshot is a cold
directory copy and is **only safe when no ingestion is running**;
the API endpoint enforces this by acquiring the ingestion lock for
the duration of the copy.
What is **not** in the backup:
- Source documents under `/srv/storage/atocore/sources/vault/` and
`/srv/storage/atocore/sources/drive/`. These are read-only
inputs and live in the user's PKM/Drive, which is backed up
separately by their own systems.
- Application code. The container image is the source of truth for
code; recovery means rebuilding the image, not restoring code from
a backup.
- Logs under `/srv/storage/atocore/logs/`.
- Embeddings cache under `/srv/storage/atocore/data/cache/`.
- Temp files under `/srv/storage/atocore/data/tmp/`.
## Backup root layout
Each backup snapshot lives in its own timestamped directory:
```
/srv/storage/atocore/backups/snapshots/
├── 20260407T060000Z/
│ ├── backup-metadata.json
│ ├── db/
│ │ └── atocore.db
│ ├── config/
│ │ └── project-registry.json
│ └── chroma/ # only if include_chroma=true
│ └── ...
├── 20260408T060000Z/
│ └── ...
└── ...
```
The timestamp is UTC, format `YYYYMMDDTHHMMSSZ`.
## Triggering a backup
### Option A — via the admin endpoint (preferred)
```bash
# DB + registry only (fast, safe at any time)
curl -fsS -X POST http://dalidou:8100/admin/backup \
-H "Content-Type: application/json" \
-d '{"include_chroma": false}'
# DB + registry + Chroma (acquires ingestion lock)
curl -fsS -X POST http://dalidou:8100/admin/backup \
-H "Content-Type: application/json" \
-d '{"include_chroma": true}'
```
The response is the backup metadata JSON. Save the `backup_root`
field — that's the directory the snapshot was written to.
### Option B — via the standalone script (when the API is down)
```bash
docker exec atocore python -m atocore.ops.backup
```
This runs `create_runtime_backup()` directly, without going through
the API or the ingestion lock. Use it only when the AtoCore service
itself is unhealthy and you can't hit the admin endpoint.
### Option C — manual file copy (last resort)
If both the API and the standalone script are unusable:
```bash
sudo systemctl stop atocore # or: docker compose stop atocore
sudo cp /srv/storage/atocore/data/db/atocore.db \
/srv/storage/atocore/backups/manual-$(date -u +%Y%m%dT%H%M%SZ).db
sudo cp /srv/storage/atocore/config/project-registry.json \
/srv/storage/atocore/backups/manual-$(date -u +%Y%m%dT%H%M%SZ).registry.json
sudo systemctl start atocore
```
This is a cold backup and requires brief downtime.
## Listing backups
```bash
curl -fsS http://dalidou:8100/admin/backup
```
Returns the configured `backup_dir` and a list of all snapshots
under it, with their full metadata if available.
Or, on the host directly:
```bash
ls -la /srv/storage/atocore/backups/snapshots/
```
## Validating a backup
Before relying on a backup for restore, validate it:
```bash
curl -fsS http://dalidou:8100/admin/backup/20260407T060000Z/validate
```
The validator:
- confirms the snapshot directory exists
- opens the SQLite snapshot and runs `PRAGMA integrity_check`
- parses the registry JSON
- confirms the Chroma directory exists (if it was included)
A valid backup returns `"valid": true` and an empty `errors` array.
A failing validation returns `"valid": false` with one or more
specific error strings (e.g. `db_integrity_check_failed`,
`registry_invalid_json`, `chroma_snapshot_missing`).
**Validate every backup at creation time.** A backup that has never
been validated is not actually a backup — it's just a hopeful copy
of bytes.
## Restore procedure
### Pre-flight (always)
1. Identify which snapshot you want to restore. List available
snapshots and pick by timestamp:
```bash
curl -fsS http://dalidou:8100/admin/backup | jq '.backups[].stamp'
```
2. Validate it. Refuse to restore an invalid backup:
```bash
STAMP=20260407T060000Z
curl -fsS http://dalidou:8100/admin/backup/$STAMP/validate | jq .
```
3. **Stop AtoCore.** SQLite cannot be hot-restored under a running
process and Chroma will not pick up new files until the process
restarts.
```bash
docker compose stop atocore
# or: sudo systemctl stop atocore
```
4. **Take a safety snapshot of the current state** before overwriting
it. This is your "if the restore makes things worse, here's the
undo" backup.
```bash
PRESERVE_STAMP=$(date -u +%Y%m%dT%H%M%SZ)
sudo cp /srv/storage/atocore/data/db/atocore.db \
/srv/storage/atocore/backups/pre-restore-$PRESERVE_STAMP.db
sudo cp /srv/storage/atocore/config/project-registry.json \
/srv/storage/atocore/backups/pre-restore-$PRESERVE_STAMP.registry.json 2>/dev/null || true
```
### Restore the SQLite database
```bash
SNAPSHOT_DIR=/srv/storage/atocore/backups/snapshots/$STAMP
sudo cp $SNAPSHOT_DIR/db/atocore.db \
/srv/storage/atocore/data/db/atocore.db
sudo chown 1000:1000 /srv/storage/atocore/data/db/atocore.db
sudo chmod 600 /srv/storage/atocore/data/db/atocore.db
```
The chown should match the gitea/atocore container user. Verify
by checking the existing perms before overwriting:
```bash
stat -c '%U:%G %a' /srv/storage/atocore/data/db/atocore.db
```
### Restore the project registry
```bash
if [ -f $SNAPSHOT_DIR/config/project-registry.json ]; then
sudo cp $SNAPSHOT_DIR/config/project-registry.json \
/srv/storage/atocore/config/project-registry.json
sudo chown 1000:1000 /srv/storage/atocore/config/project-registry.json
sudo chmod 644 /srv/storage/atocore/config/project-registry.json
fi
```
If the snapshot does not contain a registry, the current registry is
preserved. The pre-flight safety copy still gives you a recovery path
if you need to roll back.
### Restore the Chroma vector store (if it was in the snapshot)
```bash
if [ -d $SNAPSHOT_DIR/chroma ]; then
# Move the current chroma dir aside as a safety copy
sudo mv /srv/storage/atocore/data/chroma \
/srv/storage/atocore/data/chroma.pre-restore-$PRESERVE_STAMP
# Copy the snapshot in
sudo cp -a $SNAPSHOT_DIR/chroma /srv/storage/atocore/data/chroma
sudo chown -R 1000:1000 /srv/storage/atocore/data/chroma
fi
```
If the snapshot does NOT contain a Chroma dir but the SQLite
restore would leave the vector store and the SQL store inconsistent
(e.g. SQL has chunks the vector store doesn't), you have two
options:
- **Option 1: rebuild the vector store from source documents.** Run
ingestion fresh after the SQL restore. This regenerates embeddings
from the actual source files. Slow but produces a perfectly
consistent state.
- **Option 2: accept the inconsistency and live with stale-vector
filtering.** The retriever already drops vector results whose
SQL row no longer exists (`_existing_chunk_ids` filter), so the
inconsistency surfaces as missing results, not bad ones.
For an unplanned restore, Option 2 is the right immediate move.
Then schedule a fresh ingestion pass to rebuild the vector store
properly.
### Restart AtoCore
```bash
docker compose up -d atocore
# or: sudo systemctl start atocore
```
### Post-restore verification
```bash
# 1. Service is healthy
curl -fsS http://dalidou:8100/health | jq .
# 2. Stats look right
curl -fsS http://dalidou:8100/stats | jq .
# 3. Project registry loads
curl -fsS http://dalidou:8100/projects | jq '.projects | length'
# 4. A known-good context query returns non-empty results
curl -fsS -X POST http://dalidou:8100/context/build \
-H "Content-Type: application/json" \
-d '{"prompt": "what is p05 about", "project": "p05-interferometer"}' | jq '.chunks_used'
```
If any of these are wrong, the restore is bad. Roll back using the
pre-restore safety copy:
```bash
docker compose stop atocore
sudo cp /srv/storage/atocore/backups/pre-restore-$PRESERVE_STAMP.db \
/srv/storage/atocore/data/db/atocore.db
sudo cp /srv/storage/atocore/backups/pre-restore-$PRESERVE_STAMP.registry.json \
/srv/storage/atocore/config/project-registry.json 2>/dev/null || true
# If you also restored chroma:
sudo rm -rf /srv/storage/atocore/data/chroma
sudo mv /srv/storage/atocore/data/chroma.pre-restore-$PRESERVE_STAMP \
/srv/storage/atocore/data/chroma
docker compose up -d atocore
```
## Retention policy
- **Last 7 daily backups**: kept verbatim
- **Last 4 weekly backups** (Sunday): kept verbatim
- **Last 6 monthly backups** (1st of month): kept verbatim
- **Anything older**: deleted
The retention job is **not yet implemented** and is tracked as a
follow-up. Until then, the snapshots directory grows monotonically.
A simple cron-based cleanup script is the next step:
```cron
0 4 * * * /srv/storage/atocore/scripts/cleanup-old-backups.sh
```
## Drill schedule
A backup that has never been restored is theoretical. The schedule:
- **At least once per quarter**, perform a full restore drill on a
staging environment (or a temporary container with a separate
data dir) and verify the post-restore checks pass.
- **After every breaking schema migration**, perform a restore drill
to confirm the migration is reversible.
- **After any incident** that touched the storage layer (the EXDEV
bug from April 2026 is a good example), confirm the next backup
validates clean.
## Common failure modes and what to do about them
| Symptom | Likely cause | Action |
|---|---|---|
| `db_integrity_check_failed` on validation | SQLite snapshot copied while a write was in progress, or disk corruption | Take a fresh backup and validate again. If it fails twice, suspect the underlying disk. |
| `registry_invalid_json` | Registry was being edited at backup time | Take a fresh backup. The registry is small so this is cheap. |
| `chroma_snapshot_missing` after a restore | Snapshot was DB-only and the restore didn't move the existing chroma dir | Either rebuild via fresh ingestion or restore an older snapshot that includes Chroma. |
| Service won't start after restore | Permissions wrong on the restored files | Re-run `chown 1000:1000` (or whatever the gitea/atocore container user is) on the data dir. |
| `/stats` returns 0 documents after restore | The SQL store was restored but the source paths in `source_documents` don't match the current Dalidou paths | This means the backup came from a different deployment. Don't trust this restore — it's pulling from the wrong layout. |
## Open follow-ups (not yet implemented)
1. **Retention cleanup script**: see the cron entry above.
2. **Off-Dalidou backup target**: currently snapshots live on the
same disk as the live data. A real disaster-recovery story
needs at least one snapshot on a different physical machine.
The simplest first step is a periodic `rsync` to the user's
laptop or to another server.
3. **Backup encryption**: snapshots contain raw SQLite and JSON.
Consider age/gpg encryption if backups will be shipped off-site.
4. **Automatic post-backup validation**: today the validator must
be invoked manually. The `create_runtime_backup` function
should call `validate_backup` on its own output and refuse to
declare success if validation fails.
5. **Chroma backup is currently full directory copy** every time.
For large vector stores this gets expensive. A future
improvement would be incremental snapshots via filesystem-level
snapshotting (LVM, btrfs, ZFS).
## Quickstart cheat sheet
```bash
# Daily backup (DB + registry only — fast)
curl -fsS -X POST http://dalidou:8100/admin/backup \
-H "Content-Type: application/json" -d '{}'
# Weekly backup (DB + registry + Chroma — slower, holds ingestion lock)
curl -fsS -X POST http://dalidou:8100/admin/backup \
-H "Content-Type: application/json" -d '{"include_chroma": true}'
# List backups
curl -fsS http://dalidou:8100/admin/backup | jq '.backups[].stamp'
# Validate the most recent backup
LATEST=$(curl -fsS http://dalidou:8100/admin/backup | jq -r '.backups[-1].stamp')
curl -fsS http://dalidou:8100/admin/backup/$LATEST/validate | jq .
# Full restore — see the "Restore procedure" section above
```

View File

@@ -44,8 +44,9 @@ read-only additive mode.
### Engineering Layer Planning Sprint
The engineering layer is intentionally in planning, not implementation.
The architecture docs below are the current state of that planning:
**Status: complete.** All 8 architecture docs are drafted. The
engineering layer is now ready for V1 implementation against the
active project set.
- [engineering-query-catalog.md](architecture/engineering-query-catalog.md) —
the 20 v1-required queries the engineering layer must answer
@@ -55,17 +56,21 @@ The architecture docs below are the current state of that planning:
Layer 0 → Layer 2 pipeline, triggers, review queue mechanics
- [conflict-model.md](architecture/conflict-model.md) —
detection, representation, and resolution of contradictory facts
- [tool-handoff-boundaries.md](architecture/tool-handoff-boundaries.md) —
KB-CAD / KB-FEM one-way mirror stance, ingest endpoints, drift handling
- [representation-authority.md](architecture/representation-authority.md) —
canonical home matrix across PKM / KB / repos / AtoCore for 22 fact kinds
- [human-mirror-rules.md](architecture/human-mirror-rules.md) —
templates, regeneration triggers, edit flow, "do not edit" enforcement
- [engineering-v1-acceptance.md](architecture/engineering-v1-acceptance.md) —
measurable done definition with 23 acceptance criteria
- [engineering-knowledge-hybrid-architecture.md](architecture/engineering-knowledge-hybrid-architecture.md) —
the 5-layer model (from the previous planning wave)
- [engineering-ontology-v1.md](architecture/engineering-ontology-v1.md) —
the initial V1 object and relationship inventory (previous wave)
Still to draft before engineering-layer implementation begins:
- tool-handoff-boundaries.md (KB-CAD / KB-FEM read vs write)
- human-mirror-rules.md (templates, triggers, edit flow)
- representation-authority.md (PKM / KB / repo / AtoCore canonical home matrix)
- engineering-v1-acceptance.md (done definition)
The next concrete next step is the V1 implementation sprint, which
should follow engineering-v1-acceptance.md as its checklist.
## What Is Real Today