Compare commits
3 Commits
d0ff8b5738
...
d6ce6128cf
| Author | SHA1 | Date | |
|---|---|---|---|
| d6ce6128cf | |||
| 368adf2ebc | |||
| a637017900 |
123
.claude/commands/atocore-context.md
Normal file
123
.claude/commands/atocore-context.md
Normal file
@@ -0,0 +1,123 @@
|
|||||||
|
---
|
||||||
|
description: Pull a context pack from the live AtoCore service for the current prompt
|
||||||
|
argument-hint: <prompt text> [project-id]
|
||||||
|
---
|
||||||
|
|
||||||
|
You are about to enrich a user prompt with context from the live AtoCore
|
||||||
|
service. This is the daily-use entry point for AtoCore from inside Claude
|
||||||
|
Code.
|
||||||
|
|
||||||
|
## Step 1 — parse the arguments
|
||||||
|
|
||||||
|
The user invoked `/atocore-context` with the following arguments:
|
||||||
|
|
||||||
|
```
|
||||||
|
$ARGUMENTS
|
||||||
|
```
|
||||||
|
|
||||||
|
Treat the **entire argument string** as the prompt text by default. If the
|
||||||
|
last whitespace-separated token looks like a registered project id (matches
|
||||||
|
one of `atocore`, `p04-gigabit`, `p04`, `p05-interferometer`, `p05`,
|
||||||
|
`p06-polisher`, `p06`, or any case-insensitive variant), treat it as the
|
||||||
|
project hint and use the rest as the prompt text. Otherwise, leave the
|
||||||
|
project hint empty.
|
||||||
|
|
||||||
|
## Step 2 — call the AtoCore /context/build endpoint
|
||||||
|
|
||||||
|
Use the Bash tool to call AtoCore. The default endpoint is the live
|
||||||
|
Dalidou instance. Read `ATOCORE_API_BASE` from the environment if set,
|
||||||
|
otherwise default to `http://dalidou:3000` (the gitea host) — wait,
|
||||||
|
no, AtoCore lives on a different port. Default to `http://dalidou:8100`
|
||||||
|
which is the AtoCore service port from `pyproject.toml` and `config.py`.
|
||||||
|
|
||||||
|
Build the JSON body with `jq -n` so quoting is safe. Run something like:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ATOCORE_API_BASE="${ATOCORE_API_BASE:-http://dalidou:8100}"
|
||||||
|
PROMPT_TEXT='<the prompt text from step 1>'
|
||||||
|
PROJECT_HINT='<the project hint or empty string>'
|
||||||
|
|
||||||
|
if [ -n "$PROJECT_HINT" ]; then
|
||||||
|
BODY=$(jq -n --arg p "$PROMPT_TEXT" --arg proj "$PROJECT_HINT" \
|
||||||
|
'{prompt:$p, project:$proj}')
|
||||||
|
else
|
||||||
|
BODY=$(jq -n --arg p "$PROMPT_TEXT" '{prompt:$p}')
|
||||||
|
fi
|
||||||
|
|
||||||
|
curl -fsS -X POST "$ATOCORE_API_BASE/context/build" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d "$BODY"
|
||||||
|
```
|
||||||
|
|
||||||
|
If `jq` is not available on the host, fall back to a Python one-liner:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python -c "import json,sys; print(json.dumps({'prompt': sys.argv[1], 'project': sys.argv[2]} if sys.argv[2] else {'prompt': sys.argv[1]}))" "$PROMPT_TEXT" "$PROJECT_HINT"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Step 3 — present the context pack to the user
|
||||||
|
|
||||||
|
The response is JSON with at least these fields:
|
||||||
|
`formatted_context`, `chunks_used`, `total_chars`, `budget`,
|
||||||
|
`budget_remaining`, `duration_ms`, and a `chunks` array.
|
||||||
|
|
||||||
|
Print the response in a readable summary:
|
||||||
|
|
||||||
|
1. Print a one-line stats banner: `chunks=N, chars=X/budget, duration=Yms`
|
||||||
|
2. Print the `formatted_context` block verbatim inside a fenced text
|
||||||
|
code block so the user can read what AtoCore would feed an LLM
|
||||||
|
3. Print the `chunks` array as a small bulleted list with `source_file`,
|
||||||
|
`heading_path`, and `score` per chunk
|
||||||
|
|
||||||
|
If the response is empty (`chunks_used=0`, no project state, no
|
||||||
|
memories), tell the user explicitly: "AtoCore returned no context for
|
||||||
|
this prompt — either the corpus does not have relevant information or
|
||||||
|
the project hint is wrong. Try `/atocore-context <prompt> <project-id>`."
|
||||||
|
|
||||||
|
If the curl call fails:
|
||||||
|
- Network error → tell the user the AtoCore service may be down at
|
||||||
|
`$ATOCORE_API_BASE` and suggest checking `curl $ATOCORE_API_BASE/health`
|
||||||
|
- 4xx → print the error body verbatim, the API error message is usually
|
||||||
|
enough
|
||||||
|
- 5xx → print the error body and suggest checking the service logs
|
||||||
|
|
||||||
|
## Step 4 — capture the interaction (optional, opt-in)
|
||||||
|
|
||||||
|
If the user has previously asked the assistant to capture interactions
|
||||||
|
into AtoCore (or if the slash command was invoked with the trailing
|
||||||
|
literal `--capture` token), also POST the captured exchange to
|
||||||
|
`/interactions` so the Phase 9 reflection loop sees it. Skip this step
|
||||||
|
silently otherwise. The capture body is:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"prompt": "<user prompt>",
|
||||||
|
"response": "",
|
||||||
|
"response_summary": "",
|
||||||
|
"project": "<project hint or empty>",
|
||||||
|
"client": "claude-code-slash",
|
||||||
|
"session_id": "<a stable id for this Claude Code session>",
|
||||||
|
"memories_used": ["<from chunks array if available>"],
|
||||||
|
"chunks_used": ["<chunk_id from chunks array>"],
|
||||||
|
"context_pack": {"chunks_used": <N>, "total_chars": <X>}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Note that the response field stays empty here — the LLM hasn't actually
|
||||||
|
answered yet at the moment the slash command runs. A separate post-turn
|
||||||
|
hook (not part of this command) would update the same interaction with
|
||||||
|
the response, OR a follow-up `/atocore-record-response <interaction-id>`
|
||||||
|
command would do it. For now, leave that as future work.
|
||||||
|
|
||||||
|
## Notes for the assistant
|
||||||
|
|
||||||
|
- DO NOT invent project ids that aren't in the registry. If the user
|
||||||
|
passed something that doesn't match, treat it as part of the prompt.
|
||||||
|
- DO NOT silently fall back to a different endpoint. If `ATOCORE_API_BASE`
|
||||||
|
is wrong, surface the network error and let the user fix the env var.
|
||||||
|
- DO NOT hide the formatted context pack from the user. The whole point
|
||||||
|
of this command is to show what AtoCore would feed an LLM, so the user
|
||||||
|
can decide if it's relevant.
|
||||||
|
- The output goes into the user's working context as background — they
|
||||||
|
may follow up with their actual question, and the AtoCore context pack
|
||||||
|
acts as informal injected knowledge.
|
||||||
4
.gitignore
vendored
4
.gitignore
vendored
@@ -10,4 +10,6 @@ htmlcov/
|
|||||||
.coverage
|
.coverage
|
||||||
venv/
|
venv/
|
||||||
.venv/
|
.venv/
|
||||||
.claude/
|
.claude/*
|
||||||
|
!.claude/commands/
|
||||||
|
!.claude/commands/**
|
||||||
|
|||||||
434
docs/architecture/engineering-v1-acceptance.md
Normal file
434
docs/architecture/engineering-v1-acceptance.md
Normal file
@@ -0,0 +1,434 @@
|
|||||||
|
# Engineering Layer V1 Acceptance Criteria
|
||||||
|
|
||||||
|
## Why this document exists
|
||||||
|
|
||||||
|
The engineering layer planning sprint produced 7 architecture
|
||||||
|
docs. None of them on their own says "you're done with V1, ship
|
||||||
|
it". This document does. It translates the planning into
|
||||||
|
measurable, falsifiable acceptance criteria so the implementation
|
||||||
|
sprint can know unambiguously when V1 is complete.
|
||||||
|
|
||||||
|
The acceptance criteria are organized into four categories:
|
||||||
|
|
||||||
|
1. **Functional** — what the system must be able to do
|
||||||
|
2. **Quality** — how well it must do it
|
||||||
|
3. **Operational** — what running it must look like
|
||||||
|
4. **Documentation** — what must be written down
|
||||||
|
|
||||||
|
V1 is "done" only when **every criterion in this document is met
|
||||||
|
against at least one of the three active projects** (`p04-gigabit`,
|
||||||
|
`p05-interferometer`, `p06-polisher`). The choice of which
|
||||||
|
project is the test bed is up to the implementer, but the same
|
||||||
|
project must satisfy all functional criteria.
|
||||||
|
|
||||||
|
## The single-sentence definition
|
||||||
|
|
||||||
|
> AtoCore Engineering Layer V1 is done when, against one chosen
|
||||||
|
> active project, every v1-required query in
|
||||||
|
> `engineering-query-catalog.md` returns a correct result, the
|
||||||
|
> Human Mirror renders a coherent project overview, and a real
|
||||||
|
> KB-CAD or KB-FEM export round-trips through the ingest →
|
||||||
|
> review queue → active entity flow without violating any
|
||||||
|
> conflict or trust invariant.
|
||||||
|
|
||||||
|
Everything below is the operational form of that sentence.
|
||||||
|
|
||||||
|
## Category 1 — Functional acceptance
|
||||||
|
|
||||||
|
### F-1: Entity store implemented per the V1 ontology
|
||||||
|
|
||||||
|
- The 12 V1 entity types from `engineering-ontology-v1.md` exist
|
||||||
|
in the database with the schema described there
|
||||||
|
- The 4 relationship families (Structural, Intent, Validation,
|
||||||
|
Provenance) are implemented as edges with the relationship
|
||||||
|
types listed in the catalog
|
||||||
|
- Every entity has the shared header fields:
|
||||||
|
`id, type, name, project_id, status, confidence, source_refs,
|
||||||
|
created_at, updated_at, extractor_version, canonical_home`
|
||||||
|
- The status lifecycle matches the memory layer:
|
||||||
|
`candidate → active → superseded | invalid`
|
||||||
|
|
||||||
|
### F-2: All v1-required queries return correct results
|
||||||
|
|
||||||
|
For the chosen test project, every query Q-001 through Q-020 in
|
||||||
|
`engineering-query-catalog.md` must:
|
||||||
|
|
||||||
|
- be implemented as an API endpoint with the shape specified in
|
||||||
|
the catalog
|
||||||
|
- return the expected result shape against real data
|
||||||
|
- include the provenance chain when the catalog requires it
|
||||||
|
- handle the empty case (no matches) gracefully — empty array,
|
||||||
|
not 500
|
||||||
|
|
||||||
|
The "killer correctness queries" — Q-006 (orphan requirements),
|
||||||
|
Q-009 (decisions on flagged assumptions), Q-011 (unsupported
|
||||||
|
validation claims) — are non-negotiable. If any of those three
|
||||||
|
returns wrong results, V1 is not done.
|
||||||
|
|
||||||
|
### F-3: Tool ingest endpoints are live
|
||||||
|
|
||||||
|
Both endpoints from `tool-handoff-boundaries.md` are implemented:
|
||||||
|
|
||||||
|
- `POST /ingest/kb-cad/export` accepts the documented JSON
|
||||||
|
shape, validates it, and produces entity candidates
|
||||||
|
- `POST /ingest/kb-fem/export` ditto
|
||||||
|
- Both refuse exports with invalid schemas (4xx with a clear
|
||||||
|
error)
|
||||||
|
- Both return a summary of created/dropped/failed counts
|
||||||
|
- Both never auto-promote anything; everything lands as
|
||||||
|
`status="candidate"`
|
||||||
|
- Both carry source identifiers (exporter name, exporter version,
|
||||||
|
source artifact id) into the candidate's provenance fields
|
||||||
|
|
||||||
|
A real KB-CAD export — even a hand-crafted one if the actual
|
||||||
|
exporter doesn't exist yet — must round-trip through the endpoint
|
||||||
|
and produce reviewable candidates for the test project.
|
||||||
|
|
||||||
|
### F-4: Candidate review queue works end to end
|
||||||
|
|
||||||
|
Per `promotion-rules.md`:
|
||||||
|
|
||||||
|
- `GET /entities?status=candidate` lists the queue
|
||||||
|
- `POST /entities/{id}/promote` moves candidate → active
|
||||||
|
- `POST /entities/{id}/reject` moves candidate → invalid
|
||||||
|
- The same shapes work for memories (already shipped in Phase 9 C)
|
||||||
|
- The reviewer can edit a candidate's content via
|
||||||
|
`PUT /entities/{id}` before promoting
|
||||||
|
- Every promote/reject is logged with timestamp and reason
|
||||||
|
|
||||||
|
### F-5: Conflict detection fires
|
||||||
|
|
||||||
|
Per `conflict-model.md`:
|
||||||
|
|
||||||
|
- The synchronous detector runs at every active write
|
||||||
|
(create, promote, project_state set, KB import)
|
||||||
|
- A test must demonstrate that pushing a contradictory KB-CAD
|
||||||
|
export creates a `conflicts` row with both members linked
|
||||||
|
- The reviewer can resolve the conflict via
|
||||||
|
`POST /conflicts/{id}/resolve` with one of the supported
|
||||||
|
actions (supersede_others, no_action, dismiss)
|
||||||
|
- Resolution updates the underlying entities according to the
|
||||||
|
chosen action
|
||||||
|
|
||||||
|
### F-6: Human Mirror renders for the test project
|
||||||
|
|
||||||
|
Per `human-mirror-rules.md`:
|
||||||
|
|
||||||
|
- `GET /mirror/{project}/overview` returns rendered markdown
|
||||||
|
- `GET /mirror/{project}/decisions` returns rendered markdown
|
||||||
|
- `GET /mirror/{project}/subsystems/{subsystem}` returns
|
||||||
|
rendered markdown for at least one subsystem
|
||||||
|
- `POST /mirror/{project}/regenerate` triggers regeneration on
|
||||||
|
demand
|
||||||
|
- Generated files appear under `/srv/storage/atocore/data/mirror/`
|
||||||
|
with the "do not edit" header banner
|
||||||
|
- Disputed markers appear inline when conflicts exist
|
||||||
|
- Project-state overrides display with the `(curated)` annotation
|
||||||
|
- Output is deterministic (the same inputs produce the same
|
||||||
|
bytes, suitable for diffing)
|
||||||
|
|
||||||
|
### F-7: Memory-to-entity graduation works for at least one type
|
||||||
|
|
||||||
|
Per `memory-vs-entities.md`:
|
||||||
|
|
||||||
|
- `POST /memory/{id}/graduate` exists
|
||||||
|
- Graduating a memory of type `adaptation` produces a Decision
|
||||||
|
entity candidate with the memory's content as a starting point
|
||||||
|
- The original memory row stays at `status="graduated"` (a new
|
||||||
|
status added by the engineering layer migration)
|
||||||
|
- The graduated memory has a forward pointer to the entity
|
||||||
|
candidate's id
|
||||||
|
- Promoting the entity candidate does NOT delete the original
|
||||||
|
memory
|
||||||
|
- The same graduation flow works for `project` → Requirement
|
||||||
|
and `knowledge` → Fact entity types (test the path; doesn't
|
||||||
|
have to be exhaustive)
|
||||||
|
|
||||||
|
### F-8: Provenance chain is complete
|
||||||
|
|
||||||
|
For every active entity in the test project, the following must
|
||||||
|
be true:
|
||||||
|
|
||||||
|
- It links back to at least one source via `source_refs` (which
|
||||||
|
is one or more of: source_chunk_id, source_interaction_id,
|
||||||
|
source_artifact_id from KB import)
|
||||||
|
- The provenance chain can be walked from the entity to the
|
||||||
|
underlying raw text (source_chunks) or external artifact
|
||||||
|
- Q-017 (the evidence query) returns at least one row for every
|
||||||
|
active entity
|
||||||
|
|
||||||
|
If any active entity has no provenance, it's a bug — provenance
|
||||||
|
is mandatory at write time per the promotion rules.
|
||||||
|
|
||||||
|
## Category 2 — Quality acceptance
|
||||||
|
|
||||||
|
### Q-1: All existing tests still pass
|
||||||
|
|
||||||
|
The full pre-V1 test suite (currently 160 tests) must still
|
||||||
|
pass. The V1 implementation may add new tests but cannot regress
|
||||||
|
any existing test.
|
||||||
|
|
||||||
|
### Q-2: V1 has its own test coverage
|
||||||
|
|
||||||
|
For each of F-1 through F-8 above, at least one automated test
|
||||||
|
exists that:
|
||||||
|
|
||||||
|
- exercises the happy path
|
||||||
|
- covers at least one error path
|
||||||
|
- runs in CI in under 10 seconds (no real network, no real LLM)
|
||||||
|
|
||||||
|
The full V1 test suite should be under 30 seconds total runtime
|
||||||
|
to keep the development loop fast.
|
||||||
|
|
||||||
|
### Q-3: Conflict invariants are enforced by tests
|
||||||
|
|
||||||
|
Specific tests must demonstrate:
|
||||||
|
|
||||||
|
- Two contradictory KB exports produce a conflict (not silent
|
||||||
|
overwrite)
|
||||||
|
- A reviewer can't accidentally promote both members of an open
|
||||||
|
conflict to active without resolving the conflict first
|
||||||
|
- The "flag, never block" rule holds — writes still succeed
|
||||||
|
even when they create a conflict
|
||||||
|
|
||||||
|
### Q-4: Trust hierarchy is enforced by tests
|
||||||
|
|
||||||
|
Specific tests must demonstrate:
|
||||||
|
|
||||||
|
- Entity candidates can never appear in context packs
|
||||||
|
- Reinforcement only touches active memories (already covered
|
||||||
|
by Phase 9 Commit B tests, but the same property must hold
|
||||||
|
for entities once they exist)
|
||||||
|
- Nothing automatically writes to project_state ever
|
||||||
|
- Candidates can never satisfy Q-005 (only active entities count)
|
||||||
|
|
||||||
|
### Q-5: The Human Mirror is reproducible
|
||||||
|
|
||||||
|
A golden-file test exists for at least one Mirror page. Updating
|
||||||
|
the golden file is a normal part of template work (single
|
||||||
|
command, well-documented). The test fails if the renderer
|
||||||
|
produces different bytes for the same input, catching
|
||||||
|
non-determinism.
|
||||||
|
|
||||||
|
### Q-6: Killer correctness queries pass against real-ish data
|
||||||
|
|
||||||
|
The test bed for Q-006, Q-009, Q-011 is not synthetic. The
|
||||||
|
implementation must seed the test project with at least:
|
||||||
|
|
||||||
|
- One Requirement that has a satisfying Component (Q-006 should
|
||||||
|
not flag it)
|
||||||
|
- One Requirement with no satisfying Component (Q-006 must flag it)
|
||||||
|
- One Decision based on an Assumption flagged as `needs_review`
|
||||||
|
(Q-009 must flag the Decision)
|
||||||
|
- One ValidationClaim with at least one supporting Result
|
||||||
|
(Q-011 should not flag it)
|
||||||
|
- One ValidationClaim with no supporting Result (Q-011 must flag it)
|
||||||
|
|
||||||
|
These five seed cases run as a single integration test that
|
||||||
|
exercises the killer correctness queries against actual
|
||||||
|
representative data.
|
||||||
|
|
||||||
|
## Category 3 — Operational acceptance
|
||||||
|
|
||||||
|
### O-1: Migration is safe and reversible
|
||||||
|
|
||||||
|
The V1 schema migration (adding the `entities`, `relationships`,
|
||||||
|
`conflicts`, `conflict_members` tables, plus `mirror_regeneration_failures`)
|
||||||
|
must:
|
||||||
|
|
||||||
|
- run cleanly against a production-shape database
|
||||||
|
- be implemented via the same `_apply_migrations` pattern as
|
||||||
|
Phase 9 (additive only, idempotent, safe to run twice)
|
||||||
|
- be tested by spinning up a fresh DB AND running against a
|
||||||
|
copy of the live Dalidou DB taken from a backup
|
||||||
|
|
||||||
|
### O-2: Backup and restore still work
|
||||||
|
|
||||||
|
The backup endpoint must include the new tables. A restore drill
|
||||||
|
on the test project must:
|
||||||
|
|
||||||
|
- successfully back up the V1 entity state via
|
||||||
|
`POST /admin/backup`
|
||||||
|
- successfully validate the snapshot
|
||||||
|
- successfully restore from the snapshot per
|
||||||
|
`docs/backup-restore-procedure.md`
|
||||||
|
- pass post-restore verification including a Q-001 query against
|
||||||
|
the test project
|
||||||
|
|
||||||
|
The drill must be performed once before V1 is declared done.
|
||||||
|
|
||||||
|
### O-3: Performance bounds
|
||||||
|
|
||||||
|
These are starting bounds; tune later if real usage shows
|
||||||
|
problems:
|
||||||
|
|
||||||
|
- Single-entity write (`POST /entities/...`): under 100ms p99
|
||||||
|
on the production Dalidou hardware
|
||||||
|
- Single Q-001 / Q-005 / Q-008 query: under 500ms p99 against
|
||||||
|
a project with up to 1000 entities
|
||||||
|
- Mirror regeneration of one project overview: under 5 seconds
|
||||||
|
for a project with up to 1000 entities
|
||||||
|
- Conflict detector at write time: adds no more than 50ms p99
|
||||||
|
to a write that doesn't actually produce a conflict
|
||||||
|
|
||||||
|
These bounds are not tested by automated benchmarks in V1 (that
|
||||||
|
would be over-engineering). They are sanity-checked by the
|
||||||
|
developer running the operations against the test project.
|
||||||
|
|
||||||
|
### O-4: No new manual ops burden
|
||||||
|
|
||||||
|
V1 should not introduce any new "you have to remember to run X
|
||||||
|
every day" requirement. Specifically:
|
||||||
|
|
||||||
|
- Mirror regeneration is automatic (debounced async + daily
|
||||||
|
refresh), no manual cron entry needed
|
||||||
|
- Conflict detection is automatic at write time, no manual sweep
|
||||||
|
needed in V1 (the nightly sweep is V2)
|
||||||
|
- Backup retention cleanup is **still** an open follow-up from
|
||||||
|
the operational baseline; V1 does not block on it
|
||||||
|
|
||||||
|
### O-5: No regressions in Phase 9 reflection loop
|
||||||
|
|
||||||
|
The capture, reinforcement, and extraction loop from Phase 9
|
||||||
|
A/B/C must continue to work end to end with the engineering
|
||||||
|
layer in place. Specifically:
|
||||||
|
|
||||||
|
- Memories whose types are NOT in the engineering layer
|
||||||
|
(identity, preference, episodic) keep working exactly as
|
||||||
|
before
|
||||||
|
- Memories whose types ARE in the engineering layer (project,
|
||||||
|
knowledge, adaptation) can still be created hand or by
|
||||||
|
extraction; the deprecation rule from `memory-vs-entities.md`
|
||||||
|
("no new writes after V1 ships") is implemented as a
|
||||||
|
configurable warning, not a hard block, so existing
|
||||||
|
workflows aren't disrupted
|
||||||
|
|
||||||
|
## Category 4 — Documentation acceptance
|
||||||
|
|
||||||
|
### D-1: Per-entity-type spec docs
|
||||||
|
|
||||||
|
Each of the 12 V1 entity types has a short spec doc under
|
||||||
|
`docs/architecture/entities/` covering:
|
||||||
|
|
||||||
|
- the entity's purpose
|
||||||
|
- its required and optional fields
|
||||||
|
- its lifecycle quirks (if any beyond the standard
|
||||||
|
candidate/active/superseded/invalid)
|
||||||
|
- which queries it appears in (cross-reference to the catalog)
|
||||||
|
- which relationship types reference it
|
||||||
|
|
||||||
|
These docs can be terse — a page each, mostly bullet lists.
|
||||||
|
Their purpose is to make the entity model legible to a future
|
||||||
|
maintainer, not to be reference manuals.
|
||||||
|
|
||||||
|
### D-2: KB-CAD and KB-FEM export schema docs
|
||||||
|
|
||||||
|
`docs/architecture/kb-cad-export-schema.md` and
|
||||||
|
`docs/architecture/kb-fem-export-schema.md` are written and
|
||||||
|
match the implemented validators.
|
||||||
|
|
||||||
|
### D-3: V1 release notes
|
||||||
|
|
||||||
|
A `docs/v1-release-notes.md` summarizes:
|
||||||
|
|
||||||
|
- What V1 added (entities, relationships, conflicts, mirror,
|
||||||
|
ingest endpoints)
|
||||||
|
- What V1 deferred (auto-promotion, BOM/cost/manufacturing
|
||||||
|
entities, NX direct integration, cross-project rollups)
|
||||||
|
- The migration story for existing memories (graduation flow)
|
||||||
|
- Known limitations and the V2 roadmap pointers
|
||||||
|
|
||||||
|
### D-4: master-plan-status.md and current-state.md updated
|
||||||
|
|
||||||
|
Both top-level status docs reflect V1's completion:
|
||||||
|
|
||||||
|
- Phase 6 (AtoDrive) and the engineering layer are explicitly
|
||||||
|
marked as separate tracks
|
||||||
|
- The engineering planning sprint section is marked complete
|
||||||
|
- Phase 9 stays at "baseline complete" (V1 doesn't change Phase 9)
|
||||||
|
- The engineering layer V1 is added as its own line item
|
||||||
|
|
||||||
|
## What V1 explicitly does NOT need to do
|
||||||
|
|
||||||
|
To prevent scope creep, here is the negative list. None of the
|
||||||
|
following are V1 acceptance criteria:
|
||||||
|
|
||||||
|
- **No LLM extractor.** The Phase 9 C rule-based extractor is
|
||||||
|
the entity extractor for V1 too, just with new rules added for
|
||||||
|
entity types.
|
||||||
|
- **No auto-promotion of candidates.** Per `promotion-rules.md`.
|
||||||
|
- **No write-back to KB-CAD or KB-FEM.** Per
|
||||||
|
`tool-handoff-boundaries.md`.
|
||||||
|
- **No multi-user / per-reviewer auth.** Single-user assumed.
|
||||||
|
- **No real-time UI.** API + Mirror markdown is the V1 surface.
|
||||||
|
A web UI is V2+.
|
||||||
|
- **No cross-project rollups.** Per `human-mirror-rules.md`.
|
||||||
|
- **No time-travel queries** (Q-015 stays v1-stretch).
|
||||||
|
- **No nightly conflict sweep.** Synchronous detection only in V1.
|
||||||
|
- **No incremental Chroma snapshots.** The current full-copy
|
||||||
|
approach in `backup-restore-procedure.md` is fine for V1.
|
||||||
|
- **No retention cleanup script.** Still an open follow-up.
|
||||||
|
- **No backup encryption.** Still an open follow-up.
|
||||||
|
- **No off-Dalidou backup target.** Still an open follow-up.
|
||||||
|
|
||||||
|
## How to use this document during implementation
|
||||||
|
|
||||||
|
When the implementation sprint begins:
|
||||||
|
|
||||||
|
1. Read this doc once, top to bottom
|
||||||
|
2. Pick the test project (probably p05-interferometer because
|
||||||
|
the optical/structural domain has the cleanest entity model)
|
||||||
|
3. For each section, write the test or the implementation, in
|
||||||
|
roughly the order: F-1 → F-2 → F-3 → F-4 → F-5 → F-6 → F-7 → F-8
|
||||||
|
4. Each acceptance criterion's test should be written **before
|
||||||
|
or alongside** the implementation, not after
|
||||||
|
5. Run the full test suite at every commit
|
||||||
|
6. When every box is checked, write D-3 (release notes), update
|
||||||
|
D-4 (status docs), and call V1 done
|
||||||
|
|
||||||
|
The implementation sprint should not touch anything outside the
|
||||||
|
scope listed here. If a desire arises to add something not in
|
||||||
|
this doc, that's a V2 conversation, not a V1 expansion.
|
||||||
|
|
||||||
|
## Anticipated friction points
|
||||||
|
|
||||||
|
These are the things I expect will be hard during implementation:
|
||||||
|
|
||||||
|
1. **The graduation flow (F-7)** is the most cross-cutting
|
||||||
|
change because it touches the existing memory module.
|
||||||
|
Worth doing it last so the memory module is stable for
|
||||||
|
all the V1 entity work first.
|
||||||
|
2. **The Mirror's deterministic-output requirement (Q-5)** will
|
||||||
|
bite if the implementer iterates over Python dicts without
|
||||||
|
sorting. Plan to use `sorted()` literally everywhere.
|
||||||
|
3. **Conflict detection (F-5)** has subtle correctness traps:
|
||||||
|
the slot key extraction must be stable, the dedup-of-existing-conflicts
|
||||||
|
logic must be right, and the synchronous detector must not
|
||||||
|
slow writes meaningfully (Q-3 / O-3 cover this, but watch).
|
||||||
|
4. **Provenance backfill** for entities that come from the
|
||||||
|
existing memory layer via graduation (F-7) is the trickiest
|
||||||
|
part: the original memory may not have had a strict
|
||||||
|
`source_chunk_id`, in which case the graduated entity also
|
||||||
|
doesn't have one. The implementation needs an "orphan
|
||||||
|
provenance" allowance for graduated entities, with a
|
||||||
|
warning surfaced in the Mirror.
|
||||||
|
|
||||||
|
These aren't blockers, just the parts of the V1 spec I'd
|
||||||
|
attack with extra care.
|
||||||
|
|
||||||
|
## TL;DR
|
||||||
|
|
||||||
|
- Engineering V1 is done when every box in this doc is checked
|
||||||
|
against one chosen active project
|
||||||
|
- Functional: 8 criteria covering entities, queries, ingest,
|
||||||
|
review queue, conflicts, mirror, graduation, provenance
|
||||||
|
- Quality: 6 criteria covering tests, golden files, killer
|
||||||
|
correctness, trust enforcement
|
||||||
|
- Operational: 5 criteria covering migration safety, backup
|
||||||
|
drill, performance bounds, no new manual ops, Phase 9 not
|
||||||
|
regressed
|
||||||
|
- Documentation: 4 criteria covering entity specs, KB schema
|
||||||
|
docs, release notes, top-level status updates
|
||||||
|
- Negative list: a clear set of things V1 deliberately does
|
||||||
|
NOT need to do, to prevent scope creep
|
||||||
|
- The implementation sprint follows this doc as a checklist
|
||||||
384
docs/architecture/human-mirror-rules.md
Normal file
384
docs/architecture/human-mirror-rules.md
Normal file
@@ -0,0 +1,384 @@
|
|||||||
|
# Human Mirror Rules (Layer 3 → derived markdown views)
|
||||||
|
|
||||||
|
## Why this document exists
|
||||||
|
|
||||||
|
The engineering layer V1 stores facts as typed entities and
|
||||||
|
relationships in a SQL database. That representation is excellent
|
||||||
|
for queries, conflict detection, and automated reasoning, but
|
||||||
|
it's terrible for the human reading experience. People want to
|
||||||
|
read prose, not crawl JSON.
|
||||||
|
|
||||||
|
The Human Mirror is the layer that turns the typed entity store
|
||||||
|
into human-readable markdown pages. It's strictly a derived view —
|
||||||
|
nothing in the Human Mirror is canonical, every page is regenerated
|
||||||
|
from current entity state on demand.
|
||||||
|
|
||||||
|
This document defines:
|
||||||
|
|
||||||
|
- what the Human Mirror generates
|
||||||
|
- when it regenerates
|
||||||
|
- how the human edits things they see in the Mirror
|
||||||
|
- how the canonical-vs-derived rule is enforced (so editing the
|
||||||
|
derived markdown can't silently corrupt the entity store)
|
||||||
|
|
||||||
|
## The non-negotiable rule
|
||||||
|
|
||||||
|
> **The Human Mirror is read-only from the human's perspective.**
|
||||||
|
>
|
||||||
|
> If the human wants to change a fact they see in the Mirror, they
|
||||||
|
> change it in the canonical home (per `representation-authority.md`),
|
||||||
|
> NOT in the Mirror page. The next regeneration picks up the change.
|
||||||
|
|
||||||
|
This rule is what makes the whole derived-view approach safe. If
|
||||||
|
the human is allowed to edit Mirror pages directly, the
|
||||||
|
canonical-vs-derived split breaks and the Mirror becomes a second
|
||||||
|
source of truth that disagrees with the entity store.
|
||||||
|
|
||||||
|
The technical enforcement is that every Mirror page carries a
|
||||||
|
header banner that says "this file is generated from AtoCore
|
||||||
|
entity state, do not edit", and the file is regenerated from the
|
||||||
|
entity store on every change to its underlying entities. Manual
|
||||||
|
edits will be silently overwritten on the next regeneration.
|
||||||
|
|
||||||
|
## What the Mirror generates in V1
|
||||||
|
|
||||||
|
Three template families, each producing one or more pages per
|
||||||
|
project:
|
||||||
|
|
||||||
|
### 1. Project Overview
|
||||||
|
|
||||||
|
One page per registered project. Renders:
|
||||||
|
|
||||||
|
- Project header (id, aliases, description)
|
||||||
|
- Subsystem tree (from Q-001 / Q-004 in the query catalog)
|
||||||
|
- Active Decisions affecting this project (Q-008, ordered by date)
|
||||||
|
- Open Requirements with coverage status (Q-005, Q-006)
|
||||||
|
- Open ValidationClaims with support status (Q-010, Q-011)
|
||||||
|
- Currently flagged conflicts (from the conflict model)
|
||||||
|
- Recent changes (Q-013) — last 14 days
|
||||||
|
|
||||||
|
This is the most important Mirror page. It's the page someone
|
||||||
|
opens when they want to know "what's the state of this project
|
||||||
|
right now". It deliberately mirrors what `current-state.md` does
|
||||||
|
for AtoCore itself but generated entirely from typed state.
|
||||||
|
|
||||||
|
### 2. Decision Log
|
||||||
|
|
||||||
|
One page per project. Renders:
|
||||||
|
|
||||||
|
- All active Decisions in chronological order (newest first)
|
||||||
|
- Each Decision shows: id, what was decided, when, the affected
|
||||||
|
Subsystem/Component, the supporting evidence (Q-014, Q-017)
|
||||||
|
- Superseded Decisions appear as collapsed "history" entries
|
||||||
|
with a forward link to whatever superseded them
|
||||||
|
- Conflicting Decisions get a "⚠ disputed" marker
|
||||||
|
|
||||||
|
This is the human-readable form of the engineering query catalog's
|
||||||
|
Q-014 query.
|
||||||
|
|
||||||
|
### 3. Subsystem Detail
|
||||||
|
|
||||||
|
One page per Subsystem (so a few per project). Renders:
|
||||||
|
|
||||||
|
- Subsystem header
|
||||||
|
- Components contained in this subsystem (Q-001)
|
||||||
|
- Interfaces this subsystem has (Q-003)
|
||||||
|
- Constraints applying to it (Q-007)
|
||||||
|
- Decisions affecting it (Q-008)
|
||||||
|
- Validation status: which Requirements are satisfied,
|
||||||
|
which are open (Q-005, Q-006)
|
||||||
|
- Change history within this subsystem (Q-013 scoped)
|
||||||
|
|
||||||
|
Subsystem detail pages are what someone reads when they're
|
||||||
|
working on a specific part of the system and want everything
|
||||||
|
relevant in one place.
|
||||||
|
|
||||||
|
## What the Mirror does NOT generate in V1
|
||||||
|
|
||||||
|
Intentionally excluded so the V1 implementation stays scoped:
|
||||||
|
|
||||||
|
- **Per-component detail pages.** Components are listed in
|
||||||
|
Subsystem pages but don't get their own pages. Reduces page
|
||||||
|
count from hundreds to dozens.
|
||||||
|
- **Per-Decision detail pages.** Decisions appear inline in
|
||||||
|
Project Overview and Decision Log; their full text plus
|
||||||
|
evidence chain is shown there, not on a separate page.
|
||||||
|
- **Cross-project rollup pages.** No "all projects at a glance"
|
||||||
|
page in V1. Each project is its own report.
|
||||||
|
- **Time-series / historical pages.** The Mirror is always
|
||||||
|
"current state". History is accessible via Decision Log and
|
||||||
|
superseded chains, but no "what was true on date X" page exists
|
||||||
|
in V1 (Q-015 is v1-stretch in the query catalog for the same
|
||||||
|
reason).
|
||||||
|
- **Diff pages between two timestamps.** Same reasoning.
|
||||||
|
- **Render of the conflict queue itself.** Conflicts appear
|
||||||
|
inline in the relevant Mirror pages with the "⚠ disputed"
|
||||||
|
marker and a link to `/conflicts/{id}`, but there's no
|
||||||
|
Mirror page that lists all conflicts. Use `GET /conflicts`.
|
||||||
|
- **Per-memory pages.** Memories are not engineering entities;
|
||||||
|
they appear in context packs and the review queue, not in the
|
||||||
|
Human Mirror.
|
||||||
|
|
||||||
|
## Where Mirror pages live
|
||||||
|
|
||||||
|
Two options were considered. The chosen V1 path is option B:
|
||||||
|
|
||||||
|
**Option A — write Mirror pages back into the source vault.**
|
||||||
|
Generate `/srv/storage/atocore/sources/vault/mirror/p05/overview.md`
|
||||||
|
so the human reads them in their normal Obsidian / markdown
|
||||||
|
viewer. **Rejected** because writing into the source vault
|
||||||
|
violates the "sources are read-only" rule from
|
||||||
|
`tool-handoff-boundaries.md` and the operating model.
|
||||||
|
|
||||||
|
**Option B (chosen) — write Mirror pages into a dedicated AtoCore
|
||||||
|
output dir, served via the API.** Generate under
|
||||||
|
`/srv/storage/atocore/data/mirror/p05/overview.md`. The human
|
||||||
|
reads them via:
|
||||||
|
|
||||||
|
- the API endpoints `GET /mirror/{project}/overview`,
|
||||||
|
`GET /mirror/{project}/decisions`,
|
||||||
|
`GET /mirror/{project}/subsystems/{subsystem}` (all return
|
||||||
|
rendered markdown as text/markdown)
|
||||||
|
- a future "Mirror viewer" in the Claude Code slash command
|
||||||
|
`/atocore-mirror <project>` that fetches the rendered markdown
|
||||||
|
and displays it inline
|
||||||
|
- direct file access on Dalidou for power users:
|
||||||
|
`cat /srv/storage/atocore/data/mirror/p05/overview.md`
|
||||||
|
|
||||||
|
The dedicated dir keeps the Mirror clearly separated from the
|
||||||
|
canonical sources and makes regeneration safe (it's just a
|
||||||
|
directory wipe + write).
|
||||||
|
|
||||||
|
## When the Mirror regenerates
|
||||||
|
|
||||||
|
Three triggers, in order from cheapest to most expensive:
|
||||||
|
|
||||||
|
### 1. On explicit human request
|
||||||
|
|
||||||
|
```
|
||||||
|
POST /mirror/{project}/regenerate
|
||||||
|
```
|
||||||
|
|
||||||
|
Returns the timestamp of the regeneration and the list of files
|
||||||
|
written. This is the path the human takes when they've just
|
||||||
|
curated something into project_state and want to see the Mirror
|
||||||
|
reflect it immediately.
|
||||||
|
|
||||||
|
### 2. On entity write (debounced, async, per project)
|
||||||
|
|
||||||
|
When any entity in a project changes status (candidate → active,
|
||||||
|
active → superseded), a regeneration of that project's Mirror is
|
||||||
|
queued. The queue is debounced — multiple writes within a 30-second
|
||||||
|
window only trigger one regeneration. This keeps the Mirror
|
||||||
|
"close to current" without generating a Mirror update on every
|
||||||
|
single API call.
|
||||||
|
|
||||||
|
The implementation is a simple dict of "next regeneration time"
|
||||||
|
per project, checked by a background task. No cron, no message
|
||||||
|
queue, no Celery. Just a `dict[str, datetime]` and a thread.
|
||||||
|
|
||||||
|
### 3. On scheduled refresh (daily)
|
||||||
|
|
||||||
|
Once per day at a quiet hour, every project's Mirror regenerates
|
||||||
|
unconditionally. This catches any state drift from manual
|
||||||
|
project_state edits that bypassed the entity write hooks, and
|
||||||
|
provides a baseline guarantee that the Mirror is at most 24
|
||||||
|
hours stale.
|
||||||
|
|
||||||
|
The schedule runs from the same machinery as the future backup
|
||||||
|
retention job, so we get one cron-equivalent system to maintain
|
||||||
|
instead of two.
|
||||||
|
|
||||||
|
## What if regeneration fails
|
||||||
|
|
||||||
|
The Mirror has to be resilient. If regeneration fails for a
|
||||||
|
project (e.g. a query catalog query crashes, a template rendering
|
||||||
|
error), the existing Mirror files are **not** deleted. The
|
||||||
|
existing files stay in place (showing the last successful state)
|
||||||
|
and a regeneration error is recorded in:
|
||||||
|
|
||||||
|
- the API response if the trigger was explicit
|
||||||
|
- a log entry at warning level for the async path
|
||||||
|
- a `mirror_regeneration_failures` table for the daily refresh
|
||||||
|
|
||||||
|
This means the human can always read the Mirror, even if the
|
||||||
|
last 5 minutes of changes haven't made it in yet. Stale is
|
||||||
|
better than blank.
|
||||||
|
|
||||||
|
## How the human curates "around" the Mirror
|
||||||
|
|
||||||
|
The Mirror reflects the current entity state. If the human
|
||||||
|
doesn't like what they see, the right edits go into one of:
|
||||||
|
|
||||||
|
| What you want to change | Where you change it |
|
||||||
|
|---|---|
|
||||||
|
| A Decision's text | `PUT /entities/Decision/{id}` (or `PUT /memory/{id}` if it's still memory-layer) |
|
||||||
|
| A Decision's status (active → superseded) | `POST /entities/Decision/{id}/supersede` (V1 entity API) |
|
||||||
|
| Whether a Component "satisfies" a Requirement | edit the relationship directly via the entity API (V1) |
|
||||||
|
| The current trusted next focus shown on the Project Overview | `POST /project/state` with `category=status, key=next_focus` |
|
||||||
|
| A typo in a generated heading or label | edit the **template**, not the rendered file. Templates live in `templates/mirror/` (V1 implementation) |
|
||||||
|
| Source of a fact ("this came from KB-CAD on day X") | not editable by hand — it's automatically populated from provenance |
|
||||||
|
|
||||||
|
The rule is consistent: edit the canonical home, regenerate (or
|
||||||
|
let the auto-trigger fire), see the change reflected in the
|
||||||
|
Mirror.
|
||||||
|
|
||||||
|
## Templates
|
||||||
|
|
||||||
|
The Mirror uses Jinja2-style templates checked into the repo
|
||||||
|
under `templates/mirror/`. Each template is a markdown file with
|
||||||
|
placeholders that the renderer fills from query catalog results.
|
||||||
|
|
||||||
|
Template list for V1:
|
||||||
|
|
||||||
|
- `templates/mirror/project-overview.md.j2`
|
||||||
|
- `templates/mirror/decision-log.md.j2`
|
||||||
|
- `templates/mirror/subsystem-detail.md.j2`
|
||||||
|
|
||||||
|
Editing a template is a code change, reviewed via normal git PRs.
|
||||||
|
The templates are deliberately small and readable so the human
|
||||||
|
can tweak the output format without touching renderer code.
|
||||||
|
|
||||||
|
The renderer is a thin module:
|
||||||
|
|
||||||
|
```python
|
||||||
|
# src/atocore/mirror/renderer.py (V1, not yet implemented)
|
||||||
|
|
||||||
|
def render_project_overview(project: str) -> str:
|
||||||
|
"""Generate the project overview markdown for one project."""
|
||||||
|
facts = collect_project_overview_facts(project)
|
||||||
|
template = load_template("project-overview.md.j2")
|
||||||
|
return template.render(**facts)
|
||||||
|
```
|
||||||
|
|
||||||
|
## The "do not edit" header
|
||||||
|
|
||||||
|
Every generated Mirror file starts with a fixed banner:
|
||||||
|
|
||||||
|
```markdown
|
||||||
|
<!--
|
||||||
|
This file is generated by AtoCore from current entity state.
|
||||||
|
DO NOT EDIT — manual changes will be silently overwritten on
|
||||||
|
the next regeneration.
|
||||||
|
Edit the canonical home instead. See:
|
||||||
|
https://docs.atocore.../representation-authority.md
|
||||||
|
Regenerated: 2026-04-07T12:34:56Z
|
||||||
|
Source entities: <commit-like checksum of input data>
|
||||||
|
-->
|
||||||
|
```
|
||||||
|
|
||||||
|
The checksum at the end lets the renderer skip work when nothing
|
||||||
|
relevant has changed since the last regeneration. If the inputs
|
||||||
|
match the previous run's checksum, the existing file is left
|
||||||
|
untouched.
|
||||||
|
|
||||||
|
## Conflicts in the Mirror
|
||||||
|
|
||||||
|
Per the conflict model, any open conflict on a fact that appears
|
||||||
|
in the Mirror gets a visible disputed marker:
|
||||||
|
|
||||||
|
```markdown
|
||||||
|
- Lateral support material: **GF-PTFE** ⚠ disputed
|
||||||
|
- The KB-CAD import on 2026-04-07 reported PEEK; conflict #c-039.
|
||||||
|
```
|
||||||
|
|
||||||
|
The disputed marker is a hyperlink (in renderer terms; the markdown
|
||||||
|
output is a relative link) to the conflict detail page in the API
|
||||||
|
or to the conflict id for direct lookup. The reviewer follows the
|
||||||
|
link, resolves the conflict via `POST /conflicts/{id}/resolve`,
|
||||||
|
and on the next regeneration the marker disappears.
|
||||||
|
|
||||||
|
## Project-state overrides in the Mirror
|
||||||
|
|
||||||
|
When a Mirror page would show a value derived from entities, but
|
||||||
|
project_state has an override on the same key, **the Mirror shows
|
||||||
|
the project_state value** with a small annotation noting the
|
||||||
|
override:
|
||||||
|
|
||||||
|
```markdown
|
||||||
|
- Next focus: **Wave 2 trusted-operational ingestion** (curated)
|
||||||
|
```
|
||||||
|
|
||||||
|
The `(curated)` annotation tells the reader "this is from the
|
||||||
|
trusted-state Layer 3, not from extracted entities". This makes
|
||||||
|
the trust hierarchy visible in the human reading experience.
|
||||||
|
|
||||||
|
## The "Mirror diff" workflow (post-V1, but designed for)
|
||||||
|
|
||||||
|
A common workflow after V1 ships will be:
|
||||||
|
|
||||||
|
1. Reviewer has curated some new entities
|
||||||
|
2. They want to see "what changed in the Mirror as a result"
|
||||||
|
3. They want to share that diff with someone else as evidence
|
||||||
|
|
||||||
|
To support this, the Mirror generator writes its output
|
||||||
|
deterministically (sorted iteration, stable timestamp formatting)
|
||||||
|
so a `git diff` between two regenerated states is meaningful.
|
||||||
|
|
||||||
|
V1 doesn't add an explicit "diff between two Mirror snapshots"
|
||||||
|
endpoint — that's deferred. But the deterministic-output
|
||||||
|
property is a V1 requirement so future diffing works without
|
||||||
|
re-renderer-design work.
|
||||||
|
|
||||||
|
## What the Mirror enables
|
||||||
|
|
||||||
|
With the Mirror in place:
|
||||||
|
|
||||||
|
- **OpenClaw can read project state in human form.** The
|
||||||
|
read-only AtoCore helper skill on the T420 already calls
|
||||||
|
`/context/build`; in V1 it gains the option to call
|
||||||
|
`/mirror/{project}/overview` to get a fully-rendered markdown
|
||||||
|
page instead of just retrieved chunks. This is much faster
|
||||||
|
than crawling individual entities for general questions.
|
||||||
|
- **The human gets a daily-readable artifact.** Every morning,
|
||||||
|
Antoine can `cat /srv/storage/atocore/data/mirror/p05/overview.md`
|
||||||
|
and see the current state of p05 in his preferred reading
|
||||||
|
format. No API calls, no JSON parsing.
|
||||||
|
- **Cross-collaborator sharing.** If you ever want to send
|
||||||
|
someone a project overview without giving them AtoCore access,
|
||||||
|
the Mirror file is a self-contained markdown document they can
|
||||||
|
read in any markdown viewer.
|
||||||
|
- **Claude Code integration.** A future
|
||||||
|
`/atocore-mirror <project>` slash command renders the Mirror
|
||||||
|
inline, complementing the existing `/atocore-context` command
|
||||||
|
with a human-readable view of "what does AtoCore think about
|
||||||
|
this project right now".
|
||||||
|
|
||||||
|
## Open questions for V1 implementation
|
||||||
|
|
||||||
|
1. **What's the regeneration debounce window?** 30 seconds is the
|
||||||
|
starting value but should be tuned with real usage.
|
||||||
|
2. **Does the daily refresh need a separate trigger mechanism, or
|
||||||
|
is it just a long-period entry in the same in-process scheduler
|
||||||
|
that handles the debounced async refreshes?** Probably the
|
||||||
|
latter — keep it simple.
|
||||||
|
3. **How are templates tested?** Likely a small set of fixture
|
||||||
|
project states + golden output files, with a single test that
|
||||||
|
asserts `render(fixture) == golden`. Updating golden files is
|
||||||
|
a normal part of template work.
|
||||||
|
4. **Are Mirror pages discoverable via a directory listing
|
||||||
|
endpoint?** `GET /mirror/{project}` returns the list of
|
||||||
|
available pages for that project. Probably yes; cheap to add.
|
||||||
|
5. **How does the Mirror handle a project that has zero entities
|
||||||
|
yet?** Render an empty-state page that says "no curated facts
|
||||||
|
yet — add some via /memory or /entities/Decision". Better than
|
||||||
|
a blank file.
|
||||||
|
|
||||||
|
## TL;DR
|
||||||
|
|
||||||
|
- The Human Mirror generates 3 template families per project
|
||||||
|
(Overview, Decision Log, Subsystem Detail) from current entity
|
||||||
|
state
|
||||||
|
- It's strictly read-only from the human's perspective; edits go
|
||||||
|
to the canonical home and the Mirror picks them up on
|
||||||
|
regeneration
|
||||||
|
- Three regeneration triggers: explicit POST, debounced
|
||||||
|
async-on-write, daily scheduled refresh
|
||||||
|
- Mirror files live in `/srv/storage/atocore/data/mirror/`
|
||||||
|
(NOT in the source vault — sources stay read-only)
|
||||||
|
- Conflicts and project_state overrides are visible inline in
|
||||||
|
the rendered markdown so the trust hierarchy shows through
|
||||||
|
- Templates are checked into the repo and edited via PR; the
|
||||||
|
rendered files are derived and never canonical
|
||||||
|
- Deterministic output is a V1 requirement so future diffing
|
||||||
|
works without rework
|
||||||
273
docs/architecture/representation-authority.md
Normal file
273
docs/architecture/representation-authority.md
Normal file
@@ -0,0 +1,273 @@
|
|||||||
|
# Representation Authority (canonical home matrix)
|
||||||
|
|
||||||
|
## Why this document exists
|
||||||
|
|
||||||
|
The same fact about an engineering project can show up in many
|
||||||
|
places: a markdown note in the PKM, a structured field in KB-CAD,
|
||||||
|
a commit message in a Gitea repo, an active memory in AtoCore, an
|
||||||
|
entity in the engineering layer, a row in trusted project state.
|
||||||
|
**Without an explicit rule about which representation is
|
||||||
|
authoritative for which kind of fact, the system will accumulate
|
||||||
|
contradictions and the human will lose trust in all of them.**
|
||||||
|
|
||||||
|
This document is the canonical-home matrix. Every kind of fact
|
||||||
|
that AtoCore handles has exactly one authoritative representation,
|
||||||
|
and every other place that holds a copy of that fact is, by
|
||||||
|
definition, a derived view that may be stale.
|
||||||
|
|
||||||
|
## The representations in scope
|
||||||
|
|
||||||
|
Six places where facts can live in this ecosystem:
|
||||||
|
|
||||||
|
| Layer | What it is | Who edits it | How it's structured |
|
||||||
|
|---|---|---|---|
|
||||||
|
| **PKM** | Antoine's Obsidian-style markdown vault under `/srv/storage/atocore/sources/vault/` | Antoine, by hand | unstructured markdown with optional frontmatter |
|
||||||
|
| **KB project** | the engineering Knowledge Base (KB-CAD / KB-FEM repos and any companion docs) | Antoine, semi-structured | per-tool typed records |
|
||||||
|
| **Gitea repos** | source code repos under `dalidou:3000/Antoine/*` (Fullum-Interferometer, polisher-sim, ATOCore itself, ...) | Antoine via git commits | code, READMEs, repo-specific markdown |
|
||||||
|
| **AtoCore memories** | rows in the `memories` table | hand-authored or extracted from interactions | typed (identity / preference / project / episodic / knowledge / adaptation) |
|
||||||
|
| **AtoCore entities** | rows in the `entities` table (V1, not yet built) | imported from KB exports or extracted from interactions | typed entities + relationships per the V1 ontology |
|
||||||
|
| **AtoCore project state** | rows in the `project_state` table (Layer 3, trusted) | hand-curated only, never automatic | category + key + value |
|
||||||
|
|
||||||
|
## The canonical home rule
|
||||||
|
|
||||||
|
> For each kind of fact, exactly one of the six representations is
|
||||||
|
> the authoritative source. The other five may hold derived
|
||||||
|
> copies, but they are not allowed to disagree with the
|
||||||
|
> authoritative one. When they disagree, the disagreement is a
|
||||||
|
> conflict and surfaces via the conflict model.
|
||||||
|
|
||||||
|
The matrix below assigns the authoritative representation per fact
|
||||||
|
kind. It is the practical answer to the question "where does this
|
||||||
|
fact actually live?" for daily decisions.
|
||||||
|
|
||||||
|
## The canonical-home matrix
|
||||||
|
|
||||||
|
| Fact kind | Canonical home | Why | How it gets into AtoCore |
|
||||||
|
|---|---|---|---|
|
||||||
|
| **CAD geometry** (the actual model) | NX (or successor CAD tool) | the only place that can render and validate it | not in AtoCore at all in V1 |
|
||||||
|
| **CAD-side structure** (subsystem tree, component list, materials, parameters) | KB-CAD | KB-CAD is the structured wrapper around NX | KB-CAD export → `/ingest/kb-cad/export` → entities |
|
||||||
|
| **FEM mesh & solver settings** | KB-FEM (wrapping the FEM tool) | only the solver representation can run | not in AtoCore at all in V1 |
|
||||||
|
| **FEM results & validation outcomes** | KB-FEM | KB-FEM owns the outcome records | KB-FEM export → `/ingest/kb-fem/export` → entities |
|
||||||
|
| **Source code** | Gitea repos | repos are version-controlled and reviewable | indirectly via repo markdown ingestion (Phase 1) |
|
||||||
|
| **Repo-level documentation** (READMEs, design docs in the repo) | Gitea repos | lives next to the code it documents | ingested as source chunks; never hand-edited in AtoCore |
|
||||||
|
| **Project-level prose notes** (decisions in long-form, journal-style entries, working notes) | PKM | the place Antoine actually writes when thinking | ingested as source chunks; the extractor proposes candidates from these for the review queue |
|
||||||
|
| **Identity** ("the user is a mechanical engineer running AtoCore") | AtoCore memories (`identity` type) | nowhere else holds personal identity | hand-authored via `POST /memory` or extracted from interactions |
|
||||||
|
| **Preference** ("prefers small reviewable diffs", "uses SI units") | AtoCore memories (`preference` type) | nowhere else holds personal preferences | hand-authored or extracted |
|
||||||
|
| **Episodic** ("on April 6 we debugged the EXDEV bug") | AtoCore memories (`episodic` type) | nowhere else has time-bound personal recall | extracted from captured interactions |
|
||||||
|
| **Decision** (a structured engineering decision) | AtoCore **entities** (Decision) once the engineering layer ships; AtoCore memories (`adaptation`) until then | needs structured supersession, audit trail, and link to affected components | extracted from PKM or interactions; promoted via review queue |
|
||||||
|
| **Requirement** | AtoCore **entities** (Requirement) | needs structured satisfaction tracking | extracted from PKM, KB-CAD, or interactions |
|
||||||
|
| **Constraint** | AtoCore **entities** (Constraint) | needs structured link to the entity it constrains | extracted from PKM, KB-CAD, or interactions |
|
||||||
|
| **Validation claim** | AtoCore **entities** (ValidationClaim) | needs structured link to supporting Result | extracted from KB-FEM exports or interactions |
|
||||||
|
| **Material** | KB-CAD if the material is on a real component; AtoCore entity (Material) if it's a project-wide material decision not yet attached to geometry | structured properties live in KB-CAD's material database | KB-CAD export, or hand-authored as a Material entity |
|
||||||
|
| **Parameter** | KB-CAD or KB-FEM depending on whether it's a geometry or solver parameter; AtoCore entity (Parameter) if it's a higher-level project parameter not in either tool | structured numeric values with units live in their tool of origin | KB export, or hand-authored |
|
||||||
|
| **Project status / current focus / next milestone** | AtoCore **project_state** (Layer 3) | the trust hierarchy says trusted state is the highest authority for "what is the current state of the project" | hand-curated via `POST /project/state` |
|
||||||
|
| **Architectural decision records (ADRs)** | depends on form: long-form ADR markdown lives in the repo; the structured fact about which ADR was selected lives in the AtoCore Decision entity | both representations are useful for different audiences | repo ingestion provides the prose; the entity is created by extraction or hand-authored |
|
||||||
|
| **Operational runbooks** | repo (next to the code they describe) | lives with the system it operates | not promoted into AtoCore entities — runbooks are reference material, not facts |
|
||||||
|
| **Backup metadata** (snapshot timestamps, integrity status) | the backup-metadata.json files under `/srv/storage/atocore/backups/` | each snapshot is its own self-describing record | not in AtoCore's database; queried via the `/admin/backup` endpoints |
|
||||||
|
| **Conversation history with AtoCore (interactions)** | AtoCore `interactions` table | nowhere else has the prompt + context pack + response triple | written by capture (Phase 9 Commit A) |
|
||||||
|
|
||||||
|
## The supremacy rule for cross-layer facts
|
||||||
|
|
||||||
|
When the same fact has copies in multiple representations and they
|
||||||
|
disagree, the trust hierarchy applies in this order:
|
||||||
|
|
||||||
|
1. **AtoCore project_state** (Layer 3) is highest authority for any
|
||||||
|
"current state of the project" question. This is why it requires
|
||||||
|
manual curation and never gets touched by automatic processes.
|
||||||
|
2. **The tool-of-origin canonical home** is highest authority for
|
||||||
|
facts that are tool-managed: KB-CAD wins over AtoCore entities
|
||||||
|
for CAD-side structure facts; KB-FEM wins for FEM result facts.
|
||||||
|
3. **AtoCore entities** are highest authority for facts that are
|
||||||
|
AtoCore-managed: Decisions, Requirements, Constraints,
|
||||||
|
ValidationClaims (when the supporting Results are still loose).
|
||||||
|
4. **Active AtoCore memories** are highest authority for personal
|
||||||
|
facts (identity, preference, episodic).
|
||||||
|
5. **Source chunks (PKM, repos, ingested docs)** are lowest
|
||||||
|
authority — they are the raw substrate from which higher layers
|
||||||
|
are extracted, but they may be stale, contradictory among
|
||||||
|
themselves, or out of date.
|
||||||
|
|
||||||
|
This is the same hierarchy enforced by `conflict-model.md`. This
|
||||||
|
document just makes it explicit per fact kind.
|
||||||
|
|
||||||
|
## Examples
|
||||||
|
|
||||||
|
### Example 1 — "what material does the lateral support pad use?"
|
||||||
|
|
||||||
|
Possible representations:
|
||||||
|
|
||||||
|
- KB-CAD has the field `component.lateral-support-pad.material = "GF-PTFE"`
|
||||||
|
- A PKM note from last month says "considering PEEK for the
|
||||||
|
lateral support, GF-PTFE was the previous choice"
|
||||||
|
- An AtoCore Material entity says `GF-PTFE`
|
||||||
|
- An AtoCore project_state entry says `p05 / decision /
|
||||||
|
lateral_support_material = GF-PTFE`
|
||||||
|
|
||||||
|
Which one wins for the question "what's the current material"?
|
||||||
|
|
||||||
|
- **project_state wins** if the query is "what is the current
|
||||||
|
trusted answer for p05's lateral support material" (Layer 3)
|
||||||
|
- **KB-CAD wins** if project_state has not been curated for this
|
||||||
|
field yet, because KB-CAD is the canonical home for CAD-side
|
||||||
|
structure
|
||||||
|
- **The Material entity** is a derived view from KB-CAD; if it
|
||||||
|
disagrees with KB-CAD, the entity is wrong and a conflict is
|
||||||
|
surfaced
|
||||||
|
- **The PKM note** is historical context, not authoritative for
|
||||||
|
"current"
|
||||||
|
|
||||||
|
### Example 2 — "did we decide to merge the bind mounts?"
|
||||||
|
|
||||||
|
Possible representations:
|
||||||
|
|
||||||
|
- A working session interaction is captured in the `interactions`
|
||||||
|
table with the response containing `## Decision: merge the two
|
||||||
|
bind mounts into one`
|
||||||
|
- The Phase 9 Commit C extractor produced a candidate adaptation
|
||||||
|
memory from that decision
|
||||||
|
- A reviewer promoted the candidate to active
|
||||||
|
- The AtoCore source repo has the actual code change in commit
|
||||||
|
`d0ff8b5` and the docker-compose.yml is in its post-merge form
|
||||||
|
|
||||||
|
Which one wins for "is this decision real and current"?
|
||||||
|
|
||||||
|
- **The Gitea repo** wins for "is this decision implemented" —
|
||||||
|
the docker-compose.yml is the canonical home for the actual
|
||||||
|
bind mount configuration
|
||||||
|
- **The active adaptation memory** wins for "did we decide this"
|
||||||
|
— that's exactly what the Commit C lifecycle is for
|
||||||
|
- **The interaction record** is the audit trail — it's
|
||||||
|
authoritative for "when did this conversation happen and what
|
||||||
|
did the LLM say", but not for "is this decision current"
|
||||||
|
- **The source chunks** from PKM are not relevant here because no
|
||||||
|
PKM note about this decision exists yet (and that's fine —
|
||||||
|
decisions don't have to live in PKM if they live in the repo
|
||||||
|
and the AtoCore memory)
|
||||||
|
|
||||||
|
### Example 3 — "what's p05's current next focus?"
|
||||||
|
|
||||||
|
Possible representations:
|
||||||
|
|
||||||
|
- The PKM has a `current-status.md` note updated last week
|
||||||
|
- AtoCore project_state has `p05 / status / next_focus = "wave 2 ingestion"`
|
||||||
|
- A captured interaction from yesterday discussed the next focus
|
||||||
|
at length
|
||||||
|
|
||||||
|
Which one wins?
|
||||||
|
|
||||||
|
- **project_state wins**, full stop. The trust hierarchy says
|
||||||
|
Layer 3 is canonical for current state. This is exactly the
|
||||||
|
reason project_state exists.
|
||||||
|
- The PKM note is historical context.
|
||||||
|
- The interaction is conversation history.
|
||||||
|
- If project_state and the PKM disagree, the human updates one or
|
||||||
|
the other to bring them in line — usually by re-curating
|
||||||
|
project_state if the conversation revealed a real change.
|
||||||
|
|
||||||
|
## What this means for the engineering layer V1 implementation
|
||||||
|
|
||||||
|
Several concrete consequences fall out of the matrix:
|
||||||
|
|
||||||
|
1. **The Material and Parameter entity types are mostly KB-CAD
|
||||||
|
shadows in V1.** They exist in AtoCore so other entities
|
||||||
|
(Decisions, Requirements) can reference them with structured
|
||||||
|
links, but their authoritative values come from KB-CAD imports.
|
||||||
|
If KB-CAD doesn't know about a material, the AtoCore entity is
|
||||||
|
the canonical home only because nothing else is.
|
||||||
|
2. **Decisions / Requirements / Constraints / ValidationClaims
|
||||||
|
are AtoCore-canonical.** These don't have a natural home in
|
||||||
|
KB-CAD or KB-FEM. They live in AtoCore as first-class entities
|
||||||
|
with full lifecycle and supersession.
|
||||||
|
3. **The PKM is never authoritative.** It is the substrate for
|
||||||
|
extraction. The reviewer promotes things out of it; they don't
|
||||||
|
point at PKM notes as the "current truth".
|
||||||
|
4. **project_state is the override layer.** Whenever the human
|
||||||
|
wants to declare "the current truth is X regardless of what
|
||||||
|
the entities and memories and KB exports say", they curate
|
||||||
|
into project_state. Layer 3 is intentionally small and
|
||||||
|
intentionally manual.
|
||||||
|
5. **The conflict model is the enforcement mechanism.** When two
|
||||||
|
representations disagree on a fact whose canonical home rule
|
||||||
|
should pick a winner, the conflict surfaces via the
|
||||||
|
`/conflicts` endpoint and the reviewer resolves it. The
|
||||||
|
matrix in this document tells the reviewer who is supposed
|
||||||
|
to win in each scenario; they're not making the decision blind.
|
||||||
|
|
||||||
|
## What the matrix does NOT define
|
||||||
|
|
||||||
|
1. **Facts about people other than the user.** No "team member"
|
||||||
|
entity, no per-collaborator preferences. AtoCore is
|
||||||
|
single-user in V1.
|
||||||
|
2. **Facts about AtoCore itself as a project.** Those are project
|
||||||
|
memories and project_state entries under `project=atocore`,
|
||||||
|
same lifecycle as any other project's facts.
|
||||||
|
3. **Vendor / supplier / cost facts.** Out of V1 scope.
|
||||||
|
4. **Time-bounded facts** (a value that was true between two
|
||||||
|
dates and may not be true now). The current matrix treats all
|
||||||
|
active facts as currently-true and uses supersession to
|
||||||
|
represent change. Temporal facts are a V2 concern.
|
||||||
|
5. **Cross-project shared facts** (a Material that is reused across
|
||||||
|
p04, p05, and p06). Currently each project has its own copy.
|
||||||
|
Cross-project deduplication is also a V2 concern.
|
||||||
|
|
||||||
|
## The "single canonical home" invariant in practice
|
||||||
|
|
||||||
|
The hard rule that every fact has exactly one canonical home is
|
||||||
|
the load-bearing invariant of this matrix. To enforce it
|
||||||
|
operationally:
|
||||||
|
|
||||||
|
- **Extraction never duplicates.** When the extractor scans an
|
||||||
|
interaction or a source chunk and proposes a candidate, the
|
||||||
|
candidate is dropped if it duplicates an already-active record
|
||||||
|
in the canonical home (the existing extractor implementation
|
||||||
|
already does this for memories; the entity extractor will
|
||||||
|
follow the same pattern).
|
||||||
|
- **Imports never duplicate.** When KB-CAD pushes the same
|
||||||
|
Component twice with the same value, the second push is
|
||||||
|
recognized as identical and updates the `last_imported_at`
|
||||||
|
timestamp without creating a new entity.
|
||||||
|
- **Imports surface drift as conflict.** When KB-CAD pushes the
|
||||||
|
same Component with a different value, that's a conflict per
|
||||||
|
the conflict model — never a silent overwrite.
|
||||||
|
- **Hand-curation into project_state always wins.** A
|
||||||
|
project_state entry can disagree with an entity or a KB
|
||||||
|
export; the project_state entry is correct by fiat (Layer 3
|
||||||
|
trust), and the reviewer is responsible for bringing the lower
|
||||||
|
layers in line if appropriate.
|
||||||
|
|
||||||
|
## Open questions for V1 implementation
|
||||||
|
|
||||||
|
1. **How does the reviewer see the canonical home for a fact in
|
||||||
|
the UI?** Probably by including the fact's authoritative
|
||||||
|
layer in the entity / memory detail view: "this Material is
|
||||||
|
currently mirrored from KB-CAD; the canonical home is KB-CAD".
|
||||||
|
2. **Who owns running the KB-CAD / KB-FEM exporter?** The
|
||||||
|
`tool-handoff-boundaries.md` doc lists this as an open
|
||||||
|
question; same answer applies here.
|
||||||
|
3. **Do we need an explicit `canonical_home` field on entity
|
||||||
|
rows?** A field that records "this entity is canonical here"
|
||||||
|
vs "this entity is a mirror of <external system>". Probably
|
||||||
|
yes; deferred to the entity schema spec.
|
||||||
|
4. **How are project_state overrides surfaced in the engineering
|
||||||
|
layer query results?** When a query (e.g. Q-001 "what does
|
||||||
|
this subsystem contain?") would return entity rows, the result
|
||||||
|
should also flag any project_state entries that contradict the
|
||||||
|
entities — letting the reviewer see the override at query
|
||||||
|
time, not just in the conflict queue.
|
||||||
|
|
||||||
|
## TL;DR
|
||||||
|
|
||||||
|
- Six representation layers: PKM, KB project, repos, AtoCore
|
||||||
|
memories, AtoCore entities, AtoCore project_state
|
||||||
|
- Every fact kind has exactly one canonical home
|
||||||
|
- The trust hierarchy resolves cross-layer conflicts:
|
||||||
|
project_state > tool-of-origin (KB-CAD/KB-FEM) > entities >
|
||||||
|
active memories > source chunks
|
||||||
|
- Decisions / Requirements / Constraints / ValidationClaims are
|
||||||
|
AtoCore-canonical (no other system has a natural home for them)
|
||||||
|
- Materials / Parameters / CAD-side structure are KB-CAD-canonical
|
||||||
|
- FEM results / validation outcomes are KB-FEM-canonical
|
||||||
|
- project_state is the human override layer, top of the
|
||||||
|
hierarchy, manually curated only
|
||||||
|
- Conflicts surface via `/conflicts` and the reviewer applies the
|
||||||
|
matrix to pick a winner
|
||||||
339
docs/architecture/tool-handoff-boundaries.md
Normal file
339
docs/architecture/tool-handoff-boundaries.md
Normal file
@@ -0,0 +1,339 @@
|
|||||||
|
# Tool Hand-off Boundaries (KB-CAD / KB-FEM and friends)
|
||||||
|
|
||||||
|
## Why this document exists
|
||||||
|
|
||||||
|
The engineering layer V1 will accumulate typed entities about
|
||||||
|
projects, subsystems, components, materials, requirements,
|
||||||
|
constraints, decisions, parameters, analysis models, results, and
|
||||||
|
validation claims. Many of those concepts also live in real
|
||||||
|
external tools — CAD systems, FEM solvers, BOM managers, PLM
|
||||||
|
databases, vendor portals.
|
||||||
|
|
||||||
|
The first big design decision before writing any entity-layer code
|
||||||
|
is: **what is AtoCore's read/write relationship with each of those
|
||||||
|
external tools?**
|
||||||
|
|
||||||
|
The wrong answer in either direction is expensive:
|
||||||
|
|
||||||
|
- Too read-only: AtoCore becomes a stale shadow of the tools and
|
||||||
|
loses the trust battle the moment a value drifts.
|
||||||
|
- Too bidirectional: AtoCore takes on responsibilities it can't
|
||||||
|
reliably honor (live sync, conflict resolution against external
|
||||||
|
schemas, write-back validation), and the project never ships.
|
||||||
|
|
||||||
|
This document picks a position for V1.
|
||||||
|
|
||||||
|
## The position
|
||||||
|
|
||||||
|
> **AtoCore is a one-way mirror in V1.** External tools push
|
||||||
|
> structured exports into AtoCore. AtoCore never pushes back.
|
||||||
|
|
||||||
|
That position has three corollaries:
|
||||||
|
|
||||||
|
1. **External tools remain the source of truth for everything they
|
||||||
|
already manage.** A CAD model is canonical for geometry; a FEM
|
||||||
|
project is canonical for meshes and solver settings; KB-CAD is
|
||||||
|
canonical for whatever KB-CAD already calls canonical.
|
||||||
|
2. **AtoCore is the source of truth for the *AtoCore-shaped*
|
||||||
|
record** of those facts: the Decision that selected the geometry,
|
||||||
|
the Requirement the geometry satisfies, the ValidationClaim the
|
||||||
|
FEM result supports. AtoCore does not duplicate the external
|
||||||
|
tool's primary representation; it stores the structured *facts
|
||||||
|
about* it.
|
||||||
|
3. **The boundary is enforced by absence.** No write endpoint in
|
||||||
|
AtoCore ever generates a `.prt`, a `.fem`, an export to a PLM
|
||||||
|
schema, or a vendor purchase order. If we find ourselves wanting
|
||||||
|
to add such an endpoint in V1, we should stop and reconsider
|
||||||
|
the V1 scope.
|
||||||
|
|
||||||
|
## Why one-way and not bidirectional
|
||||||
|
|
||||||
|
Bidirectional sync between independent systems is one of the
|
||||||
|
hardest problems in engineering software. The honest reasons we
|
||||||
|
are not attempting it in V1:
|
||||||
|
|
||||||
|
1. **Schema drift.** External tools evolve their schemas
|
||||||
|
independently. A bidirectional sync would have to track every
|
||||||
|
schema version of every external tool we touch. That is a
|
||||||
|
permanent maintenance tax.
|
||||||
|
2. **Conflict semantics.** When AtoCore and an external tool
|
||||||
|
disagree on the same field, "who wins" is a per-tool, per-field
|
||||||
|
decision. There is no general rule. Bidirectional sync would
|
||||||
|
require us to specify that decision exhaustively.
|
||||||
|
3. **Trust hierarchy.** AtoCore's whole point is the trust
|
||||||
|
hierarchy: trusted project state > entities > memories. If we
|
||||||
|
let entities push values back into the external tools, we
|
||||||
|
silently elevate AtoCore's confidence to "high enough to write
|
||||||
|
to a CAD model", which it almost never deserves.
|
||||||
|
4. **Velocity.** A bidirectional engineering layer is a
|
||||||
|
multi-year project. A one-way mirror is a months project. The
|
||||||
|
value-to-effort ratio favors one-way for V1 by an enormous
|
||||||
|
margin.
|
||||||
|
5. **Reversibility.** We can always add bidirectional sync later
|
||||||
|
on a per-tool basis once V1 has shown itself to be useful. We
|
||||||
|
cannot easily walk back a half-finished bidirectional sync that
|
||||||
|
has already corrupted data in someone's CAD model.
|
||||||
|
|
||||||
|
## Per-tool stance for V1
|
||||||
|
|
||||||
|
| External tool | V1 stance | What AtoCore reads in | What AtoCore writes back |
|
||||||
|
|---|---|---|---|
|
||||||
|
| **KB-CAD** (Antoine's CAD knowledge base) | one-way mirror | structured exports of subsystems, components, materials, parameters via a documented JSON or CSV shape | nothing |
|
||||||
|
| **KB-FEM** (Antoine's FEM knowledge base) | one-way mirror | structured exports of analysis models, results, validation claims | nothing |
|
||||||
|
| **NX / Siemens NX** (the CAD tool itself) | not connected in V1 | nothing direct — only what KB-CAD exports about NX projects | nothing |
|
||||||
|
| **PKM (Obsidian / markdown vault)** | already connected via the ingestion pipeline (Phase 1) | full markdown/text corpus per the ingestion-waves doc | nothing |
|
||||||
|
| **Gitea repos** | already connected via the ingestion pipeline | repo markdown/text per project | nothing |
|
||||||
|
| **OpenClaw** (the LLM agent) | already connected via the read-only helper skill on the T420 | nothing — OpenClaw reads from AtoCore | nothing — OpenClaw does not write into AtoCore |
|
||||||
|
| **AtoDrive** (operational truth layer, future) | future: bidirectional with AtoDrive itself, but AtoDrive is internal to AtoCore so this isn't an external tool boundary | n/a in V1 | n/a in V1 |
|
||||||
|
| **PLM / vendor portals / cost systems** | not in V1 scope | nothing | nothing |
|
||||||
|
|
||||||
|
## What "one-way mirror" actually looks like in code
|
||||||
|
|
||||||
|
AtoCore exposes an ingestion endpoint per external tool that
|
||||||
|
accepts a structured export and turns it into entity candidates.
|
||||||
|
The endpoint is read-side from AtoCore's perspective (it reads
|
||||||
|
from a file or HTTP body), even though the external tool is the
|
||||||
|
one initiating the call.
|
||||||
|
|
||||||
|
Proposed V1 ingestion endpoints:
|
||||||
|
|
||||||
|
```
|
||||||
|
POST /ingest/kb-cad/export body: KB-CAD export JSON
|
||||||
|
POST /ingest/kb-fem/export body: KB-FEM export JSON
|
||||||
|
```
|
||||||
|
|
||||||
|
Each endpoint:
|
||||||
|
|
||||||
|
1. Validates the export against the documented schema
|
||||||
|
2. Maps each export record to an entity candidate (status="candidate")
|
||||||
|
3. Carries the export's source identifier into the candidate's
|
||||||
|
provenance fields (source_artifact_id, exporter_version, etc.)
|
||||||
|
4. Returns a summary: how many candidates were created, how many
|
||||||
|
were dropped as duplicates, how many failed schema validation
|
||||||
|
5. Does NOT auto-promote anything
|
||||||
|
|
||||||
|
The KB-CAD and KB-FEM teams (which is to say, future-you) own the
|
||||||
|
exporter scripts that produce these JSON bodies. Those scripts
|
||||||
|
live in the KB-CAD / KB-FEM repos respectively, not in AtoCore.
|
||||||
|
|
||||||
|
## The export schemas (sketch, not final)
|
||||||
|
|
||||||
|
These are starting shapes, intentionally minimal. The schemas
|
||||||
|
will be refined in `kb-cad-export-schema.md` and
|
||||||
|
`kb-fem-export-schema.md` once the V1 ontology lands.
|
||||||
|
|
||||||
|
### KB-CAD export shape (starting sketch)
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"exporter": "kb-cad",
|
||||||
|
"exporter_version": "1.0.0",
|
||||||
|
"exported_at": "2026-04-07T12:00:00Z",
|
||||||
|
"project": "p05-interferometer",
|
||||||
|
"subsystems": [
|
||||||
|
{
|
||||||
|
"id": "subsystem.optical-frame",
|
||||||
|
"name": "Optical frame",
|
||||||
|
"parent": null,
|
||||||
|
"components": [
|
||||||
|
{
|
||||||
|
"id": "component.lateral-support-pad",
|
||||||
|
"name": "Lateral support pad",
|
||||||
|
"material": "GF-PTFE",
|
||||||
|
"parameters": {
|
||||||
|
"thickness_mm": 3.0,
|
||||||
|
"preload_n": 12.0
|
||||||
|
},
|
||||||
|
"source_artifact": "kb-cad://p05/subsystems/optical-frame#lateral-support"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### KB-FEM export shape (starting sketch)
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"exporter": "kb-fem",
|
||||||
|
"exporter_version": "1.0.0",
|
||||||
|
"exported_at": "2026-04-07T12:00:00Z",
|
||||||
|
"project": "p05-interferometer",
|
||||||
|
"analysis_models": [
|
||||||
|
{
|
||||||
|
"id": "model.optical-frame-modal",
|
||||||
|
"name": "Optical frame modal analysis v3",
|
||||||
|
"subsystem": "subsystem.optical-frame",
|
||||||
|
"results": [
|
||||||
|
{
|
||||||
|
"id": "result.first-mode-frequency",
|
||||||
|
"name": "First-mode frequency",
|
||||||
|
"value": 187.4,
|
||||||
|
"unit": "Hz",
|
||||||
|
"supports_validation_claim": "claim.frame-rigidity-min-150hz",
|
||||||
|
"source_artifact": "kb-fem://p05/models/optical-frame-modal#first-mode"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
These shapes will evolve. The point of including them now is to
|
||||||
|
make the one-way mirror concrete: it is a small, well-defined
|
||||||
|
JSON shape, not "AtoCore reaches into KB-CAD's database".
|
||||||
|
|
||||||
|
## What AtoCore is allowed to do with the imported records
|
||||||
|
|
||||||
|
After ingestion, the imported records become entity candidates
|
||||||
|
in AtoCore's own table. From that point forward they follow the
|
||||||
|
exact same lifecycle as any other candidate:
|
||||||
|
|
||||||
|
- they sit at status="candidate" until a human reviews them
|
||||||
|
- the reviewer promotes them to status="active" or rejects them
|
||||||
|
- the active entities are queryable via the engineering query
|
||||||
|
catalog (Q-001 through Q-020)
|
||||||
|
- the active entities can be referenced from Decisions, Requirements,
|
||||||
|
ValidationClaims, etc. via the V1 relationship types
|
||||||
|
|
||||||
|
The imported records are never automatically pushed into trusted
|
||||||
|
project state, never modified in place after import (they are
|
||||||
|
superseded by re-imports, not edited), and never written back to
|
||||||
|
the external tool.
|
||||||
|
|
||||||
|
## What happens when KB-CAD changes a value AtoCore already has
|
||||||
|
|
||||||
|
This is the canonical "drift" scenario. The flow:
|
||||||
|
|
||||||
|
1. KB-CAD exports a fresh JSON. Component `component.lateral-support-pad`
|
||||||
|
now has `material: "PEEK"` instead of `material: "GF-PTFE"`.
|
||||||
|
2. AtoCore's ingestion endpoint sees the same `id` and a different
|
||||||
|
value.
|
||||||
|
3. The ingestion endpoint creates a new entity candidate with the
|
||||||
|
new value, **does NOT delete or modify the existing active
|
||||||
|
entity**, and creates a `conflicts` row linking the two members
|
||||||
|
(per the conflict model doc).
|
||||||
|
4. The reviewer sees an open conflict on the next visit to
|
||||||
|
`/conflicts`.
|
||||||
|
5. The reviewer either:
|
||||||
|
- **promotes the new value** (the active is superseded, the
|
||||||
|
candidate becomes the new active, the audit trail keeps both)
|
||||||
|
- **rejects the new value** (the candidate is invalidated, the
|
||||||
|
active stays — useful when the export was wrong)
|
||||||
|
- **dismisses the conflict** (declares them not actually about
|
||||||
|
the same thing, both stay active)
|
||||||
|
|
||||||
|
The reviewer never touches KB-CAD from AtoCore. If the resolution
|
||||||
|
implies a change in KB-CAD itself, the reviewer makes that change
|
||||||
|
in KB-CAD, then re-exports.
|
||||||
|
|
||||||
|
## What about NX directly?
|
||||||
|
|
||||||
|
NX (Siemens NX) is the underlying CAD tool that KB-CAD wraps.
|
||||||
|
**NX is not connected to AtoCore in V1.** Any facts about NX
|
||||||
|
projects flow through KB-CAD as the structured intermediate. This
|
||||||
|
gives us:
|
||||||
|
|
||||||
|
- **One schema to maintain.** AtoCore only has to understand the
|
||||||
|
KB-CAD export shape, not the NX API.
|
||||||
|
- **One ownership boundary.** KB-CAD owns the question of "what's
|
||||||
|
in NX". AtoCore owns the question of "what's in the typed
|
||||||
|
knowledge base".
|
||||||
|
- **Future flexibility.** When NX is replaced or upgraded, only
|
||||||
|
KB-CAD has to adapt; AtoCore doesn't notice.
|
||||||
|
|
||||||
|
The same logic applies to FEM solvers (Nastran, Abaqus, ANSYS):
|
||||||
|
KB-FEM is the structured intermediate, AtoCore never talks to the
|
||||||
|
solver directly.
|
||||||
|
|
||||||
|
## The hard-line invariants
|
||||||
|
|
||||||
|
These are the things V1 will not do, regardless of how convenient
|
||||||
|
they might seem:
|
||||||
|
|
||||||
|
1. **No write to external tools.** No POST/PUT/PATCH to any
|
||||||
|
external API, no file generation that gets written into a
|
||||||
|
CAD/FEM project tree, no email/chat sends.
|
||||||
|
2. **No live polling.** AtoCore does not poll KB-CAD or KB-FEM on
|
||||||
|
a schedule. Imports are explicit pushes from the external tool
|
||||||
|
into AtoCore's ingestion endpoint.
|
||||||
|
3. **No silent merging.** Every value drift surfaces as a
|
||||||
|
conflict for the reviewer (per the conflict model doc).
|
||||||
|
4. **No schema fan-out.** AtoCore does not store every field that
|
||||||
|
KB-CAD knows about. Only fields that map to one of the V1
|
||||||
|
entity types make it into AtoCore. Everything else is dropped
|
||||||
|
at the import boundary.
|
||||||
|
5. **No external-tool-specific logic in entity types.** A
|
||||||
|
`Component` in AtoCore is the same shape regardless of whether
|
||||||
|
it came from KB-CAD, KB-FEM, the PKM, or a hand-curated
|
||||||
|
project state entry. The source is recorded in provenance,
|
||||||
|
not in the entity shape.
|
||||||
|
|
||||||
|
## What this enables
|
||||||
|
|
||||||
|
With the one-way mirror locked in, V1 implementation can focus on:
|
||||||
|
|
||||||
|
- The entity table and its lifecycle
|
||||||
|
- The two `/ingest/kb-cad/export` and `/ingest/kb-fem/export`
|
||||||
|
endpoints with their JSON validators
|
||||||
|
- The candidate review queue extension (already designed in
|
||||||
|
`promotion-rules.md`)
|
||||||
|
- The conflict model (already designed in `conflict-model.md`)
|
||||||
|
- The query catalog implementation (already designed in
|
||||||
|
`engineering-query-catalog.md`)
|
||||||
|
|
||||||
|
None of those are unbounded. Each is a finite, well-defined
|
||||||
|
implementation task. The one-way mirror is the choice that makes
|
||||||
|
V1 finishable.
|
||||||
|
|
||||||
|
## What V2 might consider (deferred)
|
||||||
|
|
||||||
|
After V1 has been live and demonstrably useful for a quarter or
|
||||||
|
two, the questions that become reasonable to revisit:
|
||||||
|
|
||||||
|
1. **Selective write-back to KB-CAD for low-risk fields.** For
|
||||||
|
example, AtoCore could push back a "Decision id linked to this
|
||||||
|
component" annotation that KB-CAD then displays without it
|
||||||
|
being canonical there. Read-only annotations from AtoCore's
|
||||||
|
perspective, advisory metadata from KB-CAD's perspective.
|
||||||
|
2. **Live polling for very small payloads.** A daily poll of
|
||||||
|
"what subsystem ids exist in KB-CAD now" so AtoCore can flag
|
||||||
|
subsystems that disappeared from KB-CAD without an explicit
|
||||||
|
AtoCore invalidation.
|
||||||
|
3. **Direct NX integration** if the KB-CAD layer becomes a
|
||||||
|
bottleneck — but only if the friction is real, not theoretical.
|
||||||
|
4. **Cost / vendor / PLM connections** for projects where the
|
||||||
|
procurement cycle is part of the active engineering work.
|
||||||
|
|
||||||
|
None of these are V1 work and they are listed only so the V1
|
||||||
|
design intentionally leaves room for them later.
|
||||||
|
|
||||||
|
## Open questions for the V1 implementation sprint
|
||||||
|
|
||||||
|
1. **Where do the export schemas live?** Probably in
|
||||||
|
`docs/architecture/kb-cad-export-schema.md` and
|
||||||
|
`docs/architecture/kb-fem-export-schema.md`, drafted during
|
||||||
|
the implementation sprint.
|
||||||
|
2. **Who runs the exporter?** A scheduled job on the KB-CAD /
|
||||||
|
KB-FEM hosts, triggered by the human after a meaningful
|
||||||
|
change, or both?
|
||||||
|
3. **Is the export incremental or full?** Full is simpler but
|
||||||
|
more expensive. Incremental needs delta semantics. V1 starts
|
||||||
|
with full and revisits when full becomes too slow.
|
||||||
|
4. **How is the exporter authenticated to AtoCore?** Probably
|
||||||
|
the existing PAT model (one PAT per exporter, scoped to
|
||||||
|
`write:engineering-import` once that scope exists). Worth a
|
||||||
|
quick auth design pass before the endpoints exist.
|
||||||
|
|
||||||
|
## TL;DR
|
||||||
|
|
||||||
|
- AtoCore is a one-way mirror in V1: external tools push,
|
||||||
|
AtoCore reads, AtoCore never writes back
|
||||||
|
- Two import endpoints for V1: KB-CAD and KB-FEM, each with a
|
||||||
|
documented JSON export shape
|
||||||
|
- Drift surfaces as conflicts in the existing conflict model
|
||||||
|
- No NX, no FEM solvers, no PLM, no vendor portals, no
|
||||||
|
cost/BOM systems in V1
|
||||||
|
- Bidirectional sync is reserved for V2+ on a per-tool basis,
|
||||||
|
only after V1 demonstrates value
|
||||||
360
docs/backup-restore-procedure.md
Normal file
360
docs/backup-restore-procedure.md
Normal file
@@ -0,0 +1,360 @@
|
|||||||
|
# AtoCore Backup and Restore Procedure
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
This document defines the operational procedure for backing up and
|
||||||
|
restoring AtoCore's machine state on the Dalidou deployment. It is
|
||||||
|
the practical companion to `docs/backup-strategy.md` (which defines
|
||||||
|
the strategy) and `src/atocore/ops/backup.py` (which implements the
|
||||||
|
mechanics).
|
||||||
|
|
||||||
|
The intent is that this procedure can be followed by anyone with
|
||||||
|
SSH access to Dalidou and the AtoCore admin endpoints.
|
||||||
|
|
||||||
|
## What gets backed up
|
||||||
|
|
||||||
|
A `create_runtime_backup` snapshot contains, in order of importance:
|
||||||
|
|
||||||
|
| Artifact | Source path on Dalidou | Backup destination | Always included |
|
||||||
|
|---|---|---|---|
|
||||||
|
| SQLite database | `/srv/storage/atocore/data/db/atocore.db` | `<backup_root>/db/atocore.db` | yes |
|
||||||
|
| Project registry JSON | `/srv/storage/atocore/config/project-registry.json` | `<backup_root>/config/project-registry.json` | yes (if file exists) |
|
||||||
|
| Backup metadata | (generated) | `<backup_root>/backup-metadata.json` | yes |
|
||||||
|
| Chroma vector store | `/srv/storage/atocore/data/chroma/` | `<backup_root>/chroma/` | only when `include_chroma=true` |
|
||||||
|
|
||||||
|
The SQLite snapshot uses the online `conn.backup()` API and is safe
|
||||||
|
to take while the database is in use. The Chroma snapshot is a cold
|
||||||
|
directory copy and is **only safe when no ingestion is running**;
|
||||||
|
the API endpoint enforces this by acquiring the ingestion lock for
|
||||||
|
the duration of the copy.
|
||||||
|
|
||||||
|
What is **not** in the backup:
|
||||||
|
|
||||||
|
- Source documents under `/srv/storage/atocore/sources/vault/` and
|
||||||
|
`/srv/storage/atocore/sources/drive/`. These are read-only
|
||||||
|
inputs and live in the user's PKM/Drive, which is backed up
|
||||||
|
separately by their own systems.
|
||||||
|
- Application code. The container image is the source of truth for
|
||||||
|
code; recovery means rebuilding the image, not restoring code from
|
||||||
|
a backup.
|
||||||
|
- Logs under `/srv/storage/atocore/logs/`.
|
||||||
|
- Embeddings cache under `/srv/storage/atocore/data/cache/`.
|
||||||
|
- Temp files under `/srv/storage/atocore/data/tmp/`.
|
||||||
|
|
||||||
|
## Backup root layout
|
||||||
|
|
||||||
|
Each backup snapshot lives in its own timestamped directory:
|
||||||
|
|
||||||
|
```
|
||||||
|
/srv/storage/atocore/backups/snapshots/
|
||||||
|
├── 20260407T060000Z/
|
||||||
|
│ ├── backup-metadata.json
|
||||||
|
│ ├── db/
|
||||||
|
│ │ └── atocore.db
|
||||||
|
│ ├── config/
|
||||||
|
│ │ └── project-registry.json
|
||||||
|
│ └── chroma/ # only if include_chroma=true
|
||||||
|
│ └── ...
|
||||||
|
├── 20260408T060000Z/
|
||||||
|
│ └── ...
|
||||||
|
└── ...
|
||||||
|
```
|
||||||
|
|
||||||
|
The timestamp is UTC, format `YYYYMMDDTHHMMSSZ`.
|
||||||
|
|
||||||
|
## Triggering a backup
|
||||||
|
|
||||||
|
### Option A — via the admin endpoint (preferred)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# DB + registry only (fast, safe at any time)
|
||||||
|
curl -fsS -X POST http://dalidou:8100/admin/backup \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"include_chroma": false}'
|
||||||
|
|
||||||
|
# DB + registry + Chroma (acquires ingestion lock)
|
||||||
|
curl -fsS -X POST http://dalidou:8100/admin/backup \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"include_chroma": true}'
|
||||||
|
```
|
||||||
|
|
||||||
|
The response is the backup metadata JSON. Save the `backup_root`
|
||||||
|
field — that's the directory the snapshot was written to.
|
||||||
|
|
||||||
|
### Option B — via the standalone script (when the API is down)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker exec atocore python -m atocore.ops.backup
|
||||||
|
```
|
||||||
|
|
||||||
|
This runs `create_runtime_backup()` directly, without going through
|
||||||
|
the API or the ingestion lock. Use it only when the AtoCore service
|
||||||
|
itself is unhealthy and you can't hit the admin endpoint.
|
||||||
|
|
||||||
|
### Option C — manual file copy (last resort)
|
||||||
|
|
||||||
|
If both the API and the standalone script are unusable:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo systemctl stop atocore # or: docker compose stop atocore
|
||||||
|
sudo cp /srv/storage/atocore/data/db/atocore.db \
|
||||||
|
/srv/storage/atocore/backups/manual-$(date -u +%Y%m%dT%H%M%SZ).db
|
||||||
|
sudo cp /srv/storage/atocore/config/project-registry.json \
|
||||||
|
/srv/storage/atocore/backups/manual-$(date -u +%Y%m%dT%H%M%SZ).registry.json
|
||||||
|
sudo systemctl start atocore
|
||||||
|
```
|
||||||
|
|
||||||
|
This is a cold backup and requires brief downtime.
|
||||||
|
|
||||||
|
## Listing backups
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -fsS http://dalidou:8100/admin/backup
|
||||||
|
```
|
||||||
|
|
||||||
|
Returns the configured `backup_dir` and a list of all snapshots
|
||||||
|
under it, with their full metadata if available.
|
||||||
|
|
||||||
|
Or, on the host directly:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ls -la /srv/storage/atocore/backups/snapshots/
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validating a backup
|
||||||
|
|
||||||
|
Before relying on a backup for restore, validate it:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -fsS http://dalidou:8100/admin/backup/20260407T060000Z/validate
|
||||||
|
```
|
||||||
|
|
||||||
|
The validator:
|
||||||
|
- confirms the snapshot directory exists
|
||||||
|
- opens the SQLite snapshot and runs `PRAGMA integrity_check`
|
||||||
|
- parses the registry JSON
|
||||||
|
- confirms the Chroma directory exists (if it was included)
|
||||||
|
|
||||||
|
A valid backup returns `"valid": true` and an empty `errors` array.
|
||||||
|
A failing validation returns `"valid": false` with one or more
|
||||||
|
specific error strings (e.g. `db_integrity_check_failed`,
|
||||||
|
`registry_invalid_json`, `chroma_snapshot_missing`).
|
||||||
|
|
||||||
|
**Validate every backup at creation time.** A backup that has never
|
||||||
|
been validated is not actually a backup — it's just a hopeful copy
|
||||||
|
of bytes.
|
||||||
|
|
||||||
|
## Restore procedure
|
||||||
|
|
||||||
|
### Pre-flight (always)
|
||||||
|
|
||||||
|
1. Identify which snapshot you want to restore. List available
|
||||||
|
snapshots and pick by timestamp:
|
||||||
|
```bash
|
||||||
|
curl -fsS http://dalidou:8100/admin/backup | jq '.backups[].stamp'
|
||||||
|
```
|
||||||
|
2. Validate it. Refuse to restore an invalid backup:
|
||||||
|
```bash
|
||||||
|
STAMP=20260407T060000Z
|
||||||
|
curl -fsS http://dalidou:8100/admin/backup/$STAMP/validate | jq .
|
||||||
|
```
|
||||||
|
3. **Stop AtoCore.** SQLite cannot be hot-restored under a running
|
||||||
|
process and Chroma will not pick up new files until the process
|
||||||
|
restarts.
|
||||||
|
```bash
|
||||||
|
docker compose stop atocore
|
||||||
|
# or: sudo systemctl stop atocore
|
||||||
|
```
|
||||||
|
4. **Take a safety snapshot of the current state** before overwriting
|
||||||
|
it. This is your "if the restore makes things worse, here's the
|
||||||
|
undo" backup.
|
||||||
|
```bash
|
||||||
|
PRESERVE_STAMP=$(date -u +%Y%m%dT%H%M%SZ)
|
||||||
|
sudo cp /srv/storage/atocore/data/db/atocore.db \
|
||||||
|
/srv/storage/atocore/backups/pre-restore-$PRESERVE_STAMP.db
|
||||||
|
sudo cp /srv/storage/atocore/config/project-registry.json \
|
||||||
|
/srv/storage/atocore/backups/pre-restore-$PRESERVE_STAMP.registry.json 2>/dev/null || true
|
||||||
|
```
|
||||||
|
|
||||||
|
### Restore the SQLite database
|
||||||
|
|
||||||
|
```bash
|
||||||
|
SNAPSHOT_DIR=/srv/storage/atocore/backups/snapshots/$STAMP
|
||||||
|
sudo cp $SNAPSHOT_DIR/db/atocore.db \
|
||||||
|
/srv/storage/atocore/data/db/atocore.db
|
||||||
|
sudo chown 1000:1000 /srv/storage/atocore/data/db/atocore.db
|
||||||
|
sudo chmod 600 /srv/storage/atocore/data/db/atocore.db
|
||||||
|
```
|
||||||
|
|
||||||
|
The chown should match the gitea/atocore container user. Verify
|
||||||
|
by checking the existing perms before overwriting:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
stat -c '%U:%G %a' /srv/storage/atocore/data/db/atocore.db
|
||||||
|
```
|
||||||
|
|
||||||
|
### Restore the project registry
|
||||||
|
|
||||||
|
```bash
|
||||||
|
if [ -f $SNAPSHOT_DIR/config/project-registry.json ]; then
|
||||||
|
sudo cp $SNAPSHOT_DIR/config/project-registry.json \
|
||||||
|
/srv/storage/atocore/config/project-registry.json
|
||||||
|
sudo chown 1000:1000 /srv/storage/atocore/config/project-registry.json
|
||||||
|
sudo chmod 644 /srv/storage/atocore/config/project-registry.json
|
||||||
|
fi
|
||||||
|
```
|
||||||
|
|
||||||
|
If the snapshot does not contain a registry, the current registry is
|
||||||
|
preserved. The pre-flight safety copy still gives you a recovery path
|
||||||
|
if you need to roll back.
|
||||||
|
|
||||||
|
### Restore the Chroma vector store (if it was in the snapshot)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
if [ -d $SNAPSHOT_DIR/chroma ]; then
|
||||||
|
# Move the current chroma dir aside as a safety copy
|
||||||
|
sudo mv /srv/storage/atocore/data/chroma \
|
||||||
|
/srv/storage/atocore/data/chroma.pre-restore-$PRESERVE_STAMP
|
||||||
|
|
||||||
|
# Copy the snapshot in
|
||||||
|
sudo cp -a $SNAPSHOT_DIR/chroma /srv/storage/atocore/data/chroma
|
||||||
|
sudo chown -R 1000:1000 /srv/storage/atocore/data/chroma
|
||||||
|
fi
|
||||||
|
```
|
||||||
|
|
||||||
|
If the snapshot does NOT contain a Chroma dir but the SQLite
|
||||||
|
restore would leave the vector store and the SQL store inconsistent
|
||||||
|
(e.g. SQL has chunks the vector store doesn't), you have two
|
||||||
|
options:
|
||||||
|
|
||||||
|
- **Option 1: rebuild the vector store from source documents.** Run
|
||||||
|
ingestion fresh after the SQL restore. This regenerates embeddings
|
||||||
|
from the actual source files. Slow but produces a perfectly
|
||||||
|
consistent state.
|
||||||
|
- **Option 2: accept the inconsistency and live with stale-vector
|
||||||
|
filtering.** The retriever already drops vector results whose
|
||||||
|
SQL row no longer exists (`_existing_chunk_ids` filter), so the
|
||||||
|
inconsistency surfaces as missing results, not bad ones.
|
||||||
|
|
||||||
|
For an unplanned restore, Option 2 is the right immediate move.
|
||||||
|
Then schedule a fresh ingestion pass to rebuild the vector store
|
||||||
|
properly.
|
||||||
|
|
||||||
|
### Restart AtoCore
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker compose up -d atocore
|
||||||
|
# or: sudo systemctl start atocore
|
||||||
|
```
|
||||||
|
|
||||||
|
### Post-restore verification
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. Service is healthy
|
||||||
|
curl -fsS http://dalidou:8100/health | jq .
|
||||||
|
|
||||||
|
# 2. Stats look right
|
||||||
|
curl -fsS http://dalidou:8100/stats | jq .
|
||||||
|
|
||||||
|
# 3. Project registry loads
|
||||||
|
curl -fsS http://dalidou:8100/projects | jq '.projects | length'
|
||||||
|
|
||||||
|
# 4. A known-good context query returns non-empty results
|
||||||
|
curl -fsS -X POST http://dalidou:8100/context/build \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"prompt": "what is p05 about", "project": "p05-interferometer"}' | jq '.chunks_used'
|
||||||
|
```
|
||||||
|
|
||||||
|
If any of these are wrong, the restore is bad. Roll back using the
|
||||||
|
pre-restore safety copy:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker compose stop atocore
|
||||||
|
sudo cp /srv/storage/atocore/backups/pre-restore-$PRESERVE_STAMP.db \
|
||||||
|
/srv/storage/atocore/data/db/atocore.db
|
||||||
|
sudo cp /srv/storage/atocore/backups/pre-restore-$PRESERVE_STAMP.registry.json \
|
||||||
|
/srv/storage/atocore/config/project-registry.json 2>/dev/null || true
|
||||||
|
# If you also restored chroma:
|
||||||
|
sudo rm -rf /srv/storage/atocore/data/chroma
|
||||||
|
sudo mv /srv/storage/atocore/data/chroma.pre-restore-$PRESERVE_STAMP \
|
||||||
|
/srv/storage/atocore/data/chroma
|
||||||
|
docker compose up -d atocore
|
||||||
|
```
|
||||||
|
|
||||||
|
## Retention policy
|
||||||
|
|
||||||
|
- **Last 7 daily backups**: kept verbatim
|
||||||
|
- **Last 4 weekly backups** (Sunday): kept verbatim
|
||||||
|
- **Last 6 monthly backups** (1st of month): kept verbatim
|
||||||
|
- **Anything older**: deleted
|
||||||
|
|
||||||
|
The retention job is **not yet implemented** and is tracked as a
|
||||||
|
follow-up. Until then, the snapshots directory grows monotonically.
|
||||||
|
A simple cron-based cleanup script is the next step:
|
||||||
|
|
||||||
|
```cron
|
||||||
|
0 4 * * * /srv/storage/atocore/scripts/cleanup-old-backups.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
## Drill schedule
|
||||||
|
|
||||||
|
A backup that has never been restored is theoretical. The schedule:
|
||||||
|
|
||||||
|
- **At least once per quarter**, perform a full restore drill on a
|
||||||
|
staging environment (or a temporary container with a separate
|
||||||
|
data dir) and verify the post-restore checks pass.
|
||||||
|
- **After every breaking schema migration**, perform a restore drill
|
||||||
|
to confirm the migration is reversible.
|
||||||
|
- **After any incident** that touched the storage layer (the EXDEV
|
||||||
|
bug from April 2026 is a good example), confirm the next backup
|
||||||
|
validates clean.
|
||||||
|
|
||||||
|
## Common failure modes and what to do about them
|
||||||
|
|
||||||
|
| Symptom | Likely cause | Action |
|
||||||
|
|---|---|---|
|
||||||
|
| `db_integrity_check_failed` on validation | SQLite snapshot copied while a write was in progress, or disk corruption | Take a fresh backup and validate again. If it fails twice, suspect the underlying disk. |
|
||||||
|
| `registry_invalid_json` | Registry was being edited at backup time | Take a fresh backup. The registry is small so this is cheap. |
|
||||||
|
| `chroma_snapshot_missing` after a restore | Snapshot was DB-only and the restore didn't move the existing chroma dir | Either rebuild via fresh ingestion or restore an older snapshot that includes Chroma. |
|
||||||
|
| Service won't start after restore | Permissions wrong on the restored files | Re-run `chown 1000:1000` (or whatever the gitea/atocore container user is) on the data dir. |
|
||||||
|
| `/stats` returns 0 documents after restore | The SQL store was restored but the source paths in `source_documents` don't match the current Dalidou paths | This means the backup came from a different deployment. Don't trust this restore — it's pulling from the wrong layout. |
|
||||||
|
|
||||||
|
## Open follow-ups (not yet implemented)
|
||||||
|
|
||||||
|
1. **Retention cleanup script**: see the cron entry above.
|
||||||
|
2. **Off-Dalidou backup target**: currently snapshots live on the
|
||||||
|
same disk as the live data. A real disaster-recovery story
|
||||||
|
needs at least one snapshot on a different physical machine.
|
||||||
|
The simplest first step is a periodic `rsync` to the user's
|
||||||
|
laptop or to another server.
|
||||||
|
3. **Backup encryption**: snapshots contain raw SQLite and JSON.
|
||||||
|
Consider age/gpg encryption if backups will be shipped off-site.
|
||||||
|
4. **Automatic post-backup validation**: today the validator must
|
||||||
|
be invoked manually. The `create_runtime_backup` function
|
||||||
|
should call `validate_backup` on its own output and refuse to
|
||||||
|
declare success if validation fails.
|
||||||
|
5. **Chroma backup is currently full directory copy** every time.
|
||||||
|
For large vector stores this gets expensive. A future
|
||||||
|
improvement would be incremental snapshots via filesystem-level
|
||||||
|
snapshotting (LVM, btrfs, ZFS).
|
||||||
|
|
||||||
|
## Quickstart cheat sheet
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Daily backup (DB + registry only — fast)
|
||||||
|
curl -fsS -X POST http://dalidou:8100/admin/backup \
|
||||||
|
-H "Content-Type: application/json" -d '{}'
|
||||||
|
|
||||||
|
# Weekly backup (DB + registry + Chroma — slower, holds ingestion lock)
|
||||||
|
curl -fsS -X POST http://dalidou:8100/admin/backup \
|
||||||
|
-H "Content-Type: application/json" -d '{"include_chroma": true}'
|
||||||
|
|
||||||
|
# List backups
|
||||||
|
curl -fsS http://dalidou:8100/admin/backup | jq '.backups[].stamp'
|
||||||
|
|
||||||
|
# Validate the most recent backup
|
||||||
|
LATEST=$(curl -fsS http://dalidou:8100/admin/backup | jq -r '.backups[-1].stamp')
|
||||||
|
curl -fsS http://dalidou:8100/admin/backup/$LATEST/validate | jq .
|
||||||
|
|
||||||
|
# Full restore — see the "Restore procedure" section above
|
||||||
|
```
|
||||||
@@ -44,8 +44,9 @@ read-only additive mode.
|
|||||||
|
|
||||||
### Engineering Layer Planning Sprint
|
### Engineering Layer Planning Sprint
|
||||||
|
|
||||||
The engineering layer is intentionally in planning, not implementation.
|
**Status: complete.** All 8 architecture docs are drafted. The
|
||||||
The architecture docs below are the current state of that planning:
|
engineering layer is now ready for V1 implementation against the
|
||||||
|
active project set.
|
||||||
|
|
||||||
- [engineering-query-catalog.md](architecture/engineering-query-catalog.md) —
|
- [engineering-query-catalog.md](architecture/engineering-query-catalog.md) —
|
||||||
the 20 v1-required queries the engineering layer must answer
|
the 20 v1-required queries the engineering layer must answer
|
||||||
@@ -55,17 +56,21 @@ The architecture docs below are the current state of that planning:
|
|||||||
Layer 0 → Layer 2 pipeline, triggers, review queue mechanics
|
Layer 0 → Layer 2 pipeline, triggers, review queue mechanics
|
||||||
- [conflict-model.md](architecture/conflict-model.md) —
|
- [conflict-model.md](architecture/conflict-model.md) —
|
||||||
detection, representation, and resolution of contradictory facts
|
detection, representation, and resolution of contradictory facts
|
||||||
|
- [tool-handoff-boundaries.md](architecture/tool-handoff-boundaries.md) —
|
||||||
|
KB-CAD / KB-FEM one-way mirror stance, ingest endpoints, drift handling
|
||||||
|
- [representation-authority.md](architecture/representation-authority.md) —
|
||||||
|
canonical home matrix across PKM / KB / repos / AtoCore for 22 fact kinds
|
||||||
|
- [human-mirror-rules.md](architecture/human-mirror-rules.md) —
|
||||||
|
templates, regeneration triggers, edit flow, "do not edit" enforcement
|
||||||
|
- [engineering-v1-acceptance.md](architecture/engineering-v1-acceptance.md) —
|
||||||
|
measurable done definition with 23 acceptance criteria
|
||||||
- [engineering-knowledge-hybrid-architecture.md](architecture/engineering-knowledge-hybrid-architecture.md) —
|
- [engineering-knowledge-hybrid-architecture.md](architecture/engineering-knowledge-hybrid-architecture.md) —
|
||||||
the 5-layer model (from the previous planning wave)
|
the 5-layer model (from the previous planning wave)
|
||||||
- [engineering-ontology-v1.md](architecture/engineering-ontology-v1.md) —
|
- [engineering-ontology-v1.md](architecture/engineering-ontology-v1.md) —
|
||||||
the initial V1 object and relationship inventory (previous wave)
|
the initial V1 object and relationship inventory (previous wave)
|
||||||
|
|
||||||
Still to draft before engineering-layer implementation begins:
|
The next concrete next step is the V1 implementation sprint, which
|
||||||
|
should follow engineering-v1-acceptance.md as its checklist.
|
||||||
- tool-handoff-boundaries.md (KB-CAD / KB-FEM read vs write)
|
|
||||||
- human-mirror-rules.md (templates, triggers, edit flow)
|
|
||||||
- representation-authority.md (PKM / KB / repo / AtoCore canonical home matrix)
|
|
||||||
- engineering-v1-acceptance.md (done definition)
|
|
||||||
|
|
||||||
## What Is Real Today
|
## What Is Real Today
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user