Compare commits
3 Commits
d0ff8b5738
...
d6ce6128cf
| Author | SHA1 | Date | |
|---|---|---|---|
| d6ce6128cf | |||
| 368adf2ebc | |||
| a637017900 |
123
.claude/commands/atocore-context.md
Normal file
123
.claude/commands/atocore-context.md
Normal file
@@ -0,0 +1,123 @@
|
||||
---
|
||||
description: Pull a context pack from the live AtoCore service for the current prompt
|
||||
argument-hint: <prompt text> [project-id]
|
||||
---
|
||||
|
||||
You are about to enrich a user prompt with context from the live AtoCore
|
||||
service. This is the daily-use entry point for AtoCore from inside Claude
|
||||
Code.
|
||||
|
||||
## Step 1 — parse the arguments
|
||||
|
||||
The user invoked `/atocore-context` with the following arguments:
|
||||
|
||||
```
|
||||
$ARGUMENTS
|
||||
```
|
||||
|
||||
Treat the **entire argument string** as the prompt text by default. If the
|
||||
last whitespace-separated token looks like a registered project id (matches
|
||||
one of `atocore`, `p04-gigabit`, `p04`, `p05-interferometer`, `p05`,
|
||||
`p06-polisher`, `p06`, or any case-insensitive variant), treat it as the
|
||||
project hint and use the rest as the prompt text. Otherwise, leave the
|
||||
project hint empty.
|
||||
|
||||
## Step 2 — call the AtoCore /context/build endpoint
|
||||
|
||||
Use the Bash tool to call AtoCore. The default endpoint is the live
|
||||
Dalidou instance. Read `ATOCORE_API_BASE` from the environment if set,
|
||||
otherwise default to `http://dalidou:3000` (the gitea host) — wait,
|
||||
no, AtoCore lives on a different port. Default to `http://dalidou:8100`
|
||||
which is the AtoCore service port from `pyproject.toml` and `config.py`.
|
||||
|
||||
Build the JSON body with `jq -n` so quoting is safe. Run something like:
|
||||
|
||||
```bash
|
||||
ATOCORE_API_BASE="${ATOCORE_API_BASE:-http://dalidou:8100}"
|
||||
PROMPT_TEXT='<the prompt text from step 1>'
|
||||
PROJECT_HINT='<the project hint or empty string>'
|
||||
|
||||
if [ -n "$PROJECT_HINT" ]; then
|
||||
BODY=$(jq -n --arg p "$PROMPT_TEXT" --arg proj "$PROJECT_HINT" \
|
||||
'{prompt:$p, project:$proj}')
|
||||
else
|
||||
BODY=$(jq -n --arg p "$PROMPT_TEXT" '{prompt:$p}')
|
||||
fi
|
||||
|
||||
curl -fsS -X POST "$ATOCORE_API_BASE/context/build" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d "$BODY"
|
||||
```
|
||||
|
||||
If `jq` is not available on the host, fall back to a Python one-liner:
|
||||
|
||||
```bash
|
||||
python -c "import json,sys; print(json.dumps({'prompt': sys.argv[1], 'project': sys.argv[2]} if sys.argv[2] else {'prompt': sys.argv[1]}))" "$PROMPT_TEXT" "$PROJECT_HINT"
|
||||
```
|
||||
|
||||
## Step 3 — present the context pack to the user
|
||||
|
||||
The response is JSON with at least these fields:
|
||||
`formatted_context`, `chunks_used`, `total_chars`, `budget`,
|
||||
`budget_remaining`, `duration_ms`, and a `chunks` array.
|
||||
|
||||
Print the response in a readable summary:
|
||||
|
||||
1. Print a one-line stats banner: `chunks=N, chars=X/budget, duration=Yms`
|
||||
2. Print the `formatted_context` block verbatim inside a fenced text
|
||||
code block so the user can read what AtoCore would feed an LLM
|
||||
3. Print the `chunks` array as a small bulleted list with `source_file`,
|
||||
`heading_path`, and `score` per chunk
|
||||
|
||||
If the response is empty (`chunks_used=0`, no project state, no
|
||||
memories), tell the user explicitly: "AtoCore returned no context for
|
||||
this prompt — either the corpus does not have relevant information or
|
||||
the project hint is wrong. Try `/atocore-context <prompt> <project-id>`."
|
||||
|
||||
If the curl call fails:
|
||||
- Network error → tell the user the AtoCore service may be down at
|
||||
`$ATOCORE_API_BASE` and suggest checking `curl $ATOCORE_API_BASE/health`
|
||||
- 4xx → print the error body verbatim, the API error message is usually
|
||||
enough
|
||||
- 5xx → print the error body and suggest checking the service logs
|
||||
|
||||
## Step 4 — capture the interaction (optional, opt-in)
|
||||
|
||||
If the user has previously asked the assistant to capture interactions
|
||||
into AtoCore (or if the slash command was invoked with the trailing
|
||||
literal `--capture` token), also POST the captured exchange to
|
||||
`/interactions` so the Phase 9 reflection loop sees it. Skip this step
|
||||
silently otherwise. The capture body is:
|
||||
|
||||
```json
|
||||
{
|
||||
"prompt": "<user prompt>",
|
||||
"response": "",
|
||||
"response_summary": "",
|
||||
"project": "<project hint or empty>",
|
||||
"client": "claude-code-slash",
|
||||
"session_id": "<a stable id for this Claude Code session>",
|
||||
"memories_used": ["<from chunks array if available>"],
|
||||
"chunks_used": ["<chunk_id from chunks array>"],
|
||||
"context_pack": {"chunks_used": <N>, "total_chars": <X>}
|
||||
}
|
||||
```
|
||||
|
||||
Note that the response field stays empty here — the LLM hasn't actually
|
||||
answered yet at the moment the slash command runs. A separate post-turn
|
||||
hook (not part of this command) would update the same interaction with
|
||||
the response, OR a follow-up `/atocore-record-response <interaction-id>`
|
||||
command would do it. For now, leave that as future work.
|
||||
|
||||
## Notes for the assistant
|
||||
|
||||
- DO NOT invent project ids that aren't in the registry. If the user
|
||||
passed something that doesn't match, treat it as part of the prompt.
|
||||
- DO NOT silently fall back to a different endpoint. If `ATOCORE_API_BASE`
|
||||
is wrong, surface the network error and let the user fix the env var.
|
||||
- DO NOT hide the formatted context pack from the user. The whole point
|
||||
of this command is to show what AtoCore would feed an LLM, so the user
|
||||
can decide if it's relevant.
|
||||
- The output goes into the user's working context as background — they
|
||||
may follow up with their actual question, and the AtoCore context pack
|
||||
acts as informal injected knowledge.
|
||||
4
.gitignore
vendored
4
.gitignore
vendored
@@ -10,4 +10,6 @@ htmlcov/
|
||||
.coverage
|
||||
venv/
|
||||
.venv/
|
||||
.claude/
|
||||
.claude/*
|
||||
!.claude/commands/
|
||||
!.claude/commands/**
|
||||
|
||||
434
docs/architecture/engineering-v1-acceptance.md
Normal file
434
docs/architecture/engineering-v1-acceptance.md
Normal file
@@ -0,0 +1,434 @@
|
||||
# Engineering Layer V1 Acceptance Criteria
|
||||
|
||||
## Why this document exists
|
||||
|
||||
The engineering layer planning sprint produced 7 architecture
|
||||
docs. None of them on their own says "you're done with V1, ship
|
||||
it". This document does. It translates the planning into
|
||||
measurable, falsifiable acceptance criteria so the implementation
|
||||
sprint can know unambiguously when V1 is complete.
|
||||
|
||||
The acceptance criteria are organized into four categories:
|
||||
|
||||
1. **Functional** — what the system must be able to do
|
||||
2. **Quality** — how well it must do it
|
||||
3. **Operational** — what running it must look like
|
||||
4. **Documentation** — what must be written down
|
||||
|
||||
V1 is "done" only when **every criterion in this document is met
|
||||
against at least one of the three active projects** (`p04-gigabit`,
|
||||
`p05-interferometer`, `p06-polisher`). The choice of which
|
||||
project is the test bed is up to the implementer, but the same
|
||||
project must satisfy all functional criteria.
|
||||
|
||||
## The single-sentence definition
|
||||
|
||||
> AtoCore Engineering Layer V1 is done when, against one chosen
|
||||
> active project, every v1-required query in
|
||||
> `engineering-query-catalog.md` returns a correct result, the
|
||||
> Human Mirror renders a coherent project overview, and a real
|
||||
> KB-CAD or KB-FEM export round-trips through the ingest →
|
||||
> review queue → active entity flow without violating any
|
||||
> conflict or trust invariant.
|
||||
|
||||
Everything below is the operational form of that sentence.
|
||||
|
||||
## Category 1 — Functional acceptance
|
||||
|
||||
### F-1: Entity store implemented per the V1 ontology
|
||||
|
||||
- The 12 V1 entity types from `engineering-ontology-v1.md` exist
|
||||
in the database with the schema described there
|
||||
- The 4 relationship families (Structural, Intent, Validation,
|
||||
Provenance) are implemented as edges with the relationship
|
||||
types listed in the catalog
|
||||
- Every entity has the shared header fields:
|
||||
`id, type, name, project_id, status, confidence, source_refs,
|
||||
created_at, updated_at, extractor_version, canonical_home`
|
||||
- The status lifecycle matches the memory layer:
|
||||
`candidate → active → superseded | invalid`
|
||||
|
||||
### F-2: All v1-required queries return correct results
|
||||
|
||||
For the chosen test project, every query Q-001 through Q-020 in
|
||||
`engineering-query-catalog.md` must:
|
||||
|
||||
- be implemented as an API endpoint with the shape specified in
|
||||
the catalog
|
||||
- return the expected result shape against real data
|
||||
- include the provenance chain when the catalog requires it
|
||||
- handle the empty case (no matches) gracefully — empty array,
|
||||
not 500
|
||||
|
||||
The "killer correctness queries" — Q-006 (orphan requirements),
|
||||
Q-009 (decisions on flagged assumptions), Q-011 (unsupported
|
||||
validation claims) — are non-negotiable. If any of those three
|
||||
returns wrong results, V1 is not done.
|
||||
|
||||
### F-3: Tool ingest endpoints are live
|
||||
|
||||
Both endpoints from `tool-handoff-boundaries.md` are implemented:
|
||||
|
||||
- `POST /ingest/kb-cad/export` accepts the documented JSON
|
||||
shape, validates it, and produces entity candidates
|
||||
- `POST /ingest/kb-fem/export` ditto
|
||||
- Both refuse exports with invalid schemas (4xx with a clear
|
||||
error)
|
||||
- Both return a summary of created/dropped/failed counts
|
||||
- Both never auto-promote anything; everything lands as
|
||||
`status="candidate"`
|
||||
- Both carry source identifiers (exporter name, exporter version,
|
||||
source artifact id) into the candidate's provenance fields
|
||||
|
||||
A real KB-CAD export — even a hand-crafted one if the actual
|
||||
exporter doesn't exist yet — must round-trip through the endpoint
|
||||
and produce reviewable candidates for the test project.
|
||||
|
||||
### F-4: Candidate review queue works end to end
|
||||
|
||||
Per `promotion-rules.md`:
|
||||
|
||||
- `GET /entities?status=candidate` lists the queue
|
||||
- `POST /entities/{id}/promote` moves candidate → active
|
||||
- `POST /entities/{id}/reject` moves candidate → invalid
|
||||
- The same shapes work for memories (already shipped in Phase 9 C)
|
||||
- The reviewer can edit a candidate's content via
|
||||
`PUT /entities/{id}` before promoting
|
||||
- Every promote/reject is logged with timestamp and reason
|
||||
|
||||
### F-5: Conflict detection fires
|
||||
|
||||
Per `conflict-model.md`:
|
||||
|
||||
- The synchronous detector runs at every active write
|
||||
(create, promote, project_state set, KB import)
|
||||
- A test must demonstrate that pushing a contradictory KB-CAD
|
||||
export creates a `conflicts` row with both members linked
|
||||
- The reviewer can resolve the conflict via
|
||||
`POST /conflicts/{id}/resolve` with one of the supported
|
||||
actions (supersede_others, no_action, dismiss)
|
||||
- Resolution updates the underlying entities according to the
|
||||
chosen action
|
||||
|
||||
### F-6: Human Mirror renders for the test project
|
||||
|
||||
Per `human-mirror-rules.md`:
|
||||
|
||||
- `GET /mirror/{project}/overview` returns rendered markdown
|
||||
- `GET /mirror/{project}/decisions` returns rendered markdown
|
||||
- `GET /mirror/{project}/subsystems/{subsystem}` returns
|
||||
rendered markdown for at least one subsystem
|
||||
- `POST /mirror/{project}/regenerate` triggers regeneration on
|
||||
demand
|
||||
- Generated files appear under `/srv/storage/atocore/data/mirror/`
|
||||
with the "do not edit" header banner
|
||||
- Disputed markers appear inline when conflicts exist
|
||||
- Project-state overrides display with the `(curated)` annotation
|
||||
- Output is deterministic (the same inputs produce the same
|
||||
bytes, suitable for diffing)
|
||||
|
||||
### F-7: Memory-to-entity graduation works for at least one type
|
||||
|
||||
Per `memory-vs-entities.md`:
|
||||
|
||||
- `POST /memory/{id}/graduate` exists
|
||||
- Graduating a memory of type `adaptation` produces a Decision
|
||||
entity candidate with the memory's content as a starting point
|
||||
- The original memory row stays at `status="graduated"` (a new
|
||||
status added by the engineering layer migration)
|
||||
- The graduated memory has a forward pointer to the entity
|
||||
candidate's id
|
||||
- Promoting the entity candidate does NOT delete the original
|
||||
memory
|
||||
- The same graduation flow works for `project` → Requirement
|
||||
and `knowledge` → Fact entity types (test the path; doesn't
|
||||
have to be exhaustive)
|
||||
|
||||
### F-8: Provenance chain is complete
|
||||
|
||||
For every active entity in the test project, the following must
|
||||
be true:
|
||||
|
||||
- It links back to at least one source via `source_refs` (which
|
||||
is one or more of: source_chunk_id, source_interaction_id,
|
||||
source_artifact_id from KB import)
|
||||
- The provenance chain can be walked from the entity to the
|
||||
underlying raw text (source_chunks) or external artifact
|
||||
- Q-017 (the evidence query) returns at least one row for every
|
||||
active entity
|
||||
|
||||
If any active entity has no provenance, it's a bug — provenance
|
||||
is mandatory at write time per the promotion rules.
|
||||
|
||||
## Category 2 — Quality acceptance
|
||||
|
||||
### Q-1: All existing tests still pass
|
||||
|
||||
The full pre-V1 test suite (currently 160 tests) must still
|
||||
pass. The V1 implementation may add new tests but cannot regress
|
||||
any existing test.
|
||||
|
||||
### Q-2: V1 has its own test coverage
|
||||
|
||||
For each of F-1 through F-8 above, at least one automated test
|
||||
exists that:
|
||||
|
||||
- exercises the happy path
|
||||
- covers at least one error path
|
||||
- runs in CI in under 10 seconds (no real network, no real LLM)
|
||||
|
||||
The full V1 test suite should be under 30 seconds total runtime
|
||||
to keep the development loop fast.
|
||||
|
||||
### Q-3: Conflict invariants are enforced by tests
|
||||
|
||||
Specific tests must demonstrate:
|
||||
|
||||
- Two contradictory KB exports produce a conflict (not silent
|
||||
overwrite)
|
||||
- A reviewer can't accidentally promote both members of an open
|
||||
conflict to active without resolving the conflict first
|
||||
- The "flag, never block" rule holds — writes still succeed
|
||||
even when they create a conflict
|
||||
|
||||
### Q-4: Trust hierarchy is enforced by tests
|
||||
|
||||
Specific tests must demonstrate:
|
||||
|
||||
- Entity candidates can never appear in context packs
|
||||
- Reinforcement only touches active memories (already covered
|
||||
by Phase 9 Commit B tests, but the same property must hold
|
||||
for entities once they exist)
|
||||
- Nothing automatically writes to project_state ever
|
||||
- Candidates can never satisfy Q-005 (only active entities count)
|
||||
|
||||
### Q-5: The Human Mirror is reproducible
|
||||
|
||||
A golden-file test exists for at least one Mirror page. Updating
|
||||
the golden file is a normal part of template work (single
|
||||
command, well-documented). The test fails if the renderer
|
||||
produces different bytes for the same input, catching
|
||||
non-determinism.
|
||||
|
||||
### Q-6: Killer correctness queries pass against real-ish data
|
||||
|
||||
The test bed for Q-006, Q-009, Q-011 is not synthetic. The
|
||||
implementation must seed the test project with at least:
|
||||
|
||||
- One Requirement that has a satisfying Component (Q-006 should
|
||||
not flag it)
|
||||
- One Requirement with no satisfying Component (Q-006 must flag it)
|
||||
- One Decision based on an Assumption flagged as `needs_review`
|
||||
(Q-009 must flag the Decision)
|
||||
- One ValidationClaim with at least one supporting Result
|
||||
(Q-011 should not flag it)
|
||||
- One ValidationClaim with no supporting Result (Q-011 must flag it)
|
||||
|
||||
These five seed cases run as a single integration test that
|
||||
exercises the killer correctness queries against actual
|
||||
representative data.
|
||||
|
||||
## Category 3 — Operational acceptance
|
||||
|
||||
### O-1: Migration is safe and reversible
|
||||
|
||||
The V1 schema migration (adding the `entities`, `relationships`,
|
||||
`conflicts`, `conflict_members` tables, plus `mirror_regeneration_failures`)
|
||||
must:
|
||||
|
||||
- run cleanly against a production-shape database
|
||||
- be implemented via the same `_apply_migrations` pattern as
|
||||
Phase 9 (additive only, idempotent, safe to run twice)
|
||||
- be tested by spinning up a fresh DB AND running against a
|
||||
copy of the live Dalidou DB taken from a backup
|
||||
|
||||
### O-2: Backup and restore still work
|
||||
|
||||
The backup endpoint must include the new tables. A restore drill
|
||||
on the test project must:
|
||||
|
||||
- successfully back up the V1 entity state via
|
||||
`POST /admin/backup`
|
||||
- successfully validate the snapshot
|
||||
- successfully restore from the snapshot per
|
||||
`docs/backup-restore-procedure.md`
|
||||
- pass post-restore verification including a Q-001 query against
|
||||
the test project
|
||||
|
||||
The drill must be performed once before V1 is declared done.
|
||||
|
||||
### O-3: Performance bounds
|
||||
|
||||
These are starting bounds; tune later if real usage shows
|
||||
problems:
|
||||
|
||||
- Single-entity write (`POST /entities/...`): under 100ms p99
|
||||
on the production Dalidou hardware
|
||||
- Single Q-001 / Q-005 / Q-008 query: under 500ms p99 against
|
||||
a project with up to 1000 entities
|
||||
- Mirror regeneration of one project overview: under 5 seconds
|
||||
for a project with up to 1000 entities
|
||||
- Conflict detector at write time: adds no more than 50ms p99
|
||||
to a write that doesn't actually produce a conflict
|
||||
|
||||
These bounds are not tested by automated benchmarks in V1 (that
|
||||
would be over-engineering). They are sanity-checked by the
|
||||
developer running the operations against the test project.
|
||||
|
||||
### O-4: No new manual ops burden
|
||||
|
||||
V1 should not introduce any new "you have to remember to run X
|
||||
every day" requirement. Specifically:
|
||||
|
||||
- Mirror regeneration is automatic (debounced async + daily
|
||||
refresh), no manual cron entry needed
|
||||
- Conflict detection is automatic at write time, no manual sweep
|
||||
needed in V1 (the nightly sweep is V2)
|
||||
- Backup retention cleanup is **still** an open follow-up from
|
||||
the operational baseline; V1 does not block on it
|
||||
|
||||
### O-5: No regressions in Phase 9 reflection loop
|
||||
|
||||
The capture, reinforcement, and extraction loop from Phase 9
|
||||
A/B/C must continue to work end to end with the engineering
|
||||
layer in place. Specifically:
|
||||
|
||||
- Memories whose types are NOT in the engineering layer
|
||||
(identity, preference, episodic) keep working exactly as
|
||||
before
|
||||
- Memories whose types ARE in the engineering layer (project,
|
||||
knowledge, adaptation) can still be created hand or by
|
||||
extraction; the deprecation rule from `memory-vs-entities.md`
|
||||
("no new writes after V1 ships") is implemented as a
|
||||
configurable warning, not a hard block, so existing
|
||||
workflows aren't disrupted
|
||||
|
||||
## Category 4 — Documentation acceptance
|
||||
|
||||
### D-1: Per-entity-type spec docs
|
||||
|
||||
Each of the 12 V1 entity types has a short spec doc under
|
||||
`docs/architecture/entities/` covering:
|
||||
|
||||
- the entity's purpose
|
||||
- its required and optional fields
|
||||
- its lifecycle quirks (if any beyond the standard
|
||||
candidate/active/superseded/invalid)
|
||||
- which queries it appears in (cross-reference to the catalog)
|
||||
- which relationship types reference it
|
||||
|
||||
These docs can be terse — a page each, mostly bullet lists.
|
||||
Their purpose is to make the entity model legible to a future
|
||||
maintainer, not to be reference manuals.
|
||||
|
||||
### D-2: KB-CAD and KB-FEM export schema docs
|
||||
|
||||
`docs/architecture/kb-cad-export-schema.md` and
|
||||
`docs/architecture/kb-fem-export-schema.md` are written and
|
||||
match the implemented validators.
|
||||
|
||||
### D-3: V1 release notes
|
||||
|
||||
A `docs/v1-release-notes.md` summarizes:
|
||||
|
||||
- What V1 added (entities, relationships, conflicts, mirror,
|
||||
ingest endpoints)
|
||||
- What V1 deferred (auto-promotion, BOM/cost/manufacturing
|
||||
entities, NX direct integration, cross-project rollups)
|
||||
- The migration story for existing memories (graduation flow)
|
||||
- Known limitations and the V2 roadmap pointers
|
||||
|
||||
### D-4: master-plan-status.md and current-state.md updated
|
||||
|
||||
Both top-level status docs reflect V1's completion:
|
||||
|
||||
- Phase 6 (AtoDrive) and the engineering layer are explicitly
|
||||
marked as separate tracks
|
||||
- The engineering planning sprint section is marked complete
|
||||
- Phase 9 stays at "baseline complete" (V1 doesn't change Phase 9)
|
||||
- The engineering layer V1 is added as its own line item
|
||||
|
||||
## What V1 explicitly does NOT need to do
|
||||
|
||||
To prevent scope creep, here is the negative list. None of the
|
||||
following are V1 acceptance criteria:
|
||||
|
||||
- **No LLM extractor.** The Phase 9 C rule-based extractor is
|
||||
the entity extractor for V1 too, just with new rules added for
|
||||
entity types.
|
||||
- **No auto-promotion of candidates.** Per `promotion-rules.md`.
|
||||
- **No write-back to KB-CAD or KB-FEM.** Per
|
||||
`tool-handoff-boundaries.md`.
|
||||
- **No multi-user / per-reviewer auth.** Single-user assumed.
|
||||
- **No real-time UI.** API + Mirror markdown is the V1 surface.
|
||||
A web UI is V2+.
|
||||
- **No cross-project rollups.** Per `human-mirror-rules.md`.
|
||||
- **No time-travel queries** (Q-015 stays v1-stretch).
|
||||
- **No nightly conflict sweep.** Synchronous detection only in V1.
|
||||
- **No incremental Chroma snapshots.** The current full-copy
|
||||
approach in `backup-restore-procedure.md` is fine for V1.
|
||||
- **No retention cleanup script.** Still an open follow-up.
|
||||
- **No backup encryption.** Still an open follow-up.
|
||||
- **No off-Dalidou backup target.** Still an open follow-up.
|
||||
|
||||
## How to use this document during implementation
|
||||
|
||||
When the implementation sprint begins:
|
||||
|
||||
1. Read this doc once, top to bottom
|
||||
2. Pick the test project (probably p05-interferometer because
|
||||
the optical/structural domain has the cleanest entity model)
|
||||
3. For each section, write the test or the implementation, in
|
||||
roughly the order: F-1 → F-2 → F-3 → F-4 → F-5 → F-6 → F-7 → F-8
|
||||
4. Each acceptance criterion's test should be written **before
|
||||
or alongside** the implementation, not after
|
||||
5. Run the full test suite at every commit
|
||||
6. When every box is checked, write D-3 (release notes), update
|
||||
D-4 (status docs), and call V1 done
|
||||
|
||||
The implementation sprint should not touch anything outside the
|
||||
scope listed here. If a desire arises to add something not in
|
||||
this doc, that's a V2 conversation, not a V1 expansion.
|
||||
|
||||
## Anticipated friction points
|
||||
|
||||
These are the things I expect will be hard during implementation:
|
||||
|
||||
1. **The graduation flow (F-7)** is the most cross-cutting
|
||||
change because it touches the existing memory module.
|
||||
Worth doing it last so the memory module is stable for
|
||||
all the V1 entity work first.
|
||||
2. **The Mirror's deterministic-output requirement (Q-5)** will
|
||||
bite if the implementer iterates over Python dicts without
|
||||
sorting. Plan to use `sorted()` literally everywhere.
|
||||
3. **Conflict detection (F-5)** has subtle correctness traps:
|
||||
the slot key extraction must be stable, the dedup-of-existing-conflicts
|
||||
logic must be right, and the synchronous detector must not
|
||||
slow writes meaningfully (Q-3 / O-3 cover this, but watch).
|
||||
4. **Provenance backfill** for entities that come from the
|
||||
existing memory layer via graduation (F-7) is the trickiest
|
||||
part: the original memory may not have had a strict
|
||||
`source_chunk_id`, in which case the graduated entity also
|
||||
doesn't have one. The implementation needs an "orphan
|
||||
provenance" allowance for graduated entities, with a
|
||||
warning surfaced in the Mirror.
|
||||
|
||||
These aren't blockers, just the parts of the V1 spec I'd
|
||||
attack with extra care.
|
||||
|
||||
## TL;DR
|
||||
|
||||
- Engineering V1 is done when every box in this doc is checked
|
||||
against one chosen active project
|
||||
- Functional: 8 criteria covering entities, queries, ingest,
|
||||
review queue, conflicts, mirror, graduation, provenance
|
||||
- Quality: 6 criteria covering tests, golden files, killer
|
||||
correctness, trust enforcement
|
||||
- Operational: 5 criteria covering migration safety, backup
|
||||
drill, performance bounds, no new manual ops, Phase 9 not
|
||||
regressed
|
||||
- Documentation: 4 criteria covering entity specs, KB schema
|
||||
docs, release notes, top-level status updates
|
||||
- Negative list: a clear set of things V1 deliberately does
|
||||
NOT need to do, to prevent scope creep
|
||||
- The implementation sprint follows this doc as a checklist
|
||||
384
docs/architecture/human-mirror-rules.md
Normal file
384
docs/architecture/human-mirror-rules.md
Normal file
@@ -0,0 +1,384 @@
|
||||
# Human Mirror Rules (Layer 3 → derived markdown views)
|
||||
|
||||
## Why this document exists
|
||||
|
||||
The engineering layer V1 stores facts as typed entities and
|
||||
relationships in a SQL database. That representation is excellent
|
||||
for queries, conflict detection, and automated reasoning, but
|
||||
it's terrible for the human reading experience. People want to
|
||||
read prose, not crawl JSON.
|
||||
|
||||
The Human Mirror is the layer that turns the typed entity store
|
||||
into human-readable markdown pages. It's strictly a derived view —
|
||||
nothing in the Human Mirror is canonical, every page is regenerated
|
||||
from current entity state on demand.
|
||||
|
||||
This document defines:
|
||||
|
||||
- what the Human Mirror generates
|
||||
- when it regenerates
|
||||
- how the human edits things they see in the Mirror
|
||||
- how the canonical-vs-derived rule is enforced (so editing the
|
||||
derived markdown can't silently corrupt the entity store)
|
||||
|
||||
## The non-negotiable rule
|
||||
|
||||
> **The Human Mirror is read-only from the human's perspective.**
|
||||
>
|
||||
> If the human wants to change a fact they see in the Mirror, they
|
||||
> change it in the canonical home (per `representation-authority.md`),
|
||||
> NOT in the Mirror page. The next regeneration picks up the change.
|
||||
|
||||
This rule is what makes the whole derived-view approach safe. If
|
||||
the human is allowed to edit Mirror pages directly, the
|
||||
canonical-vs-derived split breaks and the Mirror becomes a second
|
||||
source of truth that disagrees with the entity store.
|
||||
|
||||
The technical enforcement is that every Mirror page carries a
|
||||
header banner that says "this file is generated from AtoCore
|
||||
entity state, do not edit", and the file is regenerated from the
|
||||
entity store on every change to its underlying entities. Manual
|
||||
edits will be silently overwritten on the next regeneration.
|
||||
|
||||
## What the Mirror generates in V1
|
||||
|
||||
Three template families, each producing one or more pages per
|
||||
project:
|
||||
|
||||
### 1. Project Overview
|
||||
|
||||
One page per registered project. Renders:
|
||||
|
||||
- Project header (id, aliases, description)
|
||||
- Subsystem tree (from Q-001 / Q-004 in the query catalog)
|
||||
- Active Decisions affecting this project (Q-008, ordered by date)
|
||||
- Open Requirements with coverage status (Q-005, Q-006)
|
||||
- Open ValidationClaims with support status (Q-010, Q-011)
|
||||
- Currently flagged conflicts (from the conflict model)
|
||||
- Recent changes (Q-013) — last 14 days
|
||||
|
||||
This is the most important Mirror page. It's the page someone
|
||||
opens when they want to know "what's the state of this project
|
||||
right now". It deliberately mirrors what `current-state.md` does
|
||||
for AtoCore itself but generated entirely from typed state.
|
||||
|
||||
### 2. Decision Log
|
||||
|
||||
One page per project. Renders:
|
||||
|
||||
- All active Decisions in chronological order (newest first)
|
||||
- Each Decision shows: id, what was decided, when, the affected
|
||||
Subsystem/Component, the supporting evidence (Q-014, Q-017)
|
||||
- Superseded Decisions appear as collapsed "history" entries
|
||||
with a forward link to whatever superseded them
|
||||
- Conflicting Decisions get a "⚠ disputed" marker
|
||||
|
||||
This is the human-readable form of the engineering query catalog's
|
||||
Q-014 query.
|
||||
|
||||
### 3. Subsystem Detail
|
||||
|
||||
One page per Subsystem (so a few per project). Renders:
|
||||
|
||||
- Subsystem header
|
||||
- Components contained in this subsystem (Q-001)
|
||||
- Interfaces this subsystem has (Q-003)
|
||||
- Constraints applying to it (Q-007)
|
||||
- Decisions affecting it (Q-008)
|
||||
- Validation status: which Requirements are satisfied,
|
||||
which are open (Q-005, Q-006)
|
||||
- Change history within this subsystem (Q-013 scoped)
|
||||
|
||||
Subsystem detail pages are what someone reads when they're
|
||||
working on a specific part of the system and want everything
|
||||
relevant in one place.
|
||||
|
||||
## What the Mirror does NOT generate in V1
|
||||
|
||||
Intentionally excluded so the V1 implementation stays scoped:
|
||||
|
||||
- **Per-component detail pages.** Components are listed in
|
||||
Subsystem pages but don't get their own pages. Reduces page
|
||||
count from hundreds to dozens.
|
||||
- **Per-Decision detail pages.** Decisions appear inline in
|
||||
Project Overview and Decision Log; their full text plus
|
||||
evidence chain is shown there, not on a separate page.
|
||||
- **Cross-project rollup pages.** No "all projects at a glance"
|
||||
page in V1. Each project is its own report.
|
||||
- **Time-series / historical pages.** The Mirror is always
|
||||
"current state". History is accessible via Decision Log and
|
||||
superseded chains, but no "what was true on date X" page exists
|
||||
in V1 (Q-015 is v1-stretch in the query catalog for the same
|
||||
reason).
|
||||
- **Diff pages between two timestamps.** Same reasoning.
|
||||
- **Render of the conflict queue itself.** Conflicts appear
|
||||
inline in the relevant Mirror pages with the "⚠ disputed"
|
||||
marker and a link to `/conflicts/{id}`, but there's no
|
||||
Mirror page that lists all conflicts. Use `GET /conflicts`.
|
||||
- **Per-memory pages.** Memories are not engineering entities;
|
||||
they appear in context packs and the review queue, not in the
|
||||
Human Mirror.
|
||||
|
||||
## Where Mirror pages live
|
||||
|
||||
Two options were considered. The chosen V1 path is option B:
|
||||
|
||||
**Option A — write Mirror pages back into the source vault.**
|
||||
Generate `/srv/storage/atocore/sources/vault/mirror/p05/overview.md`
|
||||
so the human reads them in their normal Obsidian / markdown
|
||||
viewer. **Rejected** because writing into the source vault
|
||||
violates the "sources are read-only" rule from
|
||||
`tool-handoff-boundaries.md` and the operating model.
|
||||
|
||||
**Option B (chosen) — write Mirror pages into a dedicated AtoCore
|
||||
output dir, served via the API.** Generate under
|
||||
`/srv/storage/atocore/data/mirror/p05/overview.md`. The human
|
||||
reads them via:
|
||||
|
||||
- the API endpoints `GET /mirror/{project}/overview`,
|
||||
`GET /mirror/{project}/decisions`,
|
||||
`GET /mirror/{project}/subsystems/{subsystem}` (all return
|
||||
rendered markdown as text/markdown)
|
||||
- a future "Mirror viewer" in the Claude Code slash command
|
||||
`/atocore-mirror <project>` that fetches the rendered markdown
|
||||
and displays it inline
|
||||
- direct file access on Dalidou for power users:
|
||||
`cat /srv/storage/atocore/data/mirror/p05/overview.md`
|
||||
|
||||
The dedicated dir keeps the Mirror clearly separated from the
|
||||
canonical sources and makes regeneration safe (it's just a
|
||||
directory wipe + write).
|
||||
|
||||
## When the Mirror regenerates
|
||||
|
||||
Three triggers, in order from cheapest to most expensive:
|
||||
|
||||
### 1. On explicit human request
|
||||
|
||||
```
|
||||
POST /mirror/{project}/regenerate
|
||||
```
|
||||
|
||||
Returns the timestamp of the regeneration and the list of files
|
||||
written. This is the path the human takes when they've just
|
||||
curated something into project_state and want to see the Mirror
|
||||
reflect it immediately.
|
||||
|
||||
### 2. On entity write (debounced, async, per project)
|
||||
|
||||
When any entity in a project changes status (candidate → active,
|
||||
active → superseded), a regeneration of that project's Mirror is
|
||||
queued. The queue is debounced — multiple writes within a 30-second
|
||||
window only trigger one regeneration. This keeps the Mirror
|
||||
"close to current" without generating a Mirror update on every
|
||||
single API call.
|
||||
|
||||
The implementation is a simple dict of "next regeneration time"
|
||||
per project, checked by a background task. No cron, no message
|
||||
queue, no Celery. Just a `dict[str, datetime]` and a thread.
|
||||
|
||||
### 3. On scheduled refresh (daily)
|
||||
|
||||
Once per day at a quiet hour, every project's Mirror regenerates
|
||||
unconditionally. This catches any state drift from manual
|
||||
project_state edits that bypassed the entity write hooks, and
|
||||
provides a baseline guarantee that the Mirror is at most 24
|
||||
hours stale.
|
||||
|
||||
The schedule runs from the same machinery as the future backup
|
||||
retention job, so we get one cron-equivalent system to maintain
|
||||
instead of two.
|
||||
|
||||
## What if regeneration fails
|
||||
|
||||
The Mirror has to be resilient. If regeneration fails for a
|
||||
project (e.g. a query catalog query crashes, a template rendering
|
||||
error), the existing Mirror files are **not** deleted. The
|
||||
existing files stay in place (showing the last successful state)
|
||||
and a regeneration error is recorded in:
|
||||
|
||||
- the API response if the trigger was explicit
|
||||
- a log entry at warning level for the async path
|
||||
- a `mirror_regeneration_failures` table for the daily refresh
|
||||
|
||||
This means the human can always read the Mirror, even if the
|
||||
last 5 minutes of changes haven't made it in yet. Stale is
|
||||
better than blank.
|
||||
|
||||
## How the human curates "around" the Mirror
|
||||
|
||||
The Mirror reflects the current entity state. If the human
|
||||
doesn't like what they see, the right edits go into one of:
|
||||
|
||||
| What you want to change | Where you change it |
|
||||
|---|---|
|
||||
| A Decision's text | `PUT /entities/Decision/{id}` (or `PUT /memory/{id}` if it's still memory-layer) |
|
||||
| A Decision's status (active → superseded) | `POST /entities/Decision/{id}/supersede` (V1 entity API) |
|
||||
| Whether a Component "satisfies" a Requirement | edit the relationship directly via the entity API (V1) |
|
||||
| The current trusted next focus shown on the Project Overview | `POST /project/state` with `category=status, key=next_focus` |
|
||||
| A typo in a generated heading or label | edit the **template**, not the rendered file. Templates live in `templates/mirror/` (V1 implementation) |
|
||||
| Source of a fact ("this came from KB-CAD on day X") | not editable by hand — it's automatically populated from provenance |
|
||||
|
||||
The rule is consistent: edit the canonical home, regenerate (or
|
||||
let the auto-trigger fire), see the change reflected in the
|
||||
Mirror.
|
||||
|
||||
## Templates
|
||||
|
||||
The Mirror uses Jinja2-style templates checked into the repo
|
||||
under `templates/mirror/`. Each template is a markdown file with
|
||||
placeholders that the renderer fills from query catalog results.
|
||||
|
||||
Template list for V1:
|
||||
|
||||
- `templates/mirror/project-overview.md.j2`
|
||||
- `templates/mirror/decision-log.md.j2`
|
||||
- `templates/mirror/subsystem-detail.md.j2`
|
||||
|
||||
Editing a template is a code change, reviewed via normal git PRs.
|
||||
The templates are deliberately small and readable so the human
|
||||
can tweak the output format without touching renderer code.
|
||||
|
||||
The renderer is a thin module:
|
||||
|
||||
```python
|
||||
# src/atocore/mirror/renderer.py (V1, not yet implemented)
|
||||
|
||||
def render_project_overview(project: str) -> str:
|
||||
"""Generate the project overview markdown for one project."""
|
||||
facts = collect_project_overview_facts(project)
|
||||
template = load_template("project-overview.md.j2")
|
||||
return template.render(**facts)
|
||||
```
|
||||
|
||||
## The "do not edit" header
|
||||
|
||||
Every generated Mirror file starts with a fixed banner:
|
||||
|
||||
```markdown
|
||||
<!--
|
||||
This file is generated by AtoCore from current entity state.
|
||||
DO NOT EDIT — manual changes will be silently overwritten on
|
||||
the next regeneration.
|
||||
Edit the canonical home instead. See:
|
||||
https://docs.atocore.../representation-authority.md
|
||||
Regenerated: 2026-04-07T12:34:56Z
|
||||
Source entities: <commit-like checksum of input data>
|
||||
-->
|
||||
```
|
||||
|
||||
The checksum at the end lets the renderer skip work when nothing
|
||||
relevant has changed since the last regeneration. If the inputs
|
||||
match the previous run's checksum, the existing file is left
|
||||
untouched.
|
||||
|
||||
## Conflicts in the Mirror
|
||||
|
||||
Per the conflict model, any open conflict on a fact that appears
|
||||
in the Mirror gets a visible disputed marker:
|
||||
|
||||
```markdown
|
||||
- Lateral support material: **GF-PTFE** ⚠ disputed
|
||||
- The KB-CAD import on 2026-04-07 reported PEEK; conflict #c-039.
|
||||
```
|
||||
|
||||
The disputed marker is a hyperlink (in renderer terms; the markdown
|
||||
output is a relative link) to the conflict detail page in the API
|
||||
or to the conflict id for direct lookup. The reviewer follows the
|
||||
link, resolves the conflict via `POST /conflicts/{id}/resolve`,
|
||||
and on the next regeneration the marker disappears.
|
||||
|
||||
## Project-state overrides in the Mirror
|
||||
|
||||
When a Mirror page would show a value derived from entities, but
|
||||
project_state has an override on the same key, **the Mirror shows
|
||||
the project_state value** with a small annotation noting the
|
||||
override:
|
||||
|
||||
```markdown
|
||||
- Next focus: **Wave 2 trusted-operational ingestion** (curated)
|
||||
```
|
||||
|
||||
The `(curated)` annotation tells the reader "this is from the
|
||||
trusted-state Layer 3, not from extracted entities". This makes
|
||||
the trust hierarchy visible in the human reading experience.
|
||||
|
||||
## The "Mirror diff" workflow (post-V1, but designed for)
|
||||
|
||||
A common workflow after V1 ships will be:
|
||||
|
||||
1. Reviewer has curated some new entities
|
||||
2. They want to see "what changed in the Mirror as a result"
|
||||
3. They want to share that diff with someone else as evidence
|
||||
|
||||
To support this, the Mirror generator writes its output
|
||||
deterministically (sorted iteration, stable timestamp formatting)
|
||||
so a `git diff` between two regenerated states is meaningful.
|
||||
|
||||
V1 doesn't add an explicit "diff between two Mirror snapshots"
|
||||
endpoint — that's deferred. But the deterministic-output
|
||||
property is a V1 requirement so future diffing works without
|
||||
re-renderer-design work.
|
||||
|
||||
## What the Mirror enables
|
||||
|
||||
With the Mirror in place:
|
||||
|
||||
- **OpenClaw can read project state in human form.** The
|
||||
read-only AtoCore helper skill on the T420 already calls
|
||||
`/context/build`; in V1 it gains the option to call
|
||||
`/mirror/{project}/overview` to get a fully-rendered markdown
|
||||
page instead of just retrieved chunks. This is much faster
|
||||
than crawling individual entities for general questions.
|
||||
- **The human gets a daily-readable artifact.** Every morning,
|
||||
Antoine can `cat /srv/storage/atocore/data/mirror/p05/overview.md`
|
||||
and see the current state of p05 in his preferred reading
|
||||
format. No API calls, no JSON parsing.
|
||||
- **Cross-collaborator sharing.** If you ever want to send
|
||||
someone a project overview without giving them AtoCore access,
|
||||
the Mirror file is a self-contained markdown document they can
|
||||
read in any markdown viewer.
|
||||
- **Claude Code integration.** A future
|
||||
`/atocore-mirror <project>` slash command renders the Mirror
|
||||
inline, complementing the existing `/atocore-context` command
|
||||
with a human-readable view of "what does AtoCore think about
|
||||
this project right now".
|
||||
|
||||
## Open questions for V1 implementation
|
||||
|
||||
1. **What's the regeneration debounce window?** 30 seconds is the
|
||||
starting value but should be tuned with real usage.
|
||||
2. **Does the daily refresh need a separate trigger mechanism, or
|
||||
is it just a long-period entry in the same in-process scheduler
|
||||
that handles the debounced async refreshes?** Probably the
|
||||
latter — keep it simple.
|
||||
3. **How are templates tested?** Likely a small set of fixture
|
||||
project states + golden output files, with a single test that
|
||||
asserts `render(fixture) == golden`. Updating golden files is
|
||||
a normal part of template work.
|
||||
4. **Are Mirror pages discoverable via a directory listing
|
||||
endpoint?** `GET /mirror/{project}` returns the list of
|
||||
available pages for that project. Probably yes; cheap to add.
|
||||
5. **How does the Mirror handle a project that has zero entities
|
||||
yet?** Render an empty-state page that says "no curated facts
|
||||
yet — add some via /memory or /entities/Decision". Better than
|
||||
a blank file.
|
||||
|
||||
## TL;DR
|
||||
|
||||
- The Human Mirror generates 3 template families per project
|
||||
(Overview, Decision Log, Subsystem Detail) from current entity
|
||||
state
|
||||
- It's strictly read-only from the human's perspective; edits go
|
||||
to the canonical home and the Mirror picks them up on
|
||||
regeneration
|
||||
- Three regeneration triggers: explicit POST, debounced
|
||||
async-on-write, daily scheduled refresh
|
||||
- Mirror files live in `/srv/storage/atocore/data/mirror/`
|
||||
(NOT in the source vault — sources stay read-only)
|
||||
- Conflicts and project_state overrides are visible inline in
|
||||
the rendered markdown so the trust hierarchy shows through
|
||||
- Templates are checked into the repo and edited via PR; the
|
||||
rendered files are derived and never canonical
|
||||
- Deterministic output is a V1 requirement so future diffing
|
||||
works without rework
|
||||
273
docs/architecture/representation-authority.md
Normal file
273
docs/architecture/representation-authority.md
Normal file
@@ -0,0 +1,273 @@
|
||||
# Representation Authority (canonical home matrix)
|
||||
|
||||
## Why this document exists
|
||||
|
||||
The same fact about an engineering project can show up in many
|
||||
places: a markdown note in the PKM, a structured field in KB-CAD,
|
||||
a commit message in a Gitea repo, an active memory in AtoCore, an
|
||||
entity in the engineering layer, a row in trusted project state.
|
||||
**Without an explicit rule about which representation is
|
||||
authoritative for which kind of fact, the system will accumulate
|
||||
contradictions and the human will lose trust in all of them.**
|
||||
|
||||
This document is the canonical-home matrix. Every kind of fact
|
||||
that AtoCore handles has exactly one authoritative representation,
|
||||
and every other place that holds a copy of that fact is, by
|
||||
definition, a derived view that may be stale.
|
||||
|
||||
## The representations in scope
|
||||
|
||||
Six places where facts can live in this ecosystem:
|
||||
|
||||
| Layer | What it is | Who edits it | How it's structured |
|
||||
|---|---|---|---|
|
||||
| **PKM** | Antoine's Obsidian-style markdown vault under `/srv/storage/atocore/sources/vault/` | Antoine, by hand | unstructured markdown with optional frontmatter |
|
||||
| **KB project** | the engineering Knowledge Base (KB-CAD / KB-FEM repos and any companion docs) | Antoine, semi-structured | per-tool typed records |
|
||||
| **Gitea repos** | source code repos under `dalidou:3000/Antoine/*` (Fullum-Interferometer, polisher-sim, ATOCore itself, ...) | Antoine via git commits | code, READMEs, repo-specific markdown |
|
||||
| **AtoCore memories** | rows in the `memories` table | hand-authored or extracted from interactions | typed (identity / preference / project / episodic / knowledge / adaptation) |
|
||||
| **AtoCore entities** | rows in the `entities` table (V1, not yet built) | imported from KB exports or extracted from interactions | typed entities + relationships per the V1 ontology |
|
||||
| **AtoCore project state** | rows in the `project_state` table (Layer 3, trusted) | hand-curated only, never automatic | category + key + value |
|
||||
|
||||
## The canonical home rule
|
||||
|
||||
> For each kind of fact, exactly one of the six representations is
|
||||
> the authoritative source. The other five may hold derived
|
||||
> copies, but they are not allowed to disagree with the
|
||||
> authoritative one. When they disagree, the disagreement is a
|
||||
> conflict and surfaces via the conflict model.
|
||||
|
||||
The matrix below assigns the authoritative representation per fact
|
||||
kind. It is the practical answer to the question "where does this
|
||||
fact actually live?" for daily decisions.
|
||||
|
||||
## The canonical-home matrix
|
||||
|
||||
| Fact kind | Canonical home | Why | How it gets into AtoCore |
|
||||
|---|---|---|---|
|
||||
| **CAD geometry** (the actual model) | NX (or successor CAD tool) | the only place that can render and validate it | not in AtoCore at all in V1 |
|
||||
| **CAD-side structure** (subsystem tree, component list, materials, parameters) | KB-CAD | KB-CAD is the structured wrapper around NX | KB-CAD export → `/ingest/kb-cad/export` → entities |
|
||||
| **FEM mesh & solver settings** | KB-FEM (wrapping the FEM tool) | only the solver representation can run | not in AtoCore at all in V1 |
|
||||
| **FEM results & validation outcomes** | KB-FEM | KB-FEM owns the outcome records | KB-FEM export → `/ingest/kb-fem/export` → entities |
|
||||
| **Source code** | Gitea repos | repos are version-controlled and reviewable | indirectly via repo markdown ingestion (Phase 1) |
|
||||
| **Repo-level documentation** (READMEs, design docs in the repo) | Gitea repos | lives next to the code it documents | ingested as source chunks; never hand-edited in AtoCore |
|
||||
| **Project-level prose notes** (decisions in long-form, journal-style entries, working notes) | PKM | the place Antoine actually writes when thinking | ingested as source chunks; the extractor proposes candidates from these for the review queue |
|
||||
| **Identity** ("the user is a mechanical engineer running AtoCore") | AtoCore memories (`identity` type) | nowhere else holds personal identity | hand-authored via `POST /memory` or extracted from interactions |
|
||||
| **Preference** ("prefers small reviewable diffs", "uses SI units") | AtoCore memories (`preference` type) | nowhere else holds personal preferences | hand-authored or extracted |
|
||||
| **Episodic** ("on April 6 we debugged the EXDEV bug") | AtoCore memories (`episodic` type) | nowhere else has time-bound personal recall | extracted from captured interactions |
|
||||
| **Decision** (a structured engineering decision) | AtoCore **entities** (Decision) once the engineering layer ships; AtoCore memories (`adaptation`) until then | needs structured supersession, audit trail, and link to affected components | extracted from PKM or interactions; promoted via review queue |
|
||||
| **Requirement** | AtoCore **entities** (Requirement) | needs structured satisfaction tracking | extracted from PKM, KB-CAD, or interactions |
|
||||
| **Constraint** | AtoCore **entities** (Constraint) | needs structured link to the entity it constrains | extracted from PKM, KB-CAD, or interactions |
|
||||
| **Validation claim** | AtoCore **entities** (ValidationClaim) | needs structured link to supporting Result | extracted from KB-FEM exports or interactions |
|
||||
| **Material** | KB-CAD if the material is on a real component; AtoCore entity (Material) if it's a project-wide material decision not yet attached to geometry | structured properties live in KB-CAD's material database | KB-CAD export, or hand-authored as a Material entity |
|
||||
| **Parameter** | KB-CAD or KB-FEM depending on whether it's a geometry or solver parameter; AtoCore entity (Parameter) if it's a higher-level project parameter not in either tool | structured numeric values with units live in their tool of origin | KB export, or hand-authored |
|
||||
| **Project status / current focus / next milestone** | AtoCore **project_state** (Layer 3) | the trust hierarchy says trusted state is the highest authority for "what is the current state of the project" | hand-curated via `POST /project/state` |
|
||||
| **Architectural decision records (ADRs)** | depends on form: long-form ADR markdown lives in the repo; the structured fact about which ADR was selected lives in the AtoCore Decision entity | both representations are useful for different audiences | repo ingestion provides the prose; the entity is created by extraction or hand-authored |
|
||||
| **Operational runbooks** | repo (next to the code they describe) | lives with the system it operates | not promoted into AtoCore entities — runbooks are reference material, not facts |
|
||||
| **Backup metadata** (snapshot timestamps, integrity status) | the backup-metadata.json files under `/srv/storage/atocore/backups/` | each snapshot is its own self-describing record | not in AtoCore's database; queried via the `/admin/backup` endpoints |
|
||||
| **Conversation history with AtoCore (interactions)** | AtoCore `interactions` table | nowhere else has the prompt + context pack + response triple | written by capture (Phase 9 Commit A) |
|
||||
|
||||
## The supremacy rule for cross-layer facts
|
||||
|
||||
When the same fact has copies in multiple representations and they
|
||||
disagree, the trust hierarchy applies in this order:
|
||||
|
||||
1. **AtoCore project_state** (Layer 3) is highest authority for any
|
||||
"current state of the project" question. This is why it requires
|
||||
manual curation and never gets touched by automatic processes.
|
||||
2. **The tool-of-origin canonical home** is highest authority for
|
||||
facts that are tool-managed: KB-CAD wins over AtoCore entities
|
||||
for CAD-side structure facts; KB-FEM wins for FEM result facts.
|
||||
3. **AtoCore entities** are highest authority for facts that are
|
||||
AtoCore-managed: Decisions, Requirements, Constraints,
|
||||
ValidationClaims (when the supporting Results are still loose).
|
||||
4. **Active AtoCore memories** are highest authority for personal
|
||||
facts (identity, preference, episodic).
|
||||
5. **Source chunks (PKM, repos, ingested docs)** are lowest
|
||||
authority — they are the raw substrate from which higher layers
|
||||
are extracted, but they may be stale, contradictory among
|
||||
themselves, or out of date.
|
||||
|
||||
This is the same hierarchy enforced by `conflict-model.md`. This
|
||||
document just makes it explicit per fact kind.
|
||||
|
||||
## Examples
|
||||
|
||||
### Example 1 — "what material does the lateral support pad use?"
|
||||
|
||||
Possible representations:
|
||||
|
||||
- KB-CAD has the field `component.lateral-support-pad.material = "GF-PTFE"`
|
||||
- A PKM note from last month says "considering PEEK for the
|
||||
lateral support, GF-PTFE was the previous choice"
|
||||
- An AtoCore Material entity says `GF-PTFE`
|
||||
- An AtoCore project_state entry says `p05 / decision /
|
||||
lateral_support_material = GF-PTFE`
|
||||
|
||||
Which one wins for the question "what's the current material"?
|
||||
|
||||
- **project_state wins** if the query is "what is the current
|
||||
trusted answer for p05's lateral support material" (Layer 3)
|
||||
- **KB-CAD wins** if project_state has not been curated for this
|
||||
field yet, because KB-CAD is the canonical home for CAD-side
|
||||
structure
|
||||
- **The Material entity** is a derived view from KB-CAD; if it
|
||||
disagrees with KB-CAD, the entity is wrong and a conflict is
|
||||
surfaced
|
||||
- **The PKM note** is historical context, not authoritative for
|
||||
"current"
|
||||
|
||||
### Example 2 — "did we decide to merge the bind mounts?"
|
||||
|
||||
Possible representations:
|
||||
|
||||
- A working session interaction is captured in the `interactions`
|
||||
table with the response containing `## Decision: merge the two
|
||||
bind mounts into one`
|
||||
- The Phase 9 Commit C extractor produced a candidate adaptation
|
||||
memory from that decision
|
||||
- A reviewer promoted the candidate to active
|
||||
- The AtoCore source repo has the actual code change in commit
|
||||
`d0ff8b5` and the docker-compose.yml is in its post-merge form
|
||||
|
||||
Which one wins for "is this decision real and current"?
|
||||
|
||||
- **The Gitea repo** wins for "is this decision implemented" —
|
||||
the docker-compose.yml is the canonical home for the actual
|
||||
bind mount configuration
|
||||
- **The active adaptation memory** wins for "did we decide this"
|
||||
— that's exactly what the Commit C lifecycle is for
|
||||
- **The interaction record** is the audit trail — it's
|
||||
authoritative for "when did this conversation happen and what
|
||||
did the LLM say", but not for "is this decision current"
|
||||
- **The source chunks** from PKM are not relevant here because no
|
||||
PKM note about this decision exists yet (and that's fine —
|
||||
decisions don't have to live in PKM if they live in the repo
|
||||
and the AtoCore memory)
|
||||
|
||||
### Example 3 — "what's p05's current next focus?"
|
||||
|
||||
Possible representations:
|
||||
|
||||
- The PKM has a `current-status.md` note updated last week
|
||||
- AtoCore project_state has `p05 / status / next_focus = "wave 2 ingestion"`
|
||||
- A captured interaction from yesterday discussed the next focus
|
||||
at length
|
||||
|
||||
Which one wins?
|
||||
|
||||
- **project_state wins**, full stop. The trust hierarchy says
|
||||
Layer 3 is canonical for current state. This is exactly the
|
||||
reason project_state exists.
|
||||
- The PKM note is historical context.
|
||||
- The interaction is conversation history.
|
||||
- If project_state and the PKM disagree, the human updates one or
|
||||
the other to bring them in line — usually by re-curating
|
||||
project_state if the conversation revealed a real change.
|
||||
|
||||
## What this means for the engineering layer V1 implementation
|
||||
|
||||
Several concrete consequences fall out of the matrix:
|
||||
|
||||
1. **The Material and Parameter entity types are mostly KB-CAD
|
||||
shadows in V1.** They exist in AtoCore so other entities
|
||||
(Decisions, Requirements) can reference them with structured
|
||||
links, but their authoritative values come from KB-CAD imports.
|
||||
If KB-CAD doesn't know about a material, the AtoCore entity is
|
||||
the canonical home only because nothing else is.
|
||||
2. **Decisions / Requirements / Constraints / ValidationClaims
|
||||
are AtoCore-canonical.** These don't have a natural home in
|
||||
KB-CAD or KB-FEM. They live in AtoCore as first-class entities
|
||||
with full lifecycle and supersession.
|
||||
3. **The PKM is never authoritative.** It is the substrate for
|
||||
extraction. The reviewer promotes things out of it; they don't
|
||||
point at PKM notes as the "current truth".
|
||||
4. **project_state is the override layer.** Whenever the human
|
||||
wants to declare "the current truth is X regardless of what
|
||||
the entities and memories and KB exports say", they curate
|
||||
into project_state. Layer 3 is intentionally small and
|
||||
intentionally manual.
|
||||
5. **The conflict model is the enforcement mechanism.** When two
|
||||
representations disagree on a fact whose canonical home rule
|
||||
should pick a winner, the conflict surfaces via the
|
||||
`/conflicts` endpoint and the reviewer resolves it. The
|
||||
matrix in this document tells the reviewer who is supposed
|
||||
to win in each scenario; they're not making the decision blind.
|
||||
|
||||
## What the matrix does NOT define
|
||||
|
||||
1. **Facts about people other than the user.** No "team member"
|
||||
entity, no per-collaborator preferences. AtoCore is
|
||||
single-user in V1.
|
||||
2. **Facts about AtoCore itself as a project.** Those are project
|
||||
memories and project_state entries under `project=atocore`,
|
||||
same lifecycle as any other project's facts.
|
||||
3. **Vendor / supplier / cost facts.** Out of V1 scope.
|
||||
4. **Time-bounded facts** (a value that was true between two
|
||||
dates and may not be true now). The current matrix treats all
|
||||
active facts as currently-true and uses supersession to
|
||||
represent change. Temporal facts are a V2 concern.
|
||||
5. **Cross-project shared facts** (a Material that is reused across
|
||||
p04, p05, and p06). Currently each project has its own copy.
|
||||
Cross-project deduplication is also a V2 concern.
|
||||
|
||||
## The "single canonical home" invariant in practice
|
||||
|
||||
The hard rule that every fact has exactly one canonical home is
|
||||
the load-bearing invariant of this matrix. To enforce it
|
||||
operationally:
|
||||
|
||||
- **Extraction never duplicates.** When the extractor scans an
|
||||
interaction or a source chunk and proposes a candidate, the
|
||||
candidate is dropped if it duplicates an already-active record
|
||||
in the canonical home (the existing extractor implementation
|
||||
already does this for memories; the entity extractor will
|
||||
follow the same pattern).
|
||||
- **Imports never duplicate.** When KB-CAD pushes the same
|
||||
Component twice with the same value, the second push is
|
||||
recognized as identical and updates the `last_imported_at`
|
||||
timestamp without creating a new entity.
|
||||
- **Imports surface drift as conflict.** When KB-CAD pushes the
|
||||
same Component with a different value, that's a conflict per
|
||||
the conflict model — never a silent overwrite.
|
||||
- **Hand-curation into project_state always wins.** A
|
||||
project_state entry can disagree with an entity or a KB
|
||||
export; the project_state entry is correct by fiat (Layer 3
|
||||
trust), and the reviewer is responsible for bringing the lower
|
||||
layers in line if appropriate.
|
||||
|
||||
## Open questions for V1 implementation
|
||||
|
||||
1. **How does the reviewer see the canonical home for a fact in
|
||||
the UI?** Probably by including the fact's authoritative
|
||||
layer in the entity / memory detail view: "this Material is
|
||||
currently mirrored from KB-CAD; the canonical home is KB-CAD".
|
||||
2. **Who owns running the KB-CAD / KB-FEM exporter?** The
|
||||
`tool-handoff-boundaries.md` doc lists this as an open
|
||||
question; same answer applies here.
|
||||
3. **Do we need an explicit `canonical_home` field on entity
|
||||
rows?** A field that records "this entity is canonical here"
|
||||
vs "this entity is a mirror of <external system>". Probably
|
||||
yes; deferred to the entity schema spec.
|
||||
4. **How are project_state overrides surfaced in the engineering
|
||||
layer query results?** When a query (e.g. Q-001 "what does
|
||||
this subsystem contain?") would return entity rows, the result
|
||||
should also flag any project_state entries that contradict the
|
||||
entities — letting the reviewer see the override at query
|
||||
time, not just in the conflict queue.
|
||||
|
||||
## TL;DR
|
||||
|
||||
- Six representation layers: PKM, KB project, repos, AtoCore
|
||||
memories, AtoCore entities, AtoCore project_state
|
||||
- Every fact kind has exactly one canonical home
|
||||
- The trust hierarchy resolves cross-layer conflicts:
|
||||
project_state > tool-of-origin (KB-CAD/KB-FEM) > entities >
|
||||
active memories > source chunks
|
||||
- Decisions / Requirements / Constraints / ValidationClaims are
|
||||
AtoCore-canonical (no other system has a natural home for them)
|
||||
- Materials / Parameters / CAD-side structure are KB-CAD-canonical
|
||||
- FEM results / validation outcomes are KB-FEM-canonical
|
||||
- project_state is the human override layer, top of the
|
||||
hierarchy, manually curated only
|
||||
- Conflicts surface via `/conflicts` and the reviewer applies the
|
||||
matrix to pick a winner
|
||||
339
docs/architecture/tool-handoff-boundaries.md
Normal file
339
docs/architecture/tool-handoff-boundaries.md
Normal file
@@ -0,0 +1,339 @@
|
||||
# Tool Hand-off Boundaries (KB-CAD / KB-FEM and friends)
|
||||
|
||||
## Why this document exists
|
||||
|
||||
The engineering layer V1 will accumulate typed entities about
|
||||
projects, subsystems, components, materials, requirements,
|
||||
constraints, decisions, parameters, analysis models, results, and
|
||||
validation claims. Many of those concepts also live in real
|
||||
external tools — CAD systems, FEM solvers, BOM managers, PLM
|
||||
databases, vendor portals.
|
||||
|
||||
The first big design decision before writing any entity-layer code
|
||||
is: **what is AtoCore's read/write relationship with each of those
|
||||
external tools?**
|
||||
|
||||
The wrong answer in either direction is expensive:
|
||||
|
||||
- Too read-only: AtoCore becomes a stale shadow of the tools and
|
||||
loses the trust battle the moment a value drifts.
|
||||
- Too bidirectional: AtoCore takes on responsibilities it can't
|
||||
reliably honor (live sync, conflict resolution against external
|
||||
schemas, write-back validation), and the project never ships.
|
||||
|
||||
This document picks a position for V1.
|
||||
|
||||
## The position
|
||||
|
||||
> **AtoCore is a one-way mirror in V1.** External tools push
|
||||
> structured exports into AtoCore. AtoCore never pushes back.
|
||||
|
||||
That position has three corollaries:
|
||||
|
||||
1. **External tools remain the source of truth for everything they
|
||||
already manage.** A CAD model is canonical for geometry; a FEM
|
||||
project is canonical for meshes and solver settings; KB-CAD is
|
||||
canonical for whatever KB-CAD already calls canonical.
|
||||
2. **AtoCore is the source of truth for the *AtoCore-shaped*
|
||||
record** of those facts: the Decision that selected the geometry,
|
||||
the Requirement the geometry satisfies, the ValidationClaim the
|
||||
FEM result supports. AtoCore does not duplicate the external
|
||||
tool's primary representation; it stores the structured *facts
|
||||
about* it.
|
||||
3. **The boundary is enforced by absence.** No write endpoint in
|
||||
AtoCore ever generates a `.prt`, a `.fem`, an export to a PLM
|
||||
schema, or a vendor purchase order. If we find ourselves wanting
|
||||
to add such an endpoint in V1, we should stop and reconsider
|
||||
the V1 scope.
|
||||
|
||||
## Why one-way and not bidirectional
|
||||
|
||||
Bidirectional sync between independent systems is one of the
|
||||
hardest problems in engineering software. The honest reasons we
|
||||
are not attempting it in V1:
|
||||
|
||||
1. **Schema drift.** External tools evolve their schemas
|
||||
independently. A bidirectional sync would have to track every
|
||||
schema version of every external tool we touch. That is a
|
||||
permanent maintenance tax.
|
||||
2. **Conflict semantics.** When AtoCore and an external tool
|
||||
disagree on the same field, "who wins" is a per-tool, per-field
|
||||
decision. There is no general rule. Bidirectional sync would
|
||||
require us to specify that decision exhaustively.
|
||||
3. **Trust hierarchy.** AtoCore's whole point is the trust
|
||||
hierarchy: trusted project state > entities > memories. If we
|
||||
let entities push values back into the external tools, we
|
||||
silently elevate AtoCore's confidence to "high enough to write
|
||||
to a CAD model", which it almost never deserves.
|
||||
4. **Velocity.** A bidirectional engineering layer is a
|
||||
multi-year project. A one-way mirror is a months project. The
|
||||
value-to-effort ratio favors one-way for V1 by an enormous
|
||||
margin.
|
||||
5. **Reversibility.** We can always add bidirectional sync later
|
||||
on a per-tool basis once V1 has shown itself to be useful. We
|
||||
cannot easily walk back a half-finished bidirectional sync that
|
||||
has already corrupted data in someone's CAD model.
|
||||
|
||||
## Per-tool stance for V1
|
||||
|
||||
| External tool | V1 stance | What AtoCore reads in | What AtoCore writes back |
|
||||
|---|---|---|---|
|
||||
| **KB-CAD** (Antoine's CAD knowledge base) | one-way mirror | structured exports of subsystems, components, materials, parameters via a documented JSON or CSV shape | nothing |
|
||||
| **KB-FEM** (Antoine's FEM knowledge base) | one-way mirror | structured exports of analysis models, results, validation claims | nothing |
|
||||
| **NX / Siemens NX** (the CAD tool itself) | not connected in V1 | nothing direct — only what KB-CAD exports about NX projects | nothing |
|
||||
| **PKM (Obsidian / markdown vault)** | already connected via the ingestion pipeline (Phase 1) | full markdown/text corpus per the ingestion-waves doc | nothing |
|
||||
| **Gitea repos** | already connected via the ingestion pipeline | repo markdown/text per project | nothing |
|
||||
| **OpenClaw** (the LLM agent) | already connected via the read-only helper skill on the T420 | nothing — OpenClaw reads from AtoCore | nothing — OpenClaw does not write into AtoCore |
|
||||
| **AtoDrive** (operational truth layer, future) | future: bidirectional with AtoDrive itself, but AtoDrive is internal to AtoCore so this isn't an external tool boundary | n/a in V1 | n/a in V1 |
|
||||
| **PLM / vendor portals / cost systems** | not in V1 scope | nothing | nothing |
|
||||
|
||||
## What "one-way mirror" actually looks like in code
|
||||
|
||||
AtoCore exposes an ingestion endpoint per external tool that
|
||||
accepts a structured export and turns it into entity candidates.
|
||||
The endpoint is read-side from AtoCore's perspective (it reads
|
||||
from a file or HTTP body), even though the external tool is the
|
||||
one initiating the call.
|
||||
|
||||
Proposed V1 ingestion endpoints:
|
||||
|
||||
```
|
||||
POST /ingest/kb-cad/export body: KB-CAD export JSON
|
||||
POST /ingest/kb-fem/export body: KB-FEM export JSON
|
||||
```
|
||||
|
||||
Each endpoint:
|
||||
|
||||
1. Validates the export against the documented schema
|
||||
2. Maps each export record to an entity candidate (status="candidate")
|
||||
3. Carries the export's source identifier into the candidate's
|
||||
provenance fields (source_artifact_id, exporter_version, etc.)
|
||||
4. Returns a summary: how many candidates were created, how many
|
||||
were dropped as duplicates, how many failed schema validation
|
||||
5. Does NOT auto-promote anything
|
||||
|
||||
The KB-CAD and KB-FEM teams (which is to say, future-you) own the
|
||||
exporter scripts that produce these JSON bodies. Those scripts
|
||||
live in the KB-CAD / KB-FEM repos respectively, not in AtoCore.
|
||||
|
||||
## The export schemas (sketch, not final)
|
||||
|
||||
These are starting shapes, intentionally minimal. The schemas
|
||||
will be refined in `kb-cad-export-schema.md` and
|
||||
`kb-fem-export-schema.md` once the V1 ontology lands.
|
||||
|
||||
### KB-CAD export shape (starting sketch)
|
||||
|
||||
```json
|
||||
{
|
||||
"exporter": "kb-cad",
|
||||
"exporter_version": "1.0.0",
|
||||
"exported_at": "2026-04-07T12:00:00Z",
|
||||
"project": "p05-interferometer",
|
||||
"subsystems": [
|
||||
{
|
||||
"id": "subsystem.optical-frame",
|
||||
"name": "Optical frame",
|
||||
"parent": null,
|
||||
"components": [
|
||||
{
|
||||
"id": "component.lateral-support-pad",
|
||||
"name": "Lateral support pad",
|
||||
"material": "GF-PTFE",
|
||||
"parameters": {
|
||||
"thickness_mm": 3.0,
|
||||
"preload_n": 12.0
|
||||
},
|
||||
"source_artifact": "kb-cad://p05/subsystems/optical-frame#lateral-support"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### KB-FEM export shape (starting sketch)
|
||||
|
||||
```json
|
||||
{
|
||||
"exporter": "kb-fem",
|
||||
"exporter_version": "1.0.0",
|
||||
"exported_at": "2026-04-07T12:00:00Z",
|
||||
"project": "p05-interferometer",
|
||||
"analysis_models": [
|
||||
{
|
||||
"id": "model.optical-frame-modal",
|
||||
"name": "Optical frame modal analysis v3",
|
||||
"subsystem": "subsystem.optical-frame",
|
||||
"results": [
|
||||
{
|
||||
"id": "result.first-mode-frequency",
|
||||
"name": "First-mode frequency",
|
||||
"value": 187.4,
|
||||
"unit": "Hz",
|
||||
"supports_validation_claim": "claim.frame-rigidity-min-150hz",
|
||||
"source_artifact": "kb-fem://p05/models/optical-frame-modal#first-mode"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
These shapes will evolve. The point of including them now is to
|
||||
make the one-way mirror concrete: it is a small, well-defined
|
||||
JSON shape, not "AtoCore reaches into KB-CAD's database".
|
||||
|
||||
## What AtoCore is allowed to do with the imported records
|
||||
|
||||
After ingestion, the imported records become entity candidates
|
||||
in AtoCore's own table. From that point forward they follow the
|
||||
exact same lifecycle as any other candidate:
|
||||
|
||||
- they sit at status="candidate" until a human reviews them
|
||||
- the reviewer promotes them to status="active" or rejects them
|
||||
- the active entities are queryable via the engineering query
|
||||
catalog (Q-001 through Q-020)
|
||||
- the active entities can be referenced from Decisions, Requirements,
|
||||
ValidationClaims, etc. via the V1 relationship types
|
||||
|
||||
The imported records are never automatically pushed into trusted
|
||||
project state, never modified in place after import (they are
|
||||
superseded by re-imports, not edited), and never written back to
|
||||
the external tool.
|
||||
|
||||
## What happens when KB-CAD changes a value AtoCore already has
|
||||
|
||||
This is the canonical "drift" scenario. The flow:
|
||||
|
||||
1. KB-CAD exports a fresh JSON. Component `component.lateral-support-pad`
|
||||
now has `material: "PEEK"` instead of `material: "GF-PTFE"`.
|
||||
2. AtoCore's ingestion endpoint sees the same `id` and a different
|
||||
value.
|
||||
3. The ingestion endpoint creates a new entity candidate with the
|
||||
new value, **does NOT delete or modify the existing active
|
||||
entity**, and creates a `conflicts` row linking the two members
|
||||
(per the conflict model doc).
|
||||
4. The reviewer sees an open conflict on the next visit to
|
||||
`/conflicts`.
|
||||
5. The reviewer either:
|
||||
- **promotes the new value** (the active is superseded, the
|
||||
candidate becomes the new active, the audit trail keeps both)
|
||||
- **rejects the new value** (the candidate is invalidated, the
|
||||
active stays — useful when the export was wrong)
|
||||
- **dismisses the conflict** (declares them not actually about
|
||||
the same thing, both stay active)
|
||||
|
||||
The reviewer never touches KB-CAD from AtoCore. If the resolution
|
||||
implies a change in KB-CAD itself, the reviewer makes that change
|
||||
in KB-CAD, then re-exports.
|
||||
|
||||
## What about NX directly?
|
||||
|
||||
NX (Siemens NX) is the underlying CAD tool that KB-CAD wraps.
|
||||
**NX is not connected to AtoCore in V1.** Any facts about NX
|
||||
projects flow through KB-CAD as the structured intermediate. This
|
||||
gives us:
|
||||
|
||||
- **One schema to maintain.** AtoCore only has to understand the
|
||||
KB-CAD export shape, not the NX API.
|
||||
- **One ownership boundary.** KB-CAD owns the question of "what's
|
||||
in NX". AtoCore owns the question of "what's in the typed
|
||||
knowledge base".
|
||||
- **Future flexibility.** When NX is replaced or upgraded, only
|
||||
KB-CAD has to adapt; AtoCore doesn't notice.
|
||||
|
||||
The same logic applies to FEM solvers (Nastran, Abaqus, ANSYS):
|
||||
KB-FEM is the structured intermediate, AtoCore never talks to the
|
||||
solver directly.
|
||||
|
||||
## The hard-line invariants
|
||||
|
||||
These are the things V1 will not do, regardless of how convenient
|
||||
they might seem:
|
||||
|
||||
1. **No write to external tools.** No POST/PUT/PATCH to any
|
||||
external API, no file generation that gets written into a
|
||||
CAD/FEM project tree, no email/chat sends.
|
||||
2. **No live polling.** AtoCore does not poll KB-CAD or KB-FEM on
|
||||
a schedule. Imports are explicit pushes from the external tool
|
||||
into AtoCore's ingestion endpoint.
|
||||
3. **No silent merging.** Every value drift surfaces as a
|
||||
conflict for the reviewer (per the conflict model doc).
|
||||
4. **No schema fan-out.** AtoCore does not store every field that
|
||||
KB-CAD knows about. Only fields that map to one of the V1
|
||||
entity types make it into AtoCore. Everything else is dropped
|
||||
at the import boundary.
|
||||
5. **No external-tool-specific logic in entity types.** A
|
||||
`Component` in AtoCore is the same shape regardless of whether
|
||||
it came from KB-CAD, KB-FEM, the PKM, or a hand-curated
|
||||
project state entry. The source is recorded in provenance,
|
||||
not in the entity shape.
|
||||
|
||||
## What this enables
|
||||
|
||||
With the one-way mirror locked in, V1 implementation can focus on:
|
||||
|
||||
- The entity table and its lifecycle
|
||||
- The two `/ingest/kb-cad/export` and `/ingest/kb-fem/export`
|
||||
endpoints with their JSON validators
|
||||
- The candidate review queue extension (already designed in
|
||||
`promotion-rules.md`)
|
||||
- The conflict model (already designed in `conflict-model.md`)
|
||||
- The query catalog implementation (already designed in
|
||||
`engineering-query-catalog.md`)
|
||||
|
||||
None of those are unbounded. Each is a finite, well-defined
|
||||
implementation task. The one-way mirror is the choice that makes
|
||||
V1 finishable.
|
||||
|
||||
## What V2 might consider (deferred)
|
||||
|
||||
After V1 has been live and demonstrably useful for a quarter or
|
||||
two, the questions that become reasonable to revisit:
|
||||
|
||||
1. **Selective write-back to KB-CAD for low-risk fields.** For
|
||||
example, AtoCore could push back a "Decision id linked to this
|
||||
component" annotation that KB-CAD then displays without it
|
||||
being canonical there. Read-only annotations from AtoCore's
|
||||
perspective, advisory metadata from KB-CAD's perspective.
|
||||
2. **Live polling for very small payloads.** A daily poll of
|
||||
"what subsystem ids exist in KB-CAD now" so AtoCore can flag
|
||||
subsystems that disappeared from KB-CAD without an explicit
|
||||
AtoCore invalidation.
|
||||
3. **Direct NX integration** if the KB-CAD layer becomes a
|
||||
bottleneck — but only if the friction is real, not theoretical.
|
||||
4. **Cost / vendor / PLM connections** for projects where the
|
||||
procurement cycle is part of the active engineering work.
|
||||
|
||||
None of these are V1 work and they are listed only so the V1
|
||||
design intentionally leaves room for them later.
|
||||
|
||||
## Open questions for the V1 implementation sprint
|
||||
|
||||
1. **Where do the export schemas live?** Probably in
|
||||
`docs/architecture/kb-cad-export-schema.md` and
|
||||
`docs/architecture/kb-fem-export-schema.md`, drafted during
|
||||
the implementation sprint.
|
||||
2. **Who runs the exporter?** A scheduled job on the KB-CAD /
|
||||
KB-FEM hosts, triggered by the human after a meaningful
|
||||
change, or both?
|
||||
3. **Is the export incremental or full?** Full is simpler but
|
||||
more expensive. Incremental needs delta semantics. V1 starts
|
||||
with full and revisits when full becomes too slow.
|
||||
4. **How is the exporter authenticated to AtoCore?** Probably
|
||||
the existing PAT model (one PAT per exporter, scoped to
|
||||
`write:engineering-import` once that scope exists). Worth a
|
||||
quick auth design pass before the endpoints exist.
|
||||
|
||||
## TL;DR
|
||||
|
||||
- AtoCore is a one-way mirror in V1: external tools push,
|
||||
AtoCore reads, AtoCore never writes back
|
||||
- Two import endpoints for V1: KB-CAD and KB-FEM, each with a
|
||||
documented JSON export shape
|
||||
- Drift surfaces as conflicts in the existing conflict model
|
||||
- No NX, no FEM solvers, no PLM, no vendor portals, no
|
||||
cost/BOM systems in V1
|
||||
- Bidirectional sync is reserved for V2+ on a per-tool basis,
|
||||
only after V1 demonstrates value
|
||||
360
docs/backup-restore-procedure.md
Normal file
360
docs/backup-restore-procedure.md
Normal file
@@ -0,0 +1,360 @@
|
||||
# AtoCore Backup and Restore Procedure
|
||||
|
||||
## Scope
|
||||
|
||||
This document defines the operational procedure for backing up and
|
||||
restoring AtoCore's machine state on the Dalidou deployment. It is
|
||||
the practical companion to `docs/backup-strategy.md` (which defines
|
||||
the strategy) and `src/atocore/ops/backup.py` (which implements the
|
||||
mechanics).
|
||||
|
||||
The intent is that this procedure can be followed by anyone with
|
||||
SSH access to Dalidou and the AtoCore admin endpoints.
|
||||
|
||||
## What gets backed up
|
||||
|
||||
A `create_runtime_backup` snapshot contains, in order of importance:
|
||||
|
||||
| Artifact | Source path on Dalidou | Backup destination | Always included |
|
||||
|---|---|---|---|
|
||||
| SQLite database | `/srv/storage/atocore/data/db/atocore.db` | `<backup_root>/db/atocore.db` | yes |
|
||||
| Project registry JSON | `/srv/storage/atocore/config/project-registry.json` | `<backup_root>/config/project-registry.json` | yes (if file exists) |
|
||||
| Backup metadata | (generated) | `<backup_root>/backup-metadata.json` | yes |
|
||||
| Chroma vector store | `/srv/storage/atocore/data/chroma/` | `<backup_root>/chroma/` | only when `include_chroma=true` |
|
||||
|
||||
The SQLite snapshot uses the online `conn.backup()` API and is safe
|
||||
to take while the database is in use. The Chroma snapshot is a cold
|
||||
directory copy and is **only safe when no ingestion is running**;
|
||||
the API endpoint enforces this by acquiring the ingestion lock for
|
||||
the duration of the copy.
|
||||
|
||||
What is **not** in the backup:
|
||||
|
||||
- Source documents under `/srv/storage/atocore/sources/vault/` and
|
||||
`/srv/storage/atocore/sources/drive/`. These are read-only
|
||||
inputs and live in the user's PKM/Drive, which is backed up
|
||||
separately by their own systems.
|
||||
- Application code. The container image is the source of truth for
|
||||
code; recovery means rebuilding the image, not restoring code from
|
||||
a backup.
|
||||
- Logs under `/srv/storage/atocore/logs/`.
|
||||
- Embeddings cache under `/srv/storage/atocore/data/cache/`.
|
||||
- Temp files under `/srv/storage/atocore/data/tmp/`.
|
||||
|
||||
## Backup root layout
|
||||
|
||||
Each backup snapshot lives in its own timestamped directory:
|
||||
|
||||
```
|
||||
/srv/storage/atocore/backups/snapshots/
|
||||
├── 20260407T060000Z/
|
||||
│ ├── backup-metadata.json
|
||||
│ ├── db/
|
||||
│ │ └── atocore.db
|
||||
│ ├── config/
|
||||
│ │ └── project-registry.json
|
||||
│ └── chroma/ # only if include_chroma=true
|
||||
│ └── ...
|
||||
├── 20260408T060000Z/
|
||||
│ └── ...
|
||||
└── ...
|
||||
```
|
||||
|
||||
The timestamp is UTC, format `YYYYMMDDTHHMMSSZ`.
|
||||
|
||||
## Triggering a backup
|
||||
|
||||
### Option A — via the admin endpoint (preferred)
|
||||
|
||||
```bash
|
||||
# DB + registry only (fast, safe at any time)
|
||||
curl -fsS -X POST http://dalidou:8100/admin/backup \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"include_chroma": false}'
|
||||
|
||||
# DB + registry + Chroma (acquires ingestion lock)
|
||||
curl -fsS -X POST http://dalidou:8100/admin/backup \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"include_chroma": true}'
|
||||
```
|
||||
|
||||
The response is the backup metadata JSON. Save the `backup_root`
|
||||
field — that's the directory the snapshot was written to.
|
||||
|
||||
### Option B — via the standalone script (when the API is down)
|
||||
|
||||
```bash
|
||||
docker exec atocore python -m atocore.ops.backup
|
||||
```
|
||||
|
||||
This runs `create_runtime_backup()` directly, without going through
|
||||
the API or the ingestion lock. Use it only when the AtoCore service
|
||||
itself is unhealthy and you can't hit the admin endpoint.
|
||||
|
||||
### Option C — manual file copy (last resort)
|
||||
|
||||
If both the API and the standalone script are unusable:
|
||||
|
||||
```bash
|
||||
sudo systemctl stop atocore # or: docker compose stop atocore
|
||||
sudo cp /srv/storage/atocore/data/db/atocore.db \
|
||||
/srv/storage/atocore/backups/manual-$(date -u +%Y%m%dT%H%M%SZ).db
|
||||
sudo cp /srv/storage/atocore/config/project-registry.json \
|
||||
/srv/storage/atocore/backups/manual-$(date -u +%Y%m%dT%H%M%SZ).registry.json
|
||||
sudo systemctl start atocore
|
||||
```
|
||||
|
||||
This is a cold backup and requires brief downtime.
|
||||
|
||||
## Listing backups
|
||||
|
||||
```bash
|
||||
curl -fsS http://dalidou:8100/admin/backup
|
||||
```
|
||||
|
||||
Returns the configured `backup_dir` and a list of all snapshots
|
||||
under it, with their full metadata if available.
|
||||
|
||||
Or, on the host directly:
|
||||
|
||||
```bash
|
||||
ls -la /srv/storage/atocore/backups/snapshots/
|
||||
```
|
||||
|
||||
## Validating a backup
|
||||
|
||||
Before relying on a backup for restore, validate it:
|
||||
|
||||
```bash
|
||||
curl -fsS http://dalidou:8100/admin/backup/20260407T060000Z/validate
|
||||
```
|
||||
|
||||
The validator:
|
||||
- confirms the snapshot directory exists
|
||||
- opens the SQLite snapshot and runs `PRAGMA integrity_check`
|
||||
- parses the registry JSON
|
||||
- confirms the Chroma directory exists (if it was included)
|
||||
|
||||
A valid backup returns `"valid": true` and an empty `errors` array.
|
||||
A failing validation returns `"valid": false` with one or more
|
||||
specific error strings (e.g. `db_integrity_check_failed`,
|
||||
`registry_invalid_json`, `chroma_snapshot_missing`).
|
||||
|
||||
**Validate every backup at creation time.** A backup that has never
|
||||
been validated is not actually a backup — it's just a hopeful copy
|
||||
of bytes.
|
||||
|
||||
## Restore procedure
|
||||
|
||||
### Pre-flight (always)
|
||||
|
||||
1. Identify which snapshot you want to restore. List available
|
||||
snapshots and pick by timestamp:
|
||||
```bash
|
||||
curl -fsS http://dalidou:8100/admin/backup | jq '.backups[].stamp'
|
||||
```
|
||||
2. Validate it. Refuse to restore an invalid backup:
|
||||
```bash
|
||||
STAMP=20260407T060000Z
|
||||
curl -fsS http://dalidou:8100/admin/backup/$STAMP/validate | jq .
|
||||
```
|
||||
3. **Stop AtoCore.** SQLite cannot be hot-restored under a running
|
||||
process and Chroma will not pick up new files until the process
|
||||
restarts.
|
||||
```bash
|
||||
docker compose stop atocore
|
||||
# or: sudo systemctl stop atocore
|
||||
```
|
||||
4. **Take a safety snapshot of the current state** before overwriting
|
||||
it. This is your "if the restore makes things worse, here's the
|
||||
undo" backup.
|
||||
```bash
|
||||
PRESERVE_STAMP=$(date -u +%Y%m%dT%H%M%SZ)
|
||||
sudo cp /srv/storage/atocore/data/db/atocore.db \
|
||||
/srv/storage/atocore/backups/pre-restore-$PRESERVE_STAMP.db
|
||||
sudo cp /srv/storage/atocore/config/project-registry.json \
|
||||
/srv/storage/atocore/backups/pre-restore-$PRESERVE_STAMP.registry.json 2>/dev/null || true
|
||||
```
|
||||
|
||||
### Restore the SQLite database
|
||||
|
||||
```bash
|
||||
SNAPSHOT_DIR=/srv/storage/atocore/backups/snapshots/$STAMP
|
||||
sudo cp $SNAPSHOT_DIR/db/atocore.db \
|
||||
/srv/storage/atocore/data/db/atocore.db
|
||||
sudo chown 1000:1000 /srv/storage/atocore/data/db/atocore.db
|
||||
sudo chmod 600 /srv/storage/atocore/data/db/atocore.db
|
||||
```
|
||||
|
||||
The chown should match the gitea/atocore container user. Verify
|
||||
by checking the existing perms before overwriting:
|
||||
|
||||
```bash
|
||||
stat -c '%U:%G %a' /srv/storage/atocore/data/db/atocore.db
|
||||
```
|
||||
|
||||
### Restore the project registry
|
||||
|
||||
```bash
|
||||
if [ -f $SNAPSHOT_DIR/config/project-registry.json ]; then
|
||||
sudo cp $SNAPSHOT_DIR/config/project-registry.json \
|
||||
/srv/storage/atocore/config/project-registry.json
|
||||
sudo chown 1000:1000 /srv/storage/atocore/config/project-registry.json
|
||||
sudo chmod 644 /srv/storage/atocore/config/project-registry.json
|
||||
fi
|
||||
```
|
||||
|
||||
If the snapshot does not contain a registry, the current registry is
|
||||
preserved. The pre-flight safety copy still gives you a recovery path
|
||||
if you need to roll back.
|
||||
|
||||
### Restore the Chroma vector store (if it was in the snapshot)
|
||||
|
||||
```bash
|
||||
if [ -d $SNAPSHOT_DIR/chroma ]; then
|
||||
# Move the current chroma dir aside as a safety copy
|
||||
sudo mv /srv/storage/atocore/data/chroma \
|
||||
/srv/storage/atocore/data/chroma.pre-restore-$PRESERVE_STAMP
|
||||
|
||||
# Copy the snapshot in
|
||||
sudo cp -a $SNAPSHOT_DIR/chroma /srv/storage/atocore/data/chroma
|
||||
sudo chown -R 1000:1000 /srv/storage/atocore/data/chroma
|
||||
fi
|
||||
```
|
||||
|
||||
If the snapshot does NOT contain a Chroma dir but the SQLite
|
||||
restore would leave the vector store and the SQL store inconsistent
|
||||
(e.g. SQL has chunks the vector store doesn't), you have two
|
||||
options:
|
||||
|
||||
- **Option 1: rebuild the vector store from source documents.** Run
|
||||
ingestion fresh after the SQL restore. This regenerates embeddings
|
||||
from the actual source files. Slow but produces a perfectly
|
||||
consistent state.
|
||||
- **Option 2: accept the inconsistency and live with stale-vector
|
||||
filtering.** The retriever already drops vector results whose
|
||||
SQL row no longer exists (`_existing_chunk_ids` filter), so the
|
||||
inconsistency surfaces as missing results, not bad ones.
|
||||
|
||||
For an unplanned restore, Option 2 is the right immediate move.
|
||||
Then schedule a fresh ingestion pass to rebuild the vector store
|
||||
properly.
|
||||
|
||||
### Restart AtoCore
|
||||
|
||||
```bash
|
||||
docker compose up -d atocore
|
||||
# or: sudo systemctl start atocore
|
||||
```
|
||||
|
||||
### Post-restore verification
|
||||
|
||||
```bash
|
||||
# 1. Service is healthy
|
||||
curl -fsS http://dalidou:8100/health | jq .
|
||||
|
||||
# 2. Stats look right
|
||||
curl -fsS http://dalidou:8100/stats | jq .
|
||||
|
||||
# 3. Project registry loads
|
||||
curl -fsS http://dalidou:8100/projects | jq '.projects | length'
|
||||
|
||||
# 4. A known-good context query returns non-empty results
|
||||
curl -fsS -X POST http://dalidou:8100/context/build \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"prompt": "what is p05 about", "project": "p05-interferometer"}' | jq '.chunks_used'
|
||||
```
|
||||
|
||||
If any of these are wrong, the restore is bad. Roll back using the
|
||||
pre-restore safety copy:
|
||||
|
||||
```bash
|
||||
docker compose stop atocore
|
||||
sudo cp /srv/storage/atocore/backups/pre-restore-$PRESERVE_STAMP.db \
|
||||
/srv/storage/atocore/data/db/atocore.db
|
||||
sudo cp /srv/storage/atocore/backups/pre-restore-$PRESERVE_STAMP.registry.json \
|
||||
/srv/storage/atocore/config/project-registry.json 2>/dev/null || true
|
||||
# If you also restored chroma:
|
||||
sudo rm -rf /srv/storage/atocore/data/chroma
|
||||
sudo mv /srv/storage/atocore/data/chroma.pre-restore-$PRESERVE_STAMP \
|
||||
/srv/storage/atocore/data/chroma
|
||||
docker compose up -d atocore
|
||||
```
|
||||
|
||||
## Retention policy
|
||||
|
||||
- **Last 7 daily backups**: kept verbatim
|
||||
- **Last 4 weekly backups** (Sunday): kept verbatim
|
||||
- **Last 6 monthly backups** (1st of month): kept verbatim
|
||||
- **Anything older**: deleted
|
||||
|
||||
The retention job is **not yet implemented** and is tracked as a
|
||||
follow-up. Until then, the snapshots directory grows monotonically.
|
||||
A simple cron-based cleanup script is the next step:
|
||||
|
||||
```cron
|
||||
0 4 * * * /srv/storage/atocore/scripts/cleanup-old-backups.sh
|
||||
```
|
||||
|
||||
## Drill schedule
|
||||
|
||||
A backup that has never been restored is theoretical. The schedule:
|
||||
|
||||
- **At least once per quarter**, perform a full restore drill on a
|
||||
staging environment (or a temporary container with a separate
|
||||
data dir) and verify the post-restore checks pass.
|
||||
- **After every breaking schema migration**, perform a restore drill
|
||||
to confirm the migration is reversible.
|
||||
- **After any incident** that touched the storage layer (the EXDEV
|
||||
bug from April 2026 is a good example), confirm the next backup
|
||||
validates clean.
|
||||
|
||||
## Common failure modes and what to do about them
|
||||
|
||||
| Symptom | Likely cause | Action |
|
||||
|---|---|---|
|
||||
| `db_integrity_check_failed` on validation | SQLite snapshot copied while a write was in progress, or disk corruption | Take a fresh backup and validate again. If it fails twice, suspect the underlying disk. |
|
||||
| `registry_invalid_json` | Registry was being edited at backup time | Take a fresh backup. The registry is small so this is cheap. |
|
||||
| `chroma_snapshot_missing` after a restore | Snapshot was DB-only and the restore didn't move the existing chroma dir | Either rebuild via fresh ingestion or restore an older snapshot that includes Chroma. |
|
||||
| Service won't start after restore | Permissions wrong on the restored files | Re-run `chown 1000:1000` (or whatever the gitea/atocore container user is) on the data dir. |
|
||||
| `/stats` returns 0 documents after restore | The SQL store was restored but the source paths in `source_documents` don't match the current Dalidou paths | This means the backup came from a different deployment. Don't trust this restore — it's pulling from the wrong layout. |
|
||||
|
||||
## Open follow-ups (not yet implemented)
|
||||
|
||||
1. **Retention cleanup script**: see the cron entry above.
|
||||
2. **Off-Dalidou backup target**: currently snapshots live on the
|
||||
same disk as the live data. A real disaster-recovery story
|
||||
needs at least one snapshot on a different physical machine.
|
||||
The simplest first step is a periodic `rsync` to the user's
|
||||
laptop or to another server.
|
||||
3. **Backup encryption**: snapshots contain raw SQLite and JSON.
|
||||
Consider age/gpg encryption if backups will be shipped off-site.
|
||||
4. **Automatic post-backup validation**: today the validator must
|
||||
be invoked manually. The `create_runtime_backup` function
|
||||
should call `validate_backup` on its own output and refuse to
|
||||
declare success if validation fails.
|
||||
5. **Chroma backup is currently full directory copy** every time.
|
||||
For large vector stores this gets expensive. A future
|
||||
improvement would be incremental snapshots via filesystem-level
|
||||
snapshotting (LVM, btrfs, ZFS).
|
||||
|
||||
## Quickstart cheat sheet
|
||||
|
||||
```bash
|
||||
# Daily backup (DB + registry only — fast)
|
||||
curl -fsS -X POST http://dalidou:8100/admin/backup \
|
||||
-H "Content-Type: application/json" -d '{}'
|
||||
|
||||
# Weekly backup (DB + registry + Chroma — slower, holds ingestion lock)
|
||||
curl -fsS -X POST http://dalidou:8100/admin/backup \
|
||||
-H "Content-Type: application/json" -d '{"include_chroma": true}'
|
||||
|
||||
# List backups
|
||||
curl -fsS http://dalidou:8100/admin/backup | jq '.backups[].stamp'
|
||||
|
||||
# Validate the most recent backup
|
||||
LATEST=$(curl -fsS http://dalidou:8100/admin/backup | jq -r '.backups[-1].stamp')
|
||||
curl -fsS http://dalidou:8100/admin/backup/$LATEST/validate | jq .
|
||||
|
||||
# Full restore — see the "Restore procedure" section above
|
||||
```
|
||||
@@ -44,8 +44,9 @@ read-only additive mode.
|
||||
|
||||
### Engineering Layer Planning Sprint
|
||||
|
||||
The engineering layer is intentionally in planning, not implementation.
|
||||
The architecture docs below are the current state of that planning:
|
||||
**Status: complete.** All 8 architecture docs are drafted. The
|
||||
engineering layer is now ready for V1 implementation against the
|
||||
active project set.
|
||||
|
||||
- [engineering-query-catalog.md](architecture/engineering-query-catalog.md) —
|
||||
the 20 v1-required queries the engineering layer must answer
|
||||
@@ -55,17 +56,21 @@ The architecture docs below are the current state of that planning:
|
||||
Layer 0 → Layer 2 pipeline, triggers, review queue mechanics
|
||||
- [conflict-model.md](architecture/conflict-model.md) —
|
||||
detection, representation, and resolution of contradictory facts
|
||||
- [tool-handoff-boundaries.md](architecture/tool-handoff-boundaries.md) —
|
||||
KB-CAD / KB-FEM one-way mirror stance, ingest endpoints, drift handling
|
||||
- [representation-authority.md](architecture/representation-authority.md) —
|
||||
canonical home matrix across PKM / KB / repos / AtoCore for 22 fact kinds
|
||||
- [human-mirror-rules.md](architecture/human-mirror-rules.md) —
|
||||
templates, regeneration triggers, edit flow, "do not edit" enforcement
|
||||
- [engineering-v1-acceptance.md](architecture/engineering-v1-acceptance.md) —
|
||||
measurable done definition with 23 acceptance criteria
|
||||
- [engineering-knowledge-hybrid-architecture.md](architecture/engineering-knowledge-hybrid-architecture.md) —
|
||||
the 5-layer model (from the previous planning wave)
|
||||
- [engineering-ontology-v1.md](architecture/engineering-ontology-v1.md) —
|
||||
the initial V1 object and relationship inventory (previous wave)
|
||||
|
||||
Still to draft before engineering-layer implementation begins:
|
||||
|
||||
- tool-handoff-boundaries.md (KB-CAD / KB-FEM read vs write)
|
||||
- human-mirror-rules.md (templates, triggers, edit flow)
|
||||
- representation-authority.md (PKM / KB / repo / AtoCore canonical home matrix)
|
||||
- engineering-v1-acceptance.md (done definition)
|
||||
The next concrete next step is the V1 implementation sprint, which
|
||||
should follow engineering-v1-acceptance.md as its checklist.
|
||||
|
||||
## What Is Real Today
|
||||
|
||||
|
||||
Reference in New Issue
Block a user