ATOCore/docs/architecture/knowledge-architecture.md

# AtoCore Knowledge Architecture

## The Problem

Engineering work produces two kinds of knowledge simultaneously:

1. **Applied knowledge** — specific to the project being worked on
   ("the p04 support pad layout is driven by CTE gradient analysis")
2. **Domain knowledge** — generalizable insight earned through that work
   ("Zerodur CTE gradient dominates WFE at fast focal ratios")

A system that only stores applied knowledge loses the general insight.
A system that mixes them pollutes project context with cross-project
noise. AtoCore needs both — separated, but both growing organically
from the same conversations.

## The Quality Bar

**AtoCore stores earned insight, not information.**

The test: "Would a competent engineer need experience to know this,
or could they find it in 30 seconds?"

| Store | Don't store |
|-------|-------------|
| "Preston removal model breaks down below 5N because the contact assumption fails" | "Preston's equation relates removal rate to pressure and velocity" |
| "m=1 (coma) is NOT correctable by force modulation (score 0.09)" | "Zernike polynomials describe wavefront aberrations" |
| "At F/1.2, CTE gradient costs ~3nm WFE and drives pad placement" | "Zerodur CTE is 0.05 ppm/K" |
| "Quilting limit for 16-inch tool is 234N" | "Quilting is a mid-spatial-frequency artifact in polishing" |

The bar is enforced in the LLM extraction system prompt
(`src/atocore/memory/extractor_llm.py`) and the auto-triage prompt
(`scripts/auto_triage.py`). Both explicitly list examples of what
qualifies and what doesn't.

## Architecture

### Five-tier context assembly

When AtoCore builds a context pack for any LLM query, it assembles
five tiers in strict trust order:

```
Tier 1: Trusted Project State        [project-specific, highest trust]
         Curated key-value entries from the project state API.
         Example: "decision/vendor_path: Twyman-Green preferred, 4D
         technical lead but cost-challenged"

Tier 2: Identity / Preferences       [global, always included]
         Who the user is and how they work.
         Example: "Antoine Letarte, mechanical/optical engineer at
         Atomaste" / "No API keys — uses OAuth exclusively"

Tier 3: Project Memories             [project-specific]
         Reinforced memories from the reflection loop, scoped to the
         queried project. Example: "Firmware interface contract is
         invariant: controller-job.v1 in, run-log.v1 out"

Tier 4: Domain Knowledge             [cross-project]
         Earned engineering insight with project="" and a domain tag.
         Surfaces in ALL project packs when query-relevant.
         Example: "[materials] Zerodur CTE gradient dominates WFE at
         fast focal ratios — costs ~3nm at F/1.2"

Tier 5: Retrieved Chunks             [project-boosted, lowest trust]
         Vector-similarity search over the ingested document corpus.
         Project-hinted but not filtered — cross-project docs can
         appear at lower rank.
```

### Budget allocation (at default 3000 chars)

| Tier | Budget ratio | Approx chars | Entries |
|------|-------------|-------------|---------|
| Project State | 20% | 600 | all curated entries |
| Identity/Preferences | 5% | 150 | 1 memory |
| Project Memories | 25% | 750 | 2-3 memories |
| Domain Knowledge | 10% | 300 | 1-2 memories |
| Retrieved Chunks | 40% | 1200 | 2-4 chunks |

Trim order when budget is tight: chunks first, then domain knowledge,
then project memories, then identity, then project state last.

### Knowledge domains

The LLM extractor tags domain knowledge with one of these domains:

| Domain | What qualifies |
|--------|---------------|
| `physics` | Optical physics, wave propagation, diffraction, thermal effects |
| `materials` | Material properties in context, CTE behavior, stress limits |
| `optics` | Lens/mirror design, aberration analysis, metrology techniques |
| `mechanics` | Structural FEA insights, support system design, kinematics |
| `manufacturing` | Polishing, grinding, machining, process control |
| `metrology` | Measurement systems, interferometry, calibration techniques |
| `controls` | PID tuning, force control, servo systems, real-time constraints |
| `software` | Architecture patterns, testing strategies, deployment insights |
| `math` | Numerical methods, optimization, statistical analysis |
| `finance` | Cost modeling, procurement strategy, budget optimization |

New domains can be added by updating the system prompt in
`extractor_llm.py` and `batch_llm_extract_live.py`.

### How domain knowledge is stored

Domain tags are embedded as a prefix in the memory content:

```
memory_type: knowledge
project: ""                    ← empty = cross-project
content: "[materials] Zerodur CTE gradient dominates WFE at F/1.2"
```

The `[domain]` prefix is a lightweight encoding that avoids a schema
migration. The context builder's query-relevance ranking matches on
domain terms naturally (a query about "materials" or "CTE" will rank
a `[materials]` memory higher). A future migration can parse the
prefix into a proper `domain` column.

## How knowledge flows

### Capture → Extract → Triage → Surface

```
1. CAPTURE
   Claude Code (Stop hook) or OpenClaw (plugin)
   → POST /interactions with reinforce=true
   → Interaction stored on Dalidou

2. EXTRACT (nightly cron, 03:00 UTC)
   batch_llm_extract_live.py runs claude -p sonnet
   → For each interaction, the LLM decides:
     - Is this project-specific? → candidate with project=X
     - Is this generalizable insight? → candidate with domain=Y, project=""
     - Is it both? → TWO candidates emitted
     - Is it common knowledge? → skip (quality bar)
   → Candidates persisted as status=candidate

3. TRIAGE (nightly, immediately after extraction)
   auto_triage.py runs claude -p sonnet
   → Each candidate classified: promote / reject / needs_human
   → Auto-promote at confidence ≥ 0.8 + no duplicate
   → Auto-reject stale snapshots, duplicates, common knowledge
   → Only needs_human reaches the operator

4. SURFACE (every context/build query)
   → Project-specific memories appear in Tier 3
   → Domain knowledge appears in Tier 4 (regardless of project)
   → Both are query-ranked by overlap-density
```

### Example: knowledge earned on p04 surfaces on p06

Working on p04-gigabit, you discover that Zerodur CTE gradient is
the dominant WFE contributor at fast focal ratios. The extraction
produces:

```json
[
  {"type": "project", "content": "CTE gradient analysis drove the
   M1 support pad layout — 2nd largest WFE contributor after gravity",
   "project": "p04-gigabit", "domain": "", "confidence": 0.6},

  {"type": "knowledge", "content": "Zerodur CTE gradient dominates
   WFE contribution at fast focal ratios (F/1.2 = ~3nm)",
   "project": "", "domain": "materials", "confidence": 0.6}
]
```

Two weeks later, working on p06-polisher (which also uses Zerodur):

```
Query: "thermal effects on polishing accuracy"
Project: p06-polisher

Tier 3 (Project Memories):
  [project] Calibration loop adjusts Preston kp from surface measurements...

Tier 4 (Domain Knowledge):
  [materials] Zerodur CTE gradient dominates WFE contribution at fast
  focal ratios — THIS CAME FROM P04 WORK
```

The insight crosses over without any manual curation.

## Future directions

### Personal knowledge branch
The same architecture supports personal domains (health, finance,
personal) by adding new domain tags and a trust boundary so
Atomaste project data never leaks into personal packs. The domain
system is domain-agnostic — it doesn't care whether the domain is
"optics" or "nutrition".

### Multi-model extraction
Different models can specialize: sonnet for extraction, opus or
Gemini for triage review. Independent validation reduces correlated
blind spots on what qualifies as "earned insight" vs "common
knowledge."

### Reinforcement-based domain promotion
A domain-knowledge memory that gets reinforced across multiple
projects (its content echoed in p04, p05, and p06 responses)
accumulates confidence faster than a project-specific memory.
High-confidence domain memories could auto-promote to a "verified
knowledge" tier above regular domain knowledge.