Layer 3 of the AtoCore architecture. Generates a human-readable
project overview in markdown from structured data:
- Trusted Project State (by category)
- System Architecture (systems → subsystems → components with
material and interface links)
- Decisions (with affected entities)
- Requirements & Constraints
- Materials
- Vendors
- Active Memories (with confidence and reference counts)
The mirror is DERIVED — every line traces back to an entity, state
entry, or memory. The footer stamps the generation timestamp and
the "not canonical truth" disclaimer.
API: GET /projects/{project_name}/mirror returns {project, format,
content} where content is the full markdown page. Supports project
aliases via resolve_project_name.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When a query matches a known engineering entity by name, the context
pack now includes a structured '--- Engineering Context ---' band
showing the entity's type, description, and its relationships to
other entities (subsystems, materials, requirements, decisions).
Six-tier context assembly:
1. Trusted Project State
2. Identity / Preferences
3. Project Memories
4. Domain Knowledge
5. Engineering Context (NEW)
6. Retrieved Chunks
The engineering band uses the same token-overlap scoring as memory
ranking: query tokens are matched against entity names + descriptions.
The top match gets its full relationship context included.
10% budget allocation. Trims before domain knowledge (lowest
priority of the structured tiers since the same info may appear in
chunks).
Example: query 'lateral support design' against p04-gigabit
surfaces the Lateral Support subsystem entity with its relationships
to GF-PTFE material, M1 Mirror Assembly parent system, and related
components.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Seeds the entity graph from existing project state, memories, and
vault docs across p04-gigabit (11 entities), p05-interferometer (10),
and p06-polisher (14). Covers systems, subsystems, components,
materials, decisions, requirements, constraints, vendors, and
parameters with structural and intent relationships.
Example: GET /entities/{M1 Mirror Assembly id} returns the full
context — 4 subsystems it contains, 2 requirements it's constrained
by, and the parent project — traversable in one API call.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Layer 2 of the AtoCore architecture. Adds typed engineering entities
with relationships on top of the flat memory/state/chunk substrate.
Schema:
- entities table: id, entity_type, name, project, description,
properties (JSON), status, confidence, source_refs, timestamps
- relationships table: source_entity_id, target_entity_id,
relationship_type, confidence, source_refs
15 entity types: project, system, subsystem, component, interface,
requirement, constraint, decision, material, parameter,
analysis_model, result, validation_claim, vendor, process
12 relationship types: contains, part_of, interfaces_with,
satisfies, constrained_by, affected_by_decision, analyzed_by,
validated_by, depends_on, uses_material, described_by, supersedes
Service layer: full CRUD + get_entity_with_context (returns an
entity with its relationships and all related entities in one call).
API endpoints:
- POST /entities — create entity
- GET /entities — list/filter by type, project, status, name
- GET /entities/{id} — entity + relationships + related entities
- POST /relationships — create relationship
Schema auto-initialized on app startup via init_engineering_schema().
7 tests covering entity CRUD, relationships, context traversal,
filtering, name search, and validation.
Test count: 290 -> 297.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Comprehensive architecture doc covering:
- The problem (applied vs domain knowledge separation)
- The quality bar (earned insight vs common knowledge, with examples)
- Five-tier context assembly with budget allocation
- Knowledge domains (10 domains: physics through finance)
- Domain tag encoding (prefix in content, no schema migration)
- Full flow: capture → extract → triage → surface
- Cross-project example (p04 insight surfaces in p06 context)
- Future directions: personal branch, multi-model, reinforcement
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The extraction system now produces two kinds of candidates from
the same conversation:
A. PROJECT-SPECIFIC: applied facts scoped to a named project
(unchanged behavior)
B. DOMAIN KNOWLEDGE: generalizable engineering insight earned
through project work, tagged with a domain (physics, materials,
optics, mechanics, manufacturing, metrology, controls, software,
math, finance) and stored with project="" so it surfaces across
all projects.
Critical quality bar enforced in the system prompt: "Would a
competent engineer need experience to know this, or could they
find it in 30 seconds on Google?" Textbook values, definitions,
and obvious facts are explicitly excluded. Only hard-won insight
qualifies — the kind that takes weeks of FEA or real machining
experience to discover.
Domain tags are embedded in the content as a prefix ("[physics]",
"[materials]") so they survive without a schema migration. A future
column can parse them out.
Context builder gains a new tier between project memories and
retrieved chunks:
Tier 1: Trusted Project State (project-specific)
Tier 2: Identity / Preferences (global)
Tier 3: Project Memories (project-specific)
Tier 4: Domain Knowledge (NEW) (cross-project, 10% budget)
Tier 5: Retrieved Chunks (project-boosted)
Trim order: chunks -> domain knowledge -> project memories ->
identity/preference -> project state.
Host-side extraction script updated with the same prompt and
domain-tag handling.
LLM_EXTRACTOR_VERSION bumped to llm-0.3.0.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds openclaw-plugins/atocore-capture/: a minimal OpenClaw plugin
that mirrors Claude Code's Stop hook. Captures user-triggered
assistant turns and POSTs to AtoCore /interactions with
client=openclaw, reinforce=true, fail-open.
Review verdict: functionally complete, one polish item (prompt
includes wrapper context — not blocking, extraction pipeline
handles noisy prompts). End-to-end verified on Dalidou with a
real client=openclaw interaction.
Both Claude Code and OpenClaw now feed AtoCore's reflection loop.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Wave 2 deeper ingestion:
- 6 new Trusted Project State entries from design-level docs:
p05: test rig architecture, CGH specification, procurement combos
p06: force control architecture, control channels, calibration loop
- Total state entries: ~23 (was ~17)
Observability:
- GET /admin/dashboard — one-shot system overview: memory counts
by type/project/status, reinforced count, project state entry
counts, recent interaction timestamp, extraction pipeline status.
Replaces the need to query 4+ endpoints to understand system state.
Harness: 17/18 (no regression from new state entries).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
3 identity memories (Antoine's role, projects, infrastructure) and
3 preference memories (no API keys, multi-model collab, action bias)
seeded on live Dalidou. These fill the identity/preference band
that was previously empty.
Lowered MEMORY_BUDGET_RATIO from 0.10 to 0.05 because the 10%
allocation squeezed project memories and retrieval chunks enough
to regress 4 harness fixtures. At 5% the band fits at most 1 short
memory — enough for the most relevant identity/preference fact
without starving the project-specific tiers.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Before: a model returning 'p04-gigabit' for a p06-polisher
interaction would silently override the known scope because the
project was registered. After: interaction.project always wins
when set. Model project is only a fallback for unscoped captures.
Not yet guaranteed: within-project semantic errors (model says
the right project but wrong content). That's a content-quality
concern, not a trust-hierarchy issue.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Batch 3, Days 1-3. The core R9 failure was Case F: when the model
returned a registered project DIFFERENT from the interaction's
known scope, the old code trusted the model because the project
was registered. A p06-polisher interaction could silently produce
a p04-gigabit candidate.
New rule (trust hierarchy):
1. Interaction scope always wins when set (cases A, C, E, F)
2. Model project used only for unscoped interactions AND only when
it resolves to a registered project (cases D, G)
3. Empty string when both are empty or unregistered (case B)
The rule is: interaction.project is the strongest signal because
it comes from the capture hook's project detection, which runs
before the LLM ever sees the content. The model's project guess
is only useful when the capture hook had no project context.
7 case tests (A-G) cover every combination of model/interaction
project state. Pre-existing tests updated for the new behavior.
Host-side script mirrors the same hierarchy using _known_projects
fetched from GET /projects at startup.
Test count: 286 -> 290 (+4 net, 7 new R9 cases, 3 old tests
consolidated).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Codex verified R1/R5/R7/R8 fixed, harness 17/18, auto-triage
dry-run works. R9 stays open: registered-but-wrong project from
model can still override interaction scope. Fair — the registry
check prevents hallucinated names but not misattribution between
real projects.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
R7: ranking scorer now uses overlap-density (overlap_count /
memory_token_count) as primary key instead of raw overlap count.
A 5-token memory with 3 overlapping tokens (density 0.6) now beats
a 40-token overview memory with 3 overlapping tokens (density 0.075)
at the same absolute count. Secondary: absolute overlap. Tertiary:
confidence. Targeting p06-firmware-interface harness fixture.
R9: when the LLM extractor returns a project that differs from the
interaction's known project, it now checks the project registry.
If the model's project is a registered canonical ID, trust it. If
not (hallucinated name), fall back to the interaction's project.
Uses load_project_registry() for the check. The host-side script
mirrors this via an API call to GET /projects at startup.
Two new tests: test_parser_keeps_registered_model_project and
test_parser_rejects_hallucinated_project.
Test count: 280 -> 281.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
scripts/auto_triage.py: fetches candidate memories, asks a triage
model (claude -p, default sonnet) to classify each as promote /
reject / needs_human, and executes the verdict via the API.
Trust model:
- Auto-promote: model says promote AND confidence >= 0.8 AND
dedup-checked against existing active memories for the project
- Auto-reject: model says reject
- needs_human: everything else stays in queue for manual review
The triage model receives both the candidate content AND a summary
of existing active memories for the same project, so it can detect
duplicates and near-duplicates. The system prompt explicitly lists
the rejection categories learned from the first two manual triage
passes (stale snapshots, impl details, planned-not-implemented,
process rules that belong in ledger not memory).
deploy/dalidou/batch-extract.sh now runs extraction (Step A) then
auto-triage (Step B) in sequence. The nightly cron at 03:00 UTC
will run the full pipeline: backup → cleanup → rsync → extract →
triage. Only needs_human candidates reach the human.
Supports --dry-run for preview without executing.
Supports --model override for multi-model triage (e.g. opus for
higher-quality review, or a future Gemini/Ollama backend).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Codex verified the host-side extraction pipeline works end-to-end
on Dalidou (ran it manually, produced 13 additional candidates).
R1 and R5 are now marked fixed. New findings:
- R11: container mode=llm silently returns 0 candidates
- R12: duplicated prompt/parser between host script and extractor
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Dalidou runs Claude Code 2.0.60 which does not have this flag
(added in 2.1.x). Removed from both extractor_llm.py and the
host-side batch script. --append-system-prompt and
--disable-slash-commands are supported on 2.0.60.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
GET /interactions returns response_chars but not the response body
to keep the listing lightweight. The batch extractor now lists ids
first, then fetches each interaction individually via
GET /interactions/{id} to get the full response for LLM extraction.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The host Python on Dalidou lacks pydantic_settings and other
container-only deps. Refactored batch_llm_extract_live.py to be
a standalone HTTP client + subprocess wrapper using only stdlib.
Duplicates the system prompt and JSON parser from extractor_llm.py
rather than importing them — acceptable duplication since this
is a deployment adapter, not a library.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The claude CLI is installed on the Dalidou HOST but not inside
the Docker container. The /admin/extract-batch API endpoint with
mode=llm silently returned 0 candidates because
shutil.which('claude') was None inside the container.
Fix: extraction runs host-side via deploy/dalidou/batch-extract.sh
which calls scripts/batch_llm_extract_live.py with the host's
PYTHONPATH pointing at the repo's src/. The script:
- Fetches interactions from the API (GET /interactions?since=...)
- Runs extract_candidates_llm() locally (host has claude CLI)
- POSTs candidates back to the API (POST /memory, status=candidate)
- Tracks last-run timestamp via project state
The cron now calls the host-side script instead of the container
API endpoint for LLM mode. Rule-mode extraction in the container
still works via /admin/extract-batch.
The API endpoint retains the mode=llm option for environments
where claude IS inside the container (future Docker image with
claude CLI, or a different deployment model).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Step 4 added to the daily cron: POST /admin/extract-batch with
mode=llm, persist=true, limit=50. Runs after backup + cleanup +
rsync. Fail-open: extraction failure never blocks the backup.
Gated on ATOCORE_EXTRACT_BATCH=true (defaults to true). The
endpoint uses the last_extract_batch_run timestamp from project
state to auto-resume, so the cron doesn't need to track state.
curl --max-time 600 gives the LLM extractor up to 10 minutes
for the batch (50 interactions × ~20s each worst case = ~17 min,
but most will be no-ops if already extracted).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Day 1 of the operational-reflection batch. Two changes:
1. POST /admin/extract-batch: batch extraction endpoint that fetches
recent interactions (since last run or explicit 'since' param),
runs the extractor (rule or LLM mode), and persists candidates
with status=candidate. Tracks last-run timestamp in project state
(atocore/status/last_extract_batch_run) so subsequent calls
auto-resume. This is the operational home for R1/R5 — makes the
LLM extractor an API operation, not just a script.
2. POST /interactions/{id}/extract now accepts mode: "rule" | "llm"
(default "rule" for backward compatibility). When "llm", it uses
extract_candidates_llm (claude -p sonnet, OAuth).
Both changes preserve the standing decision: extraction stays off
the capture hot path. The batch endpoint is invoked explicitly by
cron, manual curl, or CLI — never inline with POST /interactions.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Verified t420-openclaw/atocore.py against live Dalidou from both
the development machine and the T420 (clawdbot @ 192.168.86.39):
- health: returns 0.2.0 + build_sha + vector count
- auto-context: project detection + context/build produces full
packs with Trusted Project State, Project Memories band, and
retrieved chunks (tested p05 vendor query and p06 firmware query)
- fail-open: unreachable host returns {status: unavailable,
fail_open: true} without crashing or blocking the session
API surface coverage: atocore.py hits 15/33 endpoints (core
retrieval + project state + context build). Memory management,
interactions, and backup endpoints are correctly excluded — those
belong to the operator client (scripts/atocore_client.py) per the
read-only additive integration model.
No code changes needed — the April 6 atocore.py already matches
the current API surface. Wave 2 state entries and project-memory
band changes are transparent to the client (they enrich
formatted_context without requiring client-side updates).
Cloned repo to T420 at /home/papa/ATOCore for future OpenClaw use.
Updated master-plan-status.md: Phase 8 moved from Partial to
Baseline Complete.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Codex R6: the LLM extractor accepted the model's project field
verbatim. When the model returned empty string, clearly p06 memories
got promoted as project='', making them invisible to the p06
project-memory band and explaining the p06-offline-design harness
failure.
Fix: if model returns empty project but interaction.project is set,
inherit the interaction's project. Model-supplied project still takes
precedence when non-empty.
Two new tests lock the fallback and precedence behaviors.
R5 acknowledged (LLM extractor not yet wired into API — next task).
Test count: 278 -> 280. Harness re-run pending after deploy.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Codex correctly identified:
- R5 (P1): LLM extractor is script-only, not wired into the API
- R6 (P1): LLM extractor drops interaction.project when model
returns empty — caused the p06-offline-design harness failure
- R7 (P2): lexical scorer ties on overlap count, broad memories
win on confidence tiebreaker
- R8 (P2): no integration test for the persist/triage flow
Also corrected the harness-failure narrative: not all 3 are budget
contention. One is a ranking tie, one is a project-scope miss,
one is chunk bleed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Haiku was producing noisy candidates (31% accept rate on first
triage). Sonnet should give tighter extraction with fewer false
positives while still catching the same durable-fact patterns.
Override: ATOCORE_LLM_EXTRACTOR_MODEL=haiku to revert.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Mini-phase complete. Before/after deltas:
Metric Before After
─────────────────────────────────────────
Rule extractor recall 0% 0% (unchanged, deprioritized)
LLM extractor recall n/a 100% (new, claude -p haiku)
LLM candidate yield n/a 2.55/interaction
First triage accept rate n/a 31% (16/51)
Active memories 20 36 (+16)
p06-polisher memories 2 16 (+14)
atocore memories 0 5 (+5)
Retrieval harness 6/6 15/18 (expanded to 18 fixtures)
Test count 264 278 (+14)
3 remaining harness failures are budget-contention on the p06 memory
band: the specific memory a fixture targets ranks 4th+ and the 25%
budget only holds 2-3 entries. Not a ranking bug — the per-entry
250-char cap was the one justified tweak; a second budget change
risks regressing other fixtures per Codex's Day 7 hard gate.
Ledger updated: Orientation, Session Log, main_tip, harness line.
Next on the roadmap (from DEV-LEDGER Active Plan / docs/next-steps):
- Wave 2 trusted operational ingestion (p04/p05/p06 dashboards)
- Finish OpenClaw integration (Phase 8)
- Auto-triage (multi-model second pass to reduce human review)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
A 530-char program overview memory with confidence 0.96 was filling
the entire 25% project-memory budget at equal overlap score (3 tokens),
beating shorter query-relevant newly-promoted memories (confidence
0.5) on the confidence tiebreaker. The long memory legitimately
scored well, but its length starved every other memory from the band.
Fix: truncate each formatted entry to 250 chars with '...' so at
least 2-3 memories fit the ~700-char available budget. This doesn't
change ranking — the most relevant memory still goes first — but
it ensures the runner-up can also appear.
Harness fixture delta: Day 7 regression pass pending after deploy.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Added 12 new fixtures across all three active projects:
- p04: 1 short/ambiguous case ('current status')
- p05: 1 CGH calibration case with cross-project bleed guard
- p06: 7 new fixtures targeting triage-promoted memories
(firmware interface, z-axis, cam encoder, telemetry rate,
offline design, USB SSD, Tailscale)
- Adversarial: cross-project-no-bleed (p04 query must not surface
p06 telemetry rate), no-project-hint (project memories must not
appear without a hint)
First run: 14/18 passing.
4 failures (p06-firmware-interface, p06-z-axis, p06-offline-design,
p06-tailscale) share the same root cause: long pre-existing p06
memories (530+ chars, confidence 0.9+) fill the 25% project-memory
budget before the query-relevant newly-promoted memories (shorter,
confidence 0.5) get a slot. Budget contention at equal overlap
score tiebroken by confidence. Day 7 ranking tweak target.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Documents the LLM-assisted extractor's in-scope / out-of-scope
categories derived from the first live triage pass (16 promoted,
35 rejected). Five in-scope classes, six explicit out-of-scope
classes, trust model summary, multi-model future direction.
Cleaned up stale follow-up items in next-steps.md: rule expansion
marked deprioritized, LLM extractor marked done, retrieval harness
marked done with expansion pending.
Fixed docstring timeout (45s -> 90s) in extractor_llm.py.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
First end-to-end triage pass on 51 LLM-extracted candidates from
the Day 4 baseline run (extractor_llm via claude -p haiku against
a 20-interaction frozen snapshot).
Results:
- Promoted 16 memories (31% accept rate):
* p06-polisher: 9 (USB SSD, Tailscale, 10 Hz telemetry,
controller-job.v1 invariant, offline-first, z-axis engage/
retract, cam encoder read-only, spec separation)
* atocore: 7 (extraction off hot path, DEV-LEDGER adopted,
codex branching rule, Claude builds/Codex audits, alias
canonicalization, Stop hook capture, passive capture)
- Rejected 35 (stale roadmap items, duplicates with wrong project
tags, already-fixed P1 findings, process rules that live in
DEV-LEDGER/AGENTS.md not in memory, too-granular implementation
details, operational instructions)
Active memory count: 20 → 36. p06-polisher went from 2 to 16.
Candidate queue: 0.
The triage verdict is saved at
scripts/eval_data/triage_verdict_2026-04-12.json for audit.
persist_llm_candidates.py used to push candidates to Dalidou.
POST /memory now accepts a 'status' field (default 'active') so
external scripts can create candidate memories directly.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Mini-phase Day 1-4: frozen interaction snapshot, labeled extractor
eval corpus (20 labels), eval runner with --mode rule|llm, LLM-
assisted extractor via claude -p (OAuth, no API key), baseline
measurements (rule 0% recall → LLM 100% recall), status field
exposed on POST /memory, persist_llm_candidates.py script.
Day 4 gate cleared: LLM-assisted extraction is the recommended
path for conversational captures. Rule-based stays as default for
structural-cue content.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The API endpoint now passes the request's status field through to
create_memory() so external scripts can create candidate memories
directly without going through the extract endpoint. Default remains
'active' for backward compatibility.
persist_llm_candidates.py reads a saved extractor eval baseline
JSON (e.g. the Day 4 LLM run) and POSTs each candidate to Dalidou
with status=candidate. Safe to re-run — duplicate content returns
400 which the script counts as 'skipped'.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Second pass on the LLM-assisted extractor after Antoine's explicit
rule: no API key, ever. Refactored src/atocore/memory/extractor_llm.py
to shell out to the Claude Code 'claude -p' CLI via subprocess instead
of the anthropic SDK, so extraction reuses the user's existing Claude.ai
OAuth credentials and needs zero secret management.
Implementation:
- subprocess.run(["claude", "-p", "--model", "haiku",
"--append-system-prompt", <instructions>,
"--no-session-persistence", "--disable-slash-commands",
user_message], ...)
- cwd is a cached tempfile.mkdtemp() so every invocation starts with
a clean context instead of auto-discovering CLAUDE.md / AGENTS.md /
DEV-LEDGER.md from the repo root. We cannot use --bare because it
forces API-key auth, which defeats the purpose; the temp-cwd trick
is the lightest way to keep OAuth auth while skipping project
context loading.
- Silent-failure contract unchanged: missing CLI, non-zero exit,
timeout, malformed JSON — all return [] and log an error. The
capture audit trail must not break on an optional side effect.
- Default timeout bumped from 20s to 90s: Haiku + Node.js startup
+ OAuth check is ~20-40s per call in practice, plus real responses
up to 8KB take longer. 45s hit 2 timeouts on the first live run.
- tests/test_extractor_llm.py refactored: the API-key / anthropic SDK
tests are replaced by subprocess-mocking tests covering missing
CLI, timeout, non-zero exit, and a happy-path stdout parse. 14
tests, all green.
scripts/extractor_eval.py:
- New --output <path> flag writes the JSON result directly to a file,
bypassing stdout/log interleaving (structlog sends INFO to stdout
via PrintLoggerFactory, so a naive '> out.json' pollutes the file).
- Forces UTF-8 on stdout so real LLM output with em-dashes / arrows /
CJK doesn't crash the human report on Windows cp1252 consoles.
First live baseline run against the 20-interaction labeled corpus
(scripts/eval_data/extractor_llm_baseline_2026-04-11.json):
mode=llm labeled=20 recall=1.0 precision=0.357 yield_rate=2.55
total_actual_candidates=51 total_expected_candidates=7
false_negative_interactions=0 false_positive_interactions=9
Recall 0% -> 100% vs rule baseline — every human-labeled positive is
caught. Precision reads low (0.357) but inspection shows the "false
positives" are real candidates the human labels under-counted. For
example interaction a6b0d279 was labeled at 2 expected candidates,
the model caught all 6 polisher architectural facts; interaction
52c8c0f3 was labeled at 1, the model caught all 5 infra commitments.
The labels are the bottleneck, not the model.
Day 4 gate against Codex's criteria:
- candidate yield: 255% vs ≥15-25% target
- FP rate tolerable for manual triage: 51 candidates reviewable in
~10 minutes via the triage CLI
- ≥2 real non-synthetic candidates worth review: 20+ obvious wins
(polisher architecture set, p05 infra set, DEV-LEDGER protocol set)
Gate cleared. LLM-assisted extraction is the path forward for
conversational captures. Rule-based extractor stays as-is for
structured-cue inputs and remains the default mode. The next step
(Day 5 stabilize / document) will wire LLM mode behind a flag in
the public extraction endpoint and document scope.
Test count: 276 -> 278 passing. No existing tests changed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Day 2 baseline showed 0% recall for the rule-based extractor across
5 distinct miss classes. Day 4 decision gate: prototype an
LLM-assisted mode behind a flag. Option A ratified by Antoine.
New module src/atocore/memory/extractor_llm.py:
- extract_candidates_llm(interaction) returns the same MemoryCandidate
dataclass the rule extractor produces, so both paths flow through
the existing triage / candidate pipeline unchanged.
- extract_candidates_llm_verbose() also returns the raw model output
and any error string, for eval and debugging.
- Uses Claude Haiku 4.5 by default; model overridable via
ATOCORE_LLM_EXTRACTOR_MODEL env. Timeout via
ATOCORE_LLM_EXTRACTOR_TIMEOUT_S (default 20s).
- Silent-failure contract: missing API key, unreachable model,
malformed JSON — all return [] and log an error. Never raises
into the caller. The capture audit trail must not break on an
optional side effect.
- Parser tolerates markdown fences, surrounding prose, invalid
memory types, clamps confidence to [0,1], drops empty content.
- System prompt explicitly tells the model to return [] for most
conversational turns (durable-fact bar, not "extract everything").
- Trust rules unchanged: candidates are never auto-promoted,
extraction stays off the capture hot path, human triages via the
existing CLI.
scripts/extractor_eval.py: new --mode {rule,llm} flag so the same
labeled corpus can be scored against both extractors. Default
remains rule so existing invocations are unchanged.
tests/test_extractor_llm.py: 12 new unit tests covering the parser
(empty array, malformed JSON, markdown fences, surrounding prose,
invalid types, empty content, confidence clamping, version tagging),
plus contract tests for missing API key, empty response, and a
mocked api_error path so failure modes never raise.
Test count: 264 -> 276 passing. No existing tests changed.
Next step: run `python scripts/extractor_eval.py --mode llm` against
the labeled set with ANTHROPIC_API_KEY in env, record the delta,
decide whether to wire LLM mode into the API endpoint and CLI or
keep it script-only for now.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Day 2 extractor eval baseline on a 20-interaction labeled set shows
0% yield / 0% recall / 0% precision. The 5 false negatives span
5 distinct miss classes, matching the pattern Codex's Day 4 hard
gate was designed to catch but arriving two days early.
No extractor code change on main. Day 1+2 artifacts committed on
working branch 'claude/extractor-eval-loop' at 7d8d599. Day 4
decision (keep rule-expanding vs prototype LLM-assisted mode) is
escalated to Antoine for ratification before Day 3 work touches
any extractor.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Day 1 (labeled corpus):
- scripts/eval_data/interactions_snapshot_2026-04-11.json — frozen
snapshot of 64 real claude-code interactions pulled from live
Dalidou (test-client captures filtered out). This is the stable
corpus the whole mini-phase labels against, independent of future
captures.
- scripts/eval_data/extractor_labels_2026-04-11.json — 20 hand-labeled
interactions drawn by length-stratified random sample. Positives:
5/20 = ~25%, total expected candidates: 7. Plan deviation: Codex's
plan asked for 30 (10/10/10 buckets); the real corpus is heavily
skewed toward instructional/status content, so honest labeling of
20 already crosses the fail-early threshold of "at least 5 plausible
positives" without padding.
Day 2 (baseline measurement):
- scripts/extractor_eval.py — file-based eval runner that loads the
snapshot + labels, runs extract_candidates_from_interaction on each,
and reports yield / recall / precision / miss-class breakdown.
Returns exit 1 on any false positive or false negative.
Current rule extractor against the labeled set:
labeled=20 exact_match=15 positive_expected=5
yield=0.0 recall=0.0 precision=0.0
false_negatives=5 false_positives=0
miss_classes:
recommendation_prose
architectural_change_summary
spec_update_announcement
layered_recommendation
alignment_assertion
Interpretation: the rule-based extractor matches exactly zero of the
5 plausible positive interactions in the labeled set, and the misses
are spread across 5 distinct cue classes with no single dominant
pattern. This is the Day 4 hard-stop signal landing on Day 2 — a
single rule expansion cannot close a 5-way miss, and widening rules
blindly will collapse precision. The right move is to go straight to
the Day 4 decision gate and consider LLM-assisted extraction.
Escalating to DEV-LEDGER.md as R5 for human ratification before
continuing. Not skipping Day 3 silently.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The ledger is the one-file source of truth for "what is currently
true" across Claude/Codex/human sessions:
- Orientation (live SHA, main tip, test count, harness state)
- Active Plan (currently Codex's 8-day extractor + harness plan
with hard gates and fail-early thresholds)
- Open Review Findings (P1/P2, status)
- Recent Decisions (bounded to last 20)
- Session Log (bounded to last 20)
- Working Rules (no parallel work, branching rule, P1 block)
Narrative docs under docs/ sometimes lag reality; the ledger does
not. Every session MUST read it at start and append a Session Log
line before ending.
AGENTS.md: added a new "Session protocol" section at the top that
points at the ledger. Applies to any agent (Claude, Codex, future).
CLAUDE.md (new, project-local): project instructions for Claude
Code in this repo. Points at DEV-LEDGER.md and AGENTS.md, spells
out the deploy workflow and the Claude/Codex working model.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
