audit: record extraction pipeline findings

Dalidou runs Claude Code 2.0.60 which does not have this flag
(added in 2.1.x). Removed from both extractor_llm.py and the
host-side batch script. --append-system-prompt and
--disable-slash-commands are supported on 2.0.60.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

DEV-LEDGER.md

@@ -6,12 +6,13 @@
 
 ## Orientation
 
-- **live_sha** (Dalidou `/health` build_sha): `5c69f77`
-- **last_updated**: 2026-04-12 by Codex (audit branch `codex/audit-2026-04-12`)
-- **main_tip**: `146f2e4`
-- **test_count**: 278 passing
-- **harness**: `15/18 PASS` (remaining failures are mixed: p06-firmware-interface exposes a lexical-ranking tie, p06-offline-design is a live triage scoping miss, p06-tailscale still has retrieved-chunk bleed)
-- **active_memories**: 36 (was 20 before mini-phase; p06-polisher 2->16, atocore 0->5)
+- **live_sha** (Dalidou `/health` build_sha): `39d73e9`
+- **last_updated**: 2026-04-12 by Codex (audit branch `codex/audit-2026-04-12-extraction`)
+- **main_tip**: `ac7f77d`
+- **test_count**: 280 passing
+- **harness**: `16/18 PASS` (p06-firmware-interface = R7 ranking tie; p06-tailscale = chunk bleed)
+- **active_memories**: 36 (p06-polisher 16, p05-interferometer 6, p04-gigabit 5, atocore 5, other 4)
+- **project_state_entries**: p04=5, p05=6, p06=6 (Wave 2 entries present on live Dalidou; 17 total visible)
 - **off_host_backup**: `papa@192.168.86.39:/home/papa/atocore-backups/` via cron env `ATOCORE_BACKUP_RSYNC`, verified
 
 ## Active Plan
@@ -120,14 +121,18 @@ One branch `codex/extractor-eval-loop` for Day 1-5, a second `codex/retrieval-ha
 
 | id  | finder | severity | file:line                          | summary                                                                 | status       | owner  | opened_at  | resolved_by |
 |-----|--------|----------|------------------------------------|-------------------------------------------------------------------------|--------------|--------|------------|-------------|
-| R1  | Codex  | P1       | deploy/hooks/capture_stop.py:76-85 | Live Claude capture still omits `extract`, so "loop closed both sides" remains overstated in practice even though the API supports it | acknowledged | Claude | 2026-04-11 |             |
+| R1  | Codex  | P1       | deploy/hooks/capture_stop.py:76-85 | Live Claude capture still omits `extract`, so "loop closed both sides" remains overstated in practice even though the API supports it | fixed        | Claude | 2026-04-11 | c67bec0     |
 | R2  | Codex  | P1       | src/atocore/context/builder.py     | Project memories excluded from pack                                     | fixed        | Claude | 2026-04-11 | 8ea53f4     |
 | R3  | Claude | P2       | src/atocore/memory/extractor.py    | Rule cues (`## Decision:`) never fire on conversational LLM text        | open         | Claude | 2026-04-11 |             |
 | R4  | Codex  | P2       | DEV-LEDGER.md:11                   | Orientation `main_tip` was stale versus `HEAD` / `origin/main`          | fixed        | Codex  | 2026-04-11 | 81307ce     |
-| R5  | Codex  | P1       | src/atocore/interactions/service.py:157-174 | The deployed extraction path still calls only the rule extractor; the new LLM extractor is eval/script-only, so Day 4 "gate cleared" is true as a benchmark result but not as an operational extraction path | open         | Claude | 2026-04-12 |             |
-| R6  | Codex  | P1       | src/atocore/memory/extractor_llm.py:258-276 | LLM extraction accepts model-supplied `project` verbatim with no fallback to `interaction.project`; live triage promoted a clearly p06 memory (offline/network rule) as project=`""`, which explains the p06-offline-design harness miss and falsifies the current "all 3 failures are budget-contention" claim | open         | Claude | 2026-04-12 |             |
+| R5  | Codex  | P1       | src/atocore/interactions/service.py:157-174 | The deployed extraction path still calls only the rule extractor; the new LLM extractor is eval/script-only, so Day 4 "gate cleared" is true as a benchmark result but not as an operational extraction path | fixed        | Claude | 2026-04-12 | c67bec0     |
+| R6  | Codex  | P1       | src/atocore/memory/extractor_llm.py:258-276 | LLM extraction accepts model-supplied `project` verbatim with no fallback to `interaction.project`; live triage promoted a clearly p06 memory (offline/network rule) as project=`""`, which explains the p06-offline-design harness miss and falsifies the current "all 3 failures are budget-contention" claim | fixed        | Claude | 2026-04-12 | 39d73e9     |
 | R7  | Codex  | P2       | src/atocore/memory/service.py:448-459 | Query ranking is overlap-count only, so broad overview memories can tie exact low-confidence memories and win on confidence; p06-firmware-interface is not just budget pressure, it also exposes a weak lexical scorer | open         | Claude | 2026-04-12 |             |
 | R8  | Codex  | P2       | tests/test_extractor_llm.py:1-7    | LLM extractor tests stop at parser/failure contracts; there is no automated coverage for the script-only persistence/review path that produced the 16 promoted memories, including project-scope preservation | open         | Claude | 2026-04-12 |             |
+| R9  | Codex  | P2       | src/atocore/memory/extractor_llm.py:258-259 | The R6 fallback only repairs empty project output. A wrong non-empty model project still overrides the interaction's known scope, so project attribution is improved but not yet trust-preserving. | open         | Claude | 2026-04-12 |             |
+| R10 | Codex  | P2       | docs/master-plan-status.md:31-33   | "Phase 8 - OpenClaw Integration" is fair as a baseline milestone, but not as a "primary" integration claim. `t420-openclaw/atocore.py` currently covers a narrow read-oriented subset (13 request shapes vs 32 API routes) plus fail-open health, while memory/interactions/admin write paths remain out of surface. | open         | Claude | 2026-04-12 |             |
+| R11 | Codex  | P2       | src/atocore/api/routes.py:773-845  | `POST /admin/extract-batch` still accepts `mode="llm"` inside the container and returns a successful 0-candidate result instead of surfacing that host-only LLM extraction is unavailable from this runtime. That is a misleading API contract for operators. | open         | Claude | 2026-04-12 |             |
+| R12 | Codex  | P2       | scripts/batch_llm_extract_live.py:39-190 | The host-side extractor duplicates the LLM system prompt and JSON parsing logic from `src/atocore/memory/extractor_llm.py`. It works today, but this is now a prompt/parser drift risk across the container and host implementations. | open         | Claude | 2026-04-12 |             |
 
 ## Recent Decisions
 
@@ -145,6 +150,10 @@ One branch `codex/extractor-eval-loop` for Day 1-5, a second `codex/retrieval-ha
 
 ## Session Log
 
+- **2026-04-12 Codex (audit branch `codex/audit-2026-04-12-extraction`)** audited `54d84b5..ac7f77d` with live Dalidou verification. Confirmed the host-side LLM extraction pipeline is operational: nightly cron points at `deploy/dalidou/cron-backup.sh`, Step 4 calls `deploy/dalidou/batch-extract.sh`, the batch script exists/executable on Dalidou, and a manual host-side run produced candidates successfully. Updated R1 and R5 to **fixed** (`c67bec0`) because extraction now runs unattended off-container. Live state during audit: build `39d73e9`, active memories **36**, candidate queue **29** (16 existing + 13 added by manual verification run), and `last_extract_batch_run` populated in AtoCore project state. Added R11-R12 for the misleading container `mode=llm` no-op and host/container prompt-parser duplication. Security note: CLI positional prompt/response text is visible in process args while `claude -p` runs; acceptable on a single-user home host, but worth remembering if Dalidou's trust boundary changes.
+- **2026-04-12 Codex (audit branch `codex/audit-2026-04-12-final`)** audited `c5bad99..e2895b5` against origin/main, live Dalidou, and the OpenClaw client script. Live state checked: build `39d73e9`, harness reproducible at **16/18 PASS**, active memories **36**, and `t420-openclaw/atocore.py health` fails open correctly with `fail_open=true`. Spot-checks of Wave 2 project-state entries matched their cited vault docs. Updated R5-R8 status reality (R6 fixed by `39d73e9`), added R9-R10, and corrected Orientation `main_tip` to `e2895b5` because the ledger had drifted behind origin/main. Note: live Dalidou is still on `39d73e9`, so branch-truth and deploy-truth are not the same yet.
+- **2026-04-12 Claude** Wave 2 trusted operational ingestion + codex audit response. Read 6 vault docs, created 8 new Trusted Project State entries (p04 +2, p05 +3, p06 +3). Fixed R6 (project fallback in LLM extractor) per codex audit. Fixed misscoped p06 offline memory on live Dalidou. Merged codex/audit-2026-04-12. Switched default LLM model from haiku to sonnet. Harness 15/18 -> 16/18. Tests 278 -> 280. main_tip 146f2e4 -> 39d73e9.
+
 - **2026-04-12 Codex (audit branch `codex/audit-2026-04-12`)** audited `c5bad99..146f2e4` against code, live Dalidou, and the 36 active memories. Confirmed: `claude -p` invocation is not shell-injection-prone (`subprocess.run(args)` with no shell), off-host backup wiring matches the ledger, and R1 remains unresolved in practice. Added R5-R8. Corrected Orientation `main_tip` (`146f2e4`, not `5c69f77`) and tightened the harness note: p06-firmware-interface is a ranking-tie issue, p06-offline-design comes from a project-scope miss in live triage, and p06-tailscale is retrieved-chunk bleed rather than memory-band budget contention.
 - **2026-04-12 Claude** `06792d8..5c69f77` Day 5-8 close. Documented extractor scope (5 in-scope, 6 out-of-scope categories). Expanded harness from 6 to 18 fixtures (p04 +1, p05 +1, p06 +7, adversarial +2). Per-entry memory cap at 250 chars fixed 1 of 4 budget-contention failures. Final harness: 15/18 PASS. Mini-phase complete. Before/after: rule extractor 0% recall -> LLM 100%; harness 6/6 -> 15/18; active memories 20 -> 36.
 - **2026-04-12 Claude** `330ecfb..06792d8` (merged eval-loop branch + triage). Day 1-4 of the mini-phase completed in one session. Day 2 baseline: rule extractor 0% recall, 5 distinct miss classes. Day 4 gate cleared: LLM extractor (claude -p haiku, OAuth) hit 100% recall, 2.55 yield/interaction. Refactored from anthropic SDK to subprocess after "no API key" rule. First live triage: 51 candidates -> 16 promoted, 35 rejected. Active memories 20->36. p06-polisher went from 2 to 16 memories (firmware/telemetry architecture set). POST /memory now accepts status field. Test count 264->278.

deploy/dalidou/batch-extract.sh

@@ -0,0 +1,45 @@
+#!/usr/bin/env bash
+#
+# deploy/dalidou/batch-extract.sh
+# --------------------------------
+# Host-side LLM batch extraction for Dalidou.
+#
+# The claude CLI is available on the Dalidou HOST but NOT inside the
+# Docker container. This script runs on the host, fetches recent
+# interactions from the AtoCore API, runs the LLM extractor locally
+# (claude -p sonnet), and posts candidates back to the API.
+#
+# Intended to be called from cron-backup.sh after backup/cleanup/rsync,
+# or manually via:
+#
+#   bash /srv/storage/atocore/app/deploy/dalidou/batch-extract.sh
+#
+# Environment variables:
+#   ATOCORE_URL      default http://127.0.0.1:8100
+#   ATOCORE_EXTRACT_LIMIT  default 50
+
+set -euo pipefail
+
+ATOCORE_URL="${ATOCORE_URL:-http://127.0.0.1:8100}"
+LIMIT="${ATOCORE_EXTRACT_LIMIT:-50}"
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+APP_DIR="$(cd "$SCRIPT_DIR/../.." && pwd)"
+TIMESTAMP="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
+
+log() { printf '[%s] %s\n' "$TIMESTAMP" "$*"; }
+
+# The Python script needs the atocore source on PYTHONPATH
+export PYTHONPATH="$APP_DIR/src:${PYTHONPATH:-}"
+
+log "=== AtoCore batch LLM extraction starting ==="
+log "URL=$ATOCORE_URL  LIMIT=$LIMIT"
+
+# Run the host-side extraction script
+python3 "$APP_DIR/scripts/batch_llm_extract_live.py" \
+    --base-url "$ATOCORE_URL" \
+    --limit "$LIMIT" \
+    2>&1 || {
+    log "WARN: batch extraction failed (non-blocking)"
+}
+
+log "=== AtoCore batch LLM extraction complete ==="

deploy/dalidou/cron-backup.sh

@@ -82,4 +82,22 @@ else
     log "Step 3: ATOCORE_BACKUP_RSYNC not set, skipping off-host copy"
 fi
 
+# Step 4: Batch LLM extraction on recent interactions (optional).
+# Runs HOST-SIDE because claude CLI is on the host, not inside the
+# Docker container. The script fetches interactions from the API,
+# runs claude -p locally, and POSTs candidates back.
+# Fail-open: extraction failure never blocks backup.
+EXTRACT="${ATOCORE_EXTRACT_BATCH:-true}"
+if [[ "$EXTRACT" == "true" ]]; then
+    SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+    log "Step 4: running host-side batch LLM extraction"
+    bash "$SCRIPT_DIR/batch-extract.sh" 2>&1 && {
+        log "Extraction complete"
+    } || {
+        log "WARN: batch extraction failed (this is non-blocking)"
+    }
+else
+    log "Step 4: ATOCORE_EXTRACT_BATCH not set to true, skipping extraction"
+fi
+
 log "=== AtoCore daily backup complete ==="

docs/master-plan-status.md

@@ -27,7 +27,18 @@ read-only additive mode.
 ### Partial
 
 - Phase 4 - Identity / Preferences
-- Phase 8 - OpenClaw Integration
+
+### Baseline Complete
+
+- Phase 8 - OpenClaw Integration. As of 2026-04-12 the T420 OpenClaw
+  helper (`t420-openclaw/atocore.py`) is verified end-to-end against
+  live Dalidou: health check, auto-context with project detection,
+  Trusted Project State surfacing, project-memory band, fail-open on
+  unreachable host. Tested from both the development machine and the
+  T420 via SSH. The helper covers 15 of the 33 API endpoints — the
+  excluded endpoints (memory management, interactions, backup) are
+  correctly scoped to the operator client (`scripts/atocore_client.py`)
+  per the read-only additive integration model.
 
 ### Baseline Complete
 

scripts/batch_llm_extract_live.py

@@ -0,0 +1,277 @@
+"""Host-side LLM batch extraction — pure HTTP client, no atocore imports.
+
+Fetches interactions from the AtoCore API, runs ``claude -p`` locally
+for each, and POSTs candidates back. Zero dependency on atocore source
+or Python packages — only uses stdlib + the ``claude`` CLI on PATH.
+
+This is necessary because the ``claude`` CLI is on the Dalidou HOST
+but not inside the Docker container, and the host's Python doesn't
+have the container's dependencies (pydantic_settings, etc.).
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import shutil
+import subprocess
+import sys
+import tempfile
+import urllib.error
+import urllib.parse
+import urllib.request
+from datetime import datetime, timezone
+
+DEFAULT_BASE_URL = os.environ.get("ATOCORE_BASE_URL", "http://localhost:8100")
+DEFAULT_MODEL = os.environ.get("ATOCORE_LLM_EXTRACTOR_MODEL", "sonnet")
+DEFAULT_TIMEOUT_S = float(os.environ.get("ATOCORE_LLM_EXTRACTOR_TIMEOUT_S", "90"))
+MAX_RESPONSE_CHARS = 8000
+MAX_PROMPT_CHARS = 2000
+
+MEMORY_TYPES = {"identity", "preference", "project", "episodic", "knowledge", "adaptation"}
+
+SYSTEM_PROMPT = """You extract durable memory candidates from LLM conversation turns for a personal context engine called AtoCore.
+
+Your job is to read one user prompt plus the assistant's response and decide which durable facts, decisions, preferences, architectural rules, or project invariants should be remembered across future sessions.
+
+Rules:
+
+1. Only surface durable claims. Skip transient status ("deploy is still running"), instructional guidance ("here is how to run the command"), troubleshooting tactics, ephemeral recommendations ("merge this PR now"), and session recaps.
+2. A candidate is durable when a reader coming back in two weeks would still need to know it. Architectural choices, named rules, ratified decisions, invariants, procurement commitments, and project-level constraints qualify. Conversational fillers and step-by-step instructions do not.
+3. Each candidate must stand alone. Rewrite the claim in one sentence under 200 characters with enough context that a reader without the conversation understands it.
+4. Each candidate must have a type from this closed set: project, knowledge, preference, adaptation.
+5. If the conversation is clearly scoped to a project (p04-gigabit, p05-interferometer, p06-polisher, atocore), set ``project`` to that id. Otherwise leave ``project`` empty.
+6. If the response makes no durable claim, return an empty list. It is correct and expected to return [] on most conversational turns.
+7. Confidence should be 0.5 by default so human review workload is honest. Raise to 0.6 only when the response states the claim in an unambiguous, committed form (e.g. "the decision is X", "the selected approach is Y", "X is non-negotiable").
+8. Output must be a raw JSON array and nothing else. No prose before or after. No markdown fences. No explanations.
+
+Each array element has exactly this shape:
+
+{"type": "project|knowledge|preference|adaptation", "content": "...", "project": "...", "confidence": 0.5}
+
+Return [] when there is nothing to extract."""
+
+_sandbox_cwd = None
+
+
+def get_sandbox_cwd():
+    global _sandbox_cwd
+    if _sandbox_cwd is None:
+        _sandbox_cwd = tempfile.mkdtemp(prefix="ato-llm-extract-")
+    return _sandbox_cwd
+
+
+def api_get(base_url, path, timeout=10):
+    req = urllib.request.Request(f"{base_url}{path}")
+    with urllib.request.urlopen(req, timeout=timeout) as resp:
+        return json.loads(resp.read().decode("utf-8"))
+
+
+def api_post(base_url, path, body, timeout=10):
+    data = json.dumps(body).encode("utf-8")
+    req = urllib.request.Request(
+        f"{base_url}{path}", method="POST",
+        headers={"Content-Type": "application/json"}, data=data,
+    )
+    with urllib.request.urlopen(req, timeout=timeout) as resp:
+        return json.loads(resp.read().decode("utf-8"))
+
+
+def get_last_run(base_url):
+    try:
+        state = api_get(base_url, "/project/state/atocore?category=status")
+        for entry in state.get("entries", []):
+            if entry.get("key") == "last_extract_batch_run":
+                return entry["value"]
+    except Exception:
+        pass
+    return None
+
+
+def set_last_run(base_url, timestamp):
+    try:
+        api_post(base_url, "/project/state", {
+            "project": "atocore", "category": "status",
+            "key": "last_extract_batch_run", "value": timestamp,
+            "source": "batch_llm_extract_live.py",
+        })
+    except Exception:
+        pass
+
+
+def extract_one(prompt, response, project, model, timeout_s):
+    """Run claude -p on one interaction, return parsed candidates."""
+    if not shutil.which("claude"):
+        return [], "claude_cli_missing"
+
+    prompt_excerpt = prompt[:MAX_PROMPT_CHARS]
+    response_excerpt = response[:MAX_RESPONSE_CHARS]
+    user_message = (
+        f"PROJECT HINT (may be empty): {project}\n\n"
+        f"USER PROMPT:\n{prompt_excerpt}\n\n"
+        f"ASSISTANT RESPONSE:\n{response_excerpt}\n\n"
+        "Return the JSON array now."
+    )
+
+    args = [
+        "claude", "-p",
+        "--model", model,
+        "--append-system-prompt", SYSTEM_PROMPT,
+        "--disable-slash-commands",
+        user_message,
+    ]
+
+    try:
+        completed = subprocess.run(
+            args, capture_output=True, text=True,
+            timeout=timeout_s, cwd=get_sandbox_cwd(),
+            encoding="utf-8", errors="replace",
+        )
+    except subprocess.TimeoutExpired:
+        return [], "timeout"
+    except Exception as exc:
+        return [], f"subprocess_error: {exc}"
+
+    if completed.returncode != 0:
+        return [], f"exit_{completed.returncode}"
+
+    raw = (completed.stdout or "").strip()
+    return parse_candidates(raw, project), ""
+
+
+def parse_candidates(raw, interaction_project):
+    """Parse model JSON output into candidate dicts."""
+    text = raw.strip()
+    if text.startswith("```"):
+        text = text.strip("`")
+        nl = text.find("\n")
+        if nl >= 0:
+            text = text[nl + 1:]
+        if text.endswith("```"):
+            text = text[:-3]
+        text = text.strip()
+
+    if not text or text == "[]":
+        return []
+
+    if not text.lstrip().startswith("["):
+        start = text.find("[")
+        end = text.rfind("]")
+        if start >= 0 and end > start:
+            text = text[start:end + 1]
+
+    try:
+        parsed = json.loads(text)
+    except json.JSONDecodeError:
+        return []
+
+    if not isinstance(parsed, list):
+        return []
+
+    results = []
+    for item in parsed:
+        if not isinstance(item, dict):
+            continue
+        mem_type = str(item.get("type") or "").strip().lower()
+        content = str(item.get("content") or "").strip()
+        project = str(item.get("project") or "").strip()
+        if not project and interaction_project:
+            project = interaction_project
+        conf = item.get("confidence", 0.5)
+        if mem_type not in MEMORY_TYPES or not content:
+            continue
+        try:
+            conf = max(0.0, min(1.0, float(conf)))
+        except (TypeError, ValueError):
+            conf = 0.5
+        results.append({
+            "memory_type": mem_type,
+            "content": content[:1000],
+            "project": project,
+            "confidence": conf,
+        })
+    return results
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Host-side LLM batch extraction")
+    parser.add_argument("--base-url", default=DEFAULT_BASE_URL)
+    parser.add_argument("--limit", type=int, default=50)
+    parser.add_argument("--since", default=None)
+    parser.add_argument("--model", default=DEFAULT_MODEL)
+    args = parser.parse_args()
+
+    since = args.since or get_last_run(args.base_url)
+    print(f"since={since or '(first run)'}  limit={args.limit}  model={args.model}")
+
+    params = [f"limit={args.limit}"]
+    if since:
+        params.append(f"since={urllib.parse.quote(since)}")
+    listing = api_get(args.base_url, f"/interactions?{'&'.join(params)}")
+    interaction_summaries = listing.get("interactions", [])
+    print(f"listed {len(interaction_summaries)} interactions")
+
+    processed = 0
+    total_candidates = 0
+    total_persisted = 0
+    errors = 0
+
+    for summary in interaction_summaries:
+        resp_chars = summary.get("response_chars", 0) or 0
+        if resp_chars < 50:
+            continue
+        iid = summary["id"]
+        try:
+            raw = api_get(
+                args.base_url,
+                f"/interactions/{urllib.parse.quote(iid, safe='')}",
+            )
+        except Exception as exc:
+            print(f"  ! {iid[:8]}: fetch failed: {exc}", file=sys.stderr)
+            errors += 1
+            continue
+        response_text = raw.get("response", "") or ""
+        if not response_text.strip() or len(response_text) < 50:
+            continue
+
+        candidates, error = extract_one(
+            prompt=raw.get("prompt", "") or "",
+            response=response_text,
+            project=raw.get("project", "") or "",
+            model=args.model,
+            timeout_s=DEFAULT_TIMEOUT_S,
+        )
+
+        if error:
+            print(f"  ! {raw['id'][:8]}: {error}", file=sys.stderr)
+            errors += 1
+            continue
+
+        processed += 1
+        total_candidates += len(candidates)
+
+        for c in candidates:
+            try:
+                api_post(args.base_url, "/memory", {
+                    "memory_type": c["memory_type"],
+                    "content": c["content"],
+                    "project": c["project"],
+                    "confidence": c["confidence"],
+                    "status": "candidate",
+                })
+                total_persisted += 1
+            except urllib.error.HTTPError as exc:
+                if exc.code != 400:
+                    errors += 1
+            except Exception:
+                errors += 1
+
+    now = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S")
+    set_last_run(args.base_url, now)
+
+    print(f"processed={processed} candidates={total_candidates} persisted={total_persisted} errors={errors}")
+
+
+if __name__ == "__main__":
+    main()

src/atocore/api/routes.py

@@ -35,6 +35,10 @@ from atocore.memory.extractor import (
     MemoryCandidate,
     extract_candidates_from_interaction,
 )
+from atocore.memory.extractor_llm import (
+    LLM_EXTRACTOR_VERSION,
+    extract_candidates_llm,
+)
 from atocore.memory.reinforcement import reinforce_from_interaction
 from atocore.memory.service import (
     MEMORY_STATUSES,
@@ -580,6 +584,7 @@ def api_reinforce_interaction(interaction_id: str) -> dict:
 
 class InteractionExtractRequest(BaseModel):
     persist: bool = False
+    mode: str = "rule"  # "rule" or "llm"
 
 
 @router.post("/interactions/{interaction_id}/extract")
@@ -601,7 +606,10 @@ def api_extract_from_interaction(
     if interaction is None:
         raise HTTPException(status_code=404, detail=f"Interaction not found: {interaction_id}")
     payload = req or InteractionExtractRequest()
-    candidates: list[MemoryCandidate] = extract_candidates_from_interaction(interaction)
+    if payload.mode == "llm":
+        candidates: list[MemoryCandidate] = extract_candidates_llm(interaction)
+    else:
+        candidates: list[MemoryCandidate] = extract_candidates_from_interaction(interaction)
 
     persisted_ids: list[str] = []
     if payload.persist:
@@ -755,6 +763,109 @@ def api_cleanup_backups(req: BackupCleanupRequest | None = None) -> dict:
         raise HTTPException(status_code=500, detail=f"Cleanup failed: {e}")
 
 
+class ExtractBatchRequest(BaseModel):
+    since: str | None = None
+    mode: str = "llm"
+    limit: int = 50
+    persist: bool = True
+
+
+@router.post("/admin/extract-batch")
+def api_extract_batch(req: ExtractBatchRequest | None = None) -> dict:
+    """Run batch extraction across recent interactions.
+
+    Fetches interactions since ``since`` (or since the last recorded
+    batch run), runs the extractor (rule or LLM) on each, and persists
+    any candidates as ``status=candidate``. The last-run timestamp is
+    stored in project state under ``atocore / status /
+    last_extract_batch_run`` so subsequent calls without ``since``
+    automatically pick up where the last run left off.
+
+    This endpoint is the operational home for R1 / R5 — it makes the
+    LLM extractor accessible as an API operation rather than a
+    script-only eval tool. Still NOT on the capture hot path: callers
+    invoke this endpoint explicitly (cron, manual curl, CLI).
+    """
+    payload = req or ExtractBatchRequest()
+    since = payload.since
+
+    if not since:
+        state_entries = get_state("atocore")
+        for entry in state_entries:
+            if entry.category == "status" and entry.key == "last_extract_batch_run":
+                since = entry.value
+                break
+
+    interactions = list_interactions(since=since, limit=min(payload.limit, 200))
+
+    processed = 0
+    total_candidates = 0
+    total_persisted = 0
+    errors: list[dict] = []
+
+    for interaction in interactions:
+        if not (interaction.response or interaction.response_summary):
+            continue
+        try:
+            if payload.mode == "llm":
+                candidates = extract_candidates_llm(interaction)
+            else:
+                candidates = extract_candidates_from_interaction(interaction)
+        except Exception as exc:
+            errors.append({"interaction_id": interaction.id, "error": str(exc)})
+            continue
+
+        processed += 1
+        total_candidates += len(candidates)
+
+        if payload.persist and candidates:
+            for candidate in candidates:
+                try:
+                    create_memory(
+                        memory_type=candidate.memory_type,
+                        content=candidate.content,
+                        project=candidate.project,
+                        confidence=candidate.confidence,
+                        status="candidate",
+                    )
+                    total_persisted += 1
+                except ValueError:
+                    pass  # duplicate — skip silently
+
+    from datetime import datetime, timezone
+
+    now = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S")
+    try:
+        set_state(
+            project="atocore",
+            category="status",
+            key="last_extract_batch_run",
+            value=now,
+            source="admin/extract-batch endpoint",
+        )
+    except Exception:
+        pass  # best-effort timestamp tracking
+
+    log.info(
+        "extract_batch_complete",
+        mode=payload.mode,
+        processed=processed,
+        total_candidates=total_candidates,
+        total_persisted=total_persisted,
+        errors=len(errors),
+    )
+
+    return {
+        "processed": processed,
+        "total_candidates": total_candidates,
+        "total_persisted": total_persisted,
+        "mode": payload.mode,
+        "persist": payload.persist,
+        "since": since or "(first run)",
+        "errors": errors,
+    }
+
+
 @router.get("/admin/backup/{stamp}/validate")
 def api_validate_backup(stamp: str) -> dict:
     """Validate that a previously created backup is structurally usable."""

src/atocore/memory/extractor_llm.py

@@ -27,7 +27,7 @@ Configuration:
 
 - Requires the ``claude`` CLI on PATH (``claude --version`` should work).
 - ``ATOCORE_LLM_EXTRACTOR_MODEL`` overrides the model alias (default
-  ``haiku``).
+  ``sonnet``).
 - ``ATOCORE_LLM_EXTRACTOR_TIMEOUT_S`` overrides the per-call timeout
   (default 90 seconds — first invocation is slow because Node.js
   startup plus OAuth check is non-trivial).
@@ -65,7 +65,7 @@ from atocore.observability.logger import get_logger
 log = get_logger("extractor_llm")
 
 LLM_EXTRACTOR_VERSION = "llm-0.2.0"
-DEFAULT_MODEL = os.environ.get("ATOCORE_LLM_EXTRACTOR_MODEL", "haiku")
+DEFAULT_MODEL = os.environ.get("ATOCORE_LLM_EXTRACTOR_MODEL", "sonnet")
 DEFAULT_TIMEOUT_S = float(os.environ.get("ATOCORE_LLM_EXTRACTOR_TIMEOUT_S", "90"))
 MAX_RESPONSE_CHARS = 8000
 MAX_PROMPT_CHARS = 2000
@@ -168,7 +168,6 @@ def extract_candidates_llm_verbose(
         model or DEFAULT_MODEL,
         "--append-system-prompt",
         _SYSTEM_PROMPT,
-        "--no-session-persistence",
         "--disable-slash-commands",
         user_message,
     ]
@@ -256,6 +255,8 @@ def _parse_candidates(raw_output: str, interaction: Interaction) -> list[MemoryC
         mem_type = str(item.get("type") or "").strip().lower()
         content = str(item.get("content") or "").strip()
         project = str(item.get("project") or "").strip()
+        if not project and interaction.project:
+            project = interaction.project
         confidence_raw = item.get("confidence", 0.5)
         if mem_type not in MEMORY_TYPES:
             continue

tests/test_extractor_llm.py

@@ -97,6 +97,25 @@ def test_parser_tags_version_and_rule():
     assert result[0].source_interaction_id == "test-id"
 
 
+def test_parser_falls_back_to_interaction_project():
+    """R6: when the model returns empty project but the interaction
+    has one, the candidate should inherit the interaction's project."""
+    raw = '[{"type": "project", "content": "machine works offline"}]'
+    interaction = _make_interaction()
+    interaction.project = "p06-polisher"
+    result = _parse_candidates(raw, interaction)
+    assert result[0].project == "p06-polisher"
+
+
+def test_parser_keeps_model_project_when_provided():
+    """Model-supplied project takes precedence over interaction."""
+    raw = '[{"type": "project", "content": "x", "project": "p04-gigabit"}]'
+    interaction = _make_interaction()
+    interaction.project = "p06-polisher"
+    result = _parse_candidates(raw, interaction)
+    assert result[0].project == "p04-gigabit"
+
+
 def test_missing_cli_returns_empty(monkeypatch):
     """If ``claude`` is not on PATH the extractor returns empty, never raises."""
     monkeypatch.setattr(extractor_llm, "_cli_available", lambda: False)