Anto01 b309e7fd49 feat(eval-loop): Day 4 — LLM-assisted extractor path (additive, flagged) ...

Day 2 baseline showed 0% recall for the rule-based extractor across
5 distinct miss classes. Day 4 decision gate: prototype an
LLM-assisted mode behind a flag. Option A ratified by Antoine.

New module src/atocore/memory/extractor_llm.py:

- extract_candidates_llm(interaction) returns the same MemoryCandidate
  dataclass the rule extractor produces, so both paths flow through
  the existing triage / candidate pipeline unchanged.
- extract_candidates_llm_verbose() also returns the raw model output
  and any error string, for eval and debugging.
- Uses Claude Haiku 4.5 by default; model overridable via
  ATOCORE_LLM_EXTRACTOR_MODEL env. Timeout via
  ATOCORE_LLM_EXTRACTOR_TIMEOUT_S (default 20s).
- Silent-failure contract: missing API key, unreachable model,
  malformed JSON — all return [] and log an error. Never raises
  into the caller. The capture audit trail must not break on an
  optional side effect.
- Parser tolerates markdown fences, surrounding prose, invalid
  memory types, clamps confidence to [0,1], drops empty content.
- System prompt explicitly tells the model to return [] for most
  conversational turns (durable-fact bar, not "extract everything").
- Trust rules unchanged: candidates are never auto-promoted,
  extraction stays off the capture hot path, human triages via the
  existing CLI.

scripts/extractor_eval.py: new --mode {rule,llm} flag so the same
labeled corpus can be scored against both extractors. Default
remains rule so existing invocations are unchanged.

tests/test_extractor_llm.py: 12 new unit tests covering the parser
(empty array, malformed JSON, markdown fences, surrounding prose,
invalid types, empty content, confidence clamping, version tagging),
plus contract tests for missing API key, empty response, and a
mocked api_error path so failure modes never raise.

Test count: 264 -> 276 passing. No existing tests changed.

Next step: run `python scripts/extractor_eval.py --mode llm` against
the labeled set with ANTHROPIC_API_KEY in env, record the delta,
decide whether to wire LLM mode into the API endpoint and CLI or
keep it script-only for now.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-04-11 15:18:30 -04:00

8.7 KiB

Raw Blame History

View Raw