ATOCore/docs/phase9-first-real-use.md

Phase 9 First Real Use Report

What this is

The first empirical exercise of the Phase 9 reflection loop after Commits A, B, and C all landed. The goal is to find out where the extractor and the reinforcement matcher actually behave well versus where their behaviour drifts from the design intent.

The validation is reproducible. To re-run:

python scripts/phase9_first_real_use.py

This writes an isolated SQLite + Chroma store under data/validation/phase9-first-use/ (gitignored), seeds three active memories, then runs eight sample interactions through the full capture → reinforce → extract pipeline.

What we ran

Eight synthetic interactions, each paraphrased from a real working session about AtoCore itself or the active engineering projects:

#	Label	Project	Expected
1	exdev-mount-merge-decision	atocore	1 decision_heading
2	ownership-was-the-real-fix	atocore	1 fact_heading
3	memory-vs-entity-canonical-home	atocore	1 decision_heading (long)
4	auto-promotion-deferred	atocore	1 decision_heading
5	preference-rebase-workflow	atocore	1 preference_sentence
6	constraint-from-doc-cite	p05-interferometer	1 constraint_heading
7	prose-only-no-cues	atocore	0 candidates
8	multiple-cues-in-one-interaction	p06-polisher	3 distinct rules

Plus 3 seed memories were inserted before the run:

• pref_rebase: "prefers rebase-based workflows because history stays linear" (preference, 0.6)
• pref_concise: "writes commit messages focused on the why, not the what" (preference, 0.6)
• identity_runs_atocore: "mechanical engineer who runs AtoCore for context engineering" (identity, 0.9)

What happened — extraction (the good news)

Every extraction expectation was met exactly. All eight samples produced the predicted candidate count and the predicted rule classifications:

Sample	Expected	Got	Pass
exdev-mount-merge-decision	1	1	✅
ownership-was-the-real-fix	1	1	✅
memory-vs-entity-canonical-home	1	1	✅
auto-promotion-deferred	1	1	✅
preference-rebase-workflow	1	1	✅
constraint-from-doc-cite	1	1	✅
prose-only-no-cues	0	0	✅
multiple-cues-in-one-interaction	3	3	✅

Total: 9 candidates from 8 interactions, 0 false positives, 0 misses on heading patterns or sentence patterns.

The extractor's strictness is well-tuned for the kinds of structural cues we actually use. Things worth noting:

• Sample 7 (prose-only-no-cues) produced zero candidates as designed. This is the most important sanity check — it confirms the extractor won't fill the review queue with general prose when there's no structural intent.
• Sample 3's long content was preserved without truncation. The 280-char max wasn't hit, and the content kept its full meaning.
• Sample 8 produced three distinct rules in one interaction (decision_heading, constraint_heading, requirement_heading) without the dedup key collapsing them. The dedup key is (memory_type, normalized_content, rule) and the three are all different on at least one axis, so they coexist as expected.
• The prose around each heading was correctly ignored. Sample 6 has a second sentence ("the error budget allocates 6 nm to the laser source...") that does NOT have a structural cue, and the extractor correctly didn't fire on it.

What happened — reinforcement (the empirical finding)

Reinforcement matched zero seeded memories across all 8 samples, even when the response clearly echoed the seed.

Sample 5's response was:

"I prefer rebase-based workflows because the history stays linear and reviewers have an easier time."

The seeded pref_rebase memory was:

"prefers rebase-based workflows because history stays linear"

A human reading both says these are the same fact. The reinforcement matcher disagrees. After all 8 interactions:

pref_rebase:           confidence=0.6000  refs=0  last=-
pref_concise:          confidence=0.6000  refs=0  last=-
identity_runs_atocore: confidence=0.9000  refs=0  last=-

Nothing moved. This is the most important finding from this validation pass.

Why the matcher missed it

The current _memory_matches rule (in src/atocore/memory/reinforcement.py) does a normalized substring match: it lowercases both sides, collapses whitespace, then asks "does the leading 80-char window of the memory content appear as a substring in the response?"

For the rebase example:

• needle (normalized): prefers rebase-based workflows because history stays linear
• haystack (normalized): i prefer rebase-based workflows because the history stays linear and reviewers have an easier time.

The needle starts with prefers (with the trailing s), and the haystack has prefer (without the s, because of the first-person voice). And the needle has because history stays linear, while the haystack has because the history stays linear. Two small natural paraphrases, and the substring fails.

This isn't a bug in the matcher's implementation — it's doing exactly what it was specified to do. It's a design limitation: the substring rule is too brittle for real prose, where the same fact gets re-stated with different verb forms, articles, and word order.

Severity

Medium-high. Reinforcement is the entire point of Commit B. A reinforcement matcher that never fires on natural paraphrases will leave seeded memories with stale confidence forever. The reflection loop runs but it doesn't actually reinforce anything. That hollows out the value of having reinforcement at all.

It is not a critical bug because:

• Nothing breaks. The pipeline still runs cleanly.
• Reinforcement is supposed to be a signal, not the only path to high confidence — humans can still curate confidence directly.
• The candidate-extraction path (Commit C) is unaffected and works perfectly.

But it does need to be addressed before Phase 9 can be considered operationally complete.

Recommended fix (deferred to a follow-up commit)

Replace the substring matcher with a token-overlap matcher. The specification:

1. Tokenize both memory content and response into lowercase words of length >= 3, dropping a small stop list (the, a, an, and, or, of, to, is, was, that, this, with, for, from, into).
2. Stem aggressively (or at minimum, fold trailing s and ed so prefers/prefer/preferred collapse to one token).
3. A match exists if at least 70% of the memory's content tokens appear in the response token set.
4. Memory content must still be at least _MIN_MEMORY_CONTENT_LENGTH characters to be considered.

This is more permissive than the substring rule but still tight enough to avoid spurious matches on generic words. It would have caught the rebase example because:

• memory tokens (after stop-list and stemming): {prefer, rebase-bas, workflow, because, history, stay, linear}
• response tokens: {prefer, rebase-bas, workflow, because, history, stay, linear, reviewer, easi, time}
• overlap: 7 / 7 memory tokens = 100% > 70% threshold → match

Why not fix it in this report

Three reasons:

1. The validation report is supposed to be evidence, not a fix spec. A separate commit will introduce the new matcher with its own tests.
2. The token-overlap matcher needs its own design review for edge cases (very long memories, very short responses, technical abbreviations, code snippets in responses).
3. Mixing the report and the fix into one commit would muddle the audit trail. The report is the empirical evidence; the fix is the response.

The fix is queued as the next Phase 9 maintenance commit and is flagged in the next-steps section below.

Other observations

Extraction is conservative on purpose, and that's working

Sample 7 is the most important data point in the whole run. A natural prose response with no structural cues produced zero candidates. This is exactly the design intent — the extractor should be loud about explicit decisions/constraints/requirements and quiet about everything else. If the extractor were too loose the review queue would fill up with low-value items and the human would stop reviewing.

After this run I have measurably more confidence that the V0 rule set is the right starting point. Future rules can be added one at a time as we see specific patterns the extractor misses, instead of guessing at what might be useful.

Confidence on candidates

All extracted candidates landed at the default confidence=0.5, which is what the extractor is currently hardcoded to do. The promotion-rules.md doc proposes a per-rule prior with a structural-signal multiplier and freshness bonus. None of that is implemented yet. The validation didn't reveal any urgency around this — humans review the candidates either way — but it confirms that the priors-and-multipliers refinement is a reasonable next step rather than a critical one.

Multiple cues in one interaction

Sample 8 confirmed an important property: three structural cues in the same response do not collide in dedup. The dedup key is (memory_type, normalized_content, rule), and since each cue produced a distinct (type, content, rule) tuple, all three landed cleanly.

This matters because real working sessions naturally bundle multiple decisions/constraints/requirements into one summary. The extractor handles those bundles correctly.

Project scoping

Each candidate carries the project from the source interaction into its own project field. Sample 6 (p05) and sample 8 (p06) both produced candidates with the right project. This is non-obvious because the extractor module never explicitly looks at project — it inherits from the interaction it's scanning. Worth keeping in mind when the entity extractor is built: same pattern should apply.

What this validates and what it doesn't

Validates

• The Phase 9 Commit C extractor's rule set is well-tuned for hand-written structural cues
• The dedup logic does the right thing across multiple cues
• The "drop candidates that match an existing active memory" filter works (would have been visible if any seeded memory had matched one of the heading texts — none did, but the code path is the same one that's covered in tests/test_extractor.py)
• The prose-only-no-cues no-fire case is solid
• Long content is preserved without truncation
• Project scoping flows through the pipeline

Does NOT validate

• The reinforcement matcher (clearly, since it caught nothing)
• The behaviour against very long documents (each sample was under 700 chars; real interaction responses can be 10× that)
• The behaviour against responses that contain code blocks (the extractor's regex rules don't handle code-block fenced sections specially)
• Cross-interaction promotion-to-active flow (no candidate was promoted in this run; the lifecycle is covered by the unit tests but not by this empirical exercise)
• The behaviour at scale: 8 interactions is a one-shot. We need to see the queue after 50+ before judging reviewer ergonomics.

Recommended next empirical exercises

1. Real conversation capture, using a slash command from a real Claude Code session against either a local or Dalidou AtoCore instance. The synthetic responses in this script are honest paraphrases but they're still hand-curated.
2. Bulk capture from existing PKM, ingesting a few real project notes through the extractor as if they were interactions. This stresses the rules against documents that weren't written with the extractor in mind.
3. Reinforcement matcher rerun after the token-overlap matcher lands.

Action items from this report

• Fix reinforcement matcher with token-overlap rule described in the "Recommended fix" section above. Owner: next session. Severity: medium-high.
• Document the extractor's V0 strictness as a working property, not a limitation. Sample 7 makes the case.
• Build the slash command so the next validation run can use real (not synthetic) interactions. Tracked in Session 2 of the current planning sprint.
• Run a 50+ interaction batch to evaluate reviewer ergonomics. Deferred until the slash command exists.

Reproducibility

The script is deterministic. Re-running it will produce identical results because:

• the data dir is wiped on every run
• the sample interactions are constants
• the memory uuid generation is non-deterministic but the important fields (content, type, count, rule) are not
• the data/validation/phase9-first-use/ directory is gitignored, so no state leaks across runs

To reproduce this exact report:

python scripts/phase9_first_real_use.py

To get JSON output for downstream tooling:

python scripts/phase9_first_real_use.py --json