ATOCore/docs/PHASE-7-MEMORY-CONSOLIDATION.md

Phase 7 — Memory Consolidation (the "Sleep Cycle")

Status: 7A in progress · 7B-H scoped, deferred Design principle: "Like human memory while sleeping, but more robotic — never discard relevant details. Consolidate, update, supersede — don't delete."

Why

Phases 1–6 built capture + triage + graduation + emerging-project detection. What they don't solve:

#	Problem	Fix
1	Redundancy — "APM uses NX" said 5 different ways across 5 memories	7A Semantic dedup
2	Latent contradictions — "chose Zygo" + "switched from Zygo" both active	7B Pair contradiction detection
3	Tag drift — firmware, fw, firmware-control fragment retrieval	7C Tag canonicalization
4	Confidence staleness — 6-month unreferenced memory ranks as fresh	7D Confidence decay
5	No memory drill-down page	7E /wiki/memories/{id}
6	Domain knowledge siloed per project	7F /wiki/domains/{tag}
7	Prompt upgrades (llm-0.5 → 0.6) don't re-process old interactions	7G Re-extraction on version bump
8	Superseded memory vectors still in Chroma polluting retrieval	7H Vector hygiene

Collectively: the brain needs a nightly pass that looks at what it already knows and tidies up — dedup, resolve contradictions, canonicalize tags, decay stale facts — without losing information.

Subphases

7A — Semantic dedup + consolidation (this sprint)

Compute embeddings on active memories, find pairs within (project, memory_type) bucket above similarity threshold (default 0.88), cluster, draft a unified memory via LLM, human approves in triage UI. On approve: sources become superseded, new merged memory created with union of source_refs, sum of reference_count, max of confidence. Ships first because redundancy compounds — every new memory potentially duplicates an old one.

Detailed spec lives in the working plan (dapper-cooking-tower.md) and across the files listed under "Files touched" below. Key decisions:

• LLM drafts, human approves — no silent auto-merge.
• Same (project, memory_type) bucket only. Cross-project merges are rare + risky → separate flow in 7B.
• Recompute embeddings each scan (~2s / 335 memories). Persist only if scan time becomes a problem.
• Cluster-based proposals (ABC → one merge), not pair-based.
• status=superseded never deleted — still queryable with filter.

Schema: new table memory_merge_candidates (pending | approved | rejected). Cron: nightly at threshold 0.90 (tight); weekly (Sundays) at 0.85 (deeper cleanup). UI: new "🔗 Merge Candidates" section in /admin/triage.

Files touched in 7A:

• src/atocore/models/database.py — migration
• src/atocore/memory/similarity.py — new, compute_memory_similarity()
• src/atocore/memory/_dedup_prompt.py — new, shared LLM prompt
• src/atocore/memory/service.py — merge_memories()
• scripts/memory_dedup.py — new, host-side detector (HTTP-only)
• src/atocore/api/routes.py — 5 new endpoints under /admin/memory/
• src/atocore/engineering/triage_ui.py — merge cards section
• deploy/dalidou/batch-extract.sh — Step B3
• deploy/dalidou/dedup-watcher.sh — new, UI-triggered scans
• tests/test_memory_dedup.py — ~10-15 new tests

7B — Memory-to-memory contradiction detection

Same embedding-pair machinery as 7A but within a different band (similarity 0.70–0.88 — semantically related but different wording). LLM classifies each pair: duplicate | complementary | contradicts | supersedes-older. Contradictions write a memory_conflicts row + surface a triage badge. Clear supersessions (both tier 1 sonnet and tier 2 opus agree) auto-mark the older as superseded.

7C — Tag canonicalization

Weekly LLM pass over domain_tags distribution, proposes alias → canonical map (e.g. fw → firmware). Human approves via UI (one-click pattern, same as emerging-project registration). Bulk-rewrites domain_tags atomically across all memories.

7D — Confidence decay

Daily lightweight job. For memories with reference_count=0 AND last_referenced_at older than 30 days: multiply confidence by 0.97/day (~2-month half-life). Reinforcement already bumps confidence. Below 0.3 → auto-supersede with reason decayed, no references. Reversible (tune half-life), non-destructive (still searchable with status filter).

7E — Memory detail page /wiki/memories/{id}

Provenance chain: source_chunk → interaction → graduated_to_entity. Audit trail (Phase 4 has the data). Related memories (same project + tag + semantic neighbors). Decay trajectory plot (if 7D ships). Link target from every memory surfaced anywhere in the wiki.

7F — Cross-project domain view /wiki/domains/{tag}

One page per domain_tag showing all memories + graduated entities with that tag, grouped by project. "Optics across p04+p05+p06" becomes a real navigable page. Answers the long-standing question the tag system was meant to enable.

7G — Re-extraction on prompt upgrade

batch_llm_extract_live.py --force-reextract --since DATE. Dedupe key: (interaction_id, extractor_version) — same run on same interaction doesn't double-create. Triggered manually when LLM_EXTRACTOR_VERSION bumps. Not automatic (destructive).

7H — Vector store hygiene

Nightly: scan source_chunks and memory_embeddings (added in 7A V2) for status=superseded|invalid. Delete matching vectors from Chroma. Fail-open — the retrieval harness catches any real regression.

Verification & ship order

1. 7A — ship + observe 1 week → validate merge proposals are high-signal, rejection rate acceptable
2. 7D — decay is low-risk + high-compounding value; ship second
3. 7C — clean up tag fragmentation before 7F depends on canonical tags
4. 7E + 7F — UX surfaces; ship together once data is clean
5. 7B — contradictions flow (pairs harder than duplicates to classify; wait for 7A data to tune threshold)
6. 7G — on-demand; no ship until we actually bump the extractor prompt
7. 7H — housekeeping; after 7A + 7B + 7D have generated enough superseded rows to matter

Scope NOT in Phase 7

• Graduated memories (entity-descended) are frozen — exempt from dedup/decay. Entity consolidation is a separate Phase (8+).
• Auto-merging without human approval (always human-in-the-loop in V1).
• Summarization / compression — a different problem (reducing the number of chunks per memory, not the number of memories).
• Forgetting policies — there's no user-facing "delete this" flow in Phase 7. Supersede + filter covers the need.