ATOCore/docs/architecture/memory-vs-entities.md

Memory vs Entities (Engineering Layer V1 boundary)

Why this document exists

The engineering layer introduces a new representation — typed entities with explicit relationships — alongside AtoCore's existing memory system and its six memory types. The question that blocks every other engineering-layer planning doc is:

When we extract a fact from an interaction or a document, does it become a memory, an entity, or both? And if both, which one is canonical?

Without an answer, the rest of the engineering layer cannot be designed. This document is the answer.

The short version

• Memories stay. They are still the canonical home for unstructured, attributed, personal, natural-language facts.
• Entities are new. They are the canonical home for structured, typed, relational, engineering-domain facts.
• No concept lives in both at full fidelity. Every concept has exactly one canonical home. The other layer may hold a pointer or a rendered view, never a second source of truth.
• The two layers share one review queue. Candidates from extraction flow into the same status=candidate lifecycle regardless of whether they are memory-bound or entity-bound.
• Memories can "graduate" into entities when enough structure has accumulated, but the upgrade is an explicit, logged promotion, not a silent rewrite.

The split per memory type

The six memory types from the current Phase 2 implementation each map to exactly one outcome in V1:

Memory type	V1 destination	Rationale
identity	memory only	Always about the human user. No engineering domain structure. Never gets entity-shaped.
preference	memory only	Always about the human user's working style. Same reasoning.
episodic	memory only	"What happened in this conversation / this day." Attribution and time are the point, not typed structure.
knowledge	entity when possible, memory otherwise	If the knowledge maps to a typed engineering object (material property, constant, tolerance), it becomes a Fact entity with provenance. If it's loose general knowledge, stays a memory.
project	entity	Anything that belonged in the "project" memory type is really a Requirement, Constraint, Decision, Subsystem attribute, etc. It belongs in the engineering layer once entities exist.
adaptation	entity (Decision)	"We decided to X" is literally a Decision entity in the ontology. This is the clearest migration.

Practical consequence: when the engineering layer V1 ships, the project, knowledge, and adaptation memory types are deprecated as a canonical home for new facts. Existing rows are not deleted — they are backfilled as entities through the promotion-rules flow (see promotion-rules.md), and the old memory rows become frozen references pointing at their graduated entity.

The identity, preference, and episodic memory types continue to exist exactly as they do today and do not interact with the engineering layer at all.

What "canonical home" actually means

A concept's canonical home is the single place where:

• its current active value is stored
• its status lifecycle is managed (active/superseded/invalid)
• its confidence is tracked
• its provenance chain is rooted
• edits, supersessions, and invalidations are applied
• conflict resolution is arbitrated

Everything else is a derived view of that canonical row.

If a Decision entity is the canonical home for "we switched to GF-PTFE pads", then:

• there is no adaptation memory row with the same content; the extractor creates a Decision candidate directly
• the context builder, when asked to include relevant state, reaches into the entity store via the engineering layer, not the memory store
• if the user wants to see "recent decisions" they hit the entity API, never the memory API
• if they want to invalidate the decision, they do so via the entity API

The memory API remains the canonical home for identity, preference, and episodic — same rules, just a different set of types.

Why not a unified table with a kind column?

It would be simpler to implement. It is rejected for three reasons:

1. Different query shapes. Memories are queried by type, project, confidence, recency. Entities are queried by type, relationships, graph traversal, coverage gaps ("orphan requirements"). Cramming both into one table forces the schema to be the union of both worlds and makes each query slower.

2. Different lifecycles. Memories have a simple four-state lifecycle (candidate/active/superseded/invalid). Entities have the same four states plus per-relationship supersession, per-field versioning for the killer correctness queries, and structured conflict flagging. The unified table would have to carry all entity apparatus for every memory row.

3. Different provenance semantics. A preference memory is provenanced by "the user told me" — one author, one time. An entity like a Requirement is provenanced by "this source chunk + this source document + these supporting Results" — a graph. The tables want to be different because their provenance models are different.

So: two tables, one review queue, one promotion flow, one trust hierarchy.

The shared review queue

Both the memory extractor (Phase 9 Commit C, already shipped) and the future entity extractor write into the same conceptual queue: everything lands at status=candidate in its own table, and the human reviewer sees a unified list. The reviewer UI (future work) shows candidates of all kinds side by side, grouped by source interaction / source document, with the rule that fired.

From the data side this means:

• the memories table gets a candidate status (already done in Phase 9 Commit B/C)
• the future entities table will get the same candidate status
• both tables get the same promote / reject API shape: one verb per candidate, with an audit log entry

Implementation note: the API routes should evolve from POST /memory/{id}/promote to POST /candidates/{id}/promote once both tables exist, so the reviewer tooling can treat them uniformly. The current memory-only route stays in place for backward compatibility and is aliased by the unified route.

Memory-to-entity graduation

Even though the split is clean on paper, real usage will reveal memories that deserve to be entities but started as plain text. Four signals are good candidates for proposing graduation:

1. Reference count crosses a threshold. A memory that has been reinforced 5+ times across multiple interactions is a strong signal that it deserves structure.

2. Memory content matches a known entity template. If a knowledge memory's content matches the shape "X = value [unit]" it can be proposed as a Fact or Parameter entity.

3. A user explicitly asks for promotion. POST /memory/{id}/graduate is the simplest explicit path — it returns a proposal for an entity structured from the memory's content, which the user can accept or reject.

4. Extraction pass proposes an entity that happens to match an existing memory. The entity extractor, when scanning a new interaction, sees the same content already exists as a memory and proposes graduation as part of its candidate output.

The graduation flow is:

memory row (active, confidence C)
      |
      | propose_graduation()
      v
entity candidate row (candidate, confidence C)
      +
memory row gets status="graduated" and a forward pointer to the
entity candidate
      |
      | human promotes the candidate entity
      v
entity row (active)
      +
memory row stays "graduated" permanently (historical record)

The memory is never deleted. It becomes a frozen historical pointer to the entity it became. This keeps the audit trail intact and lets the Human Mirror show "this decision started life as a memory on April 2, was graduated to an entity on April 15, now has 2 supporting ValidationClaims".

The graduated status is a new memory status that gets added when the graduation flow is implemented. For now (Phase 9), only the three non-graduating types (identity/preference/episodic) would ever avoid it, and the three graduating types stay in their current memory-only state until the engineering layer ships.

Context pack assembly after the split

The context builder today (src/atocore/context/builder.py) pulls:

1. Trusted Project State
2. Identity + Preference memories
3. Retrieved chunks

After the split, it pulls:

1. Trusted Project State (unchanged)
2. Identity + Preference memories (unchanged — these stay memories)
3. Engineering-layer facts relevant to the prompt, queried through the entity API (new)
4. Retrieved chunks (unchanged, lowest trust)

Note the ordering: identity/preference memories stay above entities, because personal style information is always more trusted than extracted engineering facts. Entities sit below the personal layer but above raw retrieval, because they have structured provenance that raw chunks lack.

The budget allocation gains a new slot:

• trusted project state: 20% (unchanged, highest trust)
• identity memories: 5% (unchanged)
• preference memories: 5% (unchanged)
• engineering entities: 15% (new — pulls only V1-required objects relevant to the prompt)
• retrieval: 55% (reduced from 70% to make room)

These are starting numbers. After the engineering layer ships and real usage tunes retrieval quality, these will be revisited.

What the shipped memory types still mean after the split

Memory type	Still accepts new writes?	V1 destination for new extractions
identity	yes	memory (no change)
preference	yes	memory (no change)
episodic	yes	memory (no change)
knowledge	yes, but only for loose facts	entity (Fact / Parameter) for structured things; memory is a fallback
project	no new writes after engineering V1 ships	entity (Requirement / Constraint / Subsystem attribute)
adaptation	no new writes after engineering V1 ships	entity (Decision)

"No new writes" means the create_memory path will refuse to create new project or adaptation memories once the engineering layer V1 ships. Existing rows stay queryable and reinforceable but new facts of those kinds must become entities. This keeps the canonical-home rule clean going forward.

The deprecation is deferred: it does not happen until the engineering layer V1 is demonstrably working against the active project set. Until then, the existing memory types continue to accept writes so the Phase 9 loop can be exercised without waiting on the engineering layer.

Consequences for Phase 9 (what we just built)

The capture loop, reinforcement, and extractor we shipped today are memory-facing. They produce memory candidates, reinforce memory confidence, and respect the memory status lifecycle. None of that changes.

When the engineering layer V1 ships, the extractor in src/atocore/memory/extractor.py gets a sibling in src/atocore/entities/extractor.py that uses the same interaction-scanning approach but produces entity candidates instead. The POST /interactions/{id}/extract endpoint either:

• runs both extractors and returns a combined result, or
• gains a ?target=memory|entities|both query parameter

and the decision between those two shapes can wait until the entity extractor actually exists.

Until the entity layer is real, the memory extractor also has to cover some things that will eventually move to entities (decisions, constraints, requirements). That overlap is temporary and intentional. Rather than leave those cues unextracted for months while the entity layer is being built, the memory extractor surfaces them as memory candidates. Later, a migration pass will propose graduation on every active memory created by decision_heading, constraint_heading, and requirement_heading rules once the entity types exist to receive them.

So: no rework in Phase 9, no wasted extraction, clean handoff once the entity layer lands.

Open questions this document does NOT answer

These are deliberately deferred to later planning docs:

1. When exactly does extraction fire? (answered by promotion-rules.md)
2. How are conflicts between a memory and an entity handled during graduation? (answered by conflict-model.md)
3. Does the context builder traverse the entity graph for relationship-rich queries, or does it only surface direct facts? (answered by the context-builder spec in a future engineering-context-integration.md doc)
4. What is the exact API shape of the unified candidate review queue? (answered by a future review-queue-api.md doc when the entity extractor exists and both tables need one UI)

TL;DR

• memories = user-facing unstructured facts, still own identity/preference/episodic
• entities = engineering-facing typed facts, own project/knowledge/adaptation
• one canonical home per concept, never both
• one shared candidate-review queue, same promote/reject shape
• graduated memories stay as frozen historical pointers
• Phase 9 stays memory-only and ships today; entity V1 follows the remaining architecture docs in this planning sprint
• no rework required when the entity layer lands; the current memory extractor's structural cues get migrated forward via explicit graduation