ATOCore/docs/architecture/promotion-rules.md

Promotion Rules (Layer 0 → Layer 2 pipeline)

Purpose

AtoCore ingests raw human-authored content (markdown, repo notes, interaction transcripts) and eventually must turn some of it into typed engineering entities that the V1 query catalog can answer. The path from raw text to typed entity has to be:

• explicit: every step has a named operation, a trigger, and an audit log
• reversible: every promotion can be undone without data loss
• conservative: no automatic movement into trusted state; a human (or later, a very confident policy) always signs off
• traceable: every typed entity must carry a back-pointer to the raw source that produced it

This document defines that path.

The four layers

Promotion is described in terms of four layers, all of which exist simultaneously in the system once the engineering layer V1 ships:

Layer	Name	Canonical storage	Trust	Who writes
L0	Raw source	source_documents + source_chunks	low	ingestion pipeline
L1	Memory candidate	memories (status="candidate")	low	extractor
L1'	Active memory	memories (status="active")	med	human promotion
L2	Entity candidate	entities (status="candidate")	low	extractor + graduation
L2'	Active entity	entities (status="active")	high	human promotion
L3	Trusted state	project_state	highest	human curation

Layer 3 (trusted project state) is already implemented and stays manually curated — automatic promotion into L3 is never allowed.

The promotion graph

          [L0] source chunks
               |
               | extraction (memory extractor, Phase 9 Commit C)
               v
        [L1] memory candidate
               |
               | promote_memory()
               v
        [L1'] active memory
               |
               | (optional) propose_graduation()
               v
        [L2] entity candidate
               |
               | promote_entity()
               v
        [L2'] active entity
               |
               | (manual curation, NEVER automatic)
               v
        [L3] trusted project state

Short path (direct entity extraction, once the entity extractor exists):

          [L0] source chunks
               |
               | entity extractor
               v
        [L2] entity candidate
               |
               | promote_entity()
               v
        [L2'] active entity

A single fact can travel either path depending on what the extractor saw. The graduation path exists for facts that started life as memories before the entity layer existed, and for the memory extractor's structural cues (decisions, constraints, requirements) which are eventually entity-shaped.

Triggers (when does extraction fire?)

Phase 9 already shipped one trigger: on explicit API request (POST /interactions/{id}/extract). The V1 engineering layer adds two more:

1. On interaction capture (automatic)

   • Same event that runs reinforcement today
   • Controlled by a extract boolean flag on the record request (default: false for memory extractor, true once an engineering extractor exists and has been validated)
   • Output goes to the candidate queue; nothing auto-promotes

2. On ingestion (batched, per wave)

   • After a wave of markdown ingestion finishes, a batch extractor pass sweeps all newly-added source chunks and produces candidates from them
   • Batched per wave (not per chunk) to keep the review queue digestible and to let the reviewer see all candidates from a single ingestion in one place
   • Output: a report artifact plus a review queue entry per candidate

3. On explicit human request (existing)

   • POST /interactions/{id}/extract for a single interaction
   • Future: POST /ingestion/wave/{id}/extract for a whole wave
   • Future: POST /memory/{id}/graduate to propose graduation of one specific memory into an entity

Batch size rule: extraction passes never write more than N candidates per human review cycle, where N = 50 by default. If a pass produces more, it ranks by (rule confidence × content length × novelty) and only writes the top N. The remaining candidates are logged, not persisted. This protects the reviewer from getting buried.

Confidence and ranking of candidates

Each rule-based extraction rule carries a prior confidence based on how specific its pattern is:

Rule class	Prior	Rationale
Heading with explicit type (## Decision:)	0.7	Very specific structural cue, intentional author marker
Typed list item (- [Decision] ...)	0.65	Explicit but often embedded in looser prose
Sentence pattern (I prefer X)	0.5	Moderate structure, more false positives
Regex pattern matching a value+unit (X = 4.8 kg)	0.6	Structural but prone to coincidence
LLM-based (future)	variable	Depends on model's returned confidence

The candidate's final confidence at write time is:

final = prior * structural_signal_multiplier * freshness_bonus

Where:

• structural_signal_multiplier is 1.1 if the source chunk path contains any of _HIGH_SIGNAL_HINTS from the retriever (status, decision, requirements, charter, ...) and 0.9 if it contains _LOW_SIGNAL_HINTS (_archive, _history, ...)
• freshness_bonus is 1.05 if the source chunk was updated in the last 30 days, else 1.0

This formula is tuned later; the numbers are starting values.

Review queue mechanics

Queue population

• Each candidate writes one row into its target table (memories or entities) with status="candidate"
• Each candidate carries: rule, source_span, source_chunk_id, source_interaction_id, extractor_version
• No two candidates ever share the same (type, normalized_content, project) — if a second extraction pass produces a duplicate, it is dropped before being written

Queue surfacing

• GET /memory?status=candidate lists memory candidates
• GET /entities?status=candidate (future) lists entity candidates
• GET /candidates (future unified route) lists both

Reviewer actions

For each candidate, exactly one of:

• promote: POST /memory/{id}/promote or POST /entities/{id}/promote
  • sets status="active"
  • preserves the audit trail (source_chunk_id, rule, source_span)
• reject: POST /memory/{id}/reject or POST /entities/{id}/reject
  • sets status="invalid"
  • preserves audit trail so repeat extractions don't re-propose
• edit-then-promote: PUT /memory/{id} to adjust content, then POST /memory/{id}/promote
  • every edit is logged, original content preserved in a previous_content_log column (schema addition deferred to the first implementation sprint)
• defer: no action; candidate stays in queue indefinitely (future: add a pending_since staleness indicator to the UI)

Reviewer authentication

In V1 the review queue is single-user by convention. There is no per-reviewer authorization. Every promote/reject call is logged with the same default identity. Multi-user review is a V2 concern.

Auto-promotion policies (deferred, but designed for)

The current V1 stance is: no auto-promotion, ever. All promotions require a human reviewer.

The schema and API are designed so that automatic policies can be added later without schema changes. The anticipated policies:

1. Reference-count threshold

   • If a candidate accumulates N+ references across multiple interactions within M days AND the reviewer hasn't seen it yet (indicating the system sees it often but the human hasn't gotten to it), propose auto-promote
   • Starting thresholds: N=5, M=7 days. Never auto-promote entity candidates that affect validation claims or decisions without explicit human review — those are too consequential.

2. Confidence threshold

   • If final_confidence >= 0.85 AND the rule is a heading rule (not a sentence rule), eligible for auto-promotion

3. Identity/preference lane

   • identity and preference memories extracted from an interaction where the user explicitly says "I am X" or "I prefer X" with a first-person subject and high-signal verb could auto-promote. This is the safest lane because the user is the authoritative source for their own identity.

None of these run in V1. The APIs and data shape are designed so they can be added as a separate policy module without disrupting existing tests.

Reversibility

Every promotion step must be undoable:

Operation	How to undo
memory candidate written	delete the candidate row (low-risk, it was never in context)
memory candidate promoted	PUT /memory/{id} status=candidate (reverts to queue)
memory candidate rejected	PUT /memory/{id} status=candidate
memory graduated	memory stays as a frozen pointer; delete the entity candidate to undo
entity candidate promoted	PUT /entities/{id} status=candidate
entity promoted to active	supersede with a new active, or PUT back to candidate

The only irreversible operation is manual curation into L3 (trusted project state). That is by design — L3 is small, curated, and human-authored end to end.

Provenance (what every candidate must carry)

Every candidate row, memory or entity, MUST have:

• source_chunk_id — if extracted from ingested content, the chunk it came from
• source_interaction_id — if extracted from a captured interaction, the interaction it came from
• rule — the extractor rule id that fired
• extractor_version — a semver-ish string the extractor module carries so old candidates can be re-evaluated with a newer extractor

If both source_chunk_id and source_interaction_id are null, the candidate was hand-authored (via POST /memory directly) and must be flagged as such. Hand-authored candidates are allowed but discouraged — the preference is to extract from real content, not dictate candidates directly.

The active rows inherit all of these fields from their candidate row at promotion time. They are never overwritten.

Extractor versioning

The extractor is going to change — new rules added, old rules refined, precision/recall tuned over time. The promotion flow must survive extractor changes:

• every extractor module exposes an EXTRACTOR_VERSION = "0.1.0" constant
• every candidate row records this version
• when the extractor version changes, the change log explains what the new rules do
• old candidates are NOT automatically re-evaluated by the new extractor — that would lose the auditable history of why the old candidate was created
• future POST /memory/{id}/re-extract can optionally propose an updated candidate from the same source chunk with the new extractor, but it produces a new candidate alongside the old one, never a silent rewrite

Ingestion-wave extraction semantics

When the batched extraction pass fires on an ingestion wave, it produces a report artifact:

data/extraction-reports/<wave-id>/
  ├── report.json           # summary counts, rule distribution
  ├── candidates.ndjson     # one JSON line per persisted candidate
  ├── dropped.ndjson        # one JSON line per candidate dropped
  │                         # (over batch cap, duplicate, below
  │                         # min content length, etc.)
  └── errors.log            # any rule-level errors

The report artifact lives under the configured data_dir and is retained per the backup retention policy. The ingestion-waves doc (docs/ingestion-waves.md) is updated to include an "extract" step after each wave, with the expectation that the human reviews the candidates before the next wave fires.

Candidate-to-candidate deduplication across passes

Two extraction passes over the same chunk (or two different chunks containing the same fact) should not produce two identical candidate rows. The deduplication key is:

(memory_type_or_entity_type, normalized_content, project, status)

Normalization strips whitespace variants, lowercases, and drops trailing punctuation (same rules as the extractor's _clean_value function). If a second pass would produce a duplicate, it instead increments a re_extraction_count column on the existing candidate row and updates last_re_extracted_at. This gives the reviewer a "saw this N times" signal without flooding the queue.

This column is a future schema addition — current candidates do not track re-extraction. The promotion-rules implementation will land the column as part of its first migration.

The "never auto-promote into trusted state" invariant

Regardless of what auto-promotion policies might exist between L0 → L2', nothing ever moves into L3 (trusted project state) without explicit human action via POST /project/state. This is the one hard line in the promotion graph and it is enforced by having no API endpoint that takes a candidate id and writes to project_state.

Summary

• Four layers: L0 raw, L1 memory candidate/active, L2 entity candidate/active, L3 trusted state
• Three triggers for extraction: on capture, on ingestion wave, on explicit request
• Per-rule prior confidence, tuned by structural signals at write time
• Shared candidate review queue, promote/reject/edit/defer actions
• No auto-promotion in V1 (but the schema allows it later)
• Every candidate carries full provenance and extractor version
• Every promotion step is reversible except L3 curation
• L3 is never touched automatically