Antoine/ATOCore
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: Phase 3 V1 — Auto-Organization (domain_tags + valid_until)

Browse Source 
Adds structural metadata that the LLM triage was already implicitly
reasoning about ("stale snapshot" → reject). Phase 3 captures that
reasoning as fields so it can DRIVE retrieval, not just rejection.

Schema (src/atocore/models/database.py):
- domain_tags TEXT DEFAULT '[]'  JSON array of lowercase topic keywords
- valid_until DATETIME            ISO date; null = permanent
- idx_memories_valid_until index for efficient expiry queries

Memory service (src/atocore/memory/service.py):
- Memory dataclass gains domain_tags + valid_until
- create_memory, update_memory accept/persist both
- _row_to_memory safely reads both (JSON-decode + null handling)
- _normalize_tags helper: lowercase, dedup, strip, cap at 10
- get_memories_for_context filters expired (valid_until < today UTC)
- _rank_memories_for_query adds tag-boost: memories whose domain_tags
  appear as substrings in query text rank higher (tertiary key after
  content-overlap density + absolute overlap, before confidence)

LLM extractor (_llm_prompt.py → llm-0.5.0):
- SYSTEM_PROMPT documents domain_tags (2-5 keywords) + valid_until
  (time-bounded facts get expiry dates; durable facts stay null)
- normalize_candidate_item parses both fields from model output with
  graceful fallback for string/null/missing

LLM triage (scripts/auto_triage.py):
- TRIAGE_SYSTEM_PROMPT documents same two fields
- parse_verdict extracts them from verdict JSON
- On promote: PUT /memory/{id} with tags + valid_until BEFORE
  POST /memory/{id}/promote, so active memories carry them

API (src/atocore/api/routes.py):
- MemoryCreateRequest: adds domain_tags, valid_until
- MemoryUpdateRequest: adds domain_tags, valid_until, memory_type
- GET /memory response exposes domain_tags + valid_until + created_at

Triage UI (src/atocore/engineering/triage_ui.py):
- Renders existing tags as colored badges
- Adds inline text field for tags (comma-separated) + date picker for
  valid_until on every candidate card
- Save&Promote button persists edits via PUT then promotes
- Plain Promote (and Y shortcut) also saves tags/expiry if edited

Wiki (src/atocore/engineering/wiki.py):
- Search now matches memory content OR domain_tags
- Search results render tags as clickable badges linking to
  /wiki/search?q=<tag> for cross-project navigation
- valid_until shown as amber "valid until YYYY-MM-DD" hint

Tests: 303 → 308 (5 new for Phase 3 behavior):
- test_create_memory_with_tags_and_valid_until
- test_create_memory_normalizes_tags
- test_update_memory_sets_tags_and_valid_until
- test_get_memories_for_context_excludes_expired
- test_context_builder_tag_boost_orders_results

Deferred (explicitly): temporal_scope enum, source_refs memory graph,
HDBSCAN clustering, memory detail wiki page, backfill of existing
actives. See docs/MASTER-BRAIN-PLAN.md.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Anto01
2026-04-16 21:37:01 -04:00
parent 271ee25d99
commit bfa7dba4de
9 changed files with 444 additions and 36 deletions
Download Patch File Download Diff File Expand all files Collapse all files

142
src/atocore/memory/service.py
View File

@@ -63,6 +63,30 @@ class Memory:
updated_at: str
last_referenced_at: str = ""
reference_count: int = 0
domain_tags: list[str] | None = None
valid_until: str = "" # ISO UTC; empty = permanent
def _normalize_tags(tags) -> list[str]:
"""Coerce a tags value (list, JSON string, None) to a clean lowercase list."""
import json as _json
if tags is None:
return []
if isinstance(tags, str):
try:
tags = _json.loads(tags) if tags.strip().startswith("[") else []
except Exception:
tags = []
if not isinstance(tags, list):
return []
out = []
for t in tags:
if not isinstance(t, str):
continue
t = t.strip().lower()
if t and t not in out:
out.append(t)
return out
def create_memory(
@@ -72,34 +96,36 @@ def create_memory(
source_chunk_id: str = "",
confidence: float = 1.0,
status: str = "active",
domain_tags: list[str] | None = None,
valid_until: str = "",
) -> Memory:
"""Create a new memory entry.
``status`` defaults to ``active`` for backward compatibility. Pass
``candidate`` when the memory is being proposed by the Phase 9 Commit C
extractor and still needs human review before it can influence context.
Phase 3: ``domain_tags`` is a list of lowercase domain strings
(optics, mechanics, firmware, ...) for cross-project retrieval.
``valid_until`` is an ISO UTC timestamp; memories with valid_until
in the past are excluded from context packs (but remain queryable).
"""
import json as _json
if memory_type not in MEMORY_TYPES:
raise ValueError(f"Invalid memory type '{memory_type}'. Must be one of: {MEMORY_TYPES}")
if status not in MEMORY_STATUSES:
raise ValueError(f"Invalid status '{status}'. Must be one of: {MEMORY_STATUSES}")
_validate_confidence(confidence)
# Canonicalize the project through the registry so an alias and
# the canonical id store under the same bucket. This keeps
# reinforcement queries (which use the interaction's project) and
# context retrieval (which uses the registry-canonicalized hint)
# consistent with how memories are created.
project = resolve_project_name(project)
tags = _normalize_tags(domain_tags)
tags_json = _json.dumps(tags)
valid_until = (valid_until or "").strip() or None
memory_id = str(uuid.uuid4())
now = datetime.now(timezone.utc).isoformat()
# Check for duplicate content within the same type+project at the same status.
# Scoping by status keeps active curation separate from the candidate
# review queue: a candidate and an active memory with identical text can
# legitimately coexist if the candidate is a fresh extraction of something
# already curated.
with get_connection() as conn:
existing = conn.execute(
"SELECT id FROM memories "
@@ -118,9 +144,11 @@ def create_memory(
)
conn.execute(
"INSERT INTO memories (id, memory_type, content, project, source_chunk_id, confidence, status) "
"VALUES (?, ?, ?, ?, ?, ?, ?)",
(memory_id, memory_type, content, project, source_chunk_id or None, confidence, status),
"INSERT INTO memories (id, memory_type, content, project, source_chunk_id, "
"confidence, status, domain_tags, valid_until) "
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)",
(memory_id, memory_type, content, project, source_chunk_id or None,
confidence, status, tags_json, valid_until),
)
log.info(
@@ -128,6 +156,8 @@ def create_memory(
memory_type=memory_type,
status=status,
content_preview=content[:80],
tags=tags,
valid_until=valid_until or "",
)
return Memory(
@@ -142,6 +172,8 @@ def create_memory(
updated_at=now,
last_referenced_at="",
reference_count=0,
domain_tags=tags,
valid_until=valid_until or "",
)
@@ -200,8 +232,13 @@ def update_memory(
content: str | None = None,
confidence: float | None = None,
status: str | None = None,
memory_type: str | None = None,
domain_tags: list[str] | None = None,
valid_until: str | None = None,
) -> bool:
"""Update an existing memory."""
import json as _json
with get_connection() as conn:
existing = conn.execute("SELECT * FROM memories WHERE id = ?", (memory_id,)).fetchone()
if existing is None:
@@ -235,6 +272,17 @@ def update_memory(
raise ValueError(f"Invalid status '{status}'. Must be one of: {MEMORY_STATUSES}")
updates.append("status = ?")
params.append(status)
if memory_type is not None:
if memory_type not in MEMORY_TYPES:
raise ValueError(f"Invalid memory type '{memory_type}'. Must be one of: {MEMORY_TYPES}")
updates.append("memory_type = ?")
params.append(memory_type)
if domain_tags is not None:
updates.append("domain_tags = ?")
params.append(_json.dumps(_normalize_tags(domain_tags)))
if valid_until is not None:
updates.append("valid_until = ?")
params.append(valid_until.strip() or None)
if not updates:
return False
@@ -488,8 +536,14 @@ def get_memories_for_context(
seen_ids.add(mem.id)
pool.append(mem)
# Phase 3: filter out expired memories (valid_until in the past).
# Raw API queries still return them (for audit/history) but context
# packs must not surface stale facts.
if pool:
pool = _filter_expired(pool)
if query_tokens is not None:
pool = _rank_memories_for_query(pool, query_tokens)
pool = _rank_memories_for_query(pool, query_tokens, query=query)
# Per-entry cap prevents a single long memory from monopolizing
# the band. With 16 p06 memories competing for ~700 chars, an
@@ -518,40 +572,78 @@ def get_memories_for_context(
return text, len(text)
def _filter_expired(memories: list["Memory"]) -> list["Memory"]:
"""Drop memories whose valid_until is in the past (UTC comparison)."""
now_iso = datetime.now(timezone.utc).strftime("%Y-%m-%d")
out = []
for m in memories:
vu = (m.valid_until or "").strip()
if not vu:
out.append(m)
continue
# Compare as string (ISO dates/timestamps sort correctly lexicographically
# when they have the same format; date-only vs full ts both start YYYY-MM-DD).
if vu[:10] >= now_iso:
out.append(m)
# else: expired, drop silently
return out
def _rank_memories_for_query(
memories: list["Memory"],
query_tokens: set[str],
query: str | None = None,
) -> list["Memory"]:
"""Rerank a memory list by lexical overlap with a pre-tokenized query.
Primary key: overlap_density (overlap_count / memory_token_count),
which rewards short focused memories that match the query precisely
over long overview memories that incidentally share a few tokens.
Secondary: absolute overlap count. Tertiary: confidence.
Secondary: absolute overlap count. Tertiary: domain-tag match.
Quaternary: confidence.
R7 fix: previously overlap_count alone was the primary key, so a
40-token overview memory with 3 overlapping tokens tied a 5-token
memory with 3 overlapping tokens, and the overview won on
confidence. Now the short memory's density (0.6) beats the
overview's density (0.075).
Phase 3: domain_tags contribute a boost when they appear in the
query text. A memory tagged [optics, thermal] for a query about
"optics coating" gets promoted above a memory without those tags.
Tag boost fires AFTER content-overlap density so it only breaks
ties among content-similar candidates.
"""
from atocore.memory.reinforcement import _normalize, _tokenize
scored: list[tuple[float, int, float, Memory]] = []
query_lower = (query or "").lower()
scored: list[tuple[float, int, int, float, Memory]] = []
for mem in memories:
mem_tokens = _tokenize(_normalize(mem.content))
overlap = len(mem_tokens & query_tokens) if mem_tokens else 0
density = overlap / len(mem_tokens) if mem_tokens else 0.0
scored.append((density, overlap, mem.confidence, mem))
scored.sort(key=lambda t: (t[0], t[1], t[2]), reverse=True)
return [mem for _, _, _, mem in scored]
# Tag boost: count how many of the memory's domain_tags appear
# as substrings in the raw query. Strong signal for topical match.
tag_hits = 0
for tag in (mem.domain_tags or []):
if tag and tag in query_lower:
tag_hits += 1
scored.append((density, overlap, tag_hits, mem.confidence, mem))
scored.sort(key=lambda t: (t[0], t[1], t[2], t[3]), reverse=True)
return [mem for _, _, _, _, mem in scored]
def _row_to_memory(row) -> Memory:
"""Convert a DB row to Memory dataclass."""
import json as _json
keys = row.keys() if hasattr(row, "keys") else []
last_ref = row["last_referenced_at"] if "last_referenced_at" in keys else None
ref_count = row["reference_count"] if "reference_count" in keys else 0
tags_raw = row["domain_tags"] if "domain_tags" in keys else None
try:
tags = _json.loads(tags_raw) if tags_raw else []
if not isinstance(tags, list):
tags = []
except Exception:
tags = []
valid_until = row["valid_until"] if "valid_until" in keys else None
return Memory(
id=row["id"],
memory_type=row["memory_type"],
@@ -564,6 +656,8 @@ def _row_to_memory(row) -> Memory:
updated_at=row["updated_at"],
last_referenced_at=last_ref or "",
reference_count=int(ref_count or 0),
domain_tags=tags,
valid_until=valid_until or "",
)