feat: implement AtoCore Phase 0 + Phase 0.5 (foundation + PoC)
Complete implementation of the personal context engine foundation:
- FastAPI server with 5 endpoints (ingest, query, context/build, health, debug)
- SQLite database with 5 tables (documents, chunks, memories, projects, interactions)
- Heading-aware markdown chunker (800 char max, recursive splitting)
- Multilingual embeddings via sentence-transformers (EN/FR)
- ChromaDB vector store with cosine similarity retrieval
- Context builder with project boosting, dedup, and budget enforcement
- CLI scripts for batch ingestion and test prompt evaluation
- 19 unit tests passing, 79% coverage
- Validated on 482 real project files (8383 chunks, 0 errors)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 09:21:27 -04:00
|
|
|
"""pytest configuration and shared fixtures."""
|
|
|
|
|
|
fix(P1+P2): canonicalize project names at every trust boundary
Three findings from codex's review of the previous P1+P2 fix. The
earlier commit (f2372ef) only fixed alias resolution at the context
builder. Codex correctly pointed out that the same fragmentation
applies at every other place a project name crosses a boundary —
project_state writes/reads, interaction capture/listing/filtering,
memory create/queries, and reinforcement's downstream queries. Plus
a real bug in the interaction `since` filter where the storage
format and the documented ISO format don't compare cleanly.
The fix is one helper used at every boundary instead of duplicating
the resolution inline.
New helper: src/atocore/projects/registry.py::resolve_project_name
---------------------------------------------------------------
- Single canonicalization boundary for project names
- Returns the canonical project_id when the input matches any
registered id or alias
- Returns the input unchanged for empty/None and for unregistered
names (preserves backwards compat with hand-curated state that
predates the registry)
- Documented as the contract that every read/write at the trust
boundary should pass through
P1 — Trusted Project State endpoints
------------------------------------
src/atocore/context/project_state.py: set_state, get_state, and
invalidate_state now all canonicalize project_name through
resolve_project_name BEFORE looking up or creating the project row.
Before this fix:
- POST /project/state with project="p05" called ensure_project("p05")
which created a separate row in the projects table
- The state row was attached to that alias project_id
- Later context builds canonicalized "p05" -> "p05-interferometer"
via the builder fix from f2372ef and never found the state
- Result: trusted state silently fragmented across alias rows
After this fix:
- The alias is resolved to the canonical id at every entry point
- Two captures (one via "p05", one via "p05-interferometer") write
to the same row
- get_state via either alias or the canonical id finds the same row
Fixes the highest-priority gap codex flagged because Trusted Project
State is supposed to be the most dependable layer in the AtoCore
trust hierarchy.
P2.a — Interaction capture project canonicalization
----------------------------------------------------
src/atocore/interactions/service.py: record_interaction now
canonicalizes project before storing, so interaction.project is
always the canonical id regardless of what the client passed.
Downstream effects:
- reinforce_from_interaction queries memories by interaction.project
-> previously missed memories stored under canonical id
-> now consistent because interaction.project IS the canonical id
- the extractor stamps candidates with interaction.project
-> previously created candidates in alias buckets
-> now creates candidates in the canonical bucket
- list_interactions(project=alias) was already broken, now fixed by
canonicalizing the filter input on the read side too
Memory service applied the same fix:
- src/atocore/memory/service.py: create_memory and get_memories
both canonicalize project through resolve_project_name
- This keeps stored memory.project consistent with the
reinforcement query path
P2.b — Interaction `since` filter format normalization
------------------------------------------------------
src/atocore/interactions/service.py: new _normalize_since helper.
The bug:
- created_at is stored as 'YYYY-MM-DD HH:MM:SS' (no timezone, UTC by
convention) so it sorts lexically and compares cleanly with the
SQLite CURRENT_TIMESTAMP default
- The `since` parameter was documented as ISO 8601 but compared as
a raw string against the storage format
- The lexically-greater 'T' separator means an ISO timestamp like
'2026-04-07T12:00:00Z' is GREATER than the storage form
'2026-04-07 12:00:00' for the same instant
- Result: a client passing ISO `since` got an empty result for any
row from the same day, even though those rows existed and were
technically "after" the cutoff in real-world time
The fix:
- _normalize_since accepts ISO 8601 with T, optional Z suffix,
optional fractional seconds, optional +HH:MM offsets
- Uses datetime.fromisoformat for parsing (Python 3.11+)
- Converts to UTC and reformats as the storage format before the
SQL comparison
- The bare storage format still works (backwards compat path is a
regex match that returns the input unchanged)
- Unparseable input is returned as-is so the comparison degrades
gracefully (rows just don't match) instead of raising and
breaking the listing endpoint
builder.py refactor
-------------------
The previous P1 fix had inline canonicalization. Now it uses the
shared helper for consistency:
- import changed from get_registered_project to resolve_project_name
- the inline lookup is replaced with a single helper call
- the comment block now points at representation-authority.md for
the canonicalization contract
New shared test fixture: tests/conftest.py::project_registry
------------------------------------------------------------
- Standardizes the registry-setup pattern that was duplicated
across test_context_builder.py, test_project_state.py,
test_interactions.py, and test_reinforcement.py
- Returns a callable that takes (project_id, [aliases]) tuples
and writes them into a temp registry file with the env var
pointed at it and config.settings reloaded
- Used by all 12 new regression tests in this commit
Tests (12 new, all green on first run)
--------------------------------------
test_project_state.py:
- test_set_state_canonicalizes_alias: write via alias, read via
every alias and the canonical id, verify same row id
- test_get_state_canonicalizes_alias_after_canonical_write
- test_invalidate_state_canonicalizes_alias
- test_unregistered_project_state_still_works (backwards compat)
test_interactions.py:
- test_record_interaction_canonicalizes_project
- test_list_interactions_canonicalizes_project_filter
- test_list_interactions_since_accepts_iso_with_t_separator
- test_list_interactions_since_accepts_z_suffix
- test_list_interactions_since_accepts_offset
- test_list_interactions_since_storage_format_still_works
test_reinforcement.py:
- test_reinforcement_works_when_capture_uses_alias (end-to-end:
capture under alias, seed memory under canonical, verify
reinforcement matches)
- test_get_memories_filter_by_alias
Full suite: 174 passing (was 162), 1 warning. The +12 is the
new regression tests, no existing tests regressed.
What's still NOT canonicalized (and why)
----------------------------------------
- _rank_chunks's secondary substring boost in builder.py — the
retriever already does the right thing via its own
_project_match_boost which calls get_registered_project. The
redundant secondary boost still uses the raw hint but it's a
multiplicative factor on top of correct retrieval, not a
filter, so it can't drop relevant chunks. Tracked as a future
cleanup but not a P1.
- update_memory's project field (you can't change a memory's
project after creation in the API anyway).
- The retriever's project_hint parameter on direct /query calls
— same reasoning as the builder boost, plus the retriever's
own get_registered_project call already handles aliases there.
2026-04-07 08:29:33 -04:00
|
|
|
import json
|
feat: implement AtoCore Phase 0 + Phase 0.5 (foundation + PoC)
Complete implementation of the personal context engine foundation:
- FastAPI server with 5 endpoints (ingest, query, context/build, health, debug)
- SQLite database with 5 tables (documents, chunks, memories, projects, interactions)
- Heading-aware markdown chunker (800 char max, recursive splitting)
- Multilingual embeddings via sentence-transformers (EN/FR)
- ChromaDB vector store with cosine similarity retrieval
- Context builder with project boosting, dedup, and budget enforcement
- CLI scripts for batch ingestion and test prompt evaluation
- 19 unit tests passing, 79% coverage
- Validated on 482 real project files (8383 chunks, 0 errors)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 09:21:27 -04:00
|
|
|
import os
|
2026-04-05 17:53:23 -04:00
|
|
|
import sys
|
feat: implement AtoCore Phase 0 + Phase 0.5 (foundation + PoC)
Complete implementation of the personal context engine foundation:
- FastAPI server with 5 endpoints (ingest, query, context/build, health, debug)
- SQLite database with 5 tables (documents, chunks, memories, projects, interactions)
- Heading-aware markdown chunker (800 char max, recursive splitting)
- Multilingual embeddings via sentence-transformers (EN/FR)
- ChromaDB vector store with cosine similarity retrieval
- Context builder with project boosting, dedup, and budget enforcement
- CLI scripts for batch ingestion and test prompt evaluation
- 19 unit tests passing, 79% coverage
- Validated on 482 real project files (8383 chunks, 0 errors)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 09:21:27 -04:00
|
|
|
import tempfile
|
|
|
|
|
from pathlib import Path
|
|
|
|
|
|
|
|
|
|
import pytest
|
|
|
|
|
|
2026-04-05 17:53:23 -04:00
|
|
|
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src"))
|
|
|
|
|
|
feat: Phase 2 Memory Core — structured memory with context integration
Memory Core implementation:
- Memory service with 6 types: identity, preference, project, episodic, knowledge, adaptation
- CRUD operations: create (with dedup), get (filtered), update, invalidate, supersede
- Confidence scoring (0.0-1.0) and lifecycle management (active/superseded/invalid)
- Memory API endpoints: POST/GET/PUT/DELETE /memory
Context builder integration (trust precedence per Master Plan):
1. Trusted Project State (highest trust, 20% budget)
2. Identity + Preference memories (10% budget)
3. Retrieved chunks (remaining budget)
Also fixed database.py to use dynamic settings reference for test isolation.
45/45 tests passing.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 09:54:52 -04:00
|
|
|
# Default test data directory — overridden per-test by fixtures
|
|
|
|
|
_default_test_dir = tempfile.mkdtemp(prefix="atocore_test_")
|
|
|
|
|
os.environ["ATOCORE_DATA_DIR"] = _default_test_dir
|
feat: implement AtoCore Phase 0 + Phase 0.5 (foundation + PoC)
Complete implementation of the personal context engine foundation:
- FastAPI server with 5 endpoints (ingest, query, context/build, health, debug)
- SQLite database with 5 tables (documents, chunks, memories, projects, interactions)
- Heading-aware markdown chunker (800 char max, recursive splitting)
- Multilingual embeddings via sentence-transformers (EN/FR)
- ChromaDB vector store with cosine similarity retrieval
- Context builder with project boosting, dedup, and budget enforcement
- CLI scripts for batch ingestion and test prompt evaluation
- 19 unit tests passing, 79% coverage
- Validated on 482 real project files (8383 chunks, 0 errors)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 09:21:27 -04:00
|
|
|
os.environ["ATOCORE_DEBUG"] = "true"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@pytest.fixture
|
|
|
|
|
def tmp_data_dir(tmp_path):
|
|
|
|
|
"""Provide a temporary data directory for tests."""
|
|
|
|
|
os.environ["ATOCORE_DATA_DIR"] = str(tmp_path)
|
|
|
|
|
# Reset singletons
|
|
|
|
|
from atocore import config
|
|
|
|
|
config.settings = config.Settings()
|
|
|
|
|
|
|
|
|
|
import atocore.retrieval.vector_store as vs
|
|
|
|
|
vs._store = None
|
|
|
|
|
|
|
|
|
|
return tmp_path
|
|
|
|
|
|
|
|
|
|
|
fix(P1+P2): canonicalize project names at every trust boundary
Three findings from codex's review of the previous P1+P2 fix. The
earlier commit (f2372ef) only fixed alias resolution at the context
builder. Codex correctly pointed out that the same fragmentation
applies at every other place a project name crosses a boundary —
project_state writes/reads, interaction capture/listing/filtering,
memory create/queries, and reinforcement's downstream queries. Plus
a real bug in the interaction `since` filter where the storage
format and the documented ISO format don't compare cleanly.
The fix is one helper used at every boundary instead of duplicating
the resolution inline.
New helper: src/atocore/projects/registry.py::resolve_project_name
---------------------------------------------------------------
- Single canonicalization boundary for project names
- Returns the canonical project_id when the input matches any
registered id or alias
- Returns the input unchanged for empty/None and for unregistered
names (preserves backwards compat with hand-curated state that
predates the registry)
- Documented as the contract that every read/write at the trust
boundary should pass through
P1 — Trusted Project State endpoints
------------------------------------
src/atocore/context/project_state.py: set_state, get_state, and
invalidate_state now all canonicalize project_name through
resolve_project_name BEFORE looking up or creating the project row.
Before this fix:
- POST /project/state with project="p05" called ensure_project("p05")
which created a separate row in the projects table
- The state row was attached to that alias project_id
- Later context builds canonicalized "p05" -> "p05-interferometer"
via the builder fix from f2372ef and never found the state
- Result: trusted state silently fragmented across alias rows
After this fix:
- The alias is resolved to the canonical id at every entry point
- Two captures (one via "p05", one via "p05-interferometer") write
to the same row
- get_state via either alias or the canonical id finds the same row
Fixes the highest-priority gap codex flagged because Trusted Project
State is supposed to be the most dependable layer in the AtoCore
trust hierarchy.
P2.a — Interaction capture project canonicalization
----------------------------------------------------
src/atocore/interactions/service.py: record_interaction now
canonicalizes project before storing, so interaction.project is
always the canonical id regardless of what the client passed.
Downstream effects:
- reinforce_from_interaction queries memories by interaction.project
-> previously missed memories stored under canonical id
-> now consistent because interaction.project IS the canonical id
- the extractor stamps candidates with interaction.project
-> previously created candidates in alias buckets
-> now creates candidates in the canonical bucket
- list_interactions(project=alias) was already broken, now fixed by
canonicalizing the filter input on the read side too
Memory service applied the same fix:
- src/atocore/memory/service.py: create_memory and get_memories
both canonicalize project through resolve_project_name
- This keeps stored memory.project consistent with the
reinforcement query path
P2.b — Interaction `since` filter format normalization
------------------------------------------------------
src/atocore/interactions/service.py: new _normalize_since helper.
The bug:
- created_at is stored as 'YYYY-MM-DD HH:MM:SS' (no timezone, UTC by
convention) so it sorts lexically and compares cleanly with the
SQLite CURRENT_TIMESTAMP default
- The `since` parameter was documented as ISO 8601 but compared as
a raw string against the storage format
- The lexically-greater 'T' separator means an ISO timestamp like
'2026-04-07T12:00:00Z' is GREATER than the storage form
'2026-04-07 12:00:00' for the same instant
- Result: a client passing ISO `since` got an empty result for any
row from the same day, even though those rows existed and were
technically "after" the cutoff in real-world time
The fix:
- _normalize_since accepts ISO 8601 with T, optional Z suffix,
optional fractional seconds, optional +HH:MM offsets
- Uses datetime.fromisoformat for parsing (Python 3.11+)
- Converts to UTC and reformats as the storage format before the
SQL comparison
- The bare storage format still works (backwards compat path is a
regex match that returns the input unchanged)
- Unparseable input is returned as-is so the comparison degrades
gracefully (rows just don't match) instead of raising and
breaking the listing endpoint
builder.py refactor
-------------------
The previous P1 fix had inline canonicalization. Now it uses the
shared helper for consistency:
- import changed from get_registered_project to resolve_project_name
- the inline lookup is replaced with a single helper call
- the comment block now points at representation-authority.md for
the canonicalization contract
New shared test fixture: tests/conftest.py::project_registry
------------------------------------------------------------
- Standardizes the registry-setup pattern that was duplicated
across test_context_builder.py, test_project_state.py,
test_interactions.py, and test_reinforcement.py
- Returns a callable that takes (project_id, [aliases]) tuples
and writes them into a temp registry file with the env var
pointed at it and config.settings reloaded
- Used by all 12 new regression tests in this commit
Tests (12 new, all green on first run)
--------------------------------------
test_project_state.py:
- test_set_state_canonicalizes_alias: write via alias, read via
every alias and the canonical id, verify same row id
- test_get_state_canonicalizes_alias_after_canonical_write
- test_invalidate_state_canonicalizes_alias
- test_unregistered_project_state_still_works (backwards compat)
test_interactions.py:
- test_record_interaction_canonicalizes_project
- test_list_interactions_canonicalizes_project_filter
- test_list_interactions_since_accepts_iso_with_t_separator
- test_list_interactions_since_accepts_z_suffix
- test_list_interactions_since_accepts_offset
- test_list_interactions_since_storage_format_still_works
test_reinforcement.py:
- test_reinforcement_works_when_capture_uses_alias (end-to-end:
capture under alias, seed memory under canonical, verify
reinforcement matches)
- test_get_memories_filter_by_alias
Full suite: 174 passing (was 162), 1 warning. The +12 is the
new regression tests, no existing tests regressed.
What's still NOT canonicalized (and why)
----------------------------------------
- _rank_chunks's secondary substring boost in builder.py — the
retriever already does the right thing via its own
_project_match_boost which calls get_registered_project. The
redundant secondary boost still uses the raw hint but it's a
multiplicative factor on top of correct retrieval, not a
filter, so it can't drop relevant chunks. Tracked as a future
cleanup but not a P1.
- update_memory's project field (you can't change a memory's
project after creation in the API anyway).
- The retriever's project_hint parameter on direct /query calls
— same reasoning as the builder boost, plus the retriever's
own get_registered_project call already handles aliases there.
2026-04-07 08:29:33 -04:00
|
|
|
@pytest.fixture
|
|
|
|
|
def project_registry(tmp_path, monkeypatch):
|
|
|
|
|
"""Stand up an isolated project registry pointing at a temp file.
|
|
|
|
|
|
|
|
|
|
Returns a callable that takes one or more (project_id, [aliases])
|
|
|
|
|
tuples and writes them into the registry, then forces the in-process
|
|
|
|
|
settings singleton to re-resolve. Use this when a test needs the
|
|
|
|
|
canonicalization helpers (resolve_project_name, get_registered_project)
|
|
|
|
|
to recognize aliases.
|
|
|
|
|
"""
|
|
|
|
|
registry_path = tmp_path / "test-project-registry.json"
|
|
|
|
|
|
|
|
|
|
def _set(*projects):
|
|
|
|
|
payload = {"projects": []}
|
|
|
|
|
for entry in projects:
|
|
|
|
|
if isinstance(entry, str):
|
|
|
|
|
project_id, aliases = entry, []
|
|
|
|
|
else:
|
|
|
|
|
project_id, aliases = entry
|
|
|
|
|
payload["projects"].append(
|
|
|
|
|
{
|
|
|
|
|
"id": project_id,
|
|
|
|
|
"aliases": list(aliases),
|
|
|
|
|
"description": f"test project {project_id}",
|
|
|
|
|
"ingest_roots": [
|
|
|
|
|
{"source": "vault", "subpath": f"incoming/projects/{project_id}"}
|
|
|
|
|
],
|
|
|
|
|
}
|
|
|
|
|
)
|
|
|
|
|
registry_path.write_text(json.dumps(payload), encoding="utf-8")
|
|
|
|
|
monkeypatch.setenv("ATOCORE_PROJECT_REGISTRY_PATH", str(registry_path))
|
|
|
|
|
from atocore import config
|
|
|
|
|
|
|
|
|
|
config.settings = config.Settings()
|
|
|
|
|
return registry_path
|
|
|
|
|
|
|
|
|
|
return _set
|
|
|
|
|
|
|
|
|
|
|
feat: implement AtoCore Phase 0 + Phase 0.5 (foundation + PoC)
Complete implementation of the personal context engine foundation:
- FastAPI server with 5 endpoints (ingest, query, context/build, health, debug)
- SQLite database with 5 tables (documents, chunks, memories, projects, interactions)
- Heading-aware markdown chunker (800 char max, recursive splitting)
- Multilingual embeddings via sentence-transformers (EN/FR)
- ChromaDB vector store with cosine similarity retrieval
- Context builder with project boosting, dedup, and budget enforcement
- CLI scripts for batch ingestion and test prompt evaluation
- 19 unit tests passing, 79% coverage
- Validated on 482 real project files (8383 chunks, 0 errors)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 09:21:27 -04:00
|
|
|
@pytest.fixture
|
|
|
|
|
def sample_markdown(tmp_path) -> Path:
|
|
|
|
|
"""Create a sample markdown file for testing."""
|
|
|
|
|
md_file = tmp_path / "test_note.md"
|
|
|
|
|
md_file.write_text(
|
|
|
|
|
"""---
|
|
|
|
|
tags:
|
|
|
|
|
- atocore
|
|
|
|
|
- architecture
|
|
|
|
|
date: 2026-04-05
|
|
|
|
|
---
|
|
|
|
|
# AtoCore Architecture
|
|
|
|
|
|
|
|
|
|
## Overview
|
|
|
|
|
|
|
|
|
|
AtoCore is a personal context engine that enriches LLM interactions
|
|
|
|
|
with durable memory, structured context, and project knowledge.
|
|
|
|
|
|
|
|
|
|
## Layers
|
|
|
|
|
|
|
|
|
|
The system has these layers:
|
|
|
|
|
|
|
|
|
|
1. Main PKM (human, messy, exploratory)
|
|
|
|
|
2. AtoVault (system mirror)
|
|
|
|
|
3. AtoDrive (trusted project truth)
|
|
|
|
|
4. Structured Memory (DB)
|
|
|
|
|
5. Semantic Retrieval (vector DB)
|
|
|
|
|
|
|
|
|
|
## Memory Types
|
|
|
|
|
|
|
|
|
|
AtoCore supports these memory types:
|
|
|
|
|
|
|
|
|
|
- Identity
|
|
|
|
|
- Preferences
|
|
|
|
|
- Project Memory
|
|
|
|
|
- Episodic Memory
|
|
|
|
|
- Knowledge Objects
|
|
|
|
|
- Adaptation Memory
|
|
|
|
|
- Trusted Project State
|
|
|
|
|
|
|
|
|
|
## Trust Precedence
|
|
|
|
|
|
|
|
|
|
When sources conflict:
|
|
|
|
|
|
|
|
|
|
1. Trusted Project State wins
|
|
|
|
|
2. AtoDrive overrides PKM
|
|
|
|
|
3. Most recent confirmed wins
|
|
|
|
|
4. Higher confidence wins
|
|
|
|
|
5. Equal → flag conflict
|
|
|
|
|
|
|
|
|
|
No silent merging.
|
|
|
|
|
""",
|
|
|
|
|
encoding="utf-8",
|
|
|
|
|
)
|
|
|
|
|
return md_file
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@pytest.fixture
|
|
|
|
|
def sample_folder(tmp_path, sample_markdown) -> Path:
|
|
|
|
|
"""Create a folder with multiple markdown files."""
|
|
|
|
|
# Already has test_note.md from sample_markdown
|
|
|
|
|
second = tmp_path / "second_note.md"
|
|
|
|
|
second.write_text(
|
|
|
|
|
"""---
|
|
|
|
|
tags:
|
|
|
|
|
- chunking
|
|
|
|
|
---
|
|
|
|
|
# Chunking Strategy
|
|
|
|
|
|
|
|
|
|
## Approach
|
|
|
|
|
|
|
|
|
|
Heading-aware recursive splitting:
|
|
|
|
|
|
|
|
|
|
1. Split on H2 boundaries first
|
|
|
|
|
2. If section > 800 chars, split on H3
|
|
|
|
|
3. If still > 800 chars, split on paragraphs
|
|
|
|
|
4. Hard split at 800 chars with 100 char overlap
|
|
|
|
|
|
|
|
|
|
## Parameters
|
|
|
|
|
|
|
|
|
|
- max_chunk_size: 800 characters
|
|
|
|
|
- overlap: 100 characters
|
|
|
|
|
- min_chunk_size: 50 characters
|
|
|
|
|
""",
|
|
|
|
|
encoding="utf-8",
|
|
|
|
|
)
|
|
|
|
|
return tmp_path
|