feat: implement AtoCore Phase 0 + Phase 0.5 (foundation + PoC)
Complete implementation of the personal context engine foundation:
- FastAPI server with 5 endpoints (ingest, query, context/build, health, debug)
- SQLite database with 5 tables (documents, chunks, memories, projects, interactions)
- Heading-aware markdown chunker (800 char max, recursive splitting)
- Multilingual embeddings via sentence-transformers (EN/FR)
- ChromaDB vector store with cosine similarity retrieval
- Context builder with project boosting, dedup, and budget enforcement
- CLI scripts for batch ingestion and test prompt evaluation
- 19 unit tests passing, 79% coverage
- Validated on 482 real project files (8383 chunks, 0 errors)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 09:21:27 -04:00
|
|
|
"""Embedding model management."""
|
|
|
|
|
|
2026-04-05 17:53:23 -04:00
|
|
|
import atocore.config as _config
|
feat: implement AtoCore Phase 0 + Phase 0.5 (foundation + PoC)
Complete implementation of the personal context engine foundation:
- FastAPI server with 5 endpoints (ingest, query, context/build, health, debug)
- SQLite database with 5 tables (documents, chunks, memories, projects, interactions)
- Heading-aware markdown chunker (800 char max, recursive splitting)
- Multilingual embeddings via sentence-transformers (EN/FR)
- ChromaDB vector store with cosine similarity retrieval
- Context builder with project boosting, dedup, and budget enforcement
- CLI scripts for batch ingestion and test prompt evaluation
- 19 unit tests passing, 79% coverage
- Validated on 482 real project files (8383 chunks, 0 errors)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 09:21:27 -04:00
|
|
|
from sentence_transformers import SentenceTransformer
|
|
|
|
|
|
|
|
|
|
from atocore.observability.logger import get_logger
|
|
|
|
|
|
|
|
|
|
log = get_logger("embeddings")
|
|
|
|
|
|
|
|
|
|
_model: SentenceTransformer | None = None
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def get_model() -> SentenceTransformer:
|
|
|
|
|
"""Load and cache the embedding model."""
|
|
|
|
|
global _model
|
|
|
|
|
if _model is None:
|
2026-04-05 17:53:23 -04:00
|
|
|
log.info("loading_embedding_model", model=_config.settings.embedding_model)
|
|
|
|
|
_model = SentenceTransformer(_config.settings.embedding_model)
|
|
|
|
|
log.info("embedding_model_loaded", model=_config.settings.embedding_model)
|
feat: implement AtoCore Phase 0 + Phase 0.5 (foundation + PoC)
Complete implementation of the personal context engine foundation:
- FastAPI server with 5 endpoints (ingest, query, context/build, health, debug)
- SQLite database with 5 tables (documents, chunks, memories, projects, interactions)
- Heading-aware markdown chunker (800 char max, recursive splitting)
- Multilingual embeddings via sentence-transformers (EN/FR)
- ChromaDB vector store with cosine similarity retrieval
- Context builder with project boosting, dedup, and budget enforcement
- CLI scripts for batch ingestion and test prompt evaluation
- 19 unit tests passing, 79% coverage
- Validated on 482 real project files (8383 chunks, 0 errors)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 09:21:27 -04:00
|
|
|
return _model
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def embed_texts(texts: list[str]) -> list[list[float]]:
|
|
|
|
|
"""Generate embeddings for a list of texts."""
|
|
|
|
|
model = get_model()
|
|
|
|
|
embeddings = model.encode(texts, show_progress_bar=False, normalize_embeddings=True)
|
|
|
|
|
return embeddings.tolist()
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def embed_query(query: str) -> list[float]:
|
|
|
|
|
"""Generate embedding for a single query."""
|
|
|
|
|
return embed_texts([query])[0]
|