53 Commits

Author SHA1 Message Date
86637f8eee feat: universal LLM consumption (Phase 1 complete)
Completes the Phase 1 master brain keystone: every LLM interaction
across the ecosystem now pulls context from AtoCore automatically.

Three adapters, one HTTP backend:

1. OpenClaw plugin pull (handler.js):
   - Added before_prompt_build hook that calls /context/build and
     injects the pack via prependContext
   - Existing capture hooks (before_agent_start + llm_output)
     unchanged
   - 6s context timeout, fail-open on AtoCore unreachable
   - Deployed to T420, gateway restarted, "7 plugins loaded"

2. atocore-proxy (scripts/atocore_proxy.py):
   - Stdlib-only OpenAI-compatible HTTP middleware
   - Drop-in layer for Codex, Ollama, LiteLLM, any OpenAI-compat client
   - Intercepts /chat/completions: extracts query, pulls context,
     injects as system message, forwards to upstream, captures back
   - Fail-open: AtoCore down = passthrough without injection
   - Configurable via env: UPSTREAM, PORT, CLIENT_LABEL, INJECT, CAPTURE

3. (from prior commit c49363f) atocore-mcp:
   - stdio MCP server, stdlib Python, 7 tools exposed
   - Registered in Claude Code: "✓ Connected"

Plus quick win:
- Project synthesis moved from Sunday-only to daily cron so wiki /
  mirror pages stay fresh (Step C in batch-extract.sh). Lint stays
  weekly.

Plus docs:
- docs/universal-consumption.md: configuration guide for all 3 adapters
  with registration/env-var tables and verification checklist

Plus housekeeping:
- .gitignore: add .mypy_cache/

Tests: 303/303 passing.

This closes the consumption gap: the reinforcement feedback loop
can now actually work (memories get injected → get referenced →
reinforcement fires → auto-promotion). Every Claude, OpenClaw,
Codex, or Ollama session is automatically AtoCore-grounded.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-16 20:14:25 -04:00
c49363fccc feat: atocore-mcp server for universal LLM consumption (Phase 1)
Stdlib-only Python stdio MCP server that wraps the AtoCore HTTP
API. Makes AtoCore available as built-in tools to every MCP-aware
client (Claude Desktop, Claude Code, Cursor, Zed, Windsurf).

7 tools exposed:
- atocore_context: full context pack (state + memories + chunks)
- atocore_search: semantic retrieval with scores + sources
- atocore_memory_list: filter active memories by project/type
- atocore_memory_create: propose a candidate memory
- atocore_project_state: query Trusted Project State by category
- atocore_projects: list registered projects + aliases
- atocore_health: service status check

Design choices:
- stdlib only (no mcp SDK dep) — AtoCore philosophy
- Thin HTTP passthrough — zero business logic, zero drift risk
- Fail-open: AtoCore unreachable returns graceful error, not crash
- Protocol MCP 2024-11-05 compatible

Registered in Claude Code: `claude mcp add atocore -- python ...`
Verified: ✓ Connected, all 7 tools exposed, context/search/state
return live data from Dalidou (sha=775960c8, vectors=33253).

This is the keystone for master brain vision: every Claude session
now has AtoCore available as built-in capability without the user
or agent having to remember to invoke it.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-16 20:08:20 -04:00
33a6c61ca6 feat: daily backup to Windows main computer via pull-based scp
Third backup tier (after Dalidou local + T420 off-host): pull-based
backup to the user's Windows main computer.

- scripts/windows/atocore-backup-pull.ps1: PowerShell script using
  built-in OpenSSH scp. Fail-open: exits cleanly if Dalidou
  unreachable (e.g., laptop on the road). Pulls whole snapshots dir
  (~45MB, bounded by Dalidou's retention policy).
- docs/windows-backup-setup.md: Task Scheduler setup (automated +
  manual). Runs daily 10:00 local, catches up missed days via
  StartWhenAvailable, retries 2x on failure.

Verified: pulled 3 snapshots (45MB) to
C:\Users\antoi\Documents\ATOCore_Backups\. Task "AtoCore Backup
Pull" registered in Task Scheduler, State: Ready.

Three independent backup tiers now: Dalidou local, T420 off-host,
user Windows machine. Any two can fail without data loss.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-16 20:04:00 -04:00
33a106732f docs: master brain plan — vision, universal consumption, roadmap
Documents the path from current AtoCore (capture-only, thin
knowledge) to master brain status (universal consumption, dense
knowledge, auto-organized, self-growing, flawless).

Key strategic decisions documented:
- HTTP API is the canonical truth; every client gets a thin adapter
- MCP is for Claude ecosystem; OpenClaw plugin + middleware proxy
  handle Codex/Ollama/others
- Three-tier integration: MCP server, OpenClaw plugin, generic proxy
- Phase 1 (keystone) = universal consumption at prompt time
- 7-phase roadmap over 8-10 weeks

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-16 19:55:19 -04:00
3011aa77da fix: retry + stderr capture + pacing in triage/extractor
Both scripts now:
- Retry up to 3x with 2s/4s exponential backoff on transient
  failures (rate limits, capacity spikes)
- Capture claude CLI stderr in the error message (200 char cap)
  instead of just the exit code — diagnostics actually useful now
- Sleep 0.5s between calls to avoid bursting the backend

Context: last batch run hit 100% failure in triage (every call
exit 1) after 40% failure in extraction. claude CLI worked fine
immediately after, so the failures were capacity/rate-limit
transients. With retry + pacing these batches should complete
cleanly now. 439 candidates are already in the queue waiting
for triage.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-16 16:29:20 -04:00
ba36a28453 docs: sprint documentation — ledger + master-plan sync
Updated DEV-LEDGER orientation with post-sprint state:
- live_sha 775960c, tests 303, harness 17/18 on live
- interactions 234 (192 claude-code + 38 openclaw)
- project_state_entries 110 across 6 projects
- nightly pipeline now includes auto-promote, harness, summary

Updated master-plan-status.md "What Is Real Today" to match
actual 2026-04-16 state. Phase 10 moved from "Next" to
operational. New "Now" priorities: observe pipeline, knowledge
density, multi-model triage, fix p04-constraints.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-16 14:08:19 -04:00
999788b790 chore: OpenClaw capture handler (llm_output) + ledger sync
- openclaw-plugins/atocore-capture/handler.js: simplified version
  using before_agent_start + llm_output hooks (survives gateway
  restarts). The production copy lives on T420 at
  /tmp/atocore-openclaw-capture-plugin/openclaw-plugins/atocore-capture/
- DEV-LEDGER: updated orientation (live_sha b687e7f, capture clients)
  and session log for 2026-04-16

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-16 14:04:40 -04:00
775960c8c8 feat: "Make It Actually Useful" sprint — observability + Phase 10
Pipeline observability:
- Retrieval harness runs nightly (Step E in batch-extract.sh)
- Pipeline summary persisted to project state after each run
  (pipeline_last_run, pipeline_summary, retrieval_harness_result)
- Dashboard enhanced: interaction total + by_client, pipeline health
  (last_run, hours_since, harness results, triage stats), dynamic
  project list from registry

Phase 10 — reinforcement-based auto-promotion:
- auto_promote_reinforced(): candidates with reference_count >= 3 and
  confidence >= 0.7 auto-graduate to active
- expire_stale_candidates(): candidates unreinforced for 14+ days
  auto-rejected to prevent unbounded queue growth
- Both wired into nightly cron (Step B2)
- Batch script: scripts/auto_promote_reinforced.py (--dry-run support)

Knowledge seeding:
- scripts/seed_project_state.py: 26 curated Trusted Project State
  entries across p04-gigabit, p05-interferometer, p06-polisher,
  atomizer-v2, abb-space, atocore (decisions, requirements, facts,
  contacts, milestones)

Tests: 299 → 303 (4 new Phase 10 tests)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-16 13:59:12 -04:00
b687e7fa6f feat(capture): wire project inference from cwd
Populate _PROJECT_PATH_MAP in capture_stop.py so Claude Code
interactions get tagged with the correct project at capture time
instead of relying on the nightly LLM extractor to guess from
content. Covers 6 vault PARA sub-projects (P04, P05, P11/P06,
P08, I01, I02) and 4 local code repos (ATOCore, Polisher-Sim,
Fullum-Interferometer, Atomizer-V2).

Also sync project-registry.json with live Dalidou (adds abb-space,
atomizer-v2, and p11/polisher-fullum aliases to p06-polisher).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-16 09:01:38 -04:00
4d4d5f437a test(harness): fix p06-tailscale false positive, 18/18 PASS
The fixture's expect_absent: "GigaBIT" was catching legitimate
semantic overlap, not retrieval bleed. The p06 ARCHITECTURE.md
Overview describes the Polisher Suite as built for the GigaBIT M1
mirror — it is what the polisher is for, so the word appears
correctly in p06 content. All retrieved sources for this prompt
were genuinely p06/shared paths; zero actual p04 chunks leaked.

Narrowed the assertion to expect_absent: "[Source: p04-gigabit/",
which tests the real invariant (no p04 source chunks retrieved
into p06 context) without the false positive.

No retrieval/ranking code change. Fixture-only fix.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-15 11:23:00 -04:00
5b114baa87 docs(ledger): deploy c2e7064 live; close R10 + R13
- R10 fixed: master-plan-status Phase 8 now disclaims "primary
  integration", reports current narrow surface (14 client shapes vs
  ~44 routes, read-heavy + project-state/ingest writes).
- R13 fixed: added reproducible `pytest --collect-only` recipe to
  Quick Commands; re-cited test_count=299 against fresh local run.
- Orientation bumped: live_sha and main_tip c2e7064.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-15 11:19:55 -04:00
c2e7064238 fix(extraction): R11 container 503 + R12 shared prompt module
R11: POST /admin/extract-batch with mode=llm now returns 503 when the
claude CLI is unavailable (was silently returning success with 0
candidates), with a message pointing at the host-side script. +2 tests.

R12: extracted SYSTEM_PROMPT + parse_llm_json_array +
normalize_candidate_item + build_user_message into stdlib-only
src/atocore/memory/_llm_prompt.py. Both the container extractor and
scripts/batch_llm_extract_live.py now import from it, eliminating the
prompt/parser drift risk.

Tests 297 -> 299.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-15 10:47:01 -04:00
dc9fdd3a38 chore(ledger): end-of-session sync (2026-04-14)
Reflects today's massive work: engineering layer + wiki + Karpathy
upgrades + OpenClaw importer + auto-detection. Active memories
47 -> 84. Ready for next session to pick up cold.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-14 11:24:25 -04:00
58ea21df80 fix: triage prompt leniency for OpenClaw-curated imports (real this time)
Previous commit had the wrong message — the diff was the config
persistence fix, not triage. This properly adds rule 4 to the
triage prompt: when candidate content starts with 'From OpenClaw/',
apply a much lower bar. OpenClaw's SOUL.md, USER.md, MEMORY.md,
MODEL-ROUTING.md, and daily memory/*.md are already curated —
promote unless clearly wrong or duplicate.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-14 10:55:08 -04:00
8c0f1ff6f3 fix: triage is lenient on OpenClaw-curated content
Auto-triage was rejecting 8 of 10 OpenClaw imports as 'session log'
or 'process rule belongs elsewhere'. But OpenClaw's SOUL.md, USER.md,
MEMORY.md and daily memory/*.md files are already curated — they ARE
the canonical continuity layer we want to absorb. Applying the
conservative LLM-conversation triage bar to them discards the signal
the importer was designed to capture.

Triage prompt now has a rule 4: when candidate content starts with
'From OpenClaw/' apply a much lower bar. Session events, project
updates, stakeholder notes, and decisions from daily memory files
should promote, not reject.

The ABB-Space Schott quote that DID promote was the lucky exception
— after this fix, the other 7 daily notes (CDR execution log,
Discord migration plan, isogrid research, etc.) will promote too.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-14 10:54:17 -04:00
3db1dd99b5 fix: OpenClaw importer default path = /home/papa/clawd
The .openclaw/workspace-* dirs were empty templates. Antoine's real
OpenClaw workspace is /home/papa/clawd with SOUL.md, USER.md,
MEMORY.md, MODEL-ROUTING.md, IDENTITY.md, PROJECT_STATE.md and
rich continuity subdirs (decisions/, lessons/, knowledge/,
commitments/, preferences/, goals/, projects/, handoffs/, memory/).

First real import: 10 candidates produced from 11 files scanned.
MEMORY.md (36K chars) skipped as duplicate content; needs smarter
section-level splitting in a follow-up.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-14 10:41:49 -04:00
57b64523fb feat: OpenClaw state importer — one-way pull via SSH
scripts/import_openclaw_state.py reads the OpenClaw file continuity
layer from clawdbot (T420) via SSH and imports candidate memories
into AtoCore. Loose coupling: OpenClaw's internals don't need to
change, AtoCore pulls from stable markdown files.

Per codex's integration proposal (docs/openclaw-atocore-integration-proposal.md):

Classification:
- SOUL.md          -> identity candidate
- USER.md          -> identity candidate
- MODEL-ROUTING.md -> adaptation candidate (routing rules)
- MEMORY.md        -> memory candidate (long-term curated)
- memory/YYYY-MM-DD.md -> episodic candidate (daily logs, last 7 days)
- heartbeat-state.json -> skipped (ops metadata only, not canonical)

Delta detection: SHA-256 hash per file stored in project_state
under atocore/status/openclaw_import_hashes. Only changed files
re-import. Hashes persist across runs so no wasted work.

All imports land as status=candidate. Auto-triage filters. Nothing
auto-promotes — the importer is a signal producer, the pipeline
decides what graduates.

Discord: deferred per codex's proposal — no durable local store in
current OpenClaw snapshot. Revisit if OpenClaw exposes an export.

Wired into cron-backup.sh as Step 3a (before vault refresh +
extraction) so OpenClaw signals flow through the same pipeline.
Gated on ATOCORE_OPENCLAW_IMPORT=true (default true).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-14 10:39:27 -04:00
a13ea3b9d1 docs: propose OpenClaw one-way pull integration 2026-04-14 10:34:15 -04:00
3f23ca1bc6 feat: signal-aggressive extraction + auto vault refresh in nightly cron
Extraction prompt rewritten for signal-aggressive mode. The old prompt
rewarded silence ("durable insight only, empty is correct") which
caused quiet failures — real project signal (Schott quotes arriving,
stakeholder events, blockers) was dropped as "not architectural enough".

New prompt explicitly lists what to emit:
1. Project activity (mentions with context — quote received, blocker,
   action item)
2. Decisions and choices (architectural commitments, vendor selection)
3. Durable engineering insight (earned knowledge, generalizable)
4. Stakeholder and vendor events (emails sent, meetings scheduled)
5. Preferences and adaptations (how Antoine works)

Philosophy shift: "capture more signal, let triage filter noise"
replaces "extract only durable architectural facts". Auto-triage
already rejects noise well, so moving the filter downstream gives us
visibility into weak signals without polluting active memory.

Added 'episodic' to the candidate types list to support stakeholder
events with a timestamp feel.

LLM_EXTRACTOR_VERSION bumped to llm-0.4.0.

Also: cron-backup.sh now runs POST /ingest/sources before extraction
so new PKM files flow in automatically. Fail-open, non-blocking.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-14 10:24:50 -04:00
c1f5b3bdee feat: Karpathy-inspired upgrades — contradiction, lint, synthesis
Three additive upgrades borrowed from Karpathy's LLM Wiki pattern:

1. CONTRADICTION DETECTION: auto-triage now has a fourth verdict —
   "contradicts". When a candidate conflicts with an existing memory
   (not duplicates, genuine disagreement like "Option A selected"
   vs "Option B selected"), the triage model flags it and leaves
   it in the queue for human review instead of silently rejecting
   or double-storing. Preserves source tension rather than
   suppressing it.

2. WEEKLY LINT PASS: scripts/lint_knowledge_base.py checks for:
   - Orphan memories (active but zero references after 14 days)
   - Stale candidates (>7 days unreviewed)
   - Unused entities (no relationships)
   - Empty-state projects
   - Unregistered projects auto-detected in memories
   Runs Sundays via the cron. Outputs a report.

3. WEEKLY SYNTHESIS: scripts/synthesize_projects.py uses sonnet to
   generate a 3-5 sentence "current state" paragraph per project
   from state + memories + entities. Cached in project_state under
   status/synthesis_cache. Wiki project pages now show this at the
   top under "Current State (auto-synthesis)". Falls back to a
   deterministic summary if no cache exists.

deploy/dalidou/batch-extract.sh: added Step C (synthesis) and
Step D (lint) gated to Sundays via date check.

All additive — nothing existing changes behavior. The database
remains the source of truth; these operations just produce better
synthesized views and catch rot.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 21:08:13 -04:00
761c483474 feat: wiki homepage groups projects by stage
Projects now appear under three buckets based on their state entries:
- Active Contracts
- Leads & Prospects
- Internal Tools & Infra

Each card shows the stage as a tag on the project title, the client
as an italic subtitle, and the project description. Empty buckets
hide. Makes it obvious at a glance what's contracted vs lead vs
internal.

Paired with stage/type/client state entries added to all 6 projects
so the grouping has data to work with.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 18:47:44 -04:00
c57617f611 feat: auto-project-detection + project stages
Three changes:

1. ABB-Space registered as a lead project with stage=lead in
   Trusted Project State. Projects now have lifecycle awareness
   (lead/proposition vs active contract vs completed).

2. Extraction no longer drops unregistered project tags. When the
   LLM extractor sees a conversation about a project not in the
   registry, it keeps the model's tag on the candidate instead of
   falling back to empty. This enables auto-detection of new
   projects/leads from organic conversations. The nightly pipeline
   surfaces these candidates for triage, where the operator sees
   "hey, there's a new project called X" and can decide whether
   to register it.

3. Extraction prompt updated to tell the model: "If the conversation
   discusses a project NOT in the known list, still tag it — the
   system will auto-detect it." This removes the artificial ceiling
   that prevented new project discovery.

Updated Case D test: unregistered + unscoped now keeps the model's
tag instead of dropping to empty.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 17:16:04 -04:00
3f18ba3b35 feat: AtoCore Wiki — navigable project knowledge browser
Full wiki interface at /wiki with:

- /wiki — Homepage with project cards, search box, system stats
- /wiki/projects/{name} — Project page with clickable entity links
- /wiki/entities/{id} — Entity detail with relationships as links
- /wiki/search?q=... — Search across entities and memories

Every entity name in a project page links to its detail page.
Entity detail pages show properties, relationships as clickable
links to related entities, and breadcrumb navigation back to the
project and wiki home.

Responsive, dark-mode, mobile-friendly. Card grid for projects.
Generated on-demand from the database — always current, no static
files, source of truth is the DB.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 16:09:12 -04:00
8527c369ee fix: add markdown to pyproject.toml (container pip install reads this, not requirements.txt)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 15:37:22 -04:00
bd3dc50100 feat: HTML mirror pages — readable project dashboards in browser
GET /projects/{name}/mirror.html serves a styled HTML page rendered
from the mirror markdown. Clean typography, responsive, dark mode
support, mobile-friendly. Open from phone or desktop:

  http://dalidou:8100/projects/p04-gigabit/mirror.html
  http://dalidou:8100/projects/p05-interferometer/mirror.html
  http://dalidou:8100/projects/p06-polisher/mirror.html

Uses the markdown library for md→html conversion. Added to
requirements.txt. The JSON endpoint (/mirror) still exists for
programmatic access.

Source of truth remains the AtoCore database. The HTML page is a
derived view with a clear disclaimer.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 15:31:03 -04:00
700e3ca2c2 feat: Human Mirror — GET /projects/{name}/mirror
Layer 3 of the AtoCore architecture. Generates a human-readable
project overview in markdown from structured data:

- Trusted Project State (by category)
- System Architecture (systems → subsystems → components with
  material and interface links)
- Decisions (with affected entities)
- Requirements & Constraints
- Materials
- Vendors
- Active Memories (with confidence and reference counts)

The mirror is DERIVED — every line traces back to an entity, state
entry, or memory. The footer stamps the generation timestamp and
the "not canonical truth" disclaimer.

API: GET /projects/{project_name}/mirror returns {project, format,
content} where content is the full markdown page. Supports project
aliases via resolve_project_name.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 14:37:12 -04:00
ccc49d3a8f feat: engineering-aware context assembly
When a query matches a known engineering entity by name, the context
pack now includes a structured '--- Engineering Context ---' band
showing the entity's type, description, and its relationships to
other entities (subsystems, materials, requirements, decisions).

Six-tier context assembly:
  1. Trusted Project State
  2. Identity / Preferences
  3. Project Memories
  4. Domain Knowledge
  5. Engineering Context (NEW)
  6. Retrieved Chunks

The engineering band uses the same token-overlap scoring as memory
ranking: query tokens are matched against entity names + descriptions.
The top match gets its full relationship context included.

10% budget allocation. Trims before domain knowledge (lowest
priority of the structured tiers since the same info may appear in
chunks).

Example: query 'lateral support design' against p04-gigabit
surfaces the Lateral Support subsystem entity with its relationships
to GF-PTFE material, M1 Mirror Assembly parent system, and related
components.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 11:17:01 -04:00
3e0a357441 feat: bootstrap 35 engineering entities + relationships from project knowledge
Seeds the entity graph from existing project state, memories, and
vault docs across p04-gigabit (11 entities), p05-interferometer (10),
and p06-polisher (14). Covers systems, subsystems, components,
materials, decisions, requirements, constraints, vendors, and
parameters with structural and intent relationships.

Example: GET /entities/{M1 Mirror Assembly id} returns the full
context — 4 subsystems it contains, 2 requirements it's constrained
by, and the parent project — traversable in one API call.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 09:57:53 -04:00
dc20033a93 feat: Engineering Knowledge Layer V1 — entities + relationships
Layer 2 of the AtoCore architecture. Adds typed engineering entities
with relationships on top of the flat memory/state/chunk substrate.

Schema:
- entities table: id, entity_type, name, project, description,
  properties (JSON), status, confidence, source_refs, timestamps
- relationships table: source_entity_id, target_entity_id,
  relationship_type, confidence, source_refs

15 entity types: project, system, subsystem, component, interface,
requirement, constraint, decision, material, parameter,
analysis_model, result, validation_claim, vendor, process

12 relationship types: contains, part_of, interfaces_with,
satisfies, constrained_by, affected_by_decision, analyzed_by,
validated_by, depends_on, uses_material, described_by, supersedes

Service layer: full CRUD + get_entity_with_context (returns an
entity with its relationships and all related entities in one call).

API endpoints:
- POST /entities — create entity
- GET /entities — list/filter by type, project, status, name
- GET /entities/{id} — entity + relationships + related entities
- POST /relationships — create relationship

Schema auto-initialized on app startup via init_engineering_schema().

7 tests covering entity CRUD, relationships, context traversal,
filtering, name search, and validation.

Test count: 290 -> 297.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 09:50:58 -04:00
b86181eb6c docs: knowledge architecture — dual-layer model + domain knowledge
Comprehensive architecture doc covering:
- The problem (applied vs domain knowledge separation)
- The quality bar (earned insight vs common knowledge, with examples)
- Five-tier context assembly with budget allocation
- Knowledge domains (10 domains: physics through finance)
- Domain tag encoding (prefix in content, no schema migration)
- Full flow: capture → extract → triage → surface
- Cross-project example (p04 insight surfaces in p06 context)
- Future directions: personal branch, multi-model, reinforcement

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 09:14:32 -04:00
9118f824fa feat: dual-layer knowledge extraction + domain knowledge band
The extraction system now produces two kinds of candidates from
the same conversation:

A. PROJECT-SPECIFIC: applied facts scoped to a named project
   (unchanged behavior)
B. DOMAIN KNOWLEDGE: generalizable engineering insight earned
   through project work, tagged with a domain (physics, materials,
   optics, mechanics, manufacturing, metrology, controls, software,
   math, finance) and stored with project="" so it surfaces across
   all projects.

Critical quality bar enforced in the system prompt: "Would a
competent engineer need experience to know this, or could they
find it in 30 seconds on Google?" Textbook values, definitions,
and obvious facts are explicitly excluded. Only hard-won insight
qualifies — the kind that takes weeks of FEA or real machining
experience to discover.

Domain tags are embedded in the content as a prefix ("[physics]",
"[materials]") so they survive without a schema migration. A future
column can parse them out.

Context builder gains a new tier between project memories and
retrieved chunks:

  Tier 1: Trusted Project State     (project-specific)
  Tier 2: Identity / Preferences    (global)
  Tier 3: Project Memories          (project-specific)
  Tier 4: Domain Knowledge (NEW)    (cross-project, 10% budget)
  Tier 5: Retrieved Chunks          (project-boosted)

Trim order: chunks -> domain knowledge -> project memories ->
identity/preference -> project state.

Host-side extraction script updated with the same prompt and
domain-tag handling.

LLM_EXTRACTOR_VERSION bumped to llm-0.3.0.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 09:04:04 -04:00
db89978871 docs: full session sync — master plan + ledger + atomizer-v2 ingested
Master plan status updated to reflect current reality:
- 5 registered projects (atomizer-v2 newly ingested, 33,253 vectors)
- 47 active memories across all types
- 61 project state entries
- Nightly pipeline fully operational (both capture clients)
- 7/14 phases baseline complete
- "Now" section updated: observe/stabilize, multi-model triage,
  automated eval, atomizer state entries
- "Next" section updated: write-back, AtoDrive, hardening
- "Not Yet" items crossed off where applicable (reflection loop,
  auto-promotion, OpenClaw write-back)

DEV-LEDGER orientation fully refreshed with current vectors,
projects, pipeline state, and capture clients.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 20:32:47 -04:00
4ac4e5cc44 Merge codex/openclaw-capture-plugin — OpenClaw capture integration
Adds openclaw-plugins/atocore-capture/: a minimal OpenClaw plugin
that mirrors Claude Code's Stop hook. Captures user-triggered
assistant turns and POSTs to AtoCore /interactions with
client=openclaw, reinforce=true, fail-open.

Review verdict: functionally complete, one polish item (prompt
includes wrapper context — not blocking, extraction pipeline
handles noisy prompts). End-to-end verified on Dalidou with a
real client=openclaw interaction.

Both Claude Code and OpenClaw now feed AtoCore's reflection loop.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 18:34:47 -04:00
a6ae6166a4 feat: add OpenClaw AtoCore capture plugin 2026-04-12 22:06:07 +00:00
4f8bec7419 feat: deeper Wave 2 + observability dashboard
Wave 2 deeper ingestion:
- 6 new Trusted Project State entries from design-level docs:
  p05: test rig architecture, CGH specification, procurement combos
  p06: force control architecture, control channels, calibration loop
- Total state entries: ~23 (was ~17)

Observability:
- GET /admin/dashboard — one-shot system overview: memory counts
  by type/project/status, reinforced count, project state entry
  counts, recent interaction timestamp, extraction pipeline status.
  Replaces the need to query 4+ endpoints to understand system state.

Harness: 17/18 (no regression from new state entries).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 17:09:36 -04:00
52380a233e docs: Phase 4 baseline complete
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 16:56:24 -04:00
8b77e83f0a feat: Phase 4 — seed identity + preference memories, lower band to 5%
3 identity memories (Antoine's role, projects, infrastructure) and
3 preference memories (no API keys, multi-model collab, action bias)
seeded on live Dalidou. These fill the identity/preference band
that was previously empty.

Lowered MEMORY_BUDGET_RATIO from 0.10 to 0.05 because the 10%
allocation squeezed project memories and retrieval chunks enough
to regress 4 harness fixtures. At 5% the band fits at most 1 short
memory — enough for the most relevant identity/preference fact
without starving the project-specific tiers.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 16:48:56 -04:00
dbb8f915e2 chore(ledger): Batch 3 close — R9 fixed, before/after documented
Before: a model returning 'p04-gigabit' for a p06-polisher
interaction would silently override the known scope because the
project was registered. After: interaction.project always wins
when set. Model project is only a fallback for unscoped captures.

Not yet guaranteed: within-project semantic errors (model says
the right project but wrong content). That's a content-quality
concern, not a trust-hierarchy issue.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 15:38:19 -04:00
e5e9a9931e fix(R9): trust hierarchy for project attribution
Batch 3, Days 1-3. The core R9 failure was Case F: when the model
returned a registered project DIFFERENT from the interaction's
known scope, the old code trusted the model because the project
was registered. A p06-polisher interaction could silently produce
a p04-gigabit candidate.

New rule (trust hierarchy):
1. Interaction scope always wins when set (cases A, C, E, F)
2. Model project used only for unscoped interactions AND only when
   it resolves to a registered project (cases D, G)
3. Empty string when both are empty or unregistered (case B)

The rule is: interaction.project is the strongest signal because
it comes from the capture hook's project detection, which runs
before the LLM ever sees the content. The model's project guess
is only useful when the capture hook had no project context.

7 case tests (A-G) cover every combination of model/interaction
project state. Pre-existing tests updated for the new behavior.

Host-side script mirrors the same hierarchy using _known_projects
fetched from GET /projects at startup.

Test count: 286 -> 290 (+4 net, 7 new R9 cases, 3 old tests
consolidated).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 15:37:29 -04:00
144dbbd700 Merge codex/audit-batch2 — R7/R8 confirmed fixed, R9 stays open
Codex verified R1/R5/R7/R8 fixed, harness 17/18, auto-triage
dry-run works. R9 stays open: registered-but-wrong project from
model can still override interaction scope. Fair — the registry
check prevents hallucinated names but not misattribution between
real projects.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 15:28:00 -04:00
7650c339a2 audit: verify batch2 claims and findings 2026-04-12 19:06:51 +00:00
69c971708a feat: Day 4+5 — R7/R9 fixes + integration tests (R8)
Day 4:
- R7 fixed: overlap-density ranking. p06-firmware-interface now
  passes (was the last memory-ranking failure). Harness 16/18→17/18.
- R9 fixed: LLM extractor checks project registry before trusting
  model-supplied project. Hallucinated projects fall back to
  interaction's known scope. Registry lookup via
  load_project_registry(), matched by project_id. Host-side script
  mirrors this via GET /projects at startup.

Day 5:
- R8 addressed: 5 integration tests in test_extraction_pipeline.py
  covering the full LLM extract → persist as candidate → promote/
  reject flow, project fallback, failure handling, and dedup
  behavior. Uses mocked subprocess to avoid real claude -p calls.

Harness: 17/18 (only p06-tailscale remains — chunk bleed from
source content, not a memory/ranking issue).
Tests: 280 → 286 (+6).

Batch complete. Before/after for this batch:
  R1:  fixed (extraction pipeline operational on Dalidou)
  R5:  fixed (batch endpoint + host-side script)
  R7:  fixed (overlap-density ranking)
  R9:  fixed (project trust-preservation via registry check)
  R8:  addressed (5 integration tests)
  Harness: 16/18 → 17/18
  Active memories: 36 → 41
  Nightly pipeline: backup → cleanup → rsync → extract → auto-triage

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 14:44:02 -04:00
8951c624fe fix(R7/R9): overlap-density ranking + project trust-preservation
R7: ranking scorer now uses overlap-density (overlap_count /
memory_token_count) as primary key instead of raw overlap count.
A 5-token memory with 3 overlapping tokens (density 0.6) now beats
a 40-token overview memory with 3 overlapping tokens (density 0.075)
at the same absolute count. Secondary: absolute overlap. Tertiary:
confidence. Targeting p06-firmware-interface harness fixture.

R9: when the LLM extractor returns a project that differs from the
interaction's known project, it now checks the project registry.
If the model's project is a registered canonical ID, trust it. If
not (hallucinated name), fall back to the interaction's project.
Uses load_project_registry() for the check. The host-side script
mirrors this via an API call to GET /projects at startup.

Two new tests: test_parser_keeps_registered_model_project and
test_parser_rejects_hallucinated_project.

Test count: 280 -> 281.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 14:34:33 -04:00
1a2ee5e07f feat: Day 3 — auto-triage via LLM second pass
scripts/auto_triage.py: fetches candidate memories, asks a triage
model (claude -p, default sonnet) to classify each as promote /
reject / needs_human, and executes the verdict via the API.

Trust model:
- Auto-promote: model says promote AND confidence >= 0.8 AND
  dedup-checked against existing active memories for the project
- Auto-reject: model says reject
- needs_human: everything else stays in queue for manual review

The triage model receives both the candidate content AND a summary
of existing active memories for the same project, so it can detect
duplicates and near-duplicates. The system prompt explicitly lists
the rejection categories learned from the first two manual triage
passes (stale snapshots, impl details, planned-not-implemented,
process rules that belong in ledger not memory).

deploy/dalidou/batch-extract.sh now runs extraction (Step A) then
auto-triage (Step B) in sequence. The nightly cron at 03:00 UTC
will run the full pipeline: backup → cleanup → rsync → extract →
triage. Only needs_human candidates reach the human.

Supports --dry-run for preview without executing.
Supports --model override for multi-model triage (e.g. opus for
higher-quality review, or a future Gemini/Ollama backend).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 12:30:57 -04:00
9b149d4bfd Merge codex/audit-2026-04-12-extraction — R1+R5 fixed, R11-R12 added
Codex verified the host-side extraction pipeline works end-to-end
on Dalidou (ran it manually, produced 13 additional candidates).
R1 and R5 are now marked fixed. New findings:
- R11: container mode=llm silently returns 0 candidates
- R12: duplicated prompt/parser between host script and extractor

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 12:23:17 -04:00
abc8af5f7e audit: record extraction pipeline findings 2026-04-12 16:20:42 +00:00
ac7f77d86d fix: remove --no-session-persistence (unsupported on claude 2.0.60)
Dalidou runs Claude Code 2.0.60 which does not have this flag
(added in 2.1.x). Removed from both extractor_llm.py and the
host-side batch script. --append-system-prompt and
--disable-slash-commands are supported on 2.0.60.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 10:59:19 -04:00
719ff649a8 fix: fetch full interaction body per-id (list endpoint omits response)
GET /interactions returns response_chars but not the response body
to keep the listing lightweight. The batch extractor now lists ids
first, then fetches each interaction individually via
GET /interactions/{id} to get the full response for LLM extraction.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 10:58:00 -04:00
8af8af90d0 fix: pure-stdlib host-side extraction script (no atocore imports)
The host Python on Dalidou lacks pydantic_settings and other
container-only deps. Refactored batch_llm_extract_live.py to be
a standalone HTTP client + subprocess wrapper using only stdlib.
Duplicates the system prompt and JSON parser from extractor_llm.py
rather than importing them — acceptable duplication since this
is a deployment adapter, not a library.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 10:57:18 -04:00
cd0fd390a8 fix: host-side LLM extraction (claude CLI not in container)
The claude CLI is installed on the Dalidou HOST but not inside
the Docker container. The /admin/extract-batch API endpoint with
mode=llm silently returned 0 candidates because
shutil.which('claude') was None inside the container.

Fix: extraction runs host-side via deploy/dalidou/batch-extract.sh
which calls scripts/batch_llm_extract_live.py with the host's
PYTHONPATH pointing at the repo's src/. The script:

- Fetches interactions from the API (GET /interactions?since=...)
- Runs extract_candidates_llm() locally (host has claude CLI)
- POSTs candidates back to the API (POST /memory, status=candidate)
- Tracks last-run timestamp via project state

The cron now calls the host-side script instead of the container
API endpoint for LLM mode. Rule-mode extraction in the container
still works via /admin/extract-batch.

The API endpoint retains the mode=llm option for environments
where claude IS inside the container (future Docker image with
claude CLI, or a different deployment model).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 10:55:22 -04:00
c67bec095c feat: nightly batch extraction in cron-backup.sh (Day 2)
Step 4 added to the daily cron: POST /admin/extract-batch with
mode=llm, persist=true, limit=50. Runs after backup + cleanup +
rsync. Fail-open: extraction failure never blocks the backup.

Gated on ATOCORE_EXTRACT_BATCH=true (defaults to true). The
endpoint uses the last_extract_batch_run timestamp from project
state to auto-resume, so the cron doesn't need to track state.

curl --max-time 600 gives the LLM extractor up to 10 minutes
for the batch (50 interactions × ~20s each worst case = ~17 min,
but most will be no-ops if already extracted).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 10:51:13 -04:00
bcb7675a0d feat(R1/R5): POST /admin/extract-batch + LLM mode on single extract
Day 1 of the operational-reflection batch. Two changes:

1. POST /admin/extract-batch: batch extraction endpoint that fetches
   recent interactions (since last run or explicit 'since' param),
   runs the extractor (rule or LLM mode), and persists candidates
   with status=candidate. Tracks last-run timestamp in project state
   (atocore/status/last_extract_batch_run) so subsequent calls
   auto-resume. This is the operational home for R1/R5 — makes the
   LLM extractor an API operation, not just a script.

2. POST /interactions/{id}/extract now accepts mode: "rule" | "llm"
   (default "rule" for backward compatibility). When "llm", it uses
   extract_candidates_llm (claude -p sonnet, OAuth).

Both changes preserve the standing decision: extraction stays off
the capture hot path. The batch endpoint is invoked explicitly by
cron, manual curl, or CLI — never inline with POST /interactions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 10:45:42 -04:00
54d84b52cb Merge codex/audit-2026-04-12-final — R9-R10, state count corrections
R9 (P2): model-supplied non-empty project can override correct
interaction scope — edge case, acknowledged.
R10 (P2): Phase 8 is baseline-complete, not primary-complete —
correct characterization, already marked as Baseline Complete.
Corrected Wave 2 state counts (p04=5, p05=6, p06=6).
Confirmed live SHA drift (39d73e9 vs e2895b5) — docs-only commits
don't trigger redeploy.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 09:05:01 -04:00
48 changed files with 6555 additions and 193 deletions

1
.gitignore vendored
View File

@@ -6,6 +6,7 @@ __pycache__/
dist/ dist/
build/ build/
.pytest_cache/ .pytest_cache/
.mypy_cache/
htmlcov/ htmlcov/
.coverage .coverage
venv/ venv/

View File

@@ -6,14 +6,23 @@
## Orientation ## Orientation
- **live_sha** (Dalidou `/health` build_sha): `39d73e9` - **live_sha** (Dalidou `/health` build_sha): `775960c` (verified 2026-04-16 via /health, build_time 2026-04-16T17:59:30Z)
- **last_updated**: 2026-04-12 by Codex (audit branch `codex/audit-2026-04-12-final`) - **last_updated**: 2026-04-16 by Claude ("Make It Actually Useful" sprint — observability + Phase 10)
- **main_tip**: `e2895b5` - **main_tip**: `999788b`
- **test_count**: 280 passing - **test_count**: 303 (4 new Phase 10 tests)
- **harness**: `16/18 PASS` (p06-firmware-interface = R7 ranking tie; p06-tailscale = chunk bleed) - **harness**: `17/18 PASS` on live Dalidou (p04-constraints expects "Zerodur" — retrieval content gap, not regression)
- **active_memories**: 36 (p06-polisher 16, p05-interferometer 6, p04-gigabit 5, atocore 5, other 4) - **vectors**: 33,253
- **project_state_entries**: p04=5, p05=6, p06=6 (Wave 2 entries present on live Dalidou; 17 total visible) - **active_memories**: 84 (31 project, 23 knowledge, 10 episodic, 8 adaptation, 7 preference, 5 identity)
- **off_host_backup**: `papa@192.168.86.39:/home/papa/atocore-backups/` via cron env `ATOCORE_BACKUP_RSYNC`, verified - **candidate_memories**: 2
- **interactions**: 234 total (192 claude-code, 38 openclaw, 4 test)
- **registered_projects**: atocore, p04-gigabit, p05-interferometer, p06-polisher, atomizer-v2, abb-space (aliased p08)
- **project_state_entries**: 110 total (atocore=47, p06=19, p05=18, p04=15, abb=6, atomizer=5)
- **entities**: 35 (engineering knowledge graph, Layer 2)
- **off_host_backup**: `papa@192.168.86.39:/home/papa/atocore-backups/` via cron, verified
- **nightly_pipeline**: backup → cleanup → rsync → OpenClaw import → vault refresh → extract → auto-triage → **auto-promote/expire (NEW)** → weekly synth/lint Sundays → **retrieval harness (NEW)****pipeline summary (NEW)**
- **capture_clients**: claude-code (Stop hook + cwd project inference), openclaw (before_agent_start + llm_output plugin, verified live)
- **wiki**: http://dalidou:8100/wiki (browse), /wiki/projects/{id}, /wiki/entities/{id}, /wiki/search
- **dashboard**: http://dalidou:8100/admin/dashboard (now shows pipeline health, interaction totals by client, all registered projects)
## Active Plan ## Active Plan
@@ -121,16 +130,19 @@ One branch `codex/extractor-eval-loop` for Day 1-5, a second `codex/retrieval-ha
| id | finder | severity | file:line | summary | status | owner | opened_at | resolved_by | | id | finder | severity | file:line | summary | status | owner | opened_at | resolved_by |
|-----|--------|----------|------------------------------------|-------------------------------------------------------------------------|--------------|--------|------------|-------------| |-----|--------|----------|------------------------------------|-------------------------------------------------------------------------|--------------|--------|------------|-------------|
| R1 | Codex | P1 | deploy/hooks/capture_stop.py:76-85 | Live Claude capture still omits `extract`, so "loop closed both sides" remains overstated in practice even though the API supports it | acknowledged | Claude | 2026-04-11 | | | R1 | Codex | P1 | deploy/hooks/capture_stop.py:76-85 | Live Claude capture still omits `extract`, so "loop closed both sides" remains overstated in practice even though the API supports it | fixed | Claude | 2026-04-11 | c67bec0 |
| R2 | Codex | P1 | src/atocore/context/builder.py | Project memories excluded from pack | fixed | Claude | 2026-04-11 | 8ea53f4 | | R2 | Codex | P1 | src/atocore/context/builder.py | Project memories excluded from pack | fixed | Claude | 2026-04-11 | 8ea53f4 |
| R3 | Claude | P2 | src/atocore/memory/extractor.py | Rule cues (`## Decision:`) never fire on conversational LLM text | open | Claude | 2026-04-11 | | | R3 | Claude | P2 | src/atocore/memory/extractor.py | Rule cues (`## Decision:`) never fire on conversational LLM text | declined | Claude | 2026-04-11 | see 2026-04-14 session log |
| R4 | Codex | P2 | DEV-LEDGER.md:11 | Orientation `main_tip` was stale versus `HEAD` / `origin/main` | fixed | Codex | 2026-04-11 | 81307ce | | R4 | Codex | P2 | DEV-LEDGER.md:11 | Orientation `main_tip` was stale versus `HEAD` / `origin/main` | fixed | Codex | 2026-04-11 | 81307ce |
| R5 | Codex | P1 | src/atocore/interactions/service.py:157-174 | The deployed extraction path still calls only the rule extractor; the new LLM extractor is eval/script-only, so Day 4 "gate cleared" is true as a benchmark result but not as an operational extraction path | acknowledged | Claude | 2026-04-12 | | | R5 | Codex | P1 | src/atocore/interactions/service.py:157-174 | The deployed extraction path still calls only the rule extractor; the new LLM extractor is eval/script-only, so Day 4 "gate cleared" is true as a benchmark result but not as an operational extraction path | fixed | Claude | 2026-04-12 | c67bec0 |
| R6 | Codex | P1 | src/atocore/memory/extractor_llm.py:258-276 | LLM extraction accepts model-supplied `project` verbatim with no fallback to `interaction.project`; live triage promoted a clearly p06 memory (offline/network rule) as project=`""`, which explains the p06-offline-design harness miss and falsifies the current "all 3 failures are budget-contention" claim | fixed | Claude | 2026-04-12 | 39d73e9 | | R6 | Codex | P1 | src/atocore/memory/extractor_llm.py:258-276 | LLM extraction accepts model-supplied `project` verbatim with no fallback to `interaction.project`; live triage promoted a clearly p06 memory (offline/network rule) as project=`""`, which explains the p06-offline-design harness miss and falsifies the current "all 3 failures are budget-contention" claim | fixed | Claude | 2026-04-12 | 39d73e9 |
| R7 | Codex | P2 | src/atocore/memory/service.py:448-459 | Query ranking is overlap-count only, so broad overview memories can tie exact low-confidence memories and win on confidence; p06-firmware-interface is not just budget pressure, it also exposes a weak lexical scorer | open | Claude | 2026-04-12 | | | R7 | Codex | P2 | src/atocore/memory/service.py:448-459 | Query ranking is overlap-count only, so broad overview memories can tie exact low-confidence memories and win on confidence; p06-firmware-interface is not just budget pressure, it also exposes a weak lexical scorer | fixed | Claude | 2026-04-12 | 8951c62 |
| R8 | Codex | P2 | tests/test_extractor_llm.py:1-7 | LLM extractor tests stop at parser/failure contracts; there is no automated coverage for the script-only persistence/review path that produced the 16 promoted memories, including project-scope preservation | open | Claude | 2026-04-12 | | | R8 | Codex | P2 | tests/test_extractor_llm.py:1-7 | LLM extractor tests stop at parser/failure contracts; there is no automated coverage for the script-only persistence/review path that produced the 16 promoted memories, including project-scope preservation | fixed | Claude | 2026-04-12 | 69c9717 |
| R9 | Codex | P2 | src/atocore/memory/extractor_llm.py:258-259 | The R6 fallback only repairs empty project output. A wrong non-empty model project still overrides the interaction's known scope, so project attribution is improved but not yet trust-preserving. | open | Claude | 2026-04-12 | | | R9 | Codex | P2 | src/atocore/memory/extractor_llm.py:258-259 | The R6 fallback only repairs empty project output. A wrong non-empty model project still overrides the interaction's known scope, so project attribution is improved but not yet trust-preserving. | fixed | Claude | 2026-04-12 | e5e9a99 |
| R10 | Codex | P2 | docs/master-plan-status.md:31-33 | "Phase 8 - OpenClaw Integration" is fair as a baseline milestone, but not as a "primary" integration claim. `t420-openclaw/atocore.py` currently covers a narrow read-oriented subset (13 request shapes vs 32 API routes) plus fail-open health, while memory/interactions/admin write paths remain out of surface. | open | Claude | 2026-04-12 | | | R10 | Codex | P2 | docs/master-plan-status.md:31-33 | "Phase 8 - OpenClaw Integration" is fair as a baseline milestone, but not as a "primary" integration claim. `t420-openclaw/atocore.py` currently covers a narrow read-oriented subset (13 request shapes vs 32 API routes) plus fail-open health, while memory/interactions/admin write paths remain out of surface. | fixed | Claude | 2026-04-12 | (pending) |
| R11 | Codex | P2 | src/atocore/api/routes.py:773-845 | `POST /admin/extract-batch` still accepts `mode="llm"` inside the container and returns a successful 0-candidate result instead of surfacing that host-only LLM extraction is unavailable from this runtime. That is a misleading API contract for operators. | fixed | Claude | 2026-04-12 | (pending) |
| R12 | Codex | P2 | scripts/batch_llm_extract_live.py:39-190 | The host-side extractor duplicates the LLM system prompt and JSON parsing logic from `src/atocore/memory/extractor_llm.py`. It works today, but this is now a prompt/parser drift risk across the container and host implementations. | fixed | Claude | 2026-04-12 | (pending) |
| R13 | Codex | P2 | DEV-LEDGER.md:12 | The new `286 passing` test-count claim is not reproducibly auditable from the current audit environments: neither Dalidou nor the clean worktree has `pytest` available. The claim may be true in Claude's dev shell, but it remains unverified in this audit. | fixed | Claude | 2026-04-12 | (pending) |
## Recent Decisions ## Recent Decisions
@@ -148,6 +160,27 @@ One branch `codex/extractor-eval-loop` for Day 1-5, a second `codex/retrieval-ha
## Session Log ## Session Log
- **2026-04-16 Claude** `b687e7f..999788b` **"Make It Actually Useful" sprint.** Two-part session: ops fixes then consolidation sprint.
**Part 1 — Ops fixes:** Deployed `b687e7f` (project inference from cwd). Fixed cron logging (was `/dev/null` — redirected to `~/atocore-logs/`). Fixed OpenClaw gateway crash-loop (`discord.replyToMode: "any"` invalid → `"all"`). Deployed `atocore-capture` plugin on T420 OpenClaw using `before_agent_start` + `llm_output` hooks — verified end-to-end: 38 `client=openclaw` interactions captured. Backfilled project tags on 179/181 unscoped interactions (165 atocore, 8 p06, 6 p04).
**Part 2 — Sprint (Phase A+C):** Pipeline observability: retrieval harness now runs nightly (Step E), pipeline summary persisted to project state (Step F), dashboard enhanced with interaction totals by client + pipeline health section + dynamic project list. Phase 10 landed: `auto_promote_reinforced()` (candidate→active when reference_count≥3, confidence≥0.7) + `expire_stale_candidates()` (14-day unreinforced→auto-reject), both wired into nightly cron Step B2. Seeding script created (26 entries across 6 projects — all already existed from prior session). Tests 299→303. Harness 17/18 on live Dalidou (p04-constraints expects "Zerodur" — retrieval content gap, not regression). Deployed `775960c`.
- **2026-04-15 Claude (pm)** Closed the last harness failure honestly. **p06-tailscale fixed: 18/18 PASS.** Root-caused: not a retrieval bug — the p06 `ARCHITECTURE.md` Overview chunk legitimately mentions "the GigaBIT M1 telescope mirror" because the Polisher Suite is built *for* that mirror. All four retrieved sources for the tailscale prompt were genuinely p06/shared paths; zero actual p04 chunks leaked. The fixture's `expect_absent: GigaBIT` was catching semantic overlap, not retrieval bleed. Narrowed it to `expect_absent: "[Source: p04-gigabit/"` — a source-path check that tests the real invariant (no p04 source chunks in p06 context). Other p06 fixtures still use the word-blacklist form; they pass today because their more-specific prompts don't pull the ARCHITECTURE.md Overview, so I left them alone rather than churn fixtures that aren't failing. Did NOT change retrieval/ranking — no code change, fixture-only fix. Tests unchanged at 299.
- **2026-04-15 Claude** Deploy + doc debt sweep. Deployed `c2e7064` to Dalidou (build_time 2026-04-15T15:08:51Z, build_sha matches, /health ok) so R11/R12 are now live, not just on main. **R11 verified on live**: `POST /admin/extract-batch {"mode":"llm"}` against http://127.0.0.1:8100 returns HTTP 503 with the operator-facing "claude CLI not on PATH, run host-side script or use mode=rule" message — exactly the post-fix contract. **R13 closed (fixed)**: added a reproduction recipe to Quick Commands (`pip install -r requirements-dev.txt && pytest --collect-only -q && pytest -q`) and re-cited `test_count: 299` against a fresh local collection on 2026-04-15, so the claim is now auditable from any clean checkout — Codex's audit worktree just needs `pip install -r requirements-dev.txt`. **R10 closed (fixed)**: rewrote the `docs/master-plan-status.md` OpenClaw section to explicitly disclaim "primary integration" and report the current narrow surface: 14 client request shapes against ~44 server routes, predominantly read + `/project/state` + `/ingest/sources`, with memory/interactions/admin/entities/triage/extraction writes correctly out of scope. Open findings now: none blocking. Next natural move: the last harness failure `p06-tailscale` (chunk bleed).
- **2026-04-14 Claude (pm)** Closed R11+R12, declined R3. **R11 (fixed):** `POST /admin/extract-batch` with `mode="llm"` now returns 503 when the `claude` CLI is not on PATH, with a message pointing at the host-side script. Previously it silently returned a success-0 payload, masking host-vs-container truth. 2 new tests in `test_extraction_pipeline.py` cover the 503 path and the rule-mode-still-works path. **R12 (fixed):** extracted shared `SYSTEM_PROMPT` + `parse_llm_json_array` + `normalize_candidate_item` + `build_user_message` into stdlib-only `src/atocore/memory/_llm_prompt.py`. Both `src/atocore/memory/extractor_llm.py` (container) and `scripts/batch_llm_extract_live.py` (host) now import from it. The host script uses `sys.path` to reach the stdlib-only module without needing the full atocore package. Project-attribution policy stays path-specific (container uses registry-check; host defers to server). **R3 (declined):** rule cues not firing on conversational LLM text is by design now — the LLM extractor (llm-0.4.0) is the production path for conversational content as of the Day 4 gate (2026-04-12). Expanding rules to match conversational prose risks the FP blowup Day 2 already showed. Rule extractor stays narrow for structural PKM text. Tests 297 → 299. Live `/health` still `58ea21d`; this session's changes need deploy.
- **2026-04-14 Claude** MAJOR session: Engineering knowledge layer V1 (Layer 2) built — entity + relationship tables, 15 types, 12 relationship kinds, 35 bootstrapped entities across p04/p05/p06. Human Mirror (Layer 3) — GET /projects/{name}/mirror.html + navigable wiki at /wiki with search. Karpathy-inspired upgrades: contradiction detection in triage, weekly lint pass, weekly synthesis pass producing "current state" paragraphs at top of project pages. Auto-detection of new projects from extraction. Registry persistence fix (ATOCORE_PROJECT_REGISTRY_DIR env var). abb-space/p08 aliases added, atomizer-v2 ingested (568 docs, +12,472 vectors). Identity/preference seed (6 new), signal-aggressive extractor rewrite (llm-0.4.0), auto vault refresh in cron. **OpenClaw one-way pull importer** built per codex proposal — reads /home/papa/clawd SOUL.md, USER.md, MEMORY.md, MODEL-ROUTING.md, memory/*.md via SSH, hash-delta import, pipeline triages. First import: 10 candidates → 10 promoted with lenient triage rule. Active memories 47→84. State entries 61→78. Tests 290→297. Dashboard at /admin/dashboard. Wiki at /wiki.
- **2026-04-12 Claude** `4f8bec7..4ac4e5c` Session close. Merged OpenClaw capture plugin, ingested atomizer-v2 (568 docs, 12,472 new vectors → 33,253 total), seeded Phase 4 identity/preference memories (6 new, 47 total active), added deeper Wave 2 state entries (p05 +3, p06 +3), fixed R9 project trust hierarchy (7 case tests), built auto-triage pipeline, observability dashboard at /admin/dashboard. Updated master-plan-status.md and DEV-LEDGER.md to reflect full current state. 7/14 phases baseline complete. All P1s closed. Nightly pipeline runs unattended with both Claude Code and OpenClaw feeding the reflection loop.
- **2026-04-12 Codex (branch `codex/openclaw-capture-plugin`)** added a minimal external OpenClaw plugin at `openclaw-plugins/atocore-capture/` that mirrors Claude Code capture semantics: user-triggered assistant turns are POSTed to AtoCore `/interactions` with `client="openclaw"` and `reinforce=true`, fail-open, no extraction in-path. For live verification, temporarily added the local plugin load path to OpenClaw config and restarted the gateway so the plugin can load. Branch truth is ready; end-to-end verification still needs one fresh post-restart OpenClaw user turn to confirm new `client=openclaw` interactions appear on Dalidou.
- **2026-04-12 Claude** Batch 3 (R9 fix): `144dbbd..e5e9a99`. Trust hierarchy for project attribution — interaction scope always wins when set, model project only used for unscoped interactions + registered check. 7 case tests (A-G) cover every combination. Harness 17/18 (no regression). Tests 286->290. Before: wrong registered project could silently override interaction scope. After: interaction.project is the strongest signal; model project is only a fallback for unscoped captures. Not yet guaranteed: nothing prevents the *same* project's model output from being semantically wrong within that project. R9 marked fixed.
- **2026-04-12 Codex (audit branch `codex/audit-batch2`)** audited `69c9717..origin/main` against the current branch tip and live Dalidou. Verified: live build is `8951c62`, retrieval harness improved to **17/18 PASS**, candidate queue is now empty, active memories rose to **41**, and `python3 scripts/auto_triage.py --dry-run --base-url http://127.0.0.1:8100` runs cleanly on Dalidou but only exercised the empty-queue path. Updated R7 to **fixed** (`8951c62`) and R8 to **fixed** (`69c9717`). Kept R9 **open** because project trust-preservation still allows a wrong non-empty registered project from the model to override the interaction scope. Added R13 because the new `286 passing` claim could not be independently reproduced in this audit: `pytest` is absent on both Dalidou and the clean audit worktree. Also corrected stale Orientation fields (live SHA, main tip, harness, active/candidate memory counts).
- **2026-04-12 Codex (audit branch `codex/audit-2026-04-12-extraction`)** audited `54d84b5..ac7f77d` with live Dalidou verification. Confirmed the host-side LLM extraction pipeline is operational: nightly cron points at `deploy/dalidou/cron-backup.sh`, Step 4 calls `deploy/dalidou/batch-extract.sh`, the batch script exists/executable on Dalidou, and a manual host-side run produced candidates successfully. Updated R1 and R5 to **fixed** (`c67bec0`) because extraction now runs unattended off-container. Live state during audit: build `39d73e9`, active memories **36**, candidate queue **29** (16 existing + 13 added by manual verification run), and `last_extract_batch_run` populated in AtoCore project state. Added R11-R12 for the misleading container `mode=llm` no-op and host/container prompt-parser duplication. Security note: CLI positional prompt/response text is visible in process args while `claude -p` runs; acceptable on a single-user home host, but worth remembering if Dalidou's trust boundary changes.
- **2026-04-12 Codex (audit branch `codex/audit-2026-04-12-final`)** audited `c5bad99..e2895b5` against origin/main, live Dalidou, and the OpenClaw client script. Live state checked: build `39d73e9`, harness reproducible at **16/18 PASS**, active memories **36**, and `t420-openclaw/atocore.py health` fails open correctly with `fail_open=true`. Spot-checks of Wave 2 project-state entries matched their cited vault docs. Updated R5-R8 status reality (R6 fixed by `39d73e9`), added R9-R10, and corrected Orientation `main_tip` to `e2895b5` because the ledger had drifted behind origin/main. Note: live Dalidou is still on `39d73e9`, so branch-truth and deploy-truth are not the same yet. - **2026-04-12 Codex (audit branch `codex/audit-2026-04-12-final`)** audited `c5bad99..e2895b5` against origin/main, live Dalidou, and the OpenClaw client script. Live state checked: build `39d73e9`, harness reproducible at **16/18 PASS**, active memories **36**, and `t420-openclaw/atocore.py health` fails open correctly with `fail_open=true`. Spot-checks of Wave 2 project-state entries matched their cited vault docs. Updated R5-R8 status reality (R6 fixed by `39d73e9`), added R9-R10, and corrected Orientation `main_tip` to `e2895b5` because the ledger had drifted behind origin/main. Note: live Dalidou is still on `39d73e9`, so branch-truth and deploy-truth are not the same yet.
- **2026-04-12 Claude** Wave 2 trusted operational ingestion + codex audit response. Read 6 vault docs, created 8 new Trusted Project State entries (p04 +2, p05 +3, p06 +3). Fixed R6 (project fallback in LLM extractor) per codex audit. Fixed misscoped p06 offline memory on live Dalidou. Merged codex/audit-2026-04-12. Switched default LLM model from haiku to sonnet. Harness 15/18 -> 16/18. Tests 278 -> 280. main_tip 146f2e4 -> 39d73e9. - **2026-04-12 Claude** Wave 2 trusted operational ingestion + codex audit response. Read 6 vault docs, created 8 new Trusted Project State entries (p04 +2, p05 +3, p06 +3). Fixed R6 (project fallback in LLM extractor) per codex audit. Fixed misscoped p06 offline memory on live Dalidou. Merged codex/audit-2026-04-12. Switched default LLM model from haiku to sonnet. Harness 15/18 -> 16/18. Tests 278 -> 280. main_tip 146f2e4 -> 39d73e9.
@@ -187,4 +220,9 @@ git push origin main && ssh papa@dalidou "bash /srv/storage/atocore/app/deploy/d
python scripts/atocore_client.py batch-extract '' '' 200 false # preview python scripts/atocore_client.py batch-extract '' '' 200 false # preview
python scripts/atocore_client.py batch-extract '' '' 200 true # persist python scripts/atocore_client.py batch-extract '' '' 200 true # persist
python scripts/atocore_client.py triage python scripts/atocore_client.py triage
# Reproduce the ledger's test_count claim from a clean checkout
pip install -r requirements-dev.txt
pytest --collect-only -q | tail -1 # -> "N tests collected"
pytest -q # -> "N passed"
``` ```

View File

@@ -38,7 +38,7 @@
}, },
{ {
"id": "p06-polisher", "id": "p06-polisher",
"aliases": ["p06", "polisher"], "aliases": ["p06", "polisher", "p11", "polisher-fullum", "P11-Polisher-Fullum"],
"description": "Active P06 polisher corpus from PKM, software-suite notes, and selected repo context.", "description": "Active P06 polisher corpus from PKM, software-suite notes, and selected repo context.",
"ingest_roots": [ "ingest_roots": [
{ {
@@ -47,6 +47,30 @@
"label": "P06 staged project docs" "label": "P06 staged project docs"
} }
] ]
},
{
"id": "abb-space",
"aliases": ["abb", "abb-mirror", "p08", "p08-abb-space", "p08-abb-space-mirror"],
"description": "ABB Space mirror - lead/proposition for Atomaste. Also tracked as P08.",
"ingest_roots": [
{
"source": "vault",
"subpath": "incoming/projects/abb-space",
"label": "ABB Space docs"
}
]
},
{
"id": "atomizer-v2",
"aliases": ["atomizer", "aom", "aom-v2"],
"description": "Atomizer V2 parametric optimization platform",
"ingest_roots": [
{
"source": "vault",
"subpath": "incoming/projects/atomizer-v2/repo",
"label": "Atomizer V2 repo"
}
]
} }
] ]
} }

View File

@@ -0,0 +1,153 @@
#!/usr/bin/env bash
#
# deploy/dalidou/batch-extract.sh
# --------------------------------
# Host-side LLM batch extraction for Dalidou.
#
# The claude CLI is available on the Dalidou HOST but NOT inside the
# Docker container. This script runs on the host, fetches recent
# interactions from the AtoCore API, runs the LLM extractor locally
# (claude -p sonnet), and posts candidates back to the API.
#
# Intended to be called from cron-backup.sh after backup/cleanup/rsync,
# or manually via:
#
# bash /srv/storage/atocore/app/deploy/dalidou/batch-extract.sh
#
# Environment variables:
# ATOCORE_URL default http://127.0.0.1:8100
# ATOCORE_EXTRACT_LIMIT default 50
set -euo pipefail
ATOCORE_URL="${ATOCORE_URL:-http://127.0.0.1:8100}"
LIMIT="${ATOCORE_EXTRACT_LIMIT:-50}"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
APP_DIR="$(cd "$SCRIPT_DIR/../.." && pwd)"
TIMESTAMP="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
log() { printf '[%s] %s\n' "$TIMESTAMP" "$*"; }
# The Python script needs the atocore source on PYTHONPATH
export PYTHONPATH="$APP_DIR/src:${PYTHONPATH:-}"
log "=== AtoCore batch extraction + triage starting ==="
log "URL=$ATOCORE_URL LIMIT=$LIMIT"
# --- Pipeline stats accumulator ---
EXTRACT_OUT=""
TRIAGE_OUT=""
HARNESS_OUT=""
# Step A: Extract candidates from recent interactions
log "Step A: LLM extraction"
EXTRACT_OUT=$(python3 "$APP_DIR/scripts/batch_llm_extract_live.py" \
--base-url "$ATOCORE_URL" \
--limit "$LIMIT" \
2>&1) || {
log "WARN: batch extraction failed (non-blocking)"
}
echo "$EXTRACT_OUT"
# Step B: Auto-triage candidates in the queue
log "Step B: auto-triage"
TRIAGE_OUT=$(python3 "$APP_DIR/scripts/auto_triage.py" \
--base-url "$ATOCORE_URL" \
2>&1) || {
log "WARN: auto-triage failed (non-blocking)"
}
echo "$TRIAGE_OUT"
# Step B2: Auto-promote reinforced candidates + expire stale ones
log "Step B2: auto-promote + expire"
python3 "$APP_DIR/scripts/auto_promote_reinforced.py" \
2>&1 || {
log "WARN: auto-promote/expire failed (non-blocking)"
}
# Step C: Daily project synthesis (keeps wiki/mirror pages fresh)
log "Step C: project synthesis (daily)"
python3 "$APP_DIR/scripts/synthesize_projects.py" \
--base-url "$ATOCORE_URL" \
2>&1 || {
log "WARN: synthesis failed (non-blocking)"
}
# Step D: Weekly lint pass (Sundays only — heavier, not needed daily)
if [[ "$(date -u +%u)" == "7" ]]; then
log "Step D: weekly lint pass"
python3 "$APP_DIR/scripts/lint_knowledge_base.py" \
--base-url "$ATOCORE_URL" \
2>&1 || true
fi
# Step E: Retrieval harness (daily)
log "Step E: retrieval harness"
HARNESS_OUT=$(python3 "$APP_DIR/scripts/retrieval_eval.py" \
--json \
--base-url "$ATOCORE_URL" \
2>&1) || {
log "WARN: retrieval harness failed (non-blocking)"
}
echo "$HARNESS_OUT"
# Step F: Persist pipeline summary to project state
log "Step F: pipeline summary"
python3 -c "
import json, urllib.request, re, sys
base = '$ATOCORE_URL'
ts = '$TIMESTAMP'
def post_state(key, value):
body = json.dumps({
'project': 'atocore', 'category': 'status',
'key': key, 'value': value, 'source': 'nightly pipeline',
}).encode()
req = urllib.request.Request(
f'{base}/project/state', data=body,
headers={'Content-Type': 'application/json'}, method='POST',
)
try:
urllib.request.urlopen(req, timeout=10)
except Exception as e:
print(f'WARN: failed to persist {key}: {e}', file=sys.stderr)
# Parse harness JSON
harness = {}
try:
harness = json.loads('''$HARNESS_OUT''')
post_state('retrieval_harness_result', json.dumps({
'passed': harness.get('passed', 0),
'total': harness.get('total', 0),
'failures': [f['name'] for f in harness.get('fixtures', []) if not f.get('ok')],
'run_at': ts,
}))
p, t = harness.get('passed', '?'), harness.get('total', '?')
print(f'Harness: {p}/{t}')
except Exception:
print('WARN: could not parse harness output')
# Parse triage counts from stdout
triage_out = '''$TRIAGE_OUT'''
promoted = len(re.findall(r'promoted', triage_out, re.IGNORECASE))
rejected = len(re.findall(r'rejected', triage_out, re.IGNORECASE))
needs_human = len(re.findall(r'needs.human', triage_out, re.IGNORECASE))
# Build summary
summary = {
'run_at': ts,
'harness_passed': harness.get('passed', -1),
'harness_total': harness.get('total', -1),
'triage_promoted': promoted,
'triage_rejected': rejected,
'triage_needs_human': needs_human,
}
post_state('pipeline_last_run', ts)
post_state('pipeline_summary', json.dumps(summary))
print(f'Pipeline summary persisted: {json.dumps(summary)}')
" 2>&1 || {
log "WARN: pipeline summary persistence failed (non-blocking)"
}
log "=== AtoCore batch extraction + triage complete ==="

View File

@@ -82,4 +82,48 @@ else
log "Step 3: ATOCORE_BACKUP_RSYNC not set, skipping off-host copy" log "Step 3: ATOCORE_BACKUP_RSYNC not set, skipping off-host copy"
fi fi
# Step 3a: Pull OpenClaw state from clawdbot (one-way import of
# SOUL.md, USER.md, MODEL-ROUTING.md, MEMORY.md, recent memory/*.md).
# Loose coupling: OpenClaw's internals don't need to change.
# Fail-open: importer failure never blocks the pipeline.
log "Step 3a: pull OpenClaw state"
OPENCLAW_IMPORT="${ATOCORE_OPENCLAW_IMPORT:-true}"
if [[ "$OPENCLAW_IMPORT" == "true" ]]; then
python3 "$SCRIPT_DIR/../../scripts/import_openclaw_state.py" \
--base-url "$ATOCORE_URL" \
2>&1 | while IFS= read -r line; do log " $line"; done || {
log " WARN: OpenClaw import failed (non-blocking)"
}
else
log " skipped (ATOCORE_OPENCLAW_IMPORT != true)"
fi
# Step 3b: Auto-refresh vault sources so new PKM files flow in
# automatically. Fail-open: never blocks the rest of the pipeline.
log "Step 3b: auto-refresh vault sources"
REFRESH_RESULT=$(curl -sf -X POST --max-time 600 \
"$ATOCORE_URL/ingest/sources" 2>&1) && {
log "Sources refresh complete"
} || {
log "WARN: sources refresh failed (non-blocking): $REFRESH_RESULT"
}
# Step 4: Batch LLM extraction on recent interactions (optional).
# Runs HOST-SIDE because claude CLI is on the host, not inside the
# Docker container. The script fetches interactions from the API,
# runs claude -p locally, and POSTs candidates back.
# Fail-open: extraction failure never blocks backup.
EXTRACT="${ATOCORE_EXTRACT_BATCH:-true}"
if [[ "$EXTRACT" == "true" ]]; then
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
log "Step 4: running host-side batch LLM extraction"
bash "$SCRIPT_DIR/batch-extract.sh" 2>&1 && {
log "Extraction complete"
} || {
log "WARN: batch extraction failed (this is non-blocking)"
}
else
log "Step 4: ATOCORE_EXTRACT_BATCH not set to true, skipping extraction"
fi
log "=== AtoCore daily backup complete ===" log "=== AtoCore daily backup complete ==="

View File

@@ -166,10 +166,19 @@ def _extract_last_user_prompt(transcript_path: str) -> str:
# Project inference from working directory. # Project inference from working directory.
# Maps known repo paths to AtoCore project IDs. The user can extend # Maps known repo paths to AtoCore project IDs. The user can extend
# this table or replace it with a registry lookup later. # this table or replace it with a registry lookup later.
_VAULT = "C:\\Users\\antoi\\antoine\\My Libraries\\Antoine Brain Extension"
_PROJECT_PATH_MAP: dict[str, str] = { _PROJECT_PATH_MAP: dict[str, str] = {
# Add mappings as needed, e.g.: f"{_VAULT}\\2-Projects\\P04-GigaBIT-M1": "p04-gigabit",
# "C:\\Users\\antoi\\gigabit": "p04-gigabit", f"{_VAULT}\\2-Projects\\P10-Interferometer": "p05-interferometer",
# "C:\\Users\\antoi\\interferometer": "p05-interferometer", f"{_VAULT}\\2-Projects\\P11-Polisher-Fullum": "p06-polisher",
f"{_VAULT}\\2-Projects\\P08-ABB-Space-Mirror": "abb-space",
f"{_VAULT}\\2-Projects\\I01-Atomizer": "atomizer-v2",
f"{_VAULT}\\2-Projects\\I02-AtoCore": "atocore",
"C:\\Users\\antoi\\ATOCore": "atocore",
"C:\\Users\\antoi\\Polisher-Sim": "p06-polisher",
"C:\\Users\\antoi\\Fullum-Interferometer": "p05-interferometer",
"C:\\Users\\antoi\\Atomizer-V2": "atomizer-v2",
} }

284
docs/MASTER-BRAIN-PLAN.md Normal file
View File

@@ -0,0 +1,284 @@
# AtoCore Master Brain Plan
> Vision: AtoCore becomes the **single source of truth** that grounds every LLM
> interaction across the entire ecosystem (Claude, OpenClaw, Codex, Ollama, future
> agents). Every prompt is automatically enriched with full project context. The
> brain self-grows from daily work, auto-organizes its metadata, and stays
> flawlessly reliable.
## The Core Insight
AtoCore today is a **well-architected capture + curation system with a critical
gap on the consumption side**. We pour water into the bucket (capture from
Claude Code Stop hook + OpenClaw message hooks) but nothing is drinking from it
at prompt time. Fixing that gap is the single highest-leverage move.
**Once every LLM call is AtoCore-grounded automatically, the feedback loop
closes**: LLMs use the context → produce better responses → those responses
reference the injected memories → reinforcement fires → knowledge curates
itself. The capture side is already working. The pull side is what's missing.
## Universal Consumption Strategy
MCP is great for Claude (Claude Desktop, Claude Code, Cursor, Zed, Windsurf) but
is **not universal**. OpenClaw has its own plugin SDK. Codex, Ollama, and GPT
don't natively support MCP. The right strategy:
**HTTP API is the truth; every client gets the thinnest possible adapter.**
```
┌─────────────────────┐
│ AtoCore HTTP API │ ← canonical interface
│ /context/build │
│ /query │
│ /memory │
│ /project/state │
└──────────┬──────────┘
┌────────────┬───────────┼──────────┬────────────┐
│ │ │ │ │
┌──┴───┐ ┌────┴────┐ ┌───┴───┐ ┌───┴────┐ ┌───┴────┐
│ MCP │ │OpenClaw │ │Claude │ │ Codex │ │ Ollama │
│server│ │ plugin │ │ Code │ │ skill │ │ proxy │
│ │ │ (pull) │ │ hook │ │ │ │ │
└──┬───┘ └────┬────┘ └───┬───┘ └────┬───┘ └────┬───┘
│ │ │ │ │
Claude OpenClaw Claude Code Codex CLI Ollama
Desktop, agent local
Cursor, models
Zed,
Windsurf
```
Each adapter's only job: accept a prompt, call AtoCore HTTP, prepend the
returned context pack. The adapter itself carries no logic.
## Three Integration Tiers
### Tier 1: MCP-native clients (Claude ecosystem)
Build **atocore-mcp** — a standalone MCP server that wraps the HTTP API. Exposes:
- `context(query, project)` → context pack
- `search(query)` → raw retrieval
- `remember(type, content, project)` → create candidate memory
- `recall(project, key)` → project state lookup
- `list_projects()` → registered projects
Works with Claude Desktop, Claude Code (via `claude mcp add atocore`), Cursor,
Zed, Windsurf without any per-client work beyond config.
### Tier 2: Custom plugin ecosystems (OpenClaw)
Extend the existing `atocore-capture` plugin on T420 to also register a
**`before_prompt_build`** hook that pulls context from AtoCore and injects it
into the agent's system prompt. The plugin already has the HTTP client, the
authentication, the fail-open pattern. This is ~30 lines of added code.
### Tier 3: Everything else (Codex, Ollama, custom agents)
For clients without plugin/hook systems, ship a **thin proxy/middleware** the
user configures as the LLM endpoint:
- `atocore-proxy` listens on `localhost:PORT`
- Intercepts OpenAI-compatible chat/completion calls
- Pulls context from AtoCore, injects into system prompt
- Forwards to the real model endpoint (OpenAI, Ollama, Anthropic, etc.)
- Returns the response, then captures the interaction back to AtoCore
This makes AtoCore a "drop-in" layer for anything that speaks
OpenAI-compatible HTTP — which is nearly every modern LLM runtime.
## Knowledge Density Plan
The brain is only as smart as what it knows. Current state: 80 active memories
across 6 projects, 324 candidates in the queue being processed. Target:
**1,000+ curated memories** to become a real master brain.
Mechanisms:
1. **Finish the current triage pass** (324 → ~80 more promotions expected).
2. **Re-extract with stronger prompt on existing 236 interactions** — tune the
LLM extractor system prompt to pull more durable facts and fewer ephemeral
snapshots.
3. **Ingest all drive/vault documents as memory candidates** (not just chunks).
Every structured markdown section with a decision/fact/requirement header
becomes a candidate memory.
4. **Multi-source triangulation**: same fact in 3+ sources = auto-promote to
confidence 0.95.
5. **Cross-project synthesis**: facts appearing in multiple project contexts
get promoted to global domain knowledge.
## Auto-Organization of Metadata
Currently: `type`, `project`, `confidence`, `status`, `reference_count`. For
master brain we need more structure, inferred automatically:
| Addition | Purpose | Mechanism |
|---|---|---|
| **Domain tags** (optics, mechanics, firmware, business…) | Cross-cutting retrieval | LLM inference during triage |
| **Temporal scope** (permanent, valid_until_X, transient) | Avoid stale truth | LLM classifies during triage |
| **Source refs** (chunk_id[], interaction_id[]) | Provenance for every fact | Enforced at creation time |
| **Relationships** (contradicts, updates, depends_on) | Memory graph | Triage infers during review |
| **Semantic clusters** | Detect duplicates, find gaps | Weekly HDBSCAN pass on embeddings |
Layer these in progressively — none of them require schema rewrites, just
additional fields and batch jobs.
## Self-Growth Mechanisms
Four loops that make AtoCore grow autonomously:
### 1. Drift detection (nightly)
Compare new chunk embeddings to existing vector distribution. Centroids >X
cosine distance from any existing centroid = new knowledge area. Log to
dashboard; human decides if it's noise or a domain worth curating.
### 2. Gap identification (continuous)
Every `/context/build` logs `query + chunks_returned + memories_returned`.
Weekly report: "top 10 queries with weak coverage." Those are targeted
curation opportunities.
### 3. Multi-source triangulation (weekly)
Scan memory content similarity across sources. When a fact appears in 3+
independent sources (vault doc + drive doc + interaction), auto-promote to
high confidence and mark as "triangulated."
### 4. Active learning prompts (monthly)
Surface "you have 200 p06 memories but only 15 p04 memories. Spend 30 min
curating p04?" via dashboard digest.
## Robustness Strategy (Flawless Operation Bar)
Current: nightly backup, off-host rsync, health endpoint, 303 tests, harness,
enhanced dashboard with pipeline health (this session).
To reach "flawless":
| Gap | Fix | Priority |
|---|---|---|
| Silent pipeline failures | Alerting webhook on harness drop / pipeline skip | P1 |
| Memory mutations untracked | Append-only audit log table | P1 |
| Integrity drift | Nightly FK + vector-chunk parity checks | P1 |
| Schema migrations ad-hoc | Formal migration framework with rollback | P2 |
| Single point of failure | Daily backup to user's main computer (new) | P1 |
| No hot standby | Second instance following primary via WAL | P3 |
| No temporal history | Memory audit + valid_until fields | P2 |
### Daily Backup to Main Computer
Currently: Dalidou → T420 (192.168.86.39) via rsync.
Add: Dalidou → main computer via a pull (main computer runs the rsync,
pulls from Dalidou). Pull-based is simpler than push — no need for SSH
keys on Dalidou to reach the Windows machine.
```bash
# On main computer, daily scheduled task:
rsync -a papa@dalidou:/srv/storage/atocore/backups/snapshots/ \
/path/to/local/atocore-backups/
```
Configure via Windows Task Scheduler or a cron-like runner. Verify weekly
that the latest snapshot is present.
## Human Interface Auto-Evolution
Current: wiki at `/wiki`, regenerates on every request from DB. Synthesis
(the "current state" paragraph at top of project pages) runs **weekly on
Sundays only**. That's why it feels stalled.
Fixes:
1. **Run synthesis daily, not weekly.** It's cheap (one claude call per
project) and keeps the human-readable overview fresh.
2. **Trigger synthesis on major events** — when 5+ new memories land for a
project, regenerate its synthesis.
3. **Add "What's New" feed** — wiki homepage shows recent additions across all
projects (last 7 days of memory promotions, state entries, entities).
4. **Memory timeline view** — project page gets a chronological list of what
we learned when.
## Phased Roadmap (8-10 weeks)
### Phase 1 (week 1-2): Universal Consumption
**Goal: every LLM call is AtoCore-grounded automatically.**
- [ ] Build `atocore-mcp` server (wraps HTTP API, stdio transport)
- [ ] Publish to npm / or run via `pipx` / stdlib HTTP
- [ ] Configure in Claude Desktop (`~/.claude/mcp_servers.json`)
- [ ] Configure in Claude Code (`claude mcp add atocore …`)
- [ ] Extend OpenClaw plugin with `before_prompt_build` PULL
- [ ] Write `atocore-proxy` middleware for Codex/Ollama/generic clients
- [ ] Document configuration for each client
**Success:** open a fresh Claude Code session, ask a project question, verify
the response references AtoCore memories without manual context commands.
### Phase 2 (week 2-3): Knowledge Density + Wiki Evolution
- [ ] Finish current triage pass (324 candidates → active)
- [ ] Tune extractor prompt for higher promotion rate on durable facts
- [ ] Daily synthesis in cron (not just Sundays)
- [ ] Event-triggered synthesis on significant project changes
- [ ] Wiki "What's New" feed
- [ ] Memory timeline per project
**Target:** 300+ active memories, wiki feels alive daily.
### Phase 3 (week 3-4): Auto-Organization
- [ ] Schema: add `domain_tags`, `valid_until`, `source_refs`, `triangulated_count`
- [ ] Triage prompt upgraded: infer tags + temporal scope + relationships
- [ ] Weekly HDBSCAN clustering of embeddings → dup detection + gap reports
- [ ] Relationship edges in a new `memory_relationships` table
### Phase 4 (week 4-5): Robustness Hardening
- [ ] Append-only `memory_audit` table + retrofit mutations
- [ ] Nightly integrity checks (FK validation, orphan detection, parity)
- [ ] Alerting webhook (Discord/email) on pipeline anomalies
- [ ] Daily backup to user's main computer (pull-based)
- [ ] Formal migration framework
### Phase 5 (week 6-7): Engineering V1 Implementation
Execute the 23 acceptance criteria in `docs/architecture/engineering-v1-acceptance.md`
against p06-polisher as the test bed. The ontology and queries are designed;
this phase implements them.
### Phase 6 (week 8-9): Self-Growth Loops
- [ ] Drift detection (nightly)
- [ ] Gap identification from `/context/build` logs
- [ ] Multi-source triangulation
- [ ] Active learning digest (monthly)
- [ ] Cross-project synthesis
### Phase 7 (ongoing): Scale & Polish
- [ ] Multi-model validation (sonnet triages, opus cross-checks on disagreements)
- [ ] AtoDrive integration (Google Drive as trusted source)
- [ ] Hot standby when real production dependence materializes
- [ ] More MCP tools (write-back, memory search, entity queries)
## Success Criteria
AtoCore is a master brain when:
1. **Zero manual context commands.** A fresh Claude/OpenClaw session answering
a project question without being told "use AtoCore context."
2. **1,000+ active memories** with >90% provenance coverage (every fact
traceable to a source).
3. **Every project has a current, human-readable overview** updated within 24h
of significant changes.
4. **Harness stays >95%** across 20+ fixtures covering all active projects.
5. **Zero silent pipeline failures** for 30 consecutive days (all failures
surface via alert within the hour).
6. **Claude on any task knows what we know** — user asks "what did we decide
about X?" and the answer is grounded in AtoCore, not reconstructed from
scratch.
## Where We Are Now (2026-04-16)
- ✅ Core infrastructure: HTTP API, SQLite, Chroma, deploy pipeline
- ✅ Capture pipes: Claude Code Stop hook, OpenClaw message hooks
- ✅ Nightly pipeline: backup, extract, triage, synthesis, lint, harness, summary
- ✅ Phase 10: auto-promotion from reinforcement + candidate expiry
- ✅ Dashboard shows pipeline health + interaction totals + all projects
- ⚡ 324 candidates being triaged (down from 439), ~80 active memories, growing
- ❌ No consumption at prompt time (capture-only)
- ❌ Wiki auto-evolves only on Sundays (synthesis cadence)
- ❌ No MCP adapter
- ❌ No daily backup to main computer
- ❌ Engineering V1 not implemented
- ❌ No alerting on pipeline failures
The path is clear. Phase 1 is the keystone.

View File

@@ -0,0 +1,206 @@
# AtoCore Knowledge Architecture
## The Problem
Engineering work produces two kinds of knowledge simultaneously:
1. **Applied knowledge** — specific to the project being worked on
("the p04 support pad layout is driven by CTE gradient analysis")
2. **Domain knowledge** — generalizable insight earned through that work
("Zerodur CTE gradient dominates WFE at fast focal ratios")
A system that only stores applied knowledge loses the general insight.
A system that mixes them pollutes project context with cross-project
noise. AtoCore needs both — separated, but both growing organically
from the same conversations.
## The Quality Bar
**AtoCore stores earned insight, not information.**
The test: "Would a competent engineer need experience to know this,
or could they find it in 30 seconds?"
| Store | Don't store |
|-------|-------------|
| "Preston removal model breaks down below 5N because the contact assumption fails" | "Preston's equation relates removal rate to pressure and velocity" |
| "m=1 (coma) is NOT correctable by force modulation (score 0.09)" | "Zernike polynomials describe wavefront aberrations" |
| "At F/1.2, CTE gradient costs ~3nm WFE and drives pad placement" | "Zerodur CTE is 0.05 ppm/K" |
| "Quilting limit for 16-inch tool is 234N" | "Quilting is a mid-spatial-frequency artifact in polishing" |
The bar is enforced in the LLM extraction system prompt
(`src/atocore/memory/extractor_llm.py`) and the auto-triage prompt
(`scripts/auto_triage.py`). Both explicitly list examples of what
qualifies and what doesn't.
## Architecture
### Five-tier context assembly
When AtoCore builds a context pack for any LLM query, it assembles
five tiers in strict trust order:
```
Tier 1: Trusted Project State [project-specific, highest trust]
Curated key-value entries from the project state API.
Example: "decision/vendor_path: Twyman-Green preferred, 4D
technical lead but cost-challenged"
Tier 2: Identity / Preferences [global, always included]
Who the user is and how they work.
Example: "Antoine Letarte, mechanical/optical engineer at
Atomaste" / "No API keys — uses OAuth exclusively"
Tier 3: Project Memories [project-specific]
Reinforced memories from the reflection loop, scoped to the
queried project. Example: "Firmware interface contract is
invariant: controller-job.v1 in, run-log.v1 out"
Tier 4: Domain Knowledge [cross-project]
Earned engineering insight with project="" and a domain tag.
Surfaces in ALL project packs when query-relevant.
Example: "[materials] Zerodur CTE gradient dominates WFE at
fast focal ratios — costs ~3nm at F/1.2"
Tier 5: Retrieved Chunks [project-boosted, lowest trust]
Vector-similarity search over the ingested document corpus.
Project-hinted but not filtered — cross-project docs can
appear at lower rank.
```
### Budget allocation (at default 3000 chars)
| Tier | Budget ratio | Approx chars | Entries |
|------|-------------|-------------|---------|
| Project State | 20% | 600 | all curated entries |
| Identity/Preferences | 5% | 150 | 1 memory |
| Project Memories | 25% | 750 | 2-3 memories |
| Domain Knowledge | 10% | 300 | 1-2 memories |
| Retrieved Chunks | 40% | 1200 | 2-4 chunks |
Trim order when budget is tight: chunks first, then domain knowledge,
then project memories, then identity, then project state last.
### Knowledge domains
The LLM extractor tags domain knowledge with one of these domains:
| Domain | What qualifies |
|--------|---------------|
| `physics` | Optical physics, wave propagation, diffraction, thermal effects |
| `materials` | Material properties in context, CTE behavior, stress limits |
| `optics` | Lens/mirror design, aberration analysis, metrology techniques |
| `mechanics` | Structural FEA insights, support system design, kinematics |
| `manufacturing` | Polishing, grinding, machining, process control |
| `metrology` | Measurement systems, interferometry, calibration techniques |
| `controls` | PID tuning, force control, servo systems, real-time constraints |
| `software` | Architecture patterns, testing strategies, deployment insights |
| `math` | Numerical methods, optimization, statistical analysis |
| `finance` | Cost modeling, procurement strategy, budget optimization |
New domains can be added by updating the system prompt in
`extractor_llm.py` and `batch_llm_extract_live.py`.
### How domain knowledge is stored
Domain tags are embedded as a prefix in the memory content:
```
memory_type: knowledge
project: "" ← empty = cross-project
content: "[materials] Zerodur CTE gradient dominates WFE at F/1.2"
```
The `[domain]` prefix is a lightweight encoding that avoids a schema
migration. The context builder's query-relevance ranking matches on
domain terms naturally (a query about "materials" or "CTE" will rank
a `[materials]` memory higher). A future migration can parse the
prefix into a proper `domain` column.
## How knowledge flows
### Capture → Extract → Triage → Surface
```
1. CAPTURE
Claude Code (Stop hook) or OpenClaw (plugin)
→ POST /interactions with reinforce=true
→ Interaction stored on Dalidou
2. EXTRACT (nightly cron, 03:00 UTC)
batch_llm_extract_live.py runs claude -p sonnet
→ For each interaction, the LLM decides:
- Is this project-specific? → candidate with project=X
- Is this generalizable insight? → candidate with domain=Y, project=""
- Is it both? → TWO candidates emitted
- Is it common knowledge? → skip (quality bar)
→ Candidates persisted as status=candidate
3. TRIAGE (nightly, immediately after extraction)
auto_triage.py runs claude -p sonnet
→ Each candidate classified: promote / reject / needs_human
→ Auto-promote at confidence ≥ 0.8 + no duplicate
→ Auto-reject stale snapshots, duplicates, common knowledge
→ Only needs_human reaches the operator
4. SURFACE (every context/build query)
→ Project-specific memories appear in Tier 3
→ Domain knowledge appears in Tier 4 (regardless of project)
→ Both are query-ranked by overlap-density
```
### Example: knowledge earned on p04 surfaces on p06
Working on p04-gigabit, you discover that Zerodur CTE gradient is
the dominant WFE contributor at fast focal ratios. The extraction
produces:
```json
[
{"type": "project", "content": "CTE gradient analysis drove the
M1 support pad layout — 2nd largest WFE contributor after gravity",
"project": "p04-gigabit", "domain": "", "confidence": 0.6},
{"type": "knowledge", "content": "Zerodur CTE gradient dominates
WFE contribution at fast focal ratios (F/1.2 = ~3nm)",
"project": "", "domain": "materials", "confidence": 0.6}
]
```
Two weeks later, working on p06-polisher (which also uses Zerodur):
```
Query: "thermal effects on polishing accuracy"
Project: p06-polisher
Tier 3 (Project Memories):
[project] Calibration loop adjusts Preston kp from surface measurements...
Tier 4 (Domain Knowledge):
[materials] Zerodur CTE gradient dominates WFE contribution at fast
focal ratios — THIS CAME FROM P04 WORK
```
The insight crosses over without any manual curation.
## Future directions
### Personal knowledge branch
The same architecture supports personal domains (health, finance,
personal) by adding new domain tags and a trust boundary so
Atomaste project data never leaks into personal packs. The domain
system is domain-agnostic — it doesn't care whether the domain is
"optics" or "nutrition".
### Multi-model extraction
Different models can specialize: sonnet for extraction, opus or
Gemini for triage review. Independent validation reduces correlated
blind spots on what qualifies as "earned insight" vs "common
knowledge."
### Reinforcement-based domain promotion
A domain-knowledge memory that gets reinforced across multiple
projects (its content echoed in p04, p05, and p06 responses)
accumulates confidence faster than a project-specific memory.
High-confidence domain memories could auto-promote to a "verified
knowledge" tier above regular domain knowledge.

View File

@@ -24,21 +24,30 @@ read-only additive mode.
- Phase 5 - Project State - Phase 5 - Project State
- Phase 7 - Context Builder - Phase 7 - Context Builder
### Partial
- Phase 4 - Identity / Preferences
### Baseline Complete ### Baseline Complete
- Phase 8 - OpenClaw Integration. As of 2026-04-12 the T420 OpenClaw - Phase 4 - Identity / Preferences. As of 2026-04-12: 3 identity
helper (`t420-openclaw/atocore.py`) is verified end-to-end against memories (role, projects, infrastructure) and 3 preference memories
live Dalidou: health check, auto-context with project detection, (no API keys, multi-model collab, action-over-discussion) seeded
Trusted Project State surfacing, project-memory band, fail-open on on live Dalidou. Identity/preference band surfaces in context packs
unreachable host. Tested from both the development machine and the at 5% budget ratio. Future identity/preference extraction happens
T420 via SSH. The helper covers 15 of the 33 API endpoints — the organically via the nightly LLM extraction pipeline.
excluded endpoints (memory management, interactions, backup) are
correctly scoped to the operator client (`scripts/atocore_client.py`) - Phase 8 - OpenClaw Integration (baseline only, not primary surface).
per the read-only additive integration model. As of 2026-04-15 the T420 OpenClaw helper (`t420-openclaw/atocore.py`)
is verified end-to-end against live Dalidou: health check, auto-context
with project detection, Trusted Project State surfacing, project-memory
band, fail-open on unreachable host. Tested from both the development
machine and the T420 via SSH. Scope is narrow: **14 request shapes
against ~44 server routes**, predominantly read-oriented plus
`POST/DELETE /project/state` and `POST /ingest/sources`. Memory
management, interactions capture (covered separately by the OpenClaw
capture plugin), admin/backup, entities, triage, and extraction write
paths remain out of this client's surface by design — they are scoped
to the operator client (`scripts/atocore_client.py`) per the
read-heavy additive integration model. "Primary integration" is
therefore overclaim; "baseline read + project-state write helper" is
the accurate framing.
### Baseline Complete ### Baseline Complete
@@ -117,59 +126,58 @@ This sits implicitly between Phase 8 (OpenClaw) and Phase 11
(multi-model). Memory-review and engineering-entity commands are (multi-model). Memory-review and engineering-entity commands are
deferred from the shared client until their workflows are exercised. deferred from the shared client until their workflows are exercised.
## What Is Real Today ## What Is Real Today (updated 2026-04-16)
- canonical AtoCore runtime on Dalidou - canonical AtoCore runtime on Dalidou (`775960c`, deploy.sh verified)
- canonical machine DB and vector store on Dalidou - 33,253 vectors across 6 registered projects
- project registry with: - 234 captured interactions (192 claude-code, 38 openclaw, 4 test)
- template - 6 registered projects:
- proposal preview - `p04-gigabit` (483 docs, 15 state entries)
- register - `p05-interferometer` (109 docs, 18 state entries)
- update - `p06-polisher` (564 docs, 19 state entries)
- refresh - `atomizer-v2` (568 docs, 5 state entries)
- read-only additive OpenClaw helper on the T420 - `abb-space` (6 state entries)
- seeded project corpus for: - `atocore` (drive source, 47 state entries)
- `p04-gigabit` - 110 Trusted Project State entries across all projects (decisions, requirements, facts, contacts, milestones)
- `p05-interferometer` - 84 active memories (31 project, 23 knowledge, 10 episodic, 8 adaptation, 7 preference, 5 identity)
- `p06-polisher` - context pack assembly with 4 tiers: Trusted Project State > identity/preference > project memories > retrieved chunks
- conservative Trusted Project State for those active projects - query-relevance memory ranking with overlap-density scoring
- first operational backup foundation for SQLite + project registry - retrieval eval harness: 18 fixtures, 17/18 passing on live
- implementation-facing architecture notes for future engineering knowledge work - 303 tests passing
- first organic routing layer in OpenClaw via: - nightly pipeline: backup → cleanup → rsync → OpenClaw import → vault refresh → extract → triage → **auto-promote/expire** → weekly synth/lint → **retrieval harness****pipeline summary to project state**
- `detect-project` - Phase 10 operational: reinforcement-based auto-promotion (ref_count ≥ 3, confidence ≥ 0.7) + stale candidate expiry (14 days unreinforced)
- `auto-context` - pipeline health visible in dashboard: interaction totals by client, pipeline last_run, harness results, triage stats
- off-host backup to clawdbot (T420) via rsync
- both Claude Code and OpenClaw capture interactions to AtoCore (OpenClaw via `before_agent_start` + `llm_output` plugin, verified live)
- DEV-LEDGER.md as shared operating memory between Claude and Codex
- observability dashboard at GET /admin/dashboard
## Now ## Now
These are the current practical priorities. These are the current practical priorities.
1. Finish practical OpenClaw integration 1. **Observe the enhanced pipeline** — let the nightly pipeline run for a
- make the helper lifecycle feel natural in daily use week with the new harness + summary + auto-promote steps. Check the
- use the new organic routing layer for project-knowledge questions dashboard daily. Verify pipeline summary populates correctly.
- confirm fail-open behavior remains acceptable 2. **Knowledge density** — run batch extraction over the full 234
- keep AtoCore clearly additive interactions (`--since 2026-01-01`) to mine the backlog for knowledge.
2. Tighten retrieval quality Target: 100+ active memories.
- reduce cross-project competition 3. **Multi-model triage** (Phase 11 entry) — switch auto-triage to a
- improve ranking on short or ambiguous prompts different model than the extractor for independent validation
- add only a few anchor docs where retrieval is still weak 4. **Fix p04-constraints harness failure** — retrieval doesn't surface
3. Continue controlled ingestion "Zerodur" for p04 constraint queries. Investigate if it's a missing
- deepen active projects selectively memory or retrieval ranking issue.
- avoid noisy bulk corpus growth
4. Strengthen operational boringness
- backup and restore procedure
- Chroma rebuild / backup policy
- retention and restore validation
## Next ## Next
These are the next major layers after the current practical pass. These are the next major layers after the current stabilization pass.
1. Clarify AtoDrive as a real operational truth layer 1. Phase 6 AtoDrive — clarify Google Drive as a trusted operational
2. Mature identity / preferences handling source and ingest from it
3. Improve observability for: 2. Phase 13 Hardening — Chroma backup policy, monitoring, alerting,
- retrieval quality failure visibility beyond log files
- context-pack inspection 3. Engineering V1 implementation sprint — once knowledge density is
- comparison of behavior with and without AtoCore sufficient and the pipeline feels boring and dependable
## Later ## Later
@@ -187,11 +195,17 @@ direction, but not yet ready for immediate implementation.
These remain intentionally deferred. These remain intentionally deferred.
- automatic write-back from OpenClaw into AtoCore - ~~automatic write-back from OpenClaw into AtoCore~~ — OpenClaw capture
- automatic memory promotion plugin now exists (`openclaw-plugins/atocore-capture/`), interactions
- ~~reflection loop integration~~ — baseline now in (capture→reinforce flow. Write-back of promoted memories back to OpenClaw's own memory
auto, extract batch/manual). Extractor tuning and scheduled batch system is still deferred.
extraction still open. - ~~automatic memory promotion~~ — Phase 10 complete: auto-triage handles
extraction candidates, reinforcement-based auto-promotion graduates
candidates referenced 3+ times to active, stale candidates expire
after 14 days unreinforced.
- ~~reflection loop integration~~ — fully operational: capture (both
clients) → reinforce (automatic) → extract (nightly cron, sonnet) →
auto-triage (nightly, sonnet) → only needs_human reaches the user.
- replacing OpenClaw's own memory system - replacing OpenClaw's own memory system
- live machine-DB sync between machines - live machine-DB sync between machines
- full ontology / graph expansion before the current baseline is stable - full ontology / graph expansion before the current baseline is stable

View File

@@ -0,0 +1,56 @@
# OpenClaw -> AtoCore Integration Proposal
One-way pull is the right pattern.
**Stable surface to pull**
- Durable files in the OpenClaw workspace:
- `SOUL.md`
- `USER.md`
- `MODEL-ROUTING.md`
- `MEMORY.md`
- `memory/YYYY-MM-DD.md`
- `memory/heartbeat-state.json`
- `HEARTBEAT.md` only as operational state, not long-term truth
- These are explicitly documented in `t420-openclaw/AGENTS.md` as the continuity layer OpenClaw reads every session.
**Volatile vs durable**
- Durable:
- `SOUL.md`, `USER.md`, `MODEL-ROUTING.md`, `MEMORY.md`
- dated memory notes under `memory/`
- explicit JSON state like `memory/heartbeat-state.json`
- Volatile:
- in-session context
- ephemeral heartbeat work
- transient orchestration state
- platform response buffers
- Semi-durable:
- `HEARTBEAT.md` and operational notes; useful for importer hints, but not canonical identity/memory truth
**Formats**
- Mostly Markdown
- Some JSON (`heartbeat-state.json`)
- No stable OpenClaw-local DB or API surface is visible in this snapshot
**How pull should work**
- Start with cron-based filesystem reads, not an OpenClaw HTTP API.
- Read the durable files on a schedule, hash them, and import only deltas.
- Map them by type:
- `SOUL.md` / `USER.md` -> identity/preferences review candidates
- `MEMORY.md` -> curated long-term memory candidates
- `memory/YYYY-MM-DD.md` -> interaction/episodic import stream
- `heartbeat-state.json` -> low-priority ops metadata only if useful
**Discord**
- I do not see a documented durable Discord message store in the OpenClaw workspace snapshot.
- `AGENTS.md` references Discord behavior, but not a canonical local log/database.
- Treat Discord as transient unless OpenClaw exposes an explicit export/log file later.
**Biggest risk**
- Importing raw OpenClaw files as truth will blur curated memory and noisy session chatter.
- Mitigation: importer should classify by source tier, preserve provenance, and default to candidate/episodic ingestion rather than active memory promotion.
**Recommendation**
- Do not build two-way sync.
- Do not require OpenClaw to change architecture.
- Build one importer against the file continuity layer first.
- Add a formal export surface later only if the importer becomes too heuristic.

View File

@@ -0,0 +1,274 @@
# Universal Consumption — Connecting LLM Clients to AtoCore
Phase 1 of the Master Brain plan. Every LLM interaction across the ecosystem
pulls context from AtoCore automatically, without the user or agent having
to remember to ask for it.
## Architecture
```
┌─────────────────────┐
│ AtoCore HTTP API │ ← single source of truth
│ http://dalidou:8100│
└──────────┬──────────┘
┌────────────────────┼────────────────────┐
│ │ │
┌───┴────┐ ┌─────┴────┐ ┌────┴────┐
│ MCP │ │ OpenClaw │ │ HTTP │
│ server │ │ plugin │ │ proxy │
└───┬────┘ └──────┬───┘ └────┬────┘
│ │ │
Claude/Cursor/ OpenClaw Codex/Ollama/
Zed/Windsurf any OpenAI-compat client
```
Three adapters, one HTTP backend. Each adapter is a thin passthrough — no
business logic duplicated.
---
## Adapter 1: MCP Server (Claude Desktop, Claude Code, Cursor, Zed, Windsurf)
The MCP server is `scripts/atocore_mcp.py` — stdlib-only Python, stdio
transport, wraps the HTTP API. Claude-family clients see AtoCore as built-in
tools just like `Read` or `Bash`.
### Tools exposed
- **`atocore_context`** (most important): Full context pack for a query —
Trusted Project State + memories + retrieved chunks. Use at the start of
any project-related conversation to ground it.
- **`atocore_search`**: Semantic search over ingested documents (top-K chunks).
- **`atocore_memory_list`**: List active memories, filterable by project + type.
- **`atocore_memory_create`**: Propose a candidate memory (enters triage queue).
- **`atocore_project_state`**: Get Trusted Project State entries by category.
- **`atocore_projects`**: List registered projects + aliases.
- **`atocore_health`**: Service status check.
### Registration
#### Claude Code (CLI)
```bash
claude mcp add atocore -- python C:/Users/antoi/ATOCore/scripts/atocore_mcp.py
claude mcp list # verify: "atocore ... ✓ Connected"
```
#### Claude Desktop (GUI)
Edit `~/Library/Application Support/Claude/claude_desktop_config.json`
(macOS) or `%APPDATA%\Claude\claude_desktop_config.json` (Windows):
```json
{
"mcpServers": {
"atocore": {
"command": "python",
"args": ["C:/Users/antoi/ATOCore/scripts/atocore_mcp.py"],
"env": {
"ATOCORE_URL": "http://dalidou:8100"
}
}
}
}
```
Restart Claude Desktop.
#### Cursor / Zed / Windsurf
Similar JSON config in each tool's MCP settings. Consult their docs —
the config schema is standard MCP.
### Configuration
Environment variables the MCP server honors:
| Var | Default | Purpose |
|---|---|---|
| `ATOCORE_URL` | `http://dalidou:8100` | Where to reach AtoCore |
| `ATOCORE_TIMEOUT` | `10` | Per-request HTTP timeout (seconds) |
### Behavior
- Fail-open: if Dalidou is unreachable, tools return "AtoCore unavailable"
error messages but don't crash the client.
- Zero business logic: every tool is a direct HTTP passthrough.
- stdlib only: no MCP SDK dependency.
---
## Adapter 2: OpenClaw Plugin (`openclaw-plugins/atocore-capture/handler.js`)
The plugin on T420 OpenClaw has two responsibilities:
1. **CAPTURE**: On `before_agent_start` + `llm_output`, POST completed turns
to AtoCore `/interactions` (existing).
2. **PULL**: On `before_prompt_build`, call `/context/build` and inject the
context pack via `prependContext` so the agent's system prompt includes
AtoCore knowledge.
### Deployment
The plugin is loaded from
`/tmp/atocore-openclaw-capture-plugin/openclaw-plugins/atocore-capture/`
on the T420 (per OpenClaw's plugin config at `~/.openclaw/openclaw.json`).
To update:
```bash
scp openclaw-plugins/atocore-capture/handler.js \
papa@192.168.86.39:/tmp/atocore-openclaw-capture-plugin/openclaw-plugins/atocore-capture/index.js
ssh papa@192.168.86.39 'systemctl --user restart openclaw-gateway'
```
Verify in gateway logs: look for "ready (7 plugins: acpx, atocore-capture, ...)"
### Configuration (env vars set on T420)
| Var | Default | Purpose |
|---|---|---|
| `ATOCORE_BASE_URL` | `http://dalidou:8100` | AtoCore HTTP endpoint |
| `ATOCORE_PULL_DISABLED` | (unset) | Set to `1` to disable context pull |
### Behavior
- Fail-open: AtoCore unreachable = no injection, no capture, agent runs
normally.
- 6s timeout on context pull, 10s on capture — won't stall the agent.
- Context pack prepended as a clearly-bracketed block so the agent can see
it's auto-injected grounding info.
---
## Adapter 3: HTTP Proxy (`scripts/atocore_proxy.py`)
A stdlib-only OpenAI-compatible HTTP proxy. Sits between any
OpenAI-API-speaking client and the real provider, enriches every
`/chat/completions` request with AtoCore context.
Works with:
- **Codex CLI** (OpenAI-compatible endpoint)
- **Ollama** (has OpenAI-compatible `/v1` endpoint since 0.1.24)
- **LiteLLM**, **llama.cpp server**, custom agents
- Anything that can be pointed at a custom base URL
### Start it
```bash
# For Ollama (local models):
ATOCORE_UPSTREAM=http://localhost:11434/v1 \
python scripts/atocore_proxy.py
# For OpenAI cloud:
ATOCORE_UPSTREAM=https://api.openai.com/v1 \
ATOCORE_CLIENT_LABEL=codex \
python scripts/atocore_proxy.py
# Test:
curl http://127.0.0.1:11435/healthz
```
### Point a client at it
Set the client's OpenAI base URL to `http://127.0.0.1:11435/v1`.
#### Ollama example:
```bash
OPENAI_BASE_URL=http://127.0.0.1:11435/v1 \
some-openai-client --model llama3:8b
```
#### Codex CLI:
Set `OPENAI_BASE_URL=http://127.0.0.1:11435/v1` in your codex config.
### Configuration
| Var | Default | Purpose |
|---|---|---|
| `ATOCORE_URL` | `http://dalidou:8100` | AtoCore HTTP endpoint |
| `ATOCORE_UPSTREAM` | (required) | Real provider base URL |
| `ATOCORE_PROXY_PORT` | `11435` | Proxy listen port |
| `ATOCORE_PROXY_HOST` | `127.0.0.1` | Proxy bind address |
| `ATOCORE_CLIENT_LABEL` | `proxy` | Client id in captures |
| `ATOCORE_INJECT` | `1` | Inject context (set `0` to disable) |
| `ATOCORE_CAPTURE` | `1` | Capture interactions (set `0` to disable) |
### Behavior
- GET requests (model listing etc) pass through unchanged
- POST to `/chat/completions` (or `/v1/chat/completions`) gets enriched:
1. Last user message extracted as query
2. AtoCore `/context/build` called with 6s timeout
3. Pack injected as system message (or prepended to existing system)
4. Enriched body forwarded to upstream
5. After success, interaction POSTed to `/interactions` in background
- Fail-open: AtoCore unreachable = pass through without injection
- Streaming responses: currently buffered (not true stream). Good enough for
most cases; can be upgraded later if needed.
### Running as a service
On Linux, create `~/.config/systemd/user/atocore-proxy.service`:
```ini
[Unit]
Description=AtoCore HTTP proxy
[Service]
Environment=ATOCORE_UPSTREAM=http://localhost:11434/v1
Environment=ATOCORE_CLIENT_LABEL=ollama
ExecStart=/usr/bin/python3 /path/to/scripts/atocore_proxy.py
Restart=on-failure
[Install]
WantedBy=default.target
```
Then: `systemctl --user enable --now atocore-proxy`
On Windows, register via Task Scheduler (similar pattern to backup task)
or use NSSM to install as a service.
---
## Verification Checklist
Fresh end-to-end test to confirm Phase 1 is working:
### For Claude Code (MCP)
1. Open a new Claude Code session (not this one).
2. Ask: "what do we know about p06 polisher's control architecture?"
3. Claude should invoke `atocore_context` or `atocore_project_state`
on its own and answer grounded in AtoCore data.
### For OpenClaw (plugin pull)
1. Send a Discord message to OpenClaw: "what's the status on p04?"
2. Check T420 logs: `journalctl --user -u openclaw-gateway --since "1 min ago" | grep atocore-pull`
3. Expect: `atocore-pull:injected project=p04-gigabit chars=NNN`
### For proxy (any OpenAI-compat client)
1. Start proxy with appropriate upstream
2. Run a client query through it
3. Check stderr: `[atocore-proxy] inject: project=... chars=...`
4. Check `curl http://127.0.0.1:8100/interactions?client=proxy` — should
show the captured turn
---
## Why not just MCP everywhere?
MCP is great for Claude-family clients but:
- Not supported natively by Codex CLI, Ollama, or OpenAI's own API
- No universal "attach MCP" mechanism in all LLM runtimes
- HTTP APIs are truly universal
HTTP API is the truth, each adapter is the thinnest possible shim for its
ecosystem. When new adapters are needed (Gemini CLI, Claude Code plugin
system, etc.), they follow the same pattern.
---
## Future enhancements
- **Streaming passthrough** in the proxy (currently buffered for simplicity)
- **Response grounding check**: parse assistant output for references to
injected context, count reinforcement events
- **Per-client metrics** in the dashboard: how often each client pulls,
context pack size, injection rate
- **Smart project detection**: today we use keyword matching; could use
AtoCore's own project resolver endpoint

View File

@@ -0,0 +1,140 @@
# Windows Main-Computer Backup Setup
The AtoCore backup pipeline runs nightly on Dalidou and already pushes snapshots
off-host to the T420 (`papa@192.168.86.39`). This doc sets up a **second**,
pull-based daily backup to your Windows main computer at
`C:\Users\antoi\Documents\ATOCore_Backups\`.
Pull-based means the Windows machine pulls from Dalidou. This is simpler than
push because Dalidou doesn't need SSH keys to reach Windows, and the backup
only runs when the Windows machine is powered on and can reach Dalidou.
## Prerequisites
- Windows 10/11 with OpenSSH client (built-in since Win10 1809)
- SSH key-based auth to `papa@dalidou` already working (you're using it today)
- `C:\Users\antoi\ATOCore\scripts\windows\atocore-backup-pull.ps1` present
## Test the script manually
```powershell
powershell.exe -ExecutionPolicy Bypass -File `
C:\Users\antoi\ATOCore\scripts\windows\atocore-backup-pull.ps1
```
Expected output:
```
[timestamp] === AtoCore backup pull starting ===
[timestamp] Dalidou reachable.
[timestamp] Pulling snapshots via scp...
[timestamp] Pulled N snapshots successfully (total X MB, latest: ...)
[timestamp] === backup complete ===
```
Target directory: `C:\Users\antoi\Documents\ATOCore_Backups\snapshots\`
Logs: `C:\Users\antoi\Documents\ATOCore_Backups\_logs\backup-*.log`
## Register the Task Scheduler task
### Option A — automatic registration (recommended)
Run this PowerShell command **as your user** (no admin needed — uses HKCU task):
```powershell
$action = New-ScheduledTaskAction -Execute 'powershell.exe' `
-Argument '-ExecutionPolicy Bypass -NonInteractive -WindowStyle Hidden -File C:\Users\antoi\ATOCore\scripts\windows\atocore-backup-pull.ps1'
# Run daily at 10:00 local time; if missed (computer off), run at next logon
$trigger = New-ScheduledTaskTrigger -Daily -At 10:00AM
$trigger.StartBoundary = (Get-Date -Format 'yyyy-MM-ddTHH:mm:ss')
$settings = New-ScheduledTaskSettingsSet `
-AllowStartIfOnBatteries `
-DontStopIfGoingOnBatteries `
-StartWhenAvailable `
-ExecutionTimeLimit (New-TimeSpan -Minutes 10) `
-RestartCount 2 `
-RestartInterval (New-TimeSpan -Minutes 30)
Register-ScheduledTask -TaskName 'AtoCore Backup Pull' `
-Description 'Daily pull of AtoCore backup snapshots from Dalidou' `
-Action $action -Trigger $trigger -Settings $settings `
-User $env:USERNAME
```
Key settings:
- `-StartWhenAvailable`: if the computer was off at 10:00, run as soon as it
comes online
- `-AllowStartIfOnBatteries`: works on laptop battery too
- `-ExecutionTimeLimit 10min`: kill hung tasks
- `-RestartCount 2`: retry twice if it fails (Dalidou temporarily unreachable)
### Option B -- Task Scheduler GUI
1. Open Task Scheduler (`taskschd.msc`)
2. Create Basic Task -> name: `AtoCore Backup Pull`
3. Trigger: Daily, 10:00 AM, recur every 1 day
4. Action: Start a program
- Program: `powershell.exe`
- Arguments: `-ExecutionPolicy Bypass -NonInteractive -WindowStyle Hidden -File "C:\Users\antoi\ATOCore\scripts\windows\atocore-backup-pull.ps1"`
5. Finish, then edit the task:
- Settings tab: check "Run task as soon as possible after a scheduled start is missed"
- Settings tab: "If the task fails, restart every 30 minutes, up to 2 times"
- Conditions tab: uncheck "Start only if computer is on AC power" (if you want it on battery)
## Verify
After the first scheduled run:
```powershell
# Most recent log
Get-ChildItem C:\Users\antoi\Documents\ATOCore_Backups\_logs\ |
Sort-Object Name -Descending |
Select-Object -First 1 |
Get-Content
# Latest snapshot present?
Get-ChildItem C:\Users\antoi\Documents\ATOCore_Backups\snapshots\ |
Sort-Object Name -Descending |
Select-Object -First 3
```
## Unregister (if needed)
```powershell
Unregister-ScheduledTask -TaskName 'AtoCore Backup Pull' -Confirm:$false
```
## How it behaves
- **Computer on, Dalidou reachable**: pulls latest snapshots silently in ~15s
- **Computer on, Dalidou unreachable** (remote work, network down): fail-open,
exits without error, logs "Dalidou unreachable"
- **Computer off at scheduled time**: Task Scheduler runs it as soon as the
computer wakes up
- **Many days off**: one run catches up; scp only transfers files not already
present (snapshots are date-stamped directories, idempotent overwrites)
## What gets backed up
The snapshots tree contains:
- `YYYYMMDDTHHMMSSZ/config/` — project registry, AtoCore config
- `YYYYMMDDTHHMMSSZ/db/` — SQLite snapshot of all memory, state, interactions
- `YYYYMMDDTHHMMSSZ/backup-metadata.json` — SHA, timestamp, source info
Chroma vectors are **not** in the snapshot by default
(`ATOCORE_BACKUP_CHROMA=false` on Dalidou). They can be rebuilt from the
source documents if lost. To include them, set `ATOCORE_BACKUP_CHROMA=true`
in the Dalidou cron environment.
## Three-tier backup summary
After this setup:
| Tier | Location | Cadence | Purpose |
|---|---|---|---|
| Live | Dalidou `/srv/storage/atocore/backups/snapshots/` | Nightly 03:00 UTC | Fast restore |
| Off-host | T420 `papa@192.168.86.39:/home/papa/atocore-backups/` | Nightly after Dalidou | Dalidou dies |
| User machine | `C:\Users\antoi\Documents\ATOCore_Backups\` | Daily 10:00 local | Full home-network failure |
Three independent copies. Any two can be lost simultaneously without data loss.

View File

@@ -0,0 +1,29 @@
# AtoCore Capture Plugin for OpenClaw
Minimal OpenClaw plugin that mirrors Claude Code's `capture_stop.py` behavior:
- watches user-triggered assistant turns
- POSTs `prompt` + `response` to `POST /interactions`
- sets `client="openclaw"`
- sets `reinforce=true`
- fails open on network or API errors
## Config
Optional plugin config:
```json
{
"baseUrl": "http://dalidou:8100",
"minPromptLength": 15,
"maxResponseLength": 50000
}
```
If `baseUrl` is omitted, the plugin uses `ATOCORE_BASE_URL` or defaults to `http://dalidou:8100`.
## Notes
- Project detection is intentionally left empty for now. Unscoped capture is acceptable because AtoCore's extraction pipeline handles unscoped interactions.
- Extraction is **not** part of the capture path. This plugin only records interactions and lets AtoCore reinforcement run automatically.
- The plugin captures only user-triggered turns, not heartbeats or system-only runs.

View File

@@ -0,0 +1,146 @@
/**
* AtoCore OpenClaw plugin — capture + pull.
*
* Two responsibilities:
*
* 1. CAPTURE (existing): On before_agent_start, buffer the user prompt.
* On llm_output, POST prompt+response to AtoCore /interactions.
* This is the "write" side — OpenClaw turns feed AtoCore's memory.
*
* 2. PULL (Phase 1 master brain): On before_prompt_build, call AtoCore
* /context/build and inject the returned context via prependContext.
* Every OpenClaw response is automatically grounded in what AtoCore
* knows (project state, memories, relevant chunks).
*
* Fail-open throughout: AtoCore unreachable = no injection, no capture,
* never blocks the agent.
*/
import { definePluginEntry } from "openclaw/plugin-sdk/core";
const BASE_URL = process.env.ATOCORE_BASE_URL || "http://dalidou:8100";
const MIN_LEN = 15;
const MAX_RESP = 50000;
const CONTEXT_TIMEOUT_MS = 6000;
const CAPTURE_TIMEOUT_MS = 10000;
function trim(v) { return typeof v === "string" ? v.trim() : ""; }
function trunc(t, m) { return !t || t.length <= m ? t : t.slice(0, m) + "\n\n[truncated]"; }
function detectProject(prompt) {
const lower = (prompt || "").toLowerCase();
const hints = [
["p04", "p04-gigabit"],
["gigabit", "p04-gigabit"],
["p05", "p05-interferometer"],
["interferometer", "p05-interferometer"],
["p06", "p06-polisher"],
["polisher", "p06-polisher"],
["fullum", "p06-polisher"],
["abb", "abb-space"],
["atomizer", "atomizer-v2"],
["atocore", "atocore"],
];
for (const [token, proj] of hints) {
if (lower.includes(token)) return proj;
}
return "";
}
export default definePluginEntry({
register(api) {
const log = api.logger;
let lastPrompt = null;
// --- PULL: inject AtoCore context into every prompt ---
api.on("before_prompt_build", async (event, ctx) => {
if (process.env.ATOCORE_PULL_DISABLED === "1") return;
const prompt = trim(event?.prompt || "");
if (prompt.length < MIN_LEN) return;
const project = detectProject(prompt);
try {
const res = await fetch(BASE_URL.replace(/\/$/, "") + "/context/build", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ prompt, project }),
signal: AbortSignal.timeout(CONTEXT_TIMEOUT_MS),
});
if (!res.ok) {
log.info("atocore-pull:http_error", { status: res.status });
return;
}
const data = await res.json();
const contextPack = data.formatted_context || "";
if (!contextPack.trim()) return;
log.info("atocore-pull:injected", {
project: project || "(none)",
chars: contextPack.length,
});
return {
prependContext:
"--- AtoCore Context (auto-injected) ---\n" +
contextPack +
"\n--- End AtoCore Context ---\n",
};
} catch (err) {
log.info("atocore-pull:error", { error: String(err).slice(0, 200) });
}
});
// --- CAPTURE: buffer user prompts on agent start ---
api.on("before_agent_start", async (event, ctx) => {
const prompt = trim(event?.prompt || event?.cleanedBody || "");
if (prompt.length < MIN_LEN || prompt.startsWith("<")) {
lastPrompt = null;
return;
}
lastPrompt = { text: prompt, sessionKey: ctx?.sessionKey || "", ts: Date.now() };
log.info("atocore-capture:prompt_buffered", { len: prompt.length });
});
// --- CAPTURE: send completed turns to AtoCore ---
api.on("llm_output", async (event, ctx) => {
if (!lastPrompt) return;
const texts = Array.isArray(event?.assistantTexts) ? event.assistantTexts : [];
const response = trunc(trim(texts.join("\n\n")), MAX_RESP);
if (!response) return;
const prompt = lastPrompt.text;
const sessionKey = lastPrompt.sessionKey || ctx?.sessionKey || "";
const project = detectProject(prompt);
lastPrompt = null;
log.info("atocore-capture:posting", {
promptLen: prompt.length,
responseLen: response.length,
project: project || "(none)",
});
fetch(BASE_URL.replace(/\/$/, "") + "/interactions", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
prompt,
response,
client: "openclaw",
session_id: sessionKey,
project,
reinforce: true,
}),
signal: AbortSignal.timeout(CAPTURE_TIMEOUT_MS),
}).then(res => {
log.info("atocore-capture:posted", { status: res.status });
}).catch(err => {
log.warn("atocore-capture:post_error", { error: String(err).slice(0, 200) });
});
});
api.on("session_end", async () => {
lastPrompt = null;
});
}
});

View File

@@ -0,0 +1,94 @@
import { definePluginEntry } from "openclaw/plugin-sdk/core";
const DEFAULT_BASE_URL = process.env.ATOCORE_BASE_URL || "http://dalidou:8100";
const DEFAULT_MIN_PROMPT_LENGTH = 15;
const DEFAULT_MAX_RESPONSE_LENGTH = 50_000;
function trimText(value) {
return typeof value === "string" ? value.trim() : "";
}
function truncateResponse(text, maxLength) {
if (!text || text.length <= maxLength) return text;
return `${text.slice(0, maxLength)}\n\n[truncated]`;
}
function shouldCapturePrompt(prompt, minLength) {
const text = trimText(prompt);
if (!text) return false;
if (text.startsWith("<")) return false;
return text.length >= minLength;
}
async function postInteraction(baseUrl, payload, logger) {
try {
const res = await fetch(`${baseUrl.replace(/\/$/, "")}/interactions`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify(payload),
signal: AbortSignal.timeout(10_000)
});
if (!res.ok) {
logger?.debug?.("atocore_capture_post_failed", { status: res.status });
return false;
}
return true;
} catch (error) {
logger?.debug?.("atocore_capture_post_error", {
error: error instanceof Error ? error.message : String(error)
});
return false;
}
}
export default definePluginEntry({
register(api) {
const logger = api.logger;
const pendingBySession = new Map();
api.on("before_agent_start", async (event, ctx) => {
if (ctx?.trigger && ctx.trigger !== "user") return;
const config = api.getConfig?.() || {};
const minPromptLength = Number(config.minPromptLength || DEFAULT_MIN_PROMPT_LENGTH);
const prompt = trimText(event?.prompt || "");
if (!shouldCapturePrompt(prompt, minPromptLength)) {
pendingBySession.delete(ctx.sessionId);
return;
}
pendingBySession.set(ctx.sessionId, {
prompt,
sessionId: ctx.sessionId,
sessionKey: ctx.sessionKey || "",
project: ""
});
});
api.on("llm_output", async (event, ctx) => {
if (ctx?.trigger && ctx.trigger !== "user") return;
const pending = pendingBySession.get(ctx.sessionId);
if (!pending) return;
const assistantTexts = Array.isArray(event?.assistantTexts) ? event.assistantTexts : [];
const response = truncateResponse(trimText(assistantTexts.join("\n\n")), Number((api.getConfig?.() || {}).maxResponseLength || DEFAULT_MAX_RESPONSE_LENGTH));
if (!response) return;
const config = api.getConfig?.() || {};
const baseUrl = trimText(config.baseUrl) || DEFAULT_BASE_URL;
const payload = {
prompt: pending.prompt,
response,
client: "openclaw",
session_id: pending.sessionKey || pending.sessionId,
project: pending.project || "",
reinforce: true
};
await postInteraction(baseUrl, payload, logger);
pendingBySession.delete(ctx.sessionId);
});
api.on("session_end", async (event) => {
if (event?.sessionId) pendingBySession.delete(event.sessionId);
});
}
});

View File

@@ -0,0 +1,29 @@
{
"id": "atocore-capture",
"name": "AtoCore Capture",
"description": "Captures completed OpenClaw assistant turns to AtoCore interactions for reinforcement.",
"configSchema": {
"type": "object",
"properties": {
"baseUrl": {
"type": "string",
"description": "Override AtoCore base URL. Defaults to ATOCORE_BASE_URL or http://dalidou:8100"
},
"minPromptLength": {
"type": "integer",
"minimum": 1,
"description": "Minimum user prompt length required before capture"
},
"maxResponseLength": {
"type": "integer",
"minimum": 100,
"description": "Maximum assistant response length to store"
}
},
"additionalProperties": false
},
"uiHints": {
"category": "automation",
"displayName": "AtoCore Capture"
}
}

View File

@@ -0,0 +1,7 @@
{
"name": "@atomaste/atocore-openclaw-capture",
"private": true,
"version": "0.0.0",
"type": "module",
"description": "OpenClaw plugin that captures assistant turns to AtoCore interactions"
}

View File

@@ -16,6 +16,7 @@ dependencies = [
"pydantic>=2.6.0", "pydantic>=2.6.0",
"pydantic-settings>=2.1.0", "pydantic-settings>=2.1.0",
"structlog>=24.1.0", "structlog>=24.1.0",
"markdown>=3.5.0",
] ]
[project.optional-dependencies] [project.optional-dependencies]

View File

@@ -6,3 +6,4 @@ sentence-transformers>=2.5.0
pydantic>=2.6.0 pydantic>=2.6.0
pydantic-settings>=2.1.0 pydantic-settings>=2.1.0
structlog>=24.1.0 structlog>=24.1.0
markdown>=3.5.0

479
scripts/atocore_mcp.py Normal file
View File

@@ -0,0 +1,479 @@
#!/usr/bin/env python3
"""AtoCore MCP server — stdio transport, stdlib-only.
Exposes the AtoCore HTTP API as MCP tools so any MCP-aware client
(Claude Desktop, Claude Code, Cursor, Zed, Windsurf) can pull
context + memories automatically at prompt time.
Design:
- stdlib only (no mcp SDK dep) — MCP protocol is simple JSON-RPC
over stdio, and AtoCore's philosophy prefers stdlib.
- Thin wrapper: every tool is a direct pass-through to an HTTP
endpoint. Zero business logic here — the AtoCore server is
the single source of truth.
- Fail-open: if AtoCore is unreachable, tools return a graceful
"unavailable" message rather than crashing the client.
Protocol: MCP 2024-11-05 / 2025-03-26 compatible
https://spec.modelcontextprotocol.io/specification/
Usage (standalone test):
echo '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"test","version":"0"}}}' | python atocore_mcp.py
Register with Claude Code:
claude mcp add atocore -- python /path/to/atocore_mcp.py
Environment:
ATOCORE_URL base URL of the AtoCore HTTP API (default http://dalidou:8100)
ATOCORE_TIMEOUT per-request HTTP timeout seconds (default 10)
"""
from __future__ import annotations
import json
import os
import sys
import urllib.error
import urllib.parse
import urllib.request
# --- Configuration ---
ATOCORE_URL = os.environ.get("ATOCORE_URL", "http://dalidou:8100").rstrip("/")
HTTP_TIMEOUT = float(os.environ.get("ATOCORE_TIMEOUT", "10"))
SERVER_NAME = "atocore"
SERVER_VERSION = "0.1.0"
PROTOCOL_VERSION = "2024-11-05"
# --- stderr logging (stdout is reserved for JSON-RPC) ---
def log(msg: str) -> None:
print(f"[atocore-mcp] {msg}", file=sys.stderr, flush=True)
# --- HTTP helpers ---
def http_get(path: str, params: dict | None = None) -> dict:
"""GET a JSON response from AtoCore. Raises on HTTP error."""
url = ATOCORE_URL + path
if params:
# Drop empty params so the URL stays clean
clean = {k: v for k, v in params.items() if v not in (None, "", [], {})}
if clean:
url += "?" + urllib.parse.urlencode(clean)
req = urllib.request.Request(url, headers={"Accept": "application/json"})
with urllib.request.urlopen(req, timeout=HTTP_TIMEOUT) as resp:
return json.loads(resp.read().decode("utf-8"))
def http_post(path: str, body: dict) -> dict:
url = ATOCORE_URL + path
data = json.dumps(body).encode("utf-8")
req = urllib.request.Request(
url, data=data, method="POST",
headers={"Content-Type": "application/json", "Accept": "application/json"},
)
with urllib.request.urlopen(req, timeout=HTTP_TIMEOUT) as resp:
return json.loads(resp.read().decode("utf-8"))
def safe_call(fn, *args, **kwargs) -> tuple[dict | None, str | None]:
"""Run an HTTP call, return (result, error_message_or_None)."""
try:
return fn(*args, **kwargs), None
except urllib.error.HTTPError as e:
try:
body = e.read().decode("utf-8", errors="replace")
except Exception:
body = ""
return None, f"AtoCore HTTP {e.code}: {body[:200]}"
except urllib.error.URLError as e:
return None, f"AtoCore unreachable at {ATOCORE_URL}: {e.reason}"
except Exception as e:
return None, f"AtoCore error: {type(e).__name__}: {str(e)[:200]}"
# --- Tool definitions ---
# Each tool: name, description, inputSchema (JSON Schema), handler
def _tool_context(args: dict) -> str:
"""Build a full context pack for a query — state + memories + retrieved chunks."""
query = (args.get("query") or "").strip()
project = args.get("project") or ""
if not query:
return "Error: 'query' is required."
result, err = safe_call(http_post, "/context/build", {
"prompt": query, "project": project,
})
if err:
return f"AtoCore context unavailable: {err}"
pack = result.get("formatted_context", "") or ""
if not pack.strip():
return "(AtoCore returned an empty context pack — no matching state, memories, or chunks.)"
return pack
def _tool_search(args: dict) -> str:
"""Retrieval only — raw chunks ranked by semantic similarity."""
query = (args.get("query") or "").strip()
project = args.get("project") or ""
top_k = int(args.get("top_k") or 5)
if not query:
return "Error: 'query' is required."
result, err = safe_call(http_post, "/query", {
"prompt": query, "project": project, "top_k": top_k,
})
if err:
return f"AtoCore search unavailable: {err}"
chunks = result.get("results", []) or []
if not chunks:
return "No results."
lines = []
for i, c in enumerate(chunks, 1):
src = c.get("source_file") or c.get("title") or "unknown"
heading = c.get("heading_path") or ""
snippet = (c.get("content") or "")[:300]
score = c.get("score", 0.0)
head_str = f" ({heading})" if heading else ""
lines.append(f"[{i}] score={score:.3f} source={src}{head_str}\n{snippet}")
return "\n\n".join(lines)
def _tool_memory_list(args: dict) -> str:
"""List active memories, optionally filtered by project and type."""
params = {
"status": "active",
"limit": int(args.get("limit") or 20),
}
if args.get("project"):
params["project"] = args["project"]
if args.get("memory_type"):
params["memory_type"] = args["memory_type"]
result, err = safe_call(http_get, "/memory", params=params)
if err:
return f"AtoCore memory list unavailable: {err}"
memories = result.get("memories", []) or []
if not memories:
return "No memories match."
lines = []
for m in memories:
mt = m.get("memory_type", "?")
proj = m.get("project") or "(global)"
conf = m.get("confidence", 0.0)
refs = m.get("reference_count", 0)
content = (m.get("content") or "")[:250]
lines.append(f"[{mt}/{proj}] conf={conf:.2f} refs={refs}\n {content}")
return "\n\n".join(lines)
def _tool_memory_create(args: dict) -> str:
"""Create a candidate memory (enters the triage queue)."""
memory_type = (args.get("memory_type") or "").strip()
content = (args.get("content") or "").strip()
project = args.get("project") or ""
confidence = float(args.get("confidence") or 0.5)
if not memory_type or not content:
return "Error: 'memory_type' and 'content' are required."
valid_types = ["identity", "preference", "project", "episodic", "knowledge", "adaptation"]
if memory_type not in valid_types:
return f"Error: memory_type must be one of {valid_types}."
result, err = safe_call(http_post, "/memory", {
"memory_type": memory_type,
"content": content,
"project": project,
"confidence": confidence,
"status": "candidate",
})
if err:
return f"AtoCore memory create failed: {err}"
mid = result.get("id", "?")
return f"Candidate memory created: id={mid} type={memory_type} project={project or '(global)'}"
def _tool_project_state(args: dict) -> str:
"""Get Trusted Project State entries for a project."""
project = (args.get("project") or "").strip()
category = args.get("category") or ""
if not project:
return "Error: 'project' is required."
path = f"/project/state/{urllib.parse.quote(project)}"
params = {"category": category} if category else None
result, err = safe_call(http_get, path, params=params)
if err:
return f"AtoCore project state unavailable: {err}"
entries = result.get("entries", []) or result.get("state", []) or []
if not entries:
return f"No state entries for project '{project}'."
lines = []
for e in entries:
cat = e.get("category", "?")
key = e.get("key", "?")
value = (e.get("value") or "")[:300]
src = e.get("source") or ""
lines.append(f"[{cat}/{key}] (source: {src})\n {value}")
return "\n\n".join(lines)
def _tool_projects(args: dict) -> str:
"""List registered AtoCore projects."""
result, err = safe_call(http_get, "/projects")
if err:
return f"AtoCore projects unavailable: {err}"
projects = result.get("projects", []) or []
if not projects:
return "No projects registered."
lines = []
for p in projects:
pid = p.get("project_id") or p.get("id") or p.get("name") or "?"
aliases = p.get("aliases", []) or []
alias_str = f" (aliases: {', '.join(aliases)})" if aliases else ""
lines.append(f"- {pid}{alias_str}")
return "\n".join(lines)
def _tool_health(args: dict) -> str:
"""Check AtoCore service health."""
result, err = safe_call(http_get, "/health")
if err:
return f"AtoCore unreachable: {err}"
sha = result.get("build_sha", "?")[:8]
vectors = result.get("vectors_count", "?")
env = result.get("env", "?")
return f"AtoCore healthy: sha={sha} vectors={vectors} env={env}"
TOOLS = [
{
"name": "atocore_context",
"description": (
"Get the full AtoCore context pack for a user query. Returns "
"Trusted Project State (high trust), relevant memories, and "
"retrieved source chunks formatted for prompt injection. "
"Use this FIRST on any project-related query to ground the "
"conversation in what AtoCore already knows."
),
"inputSchema": {
"type": "object",
"properties": {
"query": {"type": "string", "description": "The user's question or task"},
"project": {"type": "string", "description": "Project hint (e.g. 'p04-gigabit'); optional"},
},
"required": ["query"],
},
"handler": _tool_context,
},
{
"name": "atocore_search",
"description": (
"Semantic search over AtoCore's ingested source documents. "
"Returns top-K ranked chunks. Use this when you need raw "
"references rather than a full context pack."
),
"inputSchema": {
"type": "object",
"properties": {
"query": {"type": "string"},
"project": {"type": "string", "description": "optional project filter"},
"top_k": {"type": "integer", "minimum": 1, "maximum": 20, "default": 5},
},
"required": ["query"],
},
"handler": _tool_search,
},
{
"name": "atocore_memory_list",
"description": (
"List active memories (curated facts, decisions, preferences). "
"Filter by project and/or memory_type. Use this to inspect what "
"AtoCore currently remembers about a topic."
),
"inputSchema": {
"type": "object",
"properties": {
"project": {"type": "string"},
"memory_type": {
"type": "string",
"enum": ["identity", "preference", "project", "episodic", "knowledge", "adaptation"],
},
"limit": {"type": "integer", "minimum": 1, "maximum": 100, "default": 20},
},
},
"handler": _tool_memory_list,
},
{
"name": "atocore_memory_create",
"description": (
"Propose a new memory for AtoCore. Creates a CANDIDATE that "
"enters the triage queue for human/auto review — not immediately "
"active. Use this to capture durable facts/decisions that "
"should persist across sessions. Do NOT use for transient state "
"or session-specific notes."
),
"inputSchema": {
"type": "object",
"properties": {
"memory_type": {
"type": "string",
"enum": ["identity", "preference", "project", "episodic", "knowledge", "adaptation"],
},
"content": {"type": "string", "description": "The fact/decision/preference to remember"},
"project": {"type": "string", "description": "project id if project-scoped; empty for global"},
"confidence": {"type": "number", "minimum": 0, "maximum": 1, "default": 0.5},
},
"required": ["memory_type", "content"],
},
"handler": _tool_memory_create,
},
{
"name": "atocore_project_state",
"description": (
"Get Trusted Project State entries for a given project — the "
"highest-trust tier with curated decisions, requirements, "
"facts, contacts, milestones. Use this to look up authoritative "
"project info."
),
"inputSchema": {
"type": "object",
"properties": {
"project": {"type": "string"},
"category": {
"type": "string",
"enum": ["status", "decision", "requirement", "contact", "milestone", "fact", "config"],
},
},
"required": ["project"],
},
"handler": _tool_project_state,
},
{
"name": "atocore_projects",
"description": "List all registered AtoCore projects (id + aliases).",
"inputSchema": {"type": "object", "properties": {}},
"handler": _tool_projects,
},
{
"name": "atocore_health",
"description": "Check AtoCore service health (build SHA, vector count, env).",
"inputSchema": {"type": "object", "properties": {}},
"handler": _tool_health,
},
]
# --- JSON-RPC handlers ---
def handle_initialize(params: dict) -> dict:
return {
"protocolVersion": PROTOCOL_VERSION,
"capabilities": {
"tools": {"listChanged": False},
},
"serverInfo": {"name": SERVER_NAME, "version": SERVER_VERSION},
}
def handle_tools_list(params: dict) -> dict:
return {
"tools": [
{"name": t["name"], "description": t["description"], "inputSchema": t["inputSchema"]}
for t in TOOLS
]
}
def handle_tools_call(params: dict) -> dict:
tool_name = params.get("name", "")
args = params.get("arguments", {}) or {}
tool = next((t for t in TOOLS if t["name"] == tool_name), None)
if tool is None:
return {
"content": [{"type": "text", "text": f"Unknown tool: {tool_name}"}],
"isError": True,
}
try:
text = tool["handler"](args)
except Exception as e:
log(f"tool {tool_name} raised: {e}")
return {
"content": [{"type": "text", "text": f"Tool error: {type(e).__name__}: {e}"}],
"isError": True,
}
return {"content": [{"type": "text", "text": text}]}
def handle_ping(params: dict) -> dict:
return {}
METHODS = {
"initialize": handle_initialize,
"tools/list": handle_tools_list,
"tools/call": handle_tools_call,
"ping": handle_ping,
}
# --- stdio main loop ---
def send(obj: dict) -> None:
"""Write a single-line JSON message to stdout and flush."""
sys.stdout.write(json.dumps(obj, ensure_ascii=False) + "\n")
sys.stdout.flush()
def make_response(req_id, result=None, error=None) -> dict:
resp = {"jsonrpc": "2.0", "id": req_id}
if error is not None:
resp["error"] = error
else:
resp["result"] = result if result is not None else {}
return resp
def main() -> int:
log(f"starting (AtoCore at {ATOCORE_URL})")
for line in sys.stdin:
line = line.strip()
if not line:
continue
try:
msg = json.loads(line)
except json.JSONDecodeError as e:
log(f"parse error: {e}")
continue
method = msg.get("method", "")
req_id = msg.get("id")
params = msg.get("params", {}) or {}
# Notifications (no id) don't need a response
if req_id is None:
if method == "notifications/initialized":
log("client initialized")
continue
handler = METHODS.get(method)
if handler is None:
send(make_response(req_id, error={
"code": -32601,
"message": f"Method not found: {method}",
}))
continue
try:
result = handler(params)
send(make_response(req_id, result=result))
except Exception as e:
log(f"handler {method} raised: {e}")
send(make_response(req_id, error={
"code": -32603,
"message": f"Internal error: {type(e).__name__}: {e}",
}))
log("stdin closed, exiting")
return 0
if __name__ == "__main__":
sys.exit(main())

321
scripts/atocore_proxy.py Normal file
View File

@@ -0,0 +1,321 @@
#!/usr/bin/env python3
"""AtoCore Proxy — OpenAI-compatible HTTP middleware.
Acts as a drop-in layer for any client that speaks the OpenAI Chat
Completions API (Codex, Ollama, LiteLLM, custom agents). Sits between
the client and the real model provider:
client -> atocore_proxy -> real_provider (OpenAI, Ollama, Anthropic, ...)
For each chat completion request:
1. Extract the user's last message as the "query"
2. Call AtoCore /context/build to get a context pack
3. Inject the pack as a system message (or prepend to existing system)
4. Forward the enriched request to the real provider
5. Capture the full interaction back to AtoCore /interactions
Fail-open: if AtoCore is unreachable, the request passes through
unchanged. If the real provider fails, the error is propagated to the
client as-is.
Configuration (env vars):
ATOCORE_URL AtoCore base URL (default http://dalidou:8100)
ATOCORE_UPSTREAM real provider base URL (e.g. http://localhost:11434/v1 for Ollama)
ATOCORE_PROXY_PORT port to listen on (default 11435)
ATOCORE_PROXY_HOST bind address (default 127.0.0.1)
ATOCORE_CLIENT_LABEL client id recorded in captures (default "proxy")
ATOCORE_CAPTURE "1" to capture interactions back (default "1")
ATOCORE_INJECT "1" to inject context (default "1")
Usage:
# Proxy for Ollama:
ATOCORE_UPSTREAM=http://localhost:11434/v1 python atocore_proxy.py
# Then point your client at http://localhost:11435/v1 instead of the
# real provider.
Stdlib only — deliberate to keep the dependency footprint at zero.
"""
from __future__ import annotations
import http.server
import json
import os
import socketserver
import sys
import threading
import urllib.error
import urllib.parse
import urllib.request
from typing import Any
ATOCORE_URL = os.environ.get("ATOCORE_URL", "http://dalidou:8100").rstrip("/")
UPSTREAM_URL = os.environ.get("ATOCORE_UPSTREAM", "").rstrip("/")
PROXY_PORT = int(os.environ.get("ATOCORE_PROXY_PORT", "11435"))
PROXY_HOST = os.environ.get("ATOCORE_PROXY_HOST", "127.0.0.1")
CLIENT_LABEL = os.environ.get("ATOCORE_CLIENT_LABEL", "proxy")
CAPTURE_ENABLED = os.environ.get("ATOCORE_CAPTURE", "1") == "1"
INJECT_ENABLED = os.environ.get("ATOCORE_INJECT", "1") == "1"
ATOCORE_TIMEOUT = float(os.environ.get("ATOCORE_TIMEOUT", "6"))
UPSTREAM_TIMEOUT = float(os.environ.get("ATOCORE_UPSTREAM_TIMEOUT", "300"))
PROJECT_HINTS = [
("p04-gigabit", ["p04", "gigabit"]),
("p05-interferometer", ["p05", "interferometer"]),
("p06-polisher", ["p06", "polisher", "fullum"]),
("abb-space", ["abb"]),
("atomizer-v2", ["atomizer"]),
("atocore", ["atocore", "dalidou"]),
]
def log(msg: str) -> None:
print(f"[atocore-proxy] {msg}", file=sys.stderr, flush=True)
def detect_project(text: str) -> str:
lower = (text or "").lower()
for proj, tokens in PROJECT_HINTS:
if any(t in lower for t in tokens):
return proj
return ""
def get_last_user_message(body: dict) -> str:
messages = body.get("messages", []) or []
for m in reversed(messages):
if m.get("role") == "user":
content = m.get("content", "")
if isinstance(content, list):
# OpenAI multi-part content: extract text parts
parts = [p.get("text", "") for p in content if p.get("type") == "text"]
return "\n".join(parts)
return str(content)
return ""
def get_assistant_text(response: dict) -> str:
"""Extract assistant text from an OpenAI-style completion response."""
choices = response.get("choices", []) or []
if not choices:
return ""
msg = choices[0].get("message", {}) or {}
content = msg.get("content", "")
if isinstance(content, list):
parts = [p.get("text", "") for p in content if p.get("type") == "text"]
return "\n".join(parts)
return str(content)
def fetch_context(query: str, project: str) -> str:
"""Pull a context pack from AtoCore. Returns '' on any failure."""
if not INJECT_ENABLED or not query:
return ""
try:
data = json.dumps({"prompt": query, "project": project}).encode("utf-8")
req = urllib.request.Request(
ATOCORE_URL + "/context/build",
data=data,
method="POST",
headers={"Content-Type": "application/json"},
)
with urllib.request.urlopen(req, timeout=ATOCORE_TIMEOUT) as resp:
result = json.loads(resp.read().decode("utf-8"))
return result.get("formatted_context", "") or ""
except Exception as e:
log(f"context fetch failed: {type(e).__name__}: {e}")
return ""
def capture_interaction(prompt: str, response: str, project: str) -> None:
"""POST the completed turn back to AtoCore. Fire-and-forget."""
if not CAPTURE_ENABLED or not prompt or not response:
return
def _post():
try:
data = json.dumps({
"prompt": prompt,
"response": response,
"client": CLIENT_LABEL,
"project": project,
"reinforce": True,
}).encode("utf-8")
req = urllib.request.Request(
ATOCORE_URL + "/interactions",
data=data,
method="POST",
headers={"Content-Type": "application/json"},
)
urllib.request.urlopen(req, timeout=ATOCORE_TIMEOUT)
except Exception as e:
log(f"capture failed: {type(e).__name__}: {e}")
threading.Thread(target=_post, daemon=True).start()
def inject_context(body: dict, context_pack: str) -> dict:
"""Prepend the AtoCore context as a system message, or augment existing."""
if not context_pack.strip():
return body
header = "--- AtoCore Context (auto-injected) ---\n"
footer = "\n--- End AtoCore Context ---\n"
injection = header + context_pack + footer
messages = list(body.get("messages", []) or [])
if messages and messages[0].get("role") == "system":
# Augment existing system message
existing = messages[0].get("content", "") or ""
if isinstance(existing, list):
# multi-part: prepend a text part
messages[0]["content"] = [{"type": "text", "text": injection}] + existing
else:
messages[0]["content"] = injection + "\n" + str(existing)
else:
messages.insert(0, {"role": "system", "content": injection})
body["messages"] = messages
return body
def forward_to_upstream(body: dict, headers: dict[str, str], path: str) -> tuple[int, dict]:
"""Forward the enriched body to the upstream provider. Returns (status, response_dict)."""
if not UPSTREAM_URL:
return 503, {"error": {"message": "ATOCORE_UPSTREAM not configured"}}
url = UPSTREAM_URL + path
data = json.dumps(body).encode("utf-8")
# Strip hop-by-hop / host-specific headers
fwd_headers = {"Content-Type": "application/json"}
for k, v in headers.items():
lk = k.lower()
if lk in ("authorization", "x-api-key", "anthropic-version"):
fwd_headers[k] = v
req = urllib.request.Request(url, data=data, method="POST", headers=fwd_headers)
try:
with urllib.request.urlopen(req, timeout=UPSTREAM_TIMEOUT) as resp:
return resp.status, json.loads(resp.read().decode("utf-8"))
except urllib.error.HTTPError as e:
try:
body_bytes = e.read()
payload = json.loads(body_bytes.decode("utf-8"))
except Exception:
payload = {"error": {"message": f"upstream HTTP {e.code}"}}
return e.code, payload
except Exception as e:
log(f"upstream error: {e}")
return 502, {"error": {"message": f"upstream unreachable: {e}"}}
class ProxyHandler(http.server.BaseHTTPRequestHandler):
# Silence default request logging (we log what matters ourselves)
def log_message(self, format: str, *args: Any) -> None:
pass
def _read_body(self) -> dict:
length = int(self.headers.get("Content-Length", "0") or "0")
if length <= 0:
return {}
raw = self.rfile.read(length)
try:
return json.loads(raw.decode("utf-8"))
except Exception:
return {}
def _send_json(self, status: int, payload: dict) -> None:
body = json.dumps(payload).encode("utf-8")
self.send_response(status)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(body)))
self.send_header("Access-Control-Allow-Origin", "*")
self.end_headers()
self.wfile.write(body)
def do_OPTIONS(self) -> None: # CORS preflight
self.send_response(204)
self.send_header("Access-Control-Allow-Origin", "*")
self.send_header("Access-Control-Allow-Methods", "POST, GET, OPTIONS")
self.send_header("Access-Control-Allow-Headers", "Content-Type, Authorization, X-API-Key")
self.end_headers()
def do_GET(self) -> None:
parsed = urllib.parse.urlparse(self.path)
if parsed.path == "/healthz":
self._send_json(200, {
"status": "ok",
"atocore": ATOCORE_URL,
"upstream": UPSTREAM_URL or "(not configured)",
"inject": INJECT_ENABLED,
"capture": CAPTURE_ENABLED,
})
return
# Pass through GET to upstream (model listing etc)
if not UPSTREAM_URL:
self._send_json(503, {"error": {"message": "ATOCORE_UPSTREAM not configured"}})
return
try:
req = urllib.request.Request(UPSTREAM_URL + parsed.path + (f"?{parsed.query}" if parsed.query else ""))
for k in ("Authorization", "X-API-Key"):
v = self.headers.get(k)
if v:
req.add_header(k, v)
with urllib.request.urlopen(req, timeout=UPSTREAM_TIMEOUT) as resp:
data = resp.read()
self.send_response(resp.status)
self.send_header("Content-Type", resp.headers.get("Content-Type", "application/json"))
self.send_header("Content-Length", str(len(data)))
self.end_headers()
self.wfile.write(data)
except Exception as e:
self._send_json(502, {"error": {"message": f"upstream error: {e}"}})
def do_POST(self) -> None:
parsed = urllib.parse.urlparse(self.path)
body = self._read_body()
# Only enrich chat completions; other endpoints pass through
if parsed.path.endswith("/chat/completions") or parsed.path == "/v1/chat/completions":
prompt = get_last_user_message(body)
project = detect_project(prompt)
context = fetch_context(prompt, project) if prompt else ""
if context:
log(f"inject: project={project or '(none)'} chars={len(context)}")
body = inject_context(body, context)
status, response = forward_to_upstream(body, dict(self.headers), parsed.path)
self._send_json(status, response)
if status == 200:
assistant_text = get_assistant_text(response)
capture_interaction(prompt, assistant_text, project)
else:
# Non-chat endpoints (embeddings, completions, etc.) — pure passthrough
status, response = forward_to_upstream(body, dict(self.headers), parsed.path)
self._send_json(status, response)
class ThreadedServer(socketserver.ThreadingMixIn, http.server.HTTPServer):
daemon_threads = True
allow_reuse_address = True
def main() -> int:
if not UPSTREAM_URL:
log("WARNING: ATOCORE_UPSTREAM not set. Chat completions will fail.")
log("Example: ATOCORE_UPSTREAM=http://localhost:11434/v1 for Ollama")
server = ThreadedServer((PROXY_HOST, PROXY_PORT), ProxyHandler)
log(f"listening on {PROXY_HOST}:{PROXY_PORT}")
log(f"AtoCore: {ATOCORE_URL} inject={INJECT_ENABLED} capture={CAPTURE_ENABLED}")
log(f"Upstream: {UPSTREAM_URL or '(not configured)'}")
log(f"Client label: {CLIENT_LABEL}")
log("Ready. Point your OpenAI-compatible client at /v1/chat/completions")
try:
server.serve_forever()
except KeyboardInterrupt:
log("stopping")
server.server_close()
return 0
if __name__ == "__main__":
sys.exit(main())

View File

@@ -0,0 +1,79 @@
#!/usr/bin/env python3
"""Auto-promote reinforced candidates + expire stale ones.
Phase 10: reinforcement-based auto-promotion. Candidates referenced
by 3+ interactions with confidence >= 0.7 graduate to active.
Candidates unreinforced for 14+ days are auto-rejected.
Usage:
python3 scripts/auto_promote_reinforced.py [--base-url URL] [--dry-run]
"""
from __future__ import annotations
import argparse
import json
import os
import sys
# Allow importing from src/ when run from repo root
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "src"))
from atocore.memory.service import auto_promote_reinforced, expire_stale_candidates
def main() -> None:
parser = argparse.ArgumentParser(description="Auto-promote + expire candidates")
parser.add_argument("--dry-run", action="store_true", help="Report only, don't change anything")
parser.add_argument("--min-refs", type=int, default=3, help="Min reference_count for promotion")
parser.add_argument("--min-confidence", type=float, default=0.7, help="Min confidence for promotion")
parser.add_argument("--expire-days", type=int, default=14, help="Days before unreinforced candidates expire")
args = parser.parse_args()
if args.dry_run:
print("DRY RUN — no changes will be made")
# For dry-run, query directly and report
from atocore.models.database import get_connection
from datetime import datetime, timedelta, timezone
cutoff_promote = (datetime.now(timezone.utc) - timedelta(days=args.expire_days)).strftime("%Y-%m-%d %H:%M:%S")
cutoff_expire = cutoff_promote
with get_connection() as conn:
promotable = conn.execute(
"SELECT id, content, memory_type, project, confidence, reference_count "
"FROM memories WHERE status = 'candidate' "
"AND COALESCE(reference_count, 0) >= ? AND confidence >= ? "
"AND last_referenced_at >= ?",
(args.min_refs, args.min_confidence, cutoff_promote),
).fetchall()
expirable = conn.execute(
"SELECT id, content, memory_type, project "
"FROM memories WHERE status = 'candidate' "
"AND COALESCE(reference_count, 0) = 0 AND created_at < ?",
(cutoff_expire,),
).fetchall()
print(f"\nWould promote {len(promotable)} candidates:")
for r in promotable:
print(f" [{r['memory_type']}] refs={r['reference_count']} conf={r['confidence']:.2f} | {r['content'][:80]}...")
print(f"\nWould expire {len(expirable)} stale candidates:")
for r in expirable:
print(f" [{r['memory_type']}] {r['project'] or 'global'} | {r['content'][:80]}...")
return
promoted = auto_promote_reinforced(
min_reference_count=args.min_refs,
min_confidence=args.min_confidence,
)
expired = expire_stale_candidates(max_age_days=args.expire_days)
print(f"promoted={len(promoted)} expired={len(expired)}")
if promoted:
print(f"Promoted IDs: {promoted}")
if expired:
print(f"Expired IDs: {expired}")
if __name__ == "__main__":
main()

284
scripts/auto_triage.py Normal file
View File

@@ -0,0 +1,284 @@
"""Auto-triage: LLM second-pass over candidate memories.
Fetches all status=candidate memories from the AtoCore API, asks
a triage model (via claude -p) to classify each as promote / reject /
needs_human, and executes the verdict via the promote/reject endpoints.
Only needs_human candidates remain in the queue for manual review.
Trust model:
- Auto-promote: model says promote AND confidence >= 0.8 AND no
duplicate content in existing active memories
- Auto-reject: model says reject
- needs_human: everything else stays in queue
Runs host-side (same as batch extraction) because it needs the
claude CLI. Intended to be called after batch-extract.sh in the
nightly cron, or manually.
Usage:
python3 scripts/auto_triage.py --base-url http://localhost:8100
python3 scripts/auto_triage.py --dry-run # preview without executing
"""
from __future__ import annotations
import argparse
import json
import os
import shutil
import subprocess
import sys
import time
import tempfile
import urllib.error
import urllib.parse
import urllib.request
DEFAULT_BASE_URL = os.environ.get("ATOCORE_BASE_URL", "http://localhost:8100")
DEFAULT_MODEL = os.environ.get("ATOCORE_TRIAGE_MODEL", "sonnet")
DEFAULT_TIMEOUT_S = float(os.environ.get("ATOCORE_TRIAGE_TIMEOUT_S", "60"))
AUTO_PROMOTE_MIN_CONFIDENCE = 0.8
TRIAGE_SYSTEM_PROMPT = """You are a memory triage reviewer for a personal context engine called AtoCore. You review candidate memories extracted from LLM conversations and decide whether each should be promoted to active status, rejected, or flagged for human review.
You will receive:
- The candidate memory content and type
- A list of existing active memories for the same project (to check for duplicates)
For each candidate, output exactly one JSON object:
{"verdict": "promote|reject|needs_human|contradicts", "confidence": 0.0-1.0, "reason": "one sentence", "conflicts_with": "id of existing memory if contradicts"}
Rules:
1. PROMOTE when the candidate states a durable architectural fact, ratified decision, standing rule, or engineering constraint that is NOT already covered by an existing active memory. Confidence should reflect how certain you are this is worth keeping.
2. REJECT when the candidate is:
- A stale point-in-time snapshot ("live SHA is X", "36 active memories")
- An implementation detail too granular to be useful as standalone context
- A planned-but-not-implemented feature description
- A duplicate or near-duplicate of an existing active memory
- A session observation or conversational filler
- A process rule that belongs in DEV-LEDGER.md or AGENTS.md, not memory
3. CONTRADICTS when the candidate *conflicts* with an existing active memory (not a duplicate, but states something that can't both be true). Set `conflicts_with` to the existing memory id. This flags the tension for human review instead of silently rejecting or double-storing. Examples: "Option A selected" vs "Option B selected" for the same decision; "uses material X" vs "uses material Y" for the same component.
4. OPENCLAW-CURATED content (candidate content starts with "From OpenClaw/"): apply a MUCH LOWER bar. OpenClaw's SOUL.md, USER.md, MEMORY.md, MODEL-ROUTING.md, and dated memory/*.md files are ALREADY curated by OpenClaw as canonical continuity. Promote unless clearly wrong or a genuine duplicate. Do NOT reject OpenClaw content as "process rule belongs elsewhere" or "session log" — that's exactly what AtoCore wants to absorb. Session events, project updates, stakeholder notes, and decisions from OpenClaw daily memory files ARE valuable context and should promote.
5. NEEDS_HUMAN when you're genuinely unsure — the candidate might be valuable but you can't tell without domain knowledge. This should be rare (< 20% of candidates).
6. Output ONLY the JSON object. No prose, no markdown, no explanation outside the reason field."""
_sandbox_cwd = None
def get_sandbox_cwd():
global _sandbox_cwd
if _sandbox_cwd is None:
_sandbox_cwd = tempfile.mkdtemp(prefix="ato-triage-")
return _sandbox_cwd
def api_get(base_url, path, timeout=10):
req = urllib.request.Request(f"{base_url}{path}")
with urllib.request.urlopen(req, timeout=timeout) as resp:
return json.loads(resp.read().decode("utf-8"))
def api_post(base_url, path, body=None, timeout=10):
data = json.dumps(body or {}).encode("utf-8")
req = urllib.request.Request(
f"{base_url}{path}", method="POST",
headers={"Content-Type": "application/json"}, data=data,
)
with urllib.request.urlopen(req, timeout=timeout) as resp:
return json.loads(resp.read().decode("utf-8"))
def fetch_active_memories_for_project(base_url, project):
"""Fetch active memories for dedup checking."""
params = "active_only=true&limit=50"
if project:
params += f"&project={urllib.parse.quote(project)}"
result = api_get(base_url, f"/memory?{params}")
return result.get("memories", [])
def triage_one(candidate, active_memories, model, timeout_s):
"""Ask the triage model to classify one candidate."""
if not shutil.which("claude"):
return {"verdict": "needs_human", "confidence": 0.0, "reason": "claude CLI not available"}
active_summary = "\n".join(
f"- [{m['memory_type']}] {m['content'][:150]}"
for m in active_memories[:20]
) or "(no active memories for this project)"
user_message = (
f"CANDIDATE TO TRIAGE:\n"
f" type: {candidate['memory_type']}\n"
f" project: {candidate.get('project') or '(none)'}\n"
f" content: {candidate['content']}\n\n"
f"EXISTING ACTIVE MEMORIES FOR THIS PROJECT:\n{active_summary}\n\n"
f"Return the JSON verdict now."
)
args = [
"claude", "-p",
"--model", model,
"--append-system-prompt", TRIAGE_SYSTEM_PROMPT,
"--disable-slash-commands",
user_message,
]
# Retry with exponential backoff on transient failures (rate limits etc)
last_error = ""
for attempt in range(3):
if attempt > 0:
time.sleep(2 ** attempt) # 2s, 4s
try:
completed = subprocess.run(
args, capture_output=True, text=True,
timeout=timeout_s, cwd=get_sandbox_cwd(),
encoding="utf-8", errors="replace",
)
except subprocess.TimeoutExpired:
last_error = "triage model timed out"
continue
except Exception as exc:
last_error = f"subprocess error: {exc}"
continue
if completed.returncode == 0:
raw = (completed.stdout or "").strip()
return parse_verdict(raw)
# Capture stderr for diagnostics (truncate to 200 chars)
stderr = (completed.stderr or "").strip()[:200]
last_error = f"claude exit {completed.returncode}: {stderr}" if stderr else f"claude exit {completed.returncode}"
return {"verdict": "needs_human", "confidence": 0.0, "reason": last_error}
def parse_verdict(raw):
"""Parse the triage model's JSON verdict."""
text = raw.strip()
if text.startswith("```"):
text = text.strip("`")
nl = text.find("\n")
if nl >= 0:
text = text[nl + 1:]
if text.endswith("```"):
text = text[:-3]
text = text.strip()
if not text.lstrip().startswith("{"):
start = text.find("{")
end = text.rfind("}")
if start >= 0 and end > start:
text = text[start:end + 1]
try:
parsed = json.loads(text)
except json.JSONDecodeError:
return {"verdict": "needs_human", "confidence": 0.0, "reason": "failed to parse triage output"}
verdict = str(parsed.get("verdict", "needs_human")).strip().lower()
if verdict not in {"promote", "reject", "needs_human", "contradicts"}:
verdict = "needs_human"
confidence = parsed.get("confidence", 0.5)
try:
confidence = max(0.0, min(1.0, float(confidence)))
except (TypeError, ValueError):
confidence = 0.5
reason = str(parsed.get("reason", "")).strip()[:200]
conflicts_with = str(parsed.get("conflicts_with", "")).strip()
return {
"verdict": verdict,
"confidence": confidence,
"reason": reason,
"conflicts_with": conflicts_with,
}
def main():
parser = argparse.ArgumentParser(description="Auto-triage candidate memories")
parser.add_argument("--base-url", default=DEFAULT_BASE_URL)
parser.add_argument("--model", default=DEFAULT_MODEL)
parser.add_argument("--dry-run", action="store_true", help="preview without executing")
args = parser.parse_args()
# Fetch candidates
result = api_get(args.base_url, "/memory?status=candidate&limit=100")
candidates = result.get("memories", [])
print(f"candidates: {len(candidates)} model: {args.model} dry_run: {args.dry_run}")
if not candidates:
print("queue empty, nothing to triage")
return
# Cache active memories per project for dedup
active_cache = {}
promoted = rejected = needs_human = errors = 0
for i, cand in enumerate(candidates, 1):
# Light rate-limit pacing: 0.5s between triage calls so a burst
# doesn't overwhelm the claude CLI's backend. With ~60s per call
# this is negligible overhead but avoids the "all-failed" pattern
# we saw on large batches.
if i > 1:
time.sleep(0.5)
project = cand.get("project") or ""
if project not in active_cache:
active_cache[project] = fetch_active_memories_for_project(args.base_url, project)
verdict_obj = triage_one(cand, active_cache[project], args.model, DEFAULT_TIMEOUT_S)
verdict = verdict_obj["verdict"]
conf = verdict_obj["confidence"]
reason = verdict_obj["reason"]
conflicts_with = verdict_obj.get("conflicts_with", "")
mid = cand["id"]
label = f"[{i:2d}/{len(candidates)}] {mid[:8]} [{cand['memory_type']}]"
if verdict == "promote" and conf >= AUTO_PROMOTE_MIN_CONFIDENCE:
if args.dry_run:
print(f" WOULD PROMOTE {label} conf={conf:.2f} {reason}")
else:
try:
api_post(args.base_url, f"/memory/{mid}/promote")
print(f" PROMOTED {label} conf={conf:.2f} {reason}")
active_cache[project].append(cand)
except Exception:
errors += 1
promoted += 1
elif verdict == "reject":
if args.dry_run:
print(f" WOULD REJECT {label} conf={conf:.2f} {reason}")
else:
try:
api_post(args.base_url, f"/memory/{mid}/reject")
print(f" REJECTED {label} conf={conf:.2f} {reason}")
except Exception:
errors += 1
rejected += 1
elif verdict == "contradicts":
# Leave candidate in queue but flag the conflict in content
# so the wiki/triage shows it. This is conservative: we
# don't silently merge or reject when sources disagree.
print(f" CONTRADICTS {label} vs {conflicts_with[:8] if conflicts_with else '?'} {reason}")
contradicts_count = locals().get('contradicts_count', 0) + 1
needs_human += 1
else:
print(f" NEEDS_HUMAN {label} conf={conf:.2f} {reason}")
needs_human += 1
print(f"\npromoted={promoted} rejected={rejected} needs_human={needs_human} errors={errors}")
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,268 @@
"""Host-side LLM batch extraction — HTTP client + shared prompt module.
Fetches interactions from the AtoCore API, runs ``claude -p`` locally
for each, and POSTs candidates back. Uses stdlib + the ``claude`` CLI
on PATH, plus the stdlib-only shared prompt/parser module at
``atocore.memory._llm_prompt`` to eliminate prompt/parser drift
against the in-container extractor (R12).
This is necessary because the ``claude`` CLI is on the Dalidou HOST
but not inside the Docker container, and the host's Python doesn't
have the container's dependencies (pydantic_settings, etc.) — so we
only import the one stdlib-only module, not the full atocore package.
"""
from __future__ import annotations
import argparse
import json
import os
import shutil
import subprocess
import sys
import tempfile
import urllib.error
import urllib.parse
import urllib.request
from datetime import datetime, timezone
# R12: share the prompt + parser with the in-container extractor so
# the two paths can't drift. The imported module is stdlib-only by
# design; see src/atocore/memory/_llm_prompt.py.
_SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
_SRC_DIR = os.path.abspath(os.path.join(_SCRIPT_DIR, "..", "src"))
if _SRC_DIR not in sys.path:
sys.path.insert(0, _SRC_DIR)
from atocore.memory._llm_prompt import ( # noqa: E402
MEMORY_TYPES,
SYSTEM_PROMPT,
build_user_message,
normalize_candidate_item,
parse_llm_json_array,
)
DEFAULT_BASE_URL = os.environ.get("ATOCORE_BASE_URL", "http://localhost:8100")
DEFAULT_MODEL = os.environ.get("ATOCORE_LLM_EXTRACTOR_MODEL", "sonnet")
DEFAULT_TIMEOUT_S = float(os.environ.get("ATOCORE_LLM_EXTRACTOR_TIMEOUT_S", "90"))
_sandbox_cwd = None
def get_sandbox_cwd():
global _sandbox_cwd
if _sandbox_cwd is None:
_sandbox_cwd = tempfile.mkdtemp(prefix="ato-llm-extract-")
return _sandbox_cwd
def api_get(base_url, path, timeout=10):
req = urllib.request.Request(f"{base_url}{path}")
with urllib.request.urlopen(req, timeout=timeout) as resp:
return json.loads(resp.read().decode("utf-8"))
def api_post(base_url, path, body, timeout=10):
data = json.dumps(body).encode("utf-8")
req = urllib.request.Request(
f"{base_url}{path}", method="POST",
headers={"Content-Type": "application/json"}, data=data,
)
with urllib.request.urlopen(req, timeout=timeout) as resp:
return json.loads(resp.read().decode("utf-8"))
def get_last_run(base_url):
try:
state = api_get(base_url, "/project/state/atocore?category=status")
for entry in state.get("entries", []):
if entry.get("key") == "last_extract_batch_run":
return entry["value"]
except Exception:
pass
return None
def set_last_run(base_url, timestamp):
try:
api_post(base_url, "/project/state", {
"project": "atocore", "category": "status",
"key": "last_extract_batch_run", "value": timestamp,
"source": "batch_llm_extract_live.py",
})
except Exception:
pass
_known_projects: set[str] = set()
def _load_known_projects(base_url):
"""Fetch registered project IDs from the API for R9 validation."""
global _known_projects
try:
data = api_get(base_url, "/projects")
_known_projects = {p["id"] for p in data.get("projects", [])}
for p in data.get("projects", []):
for alias in p.get("aliases", []):
_known_projects.add(alias)
except Exception:
pass
def extract_one(prompt, response, project, model, timeout_s):
"""Run claude -p on one interaction, return parsed candidates."""
if not shutil.which("claude"):
return [], "claude_cli_missing"
user_message = build_user_message(prompt, response, project)
args = [
"claude", "-p",
"--model", model,
"--append-system-prompt", SYSTEM_PROMPT,
"--disable-slash-commands",
user_message,
]
# Retry with exponential backoff on transient failures (rate limits etc)
import time as _time
last_error = ""
for attempt in range(3):
if attempt > 0:
_time.sleep(2 ** attempt) # 2s, 4s
try:
completed = subprocess.run(
args, capture_output=True, text=True,
timeout=timeout_s, cwd=get_sandbox_cwd(),
encoding="utf-8", errors="replace",
)
except subprocess.TimeoutExpired:
last_error = "timeout"
continue
except Exception as exc:
last_error = f"subprocess_error: {exc}"
continue
if completed.returncode == 0:
raw = (completed.stdout or "").strip()
return parse_candidates(raw, project), ""
# Capture stderr for diagnostics (truncate to 200 chars)
stderr = (completed.stderr or "").strip()[:200]
last_error = f"exit_{completed.returncode}: {stderr}" if stderr else f"exit_{completed.returncode}"
return [], last_error
def parse_candidates(raw, interaction_project):
"""Parse model JSON output into candidate dicts.
Stripping + per-item normalization come from the shared
``_llm_prompt`` module. Host-side project attribution: interaction
scope wins, otherwise keep the model's tag (the API's own R9
registry-check will happen server-side in the container on write;
here we preserve the signal instead of dropping it).
"""
results = []
for item in parse_llm_json_array(raw):
normalized = normalize_candidate_item(item)
if normalized is None:
continue
project = interaction_project or normalized["project"] or ""
results.append({
"memory_type": normalized["type"],
"content": normalized["content"],
"project": project,
"confidence": normalized["confidence"],
})
return results
def main():
parser = argparse.ArgumentParser(description="Host-side LLM batch extraction")
parser.add_argument("--base-url", default=DEFAULT_BASE_URL)
parser.add_argument("--limit", type=int, default=50)
parser.add_argument("--since", default=None)
parser.add_argument("--model", default=DEFAULT_MODEL)
args = parser.parse_args()
_load_known_projects(args.base_url)
since = args.since or get_last_run(args.base_url)
print(f"since={since or '(first run)'} limit={args.limit} model={args.model} known_projects={len(_known_projects)}")
params = [f"limit={args.limit}"]
if since:
params.append(f"since={urllib.parse.quote(since)}")
listing = api_get(args.base_url, f"/interactions?{'&'.join(params)}")
interaction_summaries = listing.get("interactions", [])
print(f"listed {len(interaction_summaries)} interactions")
processed = 0
total_candidates = 0
total_persisted = 0
errors = 0
import time as _time
for ix, summary in enumerate(interaction_summaries):
resp_chars = summary.get("response_chars", 0) or 0
if resp_chars < 50:
continue
# Light pacing between calls to avoid bursting the claude CLI
if ix > 0:
_time.sleep(0.5)
iid = summary["id"]
try:
raw = api_get(
args.base_url,
f"/interactions/{urllib.parse.quote(iid, safe='')}",
)
except Exception as exc:
print(f" ! {iid[:8]}: fetch failed: {exc}", file=sys.stderr)
errors += 1
continue
response_text = raw.get("response", "") or ""
if not response_text.strip() or len(response_text) < 50:
continue
candidates, error = extract_one(
prompt=raw.get("prompt", "") or "",
response=response_text,
project=raw.get("project", "") or "",
model=args.model,
timeout_s=DEFAULT_TIMEOUT_S,
)
if error:
print(f" ! {raw['id'][:8]}: {error}", file=sys.stderr)
errors += 1
continue
processed += 1
total_candidates += len(candidates)
for c in candidates:
try:
api_post(args.base_url, "/memory", {
"memory_type": c["memory_type"],
"content": c["content"],
"project": c["project"],
"confidence": c["confidence"],
"status": "candidate",
})
total_persisted += 1
except urllib.error.HTTPError as exc:
if exc.code != 400:
errors += 1
except Exception:
errors += 1
now = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S")
set_last_run(args.base_url, now)
print(f"processed={processed} candidates={total_candidates} persisted={total_persisted} errors={errors}")
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,188 @@
"""Bootstrap engineering entities from existing project knowledge.
One-shot script that seeds the entity/relationship graph from what
AtoCore already knows via memories, project state, and vault docs.
Safe to re-run — uses name+project dedup.
Usage:
python3 scripts/bootstrap_entities.py --base-url http://localhost:8100
"""
from __future__ import annotations
import argparse
import json
import os
import urllib.request
DEFAULT_BASE_URL = os.environ.get("ATOCORE_BASE_URL", "http://dalidou:8100")
def post(base_url, path, body):
data = json.dumps(body).encode("utf-8")
req = urllib.request.Request(
f"{base_url}{path}", method="POST",
headers={"Content-Type": "application/json"}, data=data,
)
try:
with urllib.request.urlopen(req, timeout=10) as resp:
return json.loads(resp.read().decode("utf-8"))
except Exception as e:
return {"error": str(e)}
def entity(base_url, etype, name, project="", desc="", props=None):
result = post(base_url, "/entities", {
"entity_type": etype, "name": name, "project": project,
"description": desc, "properties": props or {},
})
eid = result.get("id", "")
status = "+" if eid else "skip"
print(f" {status} [{etype}] {name}")
return eid
def rel(base_url, src, tgt, rtype):
if not src or not tgt:
return
result = post(base_url, "/relationships", {
"source_entity_id": src, "target_entity_id": tgt,
"relationship_type": rtype,
})
print(f" -> {rtype}")
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--base-url", default=DEFAULT_BASE_URL)
args = parser.parse_args()
b = args.base_url
print("=== P04 GigaBIT M1 ===")
p04 = entity(b, "project", "GigaBIT M1", "p04-gigabit",
"1.2m primary mirror for stratospheric balloon telescope")
p04_m1 = entity(b, "system", "M1 Mirror Assembly", "p04-gigabit",
"Primary mirror blank + support system + reference frame")
rel(b, p04, p04_m1, "contains")
p04_vs = entity(b, "subsystem", "Vertical Support", "p04-gigabit",
"18-point whiffletree axial support from below")
p04_ls = entity(b, "subsystem", "Lateral Support", "p04-gigabit",
"Circumferential constraint system with GF-PTFE pads")
p04_rf = entity(b, "subsystem", "Reference Frame", "p04-gigabit",
"Structural mounting interface between mirror and OTA")
p04_blank = entity(b, "component", "M1 Blank", "p04-gigabit",
"1.2m Zerodur aspheric blank from Schott",
{"material": "Zerodur", "diameter_m": 1.2, "focal_ratio": "F/1.2"})
rel(b, p04_m1, p04_vs, "contains")
rel(b, p04_m1, p04_ls, "contains")
rel(b, p04_m1, p04_rf, "contains")
rel(b, p04_m1, p04_blank, "contains")
p04_zerodur = entity(b, "material", "Zerodur", "p04-gigabit",
"Glass-ceramic with near-zero CTE for mirror blanks")
p04_ptfe = entity(b, "material", "GF-PTFE", "p04-gigabit",
"Glass-filled PTFE for thermal stability on lateral pads")
rel(b, p04_blank, p04_zerodur, "uses_material")
rel(b, p04_ls, p04_ptfe, "uses_material")
p04_optb = entity(b, "decision", "Option B Conical Back", "p04-gigabit",
"Selected mirror architecture: conical-back lightweighting")
rel(b, p04_optb, p04_blank, "affected_by_decision")
p04_wfe = entity(b, "requirement", "WFE < 15nm RMS filtered", "p04-gigabit",
"Filtered mechanical wavefront error below 15 nm across 20-60 deg elevation")
p04_mass = entity(b, "requirement", "Mass < 103.5 kg", "p04-gigabit",
"Total mirror assembly mass constraint")
rel(b, p04_m1, p04_wfe, "constrained_by")
rel(b, p04_m1, p04_mass, "constrained_by")
print("\n=== P05 Interferometer ===")
p05 = entity(b, "project", "Interferometer System", "p05-interferometer",
"Metrology system for GigaBIT M1 figuring")
p05_rig = entity(b, "system", "Test Rig", "p05-interferometer",
"Folded-beam interferometric test setup for M1 measurement")
rel(b, p05, p05_rig, "contains")
p05_ifm = entity(b, "component", "Interferometer", "p05-interferometer",
"Fixed horizontal Twyman-Green dynamic interferometer")
p05_fold = entity(b, "component", "Fold Mirror", "p05-interferometer",
"45-degree beam redirect, <= lambda/20 surface quality")
p05_cgh = entity(b, "component", "CGH Null Corrector", "p05-interferometer",
"6-inch transmission CGH for F/1.2 asphere null test",
{"diameter": "6 inch", "substrate": "fused silica", "error_budget_nm": 5.5})
p05_tilt = entity(b, "subsystem", "Tilting Platform", "p05-interferometer",
"Mirror tilting platform, co-tilts with interferometer")
rel(b, p05_rig, p05_ifm, "contains")
rel(b, p05_rig, p05_fold, "contains")
rel(b, p05_rig, p05_cgh, "contains")
rel(b, p05_rig, p05_tilt, "contains")
rel(b, p05_ifm, p05_fold, "interfaces_with")
rel(b, p05_cgh, p05_tilt, "interfaces_with")
p05_vendor_dec = entity(b, "decision", "Vendor Path: Twyman-Green preferred", "p05-interferometer",
"4D technical lead but cost-challenged; Zygo Verifire SV at 55K is value path")
p05_vendor_zygo = entity(b, "vendor", "Zygo / AMETEK", "p05-interferometer",
"Certified used Verifire SV, 55K, Nabeel Sufi contact")
p05_vendor_4d = entity(b, "vendor", "4D Technology", "p05-interferometer",
"PC6110/PC4030, above budget but strongest technical option")
p05_vendor_aom = entity(b, "vendor", "AOM (CGH)", "p05-interferometer",
"CGH design and fabrication, 28-30K package")
rel(b, p05_vendor_dec, p05_ifm, "affected_by_decision")
print("\n=== P06 Polisher ===")
p06 = entity(b, "project", "Polisher System", "p06-polisher",
"Machine overhaul + software suite for optical polishing")
p06_machine = entity(b, "system", "Polisher Machine", "p06-polisher",
"Swing-arm polishing machine with force modulation")
p06_sw = entity(b, "system", "Software Suite", "p06-polisher",
"Three-layer software: polisher-sim, polisher-post, polisher-control")
rel(b, p06, p06_machine, "contains")
rel(b, p06, p06_sw, "contains")
p06_sim = entity(b, "subsystem", "polisher-sim", "p06-polisher",
"Digital twin: surface assimilation, removal simulation, planning")
p06_post = entity(b, "subsystem", "polisher-post", "p06-polisher",
"Bridge: validation, translation, packaging for machine")
p06_ctrl = entity(b, "subsystem", "polisher-control", "p06-polisher",
"Executor: state machine, interlocks, telemetry, run logs")
rel(b, p06_sw, p06_sim, "contains")
rel(b, p06_sw, p06_post, "contains")
rel(b, p06_sw, p06_ctrl, "contains")
rel(b, p06_sim, p06_post, "interfaces_with")
rel(b, p06_post, p06_ctrl, "interfaces_with")
p06_fc = entity(b, "subsystem", "Force Control", "p06-polisher",
"Frame-grounded counterweight actuator with cable tension modulation",
{"actuator_capacity_N": "150-200", "compliance_spring_Nmm": "3-5"})
p06_zaxis = entity(b, "component", "Z-Axis", "p06-polisher",
"Binary engage/retract mechanism, not continuous position")
p06_cam = entity(b, "component", "Cam Mechanism", "p06-polisher",
"Mechanically set by operator, read by encoders, not actuated")
rel(b, p06_machine, p06_fc, "contains")
rel(b, p06_machine, p06_zaxis, "contains")
rel(b, p06_machine, p06_cam, "contains")
p06_fw = entity(b, "decision", "Firmware Interface Contract", "p06-polisher",
"controller-job.v1 in, run-log.v1 + telemetry out — invariant")
p06_offline = entity(b, "decision", "Offline-First Design", "p06-polisher",
"Machine works fully offline; network is for remote access only")
p06_usb = entity(b, "decision", "USB SSD Storage", "p06-polisher",
"USB SSD mandatory on RPi, not SD card")
p06_contracts = entity(b, "constraint", "Shared Contracts", "p06-polisher",
"Stable IDs, explicit versions, hashable artifacts, planned-vs-executed separation")
rel(b, p06_sw, p06_contracts, "constrained_by")
p06_preston = entity(b, "parameter", "Preston Coefficient kp", "p06-polisher",
"Calibrated from before/after surface measurements, multi-run inverse-variance weighting")
print(f"\nDone.")
if __name__ == "__main__":
main()

File diff suppressed because one or more lines are too long

View File

@@ -0,0 +1,29 @@
1. [project ] proj=atocore AtoCore extraction must stay off the hot capture path; batch endpoint only
2. [project ] proj=atocore Auto-promote gate: confidence ≥0.8 AND no duplicate in active memories
3. [project ] proj=atocore AtoCore LLM extraction pipeline deployed on Dalidou host, runs via cron at 03:00 UTC via scripts/batch_llm_extract_live.py
4. [project ] proj=atocore LLM extractor runs host-side (not in container) because claude CLI not available in container environment
5. [project ] proj=atocore Host-side extraction script scripts/batch_llm_extract_live.py uses pure stdlib, no atocore imports for deployment simplicity
6. [project ] proj=atocore POST /admin/extract-batch accepts mode: rule|llm, POST /interactions/{id}/extract now mode-aware
7. [knowledge ] proj=atocore claude CLI 2.0.60 removed --no-session-persistence flag, extraction sessions now persist in claude history
8. [adaptation ] proj=atocore Durable memory extraction candidates must be <200 chars, stand-alone, typed as project|knowledge|preference|adaptation
9. [adaptation ] proj=atocore Memory extraction confidence defaults to 0.5, raise to 0.6 only for unambiguous committed claims
10. [project ] proj=atocore Live Dalidou is on commit 39d73e9, not e2895b5
11. [project ] proj=atocore Live harness is reproducible at 16/18 PASS
12. [project ] proj=atocore Live active memories count is 36
13. [project ] proj=atocore Wave 2 project-state entries on live: p04=5, p05=6, p06=6
14. [project ] proj=atocore R6 is fixed by commit 39d73e9
15. [project ] proj=atocore R9: R6 fix only covers empty project fallback; wrong non-empty model project can still override known interaction scope
16. [project ] proj=atocore R10: Phase 8 is baseline-complete but not primary-complete; OpenClaw client covers narrow read-oriented slice of API
17. [project ] proj=atocore Phase 8 is decent baseline integration milestone but not primary-ready yet
18. [project ] proj=atocore 4-step roadmap complete: extractor → harness → Wave 2 → OpenClaw
19. [project ] proj=atocore Codex audit loop proven across two full round-trips in one session
20. [project ] proj=atocore Session end state: 36 active memories, 17 project-state entries, 16/18 harness, 280 tests, main at 54d84b5
21. [project ] proj=atocore AtoCore extraction stays off the hot capture path; LLM extraction runs as scheduled batch, not inline with POST /interactions.
22. [project ] proj=atocore AtoCore auto-triage trust model: auto-promote only when confidence ≥0.8 AND no duplicate active memory; else needs_human.
23. [project ] proj=atocore Multi-model triage: use different model for triage reviewer than extractor (sonnet for extract)
24. [project ] proj=atocore R9 fix: when interaction has known project, prefer it over model's non-matching project unless model's is registered
25. [project ] proj=atocore R7 ranking fix: add overlap-density as secondary signal (overlap_count / memory_token_count)
26. [project ] proj=atocore Extraction pipeline skips interactions with response_chars < 50 to avoid low-signal content
27. [project ] proj=atocore AtoCore triage uses independent model from extractor (extractor: sonnet, triage: different model or different prompt).
28. [project ] proj=atocore AtoCore ranking scorer adds overlap-density (overlap_count / memory_tokens) as secondary signal to fix short-memory ranking.
29. [project ] proj=atocore AtoCore project trust: when interaction has known project and model returns different project, prefer interaction's project unless

View File

@@ -0,0 +1,254 @@
"""OpenClaw state importer — one-way pull from clawdbot into AtoCore.
Reads OpenClaw's file continuity layer (SOUL.md, USER.md, MODEL-ROUTING.md,
MEMORY.md, memory/YYYY-MM-DD.md) from the T420 via SSH and imports them
into AtoCore as candidate memories. Hash-based delta detection — only
re-imports files that changed since the last run.
Classification per codex's integration proposal:
- SOUL.md -> identity candidates
- USER.md -> identity + preference candidates
- MODEL-ROUTING.md -> adaptation candidates (routing rules)
- MEMORY.md -> long-term memory candidates (type varies)
- memory/YYYY-MM-DD.md -> episodic memory candidates (daily logs)
- heartbeat-state.json -> skipped (ops metadata only)
All candidates land as status=candidate. Auto-triage filters noise.
This importer is conservative: it doesn't promote directly, it just
feeds signal. The triage pipeline decides what graduates to active.
Usage:
python3 scripts/import_openclaw_state.py \
--base-url http://localhost:8100 \
--openclaw-host papa@192.168.86.39 \
--openclaw-path /home/papa/openclaw-workspace
Runs nightly via cron (added as Step 2c in cron-backup.sh).
"""
from __future__ import annotations
import argparse
import hashlib
import json
import os
import shutil
import subprocess
import sys
import tempfile
import urllib.error
import urllib.request
from pathlib import Path
DEFAULT_BASE_URL = os.environ.get("ATOCORE_BASE_URL", "http://localhost:8100")
DEFAULT_OPENCLAW_HOST = os.environ.get("ATOCORE_OPENCLAW_HOST", "papa@192.168.86.39")
DEFAULT_OPENCLAW_PATH = os.environ.get("ATOCORE_OPENCLAW_PATH", "/home/papa/clawd")
# Files to pull and how to classify them
DURABLE_FILES = [
("SOUL.md", "identity"),
("USER.md", "identity"),
("MODEL-ROUTING.md", "adaptation"),
("MEMORY.md", "memory"), # type parsed from entries
]
DAILY_MEMORY_GLOB = "memory/*.md"
HASH_STATE_KEY = "openclaw_import_hashes"
def api_get(base_url, path):
try:
with urllib.request.urlopen(f"{base_url}{path}", timeout=15) as r:
return json.loads(r.read())
except Exception:
return None
def api_post(base_url, path, body):
data = json.dumps(body).encode("utf-8")
req = urllib.request.Request(
f"{base_url}{path}", method="POST",
headers={"Content-Type": "application/json"}, data=data,
)
try:
with urllib.request.urlopen(req, timeout=15) as r:
return json.loads(r.read())
except urllib.error.HTTPError as exc:
if exc.code == 400:
return {"skipped": True}
raise
def ssh_cat(host, remote_path):
"""Cat a remote file via SSH. Returns content or None if missing."""
try:
result = subprocess.run(
["ssh", "-o", "ConnectTimeout=5", "-o", "BatchMode=yes",
host, f"cat {remote_path}"],
capture_output=True, text=True, timeout=30,
encoding="utf-8", errors="replace",
)
if result.returncode == 0:
return result.stdout
except Exception:
pass
return None
def ssh_ls(host, remote_glob):
"""List files matching a glob on the remote host."""
try:
result = subprocess.run(
["ssh", "-o", "ConnectTimeout=5", "-o", "BatchMode=yes",
host, f"ls -1 {remote_glob} 2>/dev/null"],
capture_output=True, text=True, timeout=10,
encoding="utf-8", errors="replace",
)
if result.returncode == 0:
return [line.strip() for line in result.stdout.splitlines() if line.strip()]
except Exception:
pass
return []
def content_hash(text):
return hashlib.sha256(text.encode("utf-8")).hexdigest()[:16]
def load_hash_state(base_url):
"""Load the hash state from project_state so we know what's changed."""
state = api_get(base_url, "/project/state/atocore?category=status")
if not state:
return {}
for entry in state.get("entries", []):
if entry.get("key") == HASH_STATE_KEY:
try:
return json.loads(entry["value"])
except Exception:
return {}
return {}
def save_hash_state(base_url, hashes):
api_post(base_url, "/project/state", {
"project": "atocore",
"category": "status",
"key": HASH_STATE_KEY,
"value": json.dumps(hashes),
"source": "import_openclaw_state.py",
})
def import_file_as_memory(base_url, filename, content, memory_type, source_tag):
"""Import a file's content as a single candidate memory for triage."""
# Trim to reasonable size — auto-triage can handle long content but
# we don't want single mega-memories dominating the queue
trimmed = content[:2000]
if len(content) > 2000:
trimmed += f"\n\n[...truncated from {len(content)} chars]"
body = {
"memory_type": memory_type,
"content": f"From OpenClaw/{filename}: {trimmed}",
"project": "", # global/identity, not project-scoped
"confidence": 0.5,
"status": "candidate",
}
return api_post(base_url, "/memory", body)
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--base-url", default=DEFAULT_BASE_URL)
parser.add_argument("--openclaw-host", default=DEFAULT_OPENCLAW_HOST)
parser.add_argument("--openclaw-path", default=DEFAULT_OPENCLAW_PATH)
parser.add_argument("--dry-run", action="store_true")
args = parser.parse_args()
print(f"openclaw_host={args.openclaw_host} openclaw_path={args.openclaw_path}")
print(f"dry_run={args.dry_run}")
# Check SSH connectivity first
test = ssh_cat(args.openclaw_host, f"{args.openclaw_path}/SOUL.md")
if test is None:
print("ERROR: cannot reach OpenClaw workspace via SSH or SOUL.md not found")
print("Check: ssh key installed? path correct? workspace exists?")
return 1
hashes = load_hash_state(args.base_url)
imported = skipped = errors = 0
# 1. Durable files
for filename, mem_type in DURABLE_FILES:
remote = f"{args.openclaw_path}/{filename}"
content = ssh_cat(args.openclaw_host, remote)
if content is None or not content.strip():
print(f" - {filename}: not found or empty")
continue
h = content_hash(content)
if hashes.get(filename) == h:
print(f" = {filename}: unchanged (hash {h})")
skipped += 1
continue
print(f" + {filename}: changed (hash {h}, {len(content)}ch)")
if not args.dry_run:
try:
result = import_file_as_memory(
args.base_url, filename, content, mem_type,
source_tag="openclaw-durable",
)
if result.get("skipped"):
print(f" (duplicate content, skipped)")
else:
print(f" -> candidate {result.get('id', '?')[:8]}")
imported += 1
hashes[filename] = h
except Exception as e:
print(f" ! error: {e}")
errors += 1
# 2. Daily memory logs (memory/YYYY-MM-DD.md)
daily_glob = f"{args.openclaw_path}/{DAILY_MEMORY_GLOB}"
daily_files = ssh_ls(args.openclaw_host, daily_glob)
print(f"\ndaily memory files: {len(daily_files)}")
# Only process the most recent 7 daily files to avoid flooding
for remote_path in sorted(daily_files)[-7:]:
filename = Path(remote_path).name
content = ssh_cat(args.openclaw_host, remote_path)
if content is None or not content.strip():
continue
h = content_hash(content)
key = f"daily/{filename}"
if hashes.get(key) == h:
print(f" = {filename}: unchanged")
skipped += 1
continue
print(f" + {filename}: changed ({len(content)}ch)")
if not args.dry_run:
try:
result = import_file_as_memory(
args.base_url, filename, content, "episodic",
source_tag="openclaw-daily",
)
if not result.get("skipped"):
print(f" -> candidate {result.get('id', '?')[:8]}")
imported += 1
hashes[key] = h
except Exception as e:
print(f" ! error: {e}")
errors += 1
# Save hash state
if not args.dry_run and imported > 0:
save_hash_state(args.base_url, hashes)
print(f"\nimported={imported} skipped={skipped} errors={errors}")
print("Candidates queued — auto-triage will filter them on next run.")
if __name__ == "__main__":
raise SystemExit(main() or 0)

View File

@@ -0,0 +1,170 @@
"""Weekly lint pass — health check for the AtoCore knowledge base.
Inspired by Karpathy's LLM Wiki pattern (the 'lint' operation).
Checks for orphans, stale claims, contradictions, and gaps.
Outputs a report that can be posted to the wiki as needs_review.
Usage:
python3 scripts/lint_knowledge_base.py --base-url http://dalidou:8100
Run weekly via cron, or on-demand when the knowledge base feels stale.
"""
from __future__ import annotations
import argparse
import json
import os
import urllib.request
from datetime import datetime, timezone, timedelta
DEFAULT_BASE_URL = os.environ.get("ATOCORE_BASE_URL", "http://localhost:8100")
ORPHAN_AGE_DAYS = 14
def api_get(base_url: str, path: str):
with urllib.request.urlopen(f"{base_url}{path}", timeout=15) as r:
return json.loads(r.read())
def parse_ts(ts: str) -> datetime | None:
if not ts:
return None
try:
return datetime.strptime(ts[:19], "%Y-%m-%d %H:%M:%S").replace(tzinfo=timezone.utc)
except Exception:
return None
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--base-url", default=DEFAULT_BASE_URL)
args = parser.parse_args()
b = args.base_url
now = datetime.now(timezone.utc)
orphan_threshold = now - timedelta(days=ORPHAN_AGE_DAYS)
print(f"=== AtoCore Lint — {now.strftime('%Y-%m-%d %H:%M UTC')} ===\n")
findings = {
"orphan_memories": [],
"stale_candidates": [],
"unused_entities": [],
"empty_state_projects": [],
"unregistered_projects": [],
}
# 1. Orphan memories: active but never reinforced after N days
memories = api_get(b, "/memory?active_only=true&limit=500").get("memories", [])
for m in memories:
updated = parse_ts(m.get("updated_at", ""))
if m.get("reference_count", 0) == 0 and updated and updated < orphan_threshold:
findings["orphan_memories"].append({
"id": m["id"],
"type": m["memory_type"],
"project": m.get("project") or "(none)",
"age_days": (now - updated).days,
"content": m["content"][:120],
})
# 2. Stale candidates: been in queue > 7 days without triage
candidates = api_get(b, "/memory?status=candidate&limit=500").get("memories", [])
stale_threshold = now - timedelta(days=7)
for c in candidates:
updated = parse_ts(c.get("updated_at", ""))
if updated and updated < stale_threshold:
findings["stale_candidates"].append({
"id": c["id"],
"age_days": (now - updated).days,
"content": c["content"][:120],
})
# 3. Unused entities: no relationships in either direction
entities = api_get(b, "/entities?limit=500").get("entities", [])
for e in entities:
try:
detail = api_get(b, f"/entities/{e['id']}")
if not detail.get("relationships"):
findings["unused_entities"].append({
"id": e["id"],
"type": e["entity_type"],
"name": e["name"],
"project": e.get("project") or "(none)",
})
except Exception:
pass
# 4. Registered projects with no state entries
try:
projects = api_get(b, "/projects").get("projects", [])
for p in projects:
state = api_get(b, f"/project/state/{p['id']}").get("entries", [])
if not state:
findings["empty_state_projects"].append(p["id"])
except Exception:
pass
# 5. Memories tagged to unregistered projects (auto-detection candidates)
registered_ids = {p["id"] for p in projects} | {
a for p in projects for a in p.get("aliases", [])
}
all_mems = api_get(b, "/memory?limit=500").get("memories", [])
for m in all_mems:
proj = m.get("project", "")
if proj and proj not in registered_ids and proj != "(none)":
if proj not in findings["unregistered_projects"]:
findings["unregistered_projects"].append(proj)
# Print report
print(f"## Orphan memories (active, no reinforcement, >{ORPHAN_AGE_DAYS} days old)")
if findings["orphan_memories"]:
print(f" Found: {len(findings['orphan_memories'])}")
for o in findings["orphan_memories"][:10]:
print(f" - [{o['type']}] {o['project']} ({o['age_days']}d): {o['content']}")
else:
print(" (none)")
print(f"\n## Stale candidates (>7 days in queue)")
if findings["stale_candidates"]:
print(f" Found: {len(findings['stale_candidates'])}")
for s in findings["stale_candidates"][:10]:
print(f" - ({s['age_days']}d): {s['content']}")
else:
print(" (none)")
print(f"\n## Unused entities (no relationships)")
if findings["unused_entities"]:
print(f" Found: {len(findings['unused_entities'])}")
for u in findings["unused_entities"][:10]:
print(f" - [{u['type']}] {u['project']}: {u['name']}")
else:
print(" (none)")
print(f"\n## Empty-state projects")
if findings["empty_state_projects"]:
print(f" Found: {len(findings['empty_state_projects'])}")
for p in findings["empty_state_projects"]:
print(f" - {p}")
else:
print(" (none)")
print(f"\n## Unregistered projects detected in memories")
if findings["unregistered_projects"]:
print(f" Found: {len(findings['unregistered_projects'])}")
print(" These were auto-detected by extraction — consider registering them:")
for p in findings["unregistered_projects"]:
print(f" - {p}")
else:
print(" (none)")
total_findings = sum(
len(v) if isinstance(v, list) else 0 for v in findings.values()
)
print(f"\n=== Total findings: {total_findings} ===")
# Return exit code based on findings count (for CI)
return 0 if total_findings == 0 else 1
if __name__ == "__main__":
raise SystemExit(main())

View File

@@ -218,8 +218,8 @@
"Tailscale" "Tailscale"
], ],
"expect_absent": [ "expect_absent": [
"GigaBIT" "[Source: p04-gigabit/"
], ],
"notes": "New p06 memory: Tailscale mesh for RPi remote access" "notes": "New p06 memory: Tailscale mesh for RPi remote access. Cross-project guard is a source-path check, not a word blacklist: the polisher ARCHITECTURE.md legitimately mentions the GigaBIT M1 mirror (it is what the polisher is built for), so testing for absence of that word produces false positives. The real invariant is that no p04 source chunks are retrieved into p06 context."
} }
] ]

View File

@@ -0,0 +1,159 @@
#!/usr/bin/env python3
"""Seed Trusted Project State entries for all active projects.
Populates the project_state table with curated decisions, requirements,
facts, contacts, and milestones so context packs have real content
in the highest-trust tier.
Usage:
python3 scripts/seed_project_state.py --base-url http://dalidou:8100
python3 scripts/seed_project_state.py --base-url http://dalidou:8100 --dry-run
"""
from __future__ import annotations
import argparse
import json
import urllib.request
import sys
# Each entry: (project, category, key, value, source)
SEED_ENTRIES: list[tuple[str, str, str, str, str]] = [
# ---- p04-gigabit (GigaBIT M1 1.2m Primary Mirror) ----
("p04-gigabit", "fact", "mirror-spec",
"1.2m borosilicate primary mirror for GigaBIT telescope. F/1.5, lightweight isogrid back structure.",
"CDR docs + vault"),
("p04-gigabit", "decision", "back-structure",
"Option B selected: conical isogrid back structure with variable rib density. Chosen over flat-back for stiffness-to-weight ratio.",
"CDR 2026-01"),
("p04-gigabit", "decision", "polishing-vendor",
"ABB Space (formerly INO) selected as polishing vendor. Contract includes computer-controlled polishing (CCP) and ion beam figuring (IBF).",
"Entente de service 2026-01"),
("p04-gigabit", "requirement", "surface-quality",
"Surface figure accuracy: < 25nm RMS after final figuring. Microroughness: < 2nm RMS.",
"CDR requirements"),
("p04-gigabit", "contact", "abb-space",
"ABB Space (INO), Quebec City. Primary contact for mirror polishing, CCP, and IBF. Project lead: coordinating FDR deliverables.",
"vendor records"),
("p04-gigabit", "milestone", "fdr",
"Final Design Review (FDR) in preparation. Deliverables include interface drawings, thermal analysis, and updated error budget.",
"project timeline"),
# ---- p05-interferometer (Fullum Interferometer) ----
("p05-interferometer", "fact", "system-overview",
"Custom Fizeau interferometer for in-situ metrology of large optics. Designed for the Fullum observatory polishing facility.",
"vault docs"),
("p05-interferometer", "decision", "cgh-design",
"Computer-generated hologram (CGH) selected for null testing of the 1.2m mirror. Vendor: Diffraction International.",
"vendor correspondence"),
("p05-interferometer", "requirement", "measurement-accuracy",
"Measurement accuracy target: lambda/20 (< 30nm PV) for surface figure verification.",
"system requirements"),
("p05-interferometer", "fact", "laser-source",
"HeNe laser source at 632.8nm. Beam expansion to cover full 1.2m aperture via diverger + CGH.",
"optical design docs"),
("p05-interferometer", "contact", "diffraction-intl",
"Diffraction International: CGH vendor. Fabricates the computer-generated hologram for null testing.",
"vendor records"),
# ---- p06-polisher (Polisher Suite / P11-Polisher-Fullum) ----
("p06-polisher", "fact", "suite-overview",
"Integrated CNC polishing suite for the Fullum observatory. Includes 3-axis polishing machine, metrology integration, and real-time process control.",
"vault docs"),
("p06-polisher", "decision", "control-architecture",
"Beckhoff TwinCAT 3 selected for real-time motion control. EtherCAT fieldbus for servo drives and I/O.",
"architecture docs"),
("p06-polisher", "decision", "firmware-split",
"Firmware split into safety layer (PLC-level interlocks) and application layer (trajectory generation, adaptive dwell-time).",
"architecture docs"),
("p06-polisher", "requirement", "axis-travel",
"Z-axis: 200mm travel for tool engagement. X/Y: covers 1.2m mirror diameter plus overshoot margin.",
"mechanical requirements"),
("p06-polisher", "fact", "telemetry",
"Real-time telemetry via MQTT. Metrics: spindle RPM, force sensor, temperature probes, position feedback at 1kHz.",
"control design docs"),
("p06-polisher", "contact", "fullum-observatory",
"Fullum Observatory: site where the polishing suite will be installed. Provides infrastructure (power, vibration isolation, clean environment).",
"project records"),
# ---- atomizer-v2 ----
("atomizer-v2", "fact", "product-overview",
"Atomizer V2: internal project management and multi-agent orchestration platform. War-room based task coordination.",
"repo docs"),
("atomizer-v2", "decision", "projects-first-architecture",
"Migration to projects-first architecture: each project is a workspace with its own agents, tasks, and knowledge.",
"war-room-migration-plan-v2.md"),
# ---- abb-space (P08) ----
("abb-space", "fact", "contract-overview",
"ABB Space mirror polishing contract. Phase 1: spherical mirror polishing (200mm). Schott Zerodur substrate.",
"quotes + correspondence"),
("abb-space", "contact", "schott",
"Schott AG: substrate supplier for Zerodur mirror blanks. Quote received for 200mm blank.",
"vendor records"),
# ---- atocore ----
("atocore", "fact", "architecture",
"AtoCore: runtime memory and knowledge layer. FastAPI + SQLite + ChromaDB. Hosted on Dalidou (Docker). Nightly pipeline: backup, extract, triage, synthesis.",
"codebase"),
("atocore", "decision", "no-api-keys",
"No API keys allowed in AtoCore. LLM-assisted features use OAuth via 'claude -p' CLI or equivalent CLI-authenticated paths.",
"DEV-LEDGER 2026-04-12"),
("atocore", "decision", "storage-separation",
"Human-readable sources (vault, drive) and machine operational storage (SQLite, ChromaDB) must remain separate. Machine DB is derived state.",
"AGENTS.md"),
("atocore", "decision", "extraction-off-hot-path",
"Extraction stays off the capture hot path. Batch/manual only. Never block interaction recording with extraction.",
"DEV-LEDGER 2026-04-11"),
]
def main() -> None:
parser = argparse.ArgumentParser(description="Seed Trusted Project State")
parser.add_argument("--base-url", default="http://dalidou:8100")
parser.add_argument("--dry-run", action="store_true")
args = parser.parse_args()
base = args.base_url.rstrip("/")
created = 0
skipped = 0
errors = 0
for project, category, key, value, source in SEED_ENTRIES:
if args.dry_run:
print(f" [DRY] {project}/{category}/{key}: {value[:60]}...")
created += 1
continue
body = json.dumps({
"project": project,
"category": category,
"key": key,
"value": value,
"source": source,
"confidence": 1.0,
}).encode()
req = urllib.request.Request(
f"{base}/project/state",
data=body,
headers={"Content-Type": "application/json"},
method="POST",
)
try:
resp = urllib.request.urlopen(req, timeout=10)
result = json.loads(resp.read())
if result.get("created"):
created += 1
print(f" + {project}/{category}/{key}")
else:
skipped += 1
print(f" = {project}/{category}/{key} (already exists)")
except Exception as e:
errors += 1
print(f" ! {project}/{category}/{key}: {e}", file=sys.stderr)
print(f"\nDone: {created} created, {skipped} skipped, {errors} errors")
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,168 @@
"""Weekly project synthesis — LLM-generated 'current state' paragraph per project.
Reads each registered project's state entries, memories, and entities,
asks sonnet for a 3-5 sentence synthesis, and caches it under
project_state/status/synthesis_cache. The wiki's project page reads
this cached synthesis as the top band.
Runs weekly via cron (or manually). Cheap — one LLM call per project.
Usage:
python3 scripts/synthesize_projects.py --base-url http://localhost:8100
"""
from __future__ import annotations
import argparse
import json
import os
import shutil
import subprocess
import tempfile
import urllib.request
DEFAULT_BASE_URL = os.environ.get("ATOCORE_BASE_URL", "http://localhost:8100")
DEFAULT_MODEL = os.environ.get("ATOCORE_SYNTHESIS_MODEL", "sonnet")
TIMEOUT_S = 60
SYSTEM_PROMPT = """You are summarizing the current state of an engineering project for a personal context engine called AtoCore.
You will receive:
- Project state entries (decisions, requirements, status)
- Active memories tagged to this project
- Entity graph (subsystems, components, materials, decisions)
Write a 3-5 sentence synthesis covering:
1. What the project is and its current stage
2. The key locked-in decisions and architecture
3. What the next focus is
Rules:
- Plain prose, no bullet lists
- Factual, grounded in what the data says — don't invent or speculate
- Present tense
- Under 500 characters total
- No markdown formatting, just prose
- If the data is sparse, say so honestly ("limited project data available")
Output ONLY the synthesis paragraph. No preamble, no JSON, no markdown headers."""
_cwd = None
def get_cwd():
global _cwd
if _cwd is None:
_cwd = tempfile.mkdtemp(prefix="ato-synth-")
return _cwd
def api_get(base_url, path):
with urllib.request.urlopen(f"{base_url}{path}", timeout=15) as r:
return json.loads(r.read())
def api_post(base_url, path, body):
data = json.dumps(body).encode("utf-8")
req = urllib.request.Request(
f"{base_url}{path}", method="POST",
headers={"Content-Type": "application/json"}, data=data,
)
with urllib.request.urlopen(req, timeout=15) as r:
return json.loads(r.read())
def synthesize_project(base_url, project_id, model):
# Gather context
state = api_get(base_url, f"/project/state/{project_id}").get("entries", [])
memories = api_get(base_url, f"/memory?project={project_id}&active_only=true&limit=20").get("memories", [])
entities = api_get(base_url, f"/entities?project={project_id}&limit=50").get("entities", [])
if not (state or memories or entities):
return None
lines = [f"PROJECT: {project_id}\n"]
if state:
lines.append("STATE ENTRIES:")
for e in state[:15]:
if e.get("key") == "synthesis_cache":
continue
lines.append(f" [{e['category']}] {e['key']}: {e['value'][:200]}")
if memories:
lines.append("\nACTIVE MEMORIES:")
for m in memories[:10]:
lines.append(f" [{m['memory_type']}] {m['content'][:200]}")
if entities:
lines.append("\nENTITIES:")
by_type = {}
for e in entities:
by_type.setdefault(e["entity_type"], []).append(e["name"])
for t, names in by_type.items():
lines.append(f" {t}: {', '.join(names[:8])}")
user_msg = "\n".join(lines) + "\n\nWrite the synthesis paragraph now."
if not shutil.which("claude"):
print(f" ! claude CLI not available, skipping {project_id}")
return None
try:
result = subprocess.run(
["claude", "-p", "--model", model,
"--append-system-prompt", SYSTEM_PROMPT,
"--disable-slash-commands",
user_msg],
capture_output=True, text=True, timeout=TIMEOUT_S,
cwd=get_cwd(), encoding="utf-8", errors="replace",
)
except Exception as e:
print(f" ! subprocess failed for {project_id}: {e}")
return None
if result.returncode != 0:
print(f" ! claude exit {result.returncode} for {project_id}")
return None
synthesis = (result.stdout or "").strip()
if not synthesis or len(synthesis) < 50:
return None
return synthesis[:1000]
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--base-url", default=DEFAULT_BASE_URL)
parser.add_argument("--model", default=DEFAULT_MODEL)
parser.add_argument("--project", default=None, help="single project to synthesize")
args = parser.parse_args()
projects = api_get(args.base_url, "/projects").get("projects", [])
if args.project:
projects = [p for p in projects if p["id"] == args.project]
print(f"Synthesizing {len(projects)} project(s) with {args.model}...")
for p in projects:
pid = p["id"]
print(f"\n- {pid}")
synthesis = synthesize_project(args.base_url, pid, args.model)
if synthesis:
print(f" {synthesis[:200]}...")
try:
api_post(args.base_url, "/project/state", {
"project": pid,
"category": "status",
"key": "synthesis_cache",
"value": synthesis,
"source": "weekly synthesis pass",
})
print(f" + cached")
except Exception as e:
print(f" ! save failed: {e}")
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,87 @@
# atocore-backup-pull.ps1
#
# Pull the latest AtoCore backup snapshot from Dalidou to this Windows machine.
# Designed to be run by Windows Task Scheduler. Fail-open by design -- if
# Dalidou is unreachable (laptop on the road, etc.), exit cleanly without error.
#
# Usage (manual test):
# powershell.exe -ExecutionPolicy Bypass -File atocore-backup-pull.ps1
#
# Scheduled task: see docs/windows-backup-setup.md for Task Scheduler config.
$ErrorActionPreference = "Continue"
# --- Configuration ---
$Remote = "papa@dalidou"
$RemoteSnapshots = "/srv/storage/atocore/backups/snapshots"
$LocalBackupDir = "$env:USERPROFILE\Documents\ATOCore_Backups"
$LogDir = "$LocalBackupDir\_logs"
$ReachabilityTest = 5 # seconds timeout for SSH probe
# --- Setup ---
if (-not (Test-Path $LocalBackupDir)) {
New-Item -ItemType Directory -Path $LocalBackupDir -Force | Out-Null
}
if (-not (Test-Path $LogDir)) {
New-Item -ItemType Directory -Path $LogDir -Force | Out-Null
}
$Timestamp = Get-Date -Format "yyyy-MM-dd_HHmmss"
$LogFile = "$LogDir\backup-$Timestamp.log"
function Log($msg) {
$line = "[{0}] {1}" -f (Get-Date -Format "yyyy-MM-dd HH:mm:ss"), $msg
Write-Host $line
Add-Content -Path $LogFile -Value $line
}
Log "=== AtoCore backup pull starting ==="
Log "Remote: $Remote"
Log "Local target: $LocalBackupDir"
# --- Reachability check: fail open if Dalidou is offline ---
Log "Checking Dalidou reachability..."
$probe = & ssh -o ConnectTimeout=$ReachabilityTest -o BatchMode=yes `
-o StrictHostKeyChecking=accept-new `
$Remote "echo ok" 2>&1
if ($LASTEXITCODE -ne 0 -or $probe -ne "ok") {
Log "Dalidou unreachable ($probe) -- fail-open exit"
exit 0
}
Log "Dalidou reachable."
# --- Pull the entire snapshots directory ---
# Dalidou's retention policy (7 daily + 4 weekly + 6 monthly) already caps
# the snapshot count, so pulling the whole dir is bounded and simple. scp
# will overwrite local files -- we rely on this to pick up new snapshots.
Log "Pulling snapshots via scp..."
$LocalSnapshotsDir = Join-Path $LocalBackupDir "snapshots"
if (-not (Test-Path $LocalSnapshotsDir)) {
New-Item -ItemType Directory -Path $LocalSnapshotsDir -Force | Out-Null
}
& scp -o BatchMode=yes -r "${Remote}:${RemoteSnapshots}/*" "$LocalSnapshotsDir\" 2>&1 |
ForEach-Object { Add-Content -Path $LogFile -Value $_ }
if ($LASTEXITCODE -ne 0) {
Log "scp failed with exit $LASTEXITCODE"
exit 0 # fail-open
}
# --- Stats ---
$snapshots = Get-ChildItem -Path $LocalSnapshotsDir -Directory |
Where-Object { $_.Name -match "^\d{8}T\d{6}Z$" } |
Sort-Object Name -Descending
$totalSize = (Get-ChildItem $LocalSnapshotsDir -Recurse -File | Measure-Object -Property Length -Sum).Sum
$SizeMB = [math]::Round($totalSize / 1MB, 2)
$latest = if ($snapshots.Count -gt 0) { $snapshots[0].Name } else { "(none)" }
Log ("Pulled {0} snapshots successfully (total {1} MB, latest: {2})" -f $snapshots.Count, $SizeMB, $latest)
Log "=== backup complete ==="
# --- Log retention: keep last 30 log files ---
Get-ChildItem -Path $LogDir -Filter "backup-*.log" |
Sort-Object Name -Descending |
Select-Object -Skip 30 |
ForEach-Object { Remove-Item $_.FullName -Force -ErrorAction SilentlyContinue }

View File

@@ -3,6 +3,7 @@
from pathlib import Path from pathlib import Path
from fastapi import APIRouter, HTTPException from fastapi import APIRouter, HTTPException
from fastapi.responses import HTMLResponse
from pydantic import BaseModel from pydantic import BaseModel
import atocore.config as _config import atocore.config as _config
@@ -30,11 +31,33 @@ from atocore.interactions.service import (
list_interactions, list_interactions,
record_interaction, record_interaction,
) )
from atocore.engineering.mirror import generate_project_overview
from atocore.engineering.wiki import (
render_entity,
render_homepage,
render_project,
render_search,
)
from atocore.engineering.service import (
ENTITY_TYPES,
RELATIONSHIP_TYPES,
create_entity,
create_relationship,
get_entities,
get_entity,
get_entity_with_context,
get_relationships,
)
from atocore.memory.extractor import ( from atocore.memory.extractor import (
EXTRACTOR_VERSION, EXTRACTOR_VERSION,
MemoryCandidate, MemoryCandidate,
extract_candidates_from_interaction, extract_candidates_from_interaction,
) )
from atocore.memory.extractor_llm import (
LLM_EXTRACTOR_VERSION,
_cli_available as _llm_cli_available,
extract_candidates_llm,
)
from atocore.memory.reinforcement import reinforce_from_interaction from atocore.memory.reinforcement import reinforce_from_interaction
from atocore.memory.service import ( from atocore.memory.service import (
MEMORY_STATUSES, MEMORY_STATUSES,
@@ -69,6 +92,33 @@ router = APIRouter()
log = get_logger("api") log = get_logger("api")
# --- Wiki routes (HTML, served first for clean URLs) ---
@router.get("/wiki", response_class=HTMLResponse)
def wiki_home() -> HTMLResponse:
return HTMLResponse(content=render_homepage())
@router.get("/wiki/projects/{project_name}", response_class=HTMLResponse)
def wiki_project(project_name: str) -> HTMLResponse:
from atocore.projects.registry import resolve_project_name as _resolve
return HTMLResponse(content=render_project(_resolve(project_name)))
@router.get("/wiki/entities/{entity_id}", response_class=HTMLResponse)
def wiki_entity(entity_id: str) -> HTMLResponse:
html = render_entity(entity_id)
if html is None:
raise HTTPException(status_code=404, detail="Entity not found")
return HTMLResponse(content=html)
@router.get("/wiki/search", response_class=HTMLResponse)
def wiki_search(q: str = "") -> HTMLResponse:
return HTMLResponse(content=render_search(q))
# --- Request/Response models --- # --- Request/Response models ---
@@ -580,6 +630,7 @@ def api_reinforce_interaction(interaction_id: str) -> dict:
class InteractionExtractRequest(BaseModel): class InteractionExtractRequest(BaseModel):
persist: bool = False persist: bool = False
mode: str = "rule" # "rule" or "llm"
@router.post("/interactions/{interaction_id}/extract") @router.post("/interactions/{interaction_id}/extract")
@@ -601,7 +652,10 @@ def api_extract_from_interaction(
if interaction is None: if interaction is None:
raise HTTPException(status_code=404, detail=f"Interaction not found: {interaction_id}") raise HTTPException(status_code=404, detail=f"Interaction not found: {interaction_id}")
payload = req or InteractionExtractRequest() payload = req or InteractionExtractRequest()
candidates: list[MemoryCandidate] = extract_candidates_from_interaction(interaction) if payload.mode == "llm":
candidates: list[MemoryCandidate] = extract_candidates_llm(interaction)
else:
candidates: list[MemoryCandidate] = extract_candidates_from_interaction(interaction)
persisted_ids: list[str] = [] persisted_ids: list[str] = []
if payload.persist: if payload.persist:
@@ -755,6 +809,460 @@ def api_cleanup_backups(req: BackupCleanupRequest | None = None) -> dict:
raise HTTPException(status_code=500, detail=f"Cleanup failed: {e}") raise HTTPException(status_code=500, detail=f"Cleanup failed: {e}")
class ExtractBatchRequest(BaseModel):
since: str | None = None
mode: str = "llm"
limit: int = 50
persist: bool = True
@router.post("/admin/extract-batch")
def api_extract_batch(req: ExtractBatchRequest | None = None) -> dict:
"""Run batch extraction across recent interactions.
Fetches interactions since ``since`` (or since the last recorded
batch run), runs the extractor (rule or LLM) on each, and persists
any candidates as ``status=candidate``. The last-run timestamp is
stored in project state under ``atocore / status /
last_extract_batch_run`` so subsequent calls without ``since``
automatically pick up where the last run left off.
This endpoint is the operational home for R1 / R5 — it makes the
LLM extractor accessible as an API operation rather than a
script-only eval tool. Still NOT on the capture hot path: callers
invoke this endpoint explicitly (cron, manual curl, CLI).
"""
payload = req or ExtractBatchRequest()
if payload.mode == "llm" and not _llm_cli_available():
raise HTTPException(
status_code=503,
detail=(
"LLM extraction unavailable in this runtime: the `claude` CLI "
"is not on PATH. Run host-side via "
"`scripts/batch_llm_extract_live.py` instead, or call this "
"endpoint with mode=\"rule\"."
),
)
since = payload.since
if not since:
state_entries = get_state("atocore")
for entry in state_entries:
if entry.category == "status" and entry.key == "last_extract_batch_run":
since = entry.value
break
interactions = list_interactions(since=since, limit=min(payload.limit, 200))
processed = 0
total_candidates = 0
total_persisted = 0
errors: list[dict] = []
for interaction in interactions:
if not (interaction.response or interaction.response_summary):
continue
try:
if payload.mode == "llm":
candidates = extract_candidates_llm(interaction)
else:
candidates = extract_candidates_from_interaction(interaction)
except Exception as exc:
errors.append({"interaction_id": interaction.id, "error": str(exc)})
continue
processed += 1
total_candidates += len(candidates)
if payload.persist and candidates:
for candidate in candidates:
try:
create_memory(
memory_type=candidate.memory_type,
content=candidate.content,
project=candidate.project,
confidence=candidate.confidence,
status="candidate",
)
total_persisted += 1
except ValueError:
pass # duplicate — skip silently
from datetime import datetime, timezone
now = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S")
try:
set_state(
project="atocore",
category="status",
key="last_extract_batch_run",
value=now,
source="admin/extract-batch endpoint",
)
except Exception:
pass # best-effort timestamp tracking
log.info(
"extract_batch_complete",
mode=payload.mode,
processed=processed,
total_candidates=total_candidates,
total_persisted=total_persisted,
errors=len(errors),
)
return {
"processed": processed,
"total_candidates": total_candidates,
"total_persisted": total_persisted,
"mode": payload.mode,
"persist": payload.persist,
"since": since or "(first run)",
"errors": errors,
}
@router.get("/admin/dashboard")
def api_dashboard() -> dict:
"""One-shot system observability dashboard.
Returns memory counts by type/project/status, project state
entry counts, interaction volume by client, pipeline health
(harness, triage stats, last run), and extraction pipeline
status — everything an operator needs to understand AtoCore's
health beyond the basic /health endpoint.
"""
import json as _json
from collections import Counter
from datetime import datetime as _dt, timezone as _tz
all_memories = get_memories(active_only=False, limit=500)
active = [m for m in all_memories if m.status == "active"]
candidates = [m for m in all_memories if m.status == "candidate"]
type_counts = dict(Counter(m.memory_type for m in active))
project_counts = dict(Counter(m.project or "(none)" for m in active))
reinforced = [m for m in active if m.reference_count > 0]
# Interaction stats — total + by_client from DB directly
interaction_stats: dict = {"most_recent": None, "total": 0, "by_client": {}}
try:
from atocore.models.database import get_connection as _gc
with _gc() as conn:
row = conn.execute("SELECT count(*) FROM interactions").fetchone()
interaction_stats["total"] = row[0] if row else 0
rows = conn.execute(
"SELECT client, count(*) FROM interactions GROUP BY client"
).fetchall()
interaction_stats["by_client"] = {r[0]: r[1] for r in rows}
row = conn.execute(
"SELECT created_at FROM interactions ORDER BY created_at DESC LIMIT 1"
).fetchone()
interaction_stats["most_recent"] = row[0] if row else None
except Exception:
interactions = list_interactions(limit=1)
interaction_stats["most_recent"] = (
interactions[0].created_at if interactions else None
)
# Pipeline health from project state
pipeline: dict = {}
extract_state: dict = {}
try:
state_entries = get_state("atocore")
for entry in state_entries:
if entry.category != "status":
continue
if entry.key == "last_extract_batch_run":
extract_state["last_run"] = entry.value
elif entry.key == "pipeline_last_run":
pipeline["last_run"] = entry.value
try:
last = _dt.fromisoformat(entry.value.replace("Z", "+00:00"))
delta = _dt.now(_tz.utc) - last
pipeline["hours_since_last_run"] = round(
delta.total_seconds() / 3600, 1
)
except Exception:
pass
elif entry.key == "pipeline_summary":
try:
pipeline["summary"] = _json.loads(entry.value)
except Exception:
pipeline["summary_raw"] = entry.value
elif entry.key == "retrieval_harness_result":
try:
pipeline["harness"] = _json.loads(entry.value)
except Exception:
pipeline["harness_raw"] = entry.value
except Exception:
pass
# Project state counts — include all registered projects
ps_counts = {}
try:
from atocore.projects.registry import load_project_registry as _lpr
for proj in _lpr():
try:
entries = get_state(proj.project_id)
ps_counts[proj.project_id] = len(entries)
except Exception:
pass
except Exception:
for proj_id in [
"p04-gigabit", "p05-interferometer", "p06-polisher", "atocore",
]:
try:
entries = get_state(proj_id)
ps_counts[proj_id] = len(entries)
except Exception:
pass
return {
"memories": {
"active": len(active),
"candidates": len(candidates),
"by_type": type_counts,
"by_project": project_counts,
"reinforced": len(reinforced),
},
"project_state": {
"counts": ps_counts,
"total": sum(ps_counts.values()),
},
"interactions": interaction_stats,
"extraction_pipeline": extract_state,
"pipeline": pipeline,
}
# --- Engineering Knowledge Layer (Layer 2) ---
class EntityCreateRequest(BaseModel):
entity_type: str
name: str
project: str = ""
description: str = ""
properties: dict | None = None
status: str = "active"
confidence: float = 1.0
source_refs: list[str] | None = None
class RelationshipCreateRequest(BaseModel):
source_entity_id: str
target_entity_id: str
relationship_type: str
confidence: float = 1.0
source_refs: list[str] | None = None
@router.post("/entities")
def api_create_entity(req: EntityCreateRequest) -> dict:
"""Create a new engineering entity."""
try:
entity = create_entity(
entity_type=req.entity_type,
name=req.name,
project=req.project,
description=req.description,
properties=req.properties,
status=req.status,
confidence=req.confidence,
source_refs=req.source_refs,
)
except ValueError as e:
raise HTTPException(status_code=400, detail=str(e))
return {"status": "ok", "id": entity.id, "entity_type": entity.entity_type, "name": entity.name}
@router.get("/entities")
def api_list_entities(
entity_type: str | None = None,
project: str | None = None,
status: str = "active",
name_contains: str | None = None,
limit: int = 100,
) -> dict:
"""List engineering entities with optional filters."""
entities = get_entities(
entity_type=entity_type,
project=project,
status=status,
name_contains=name_contains,
limit=limit,
)
return {
"entities": [
{
"id": e.id,
"entity_type": e.entity_type,
"name": e.name,
"project": e.project,
"description": e.description,
"properties": e.properties,
"status": e.status,
"confidence": e.confidence,
}
for e in entities
],
"count": len(entities),
}
@router.get("/entities/{entity_id}")
def api_get_entity(entity_id: str) -> dict:
"""Get an entity with its relationships and related entities."""
result = get_entity_with_context(entity_id)
if result is None:
raise HTTPException(status_code=404, detail=f"Entity not found: {entity_id}")
entity = result["entity"]
return {
"entity": {
"id": entity.id,
"entity_type": entity.entity_type,
"name": entity.name,
"project": entity.project,
"description": entity.description,
"properties": entity.properties,
"status": entity.status,
"confidence": entity.confidence,
"source_refs": entity.source_refs,
"created_at": entity.created_at,
"updated_at": entity.updated_at,
},
"relationships": [
{
"id": r.id,
"source_entity_id": r.source_entity_id,
"target_entity_id": r.target_entity_id,
"relationship_type": r.relationship_type,
"confidence": r.confidence,
}
for r in result["relationships"]
],
"related_entities": {
eid: {
"entity_type": e.entity_type,
"name": e.name,
"project": e.project,
"description": e.description[:200],
}
for eid, e in result["related_entities"].items()
},
}
@router.post("/relationships")
def api_create_relationship(req: RelationshipCreateRequest) -> dict:
"""Create a relationship between two entities."""
try:
rel = create_relationship(
source_entity_id=req.source_entity_id,
target_entity_id=req.target_entity_id,
relationship_type=req.relationship_type,
confidence=req.confidence,
source_refs=req.source_refs,
)
except ValueError as e:
raise HTTPException(status_code=400, detail=str(e))
return {
"status": "ok",
"id": rel.id,
"relationship_type": rel.relationship_type,
}
@router.get("/projects/{project_name}/mirror.html", response_class=HTMLResponse)
def api_project_mirror_html(project_name: str) -> HTMLResponse:
"""Serve a readable HTML project overview page.
Open in a browser for a clean, styled project dashboard derived
from AtoCore's structured data. Source of truth is the database —
this page is a derived view.
"""
from atocore.projects.registry import resolve_project_name as _resolve
canonical = _resolve(project_name)
try:
md_content = generate_project_overview(canonical)
except Exception as e:
raise HTTPException(status_code=500, detail=f"Mirror generation failed: {e}")
import markdown
html_body = markdown.markdown(md_content, extensions=["tables", "fenced_code"])
html = _MIRROR_HTML_TEMPLATE.replace("{{title}}", f"{canonical} — AtoCore Mirror")
html = html.replace("{{body}}", html_body)
return HTMLResponse(content=html)
_MIRROR_HTML_TEMPLATE = """<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>{{title}}</title>
<style>
:root { --bg: #fafafa; --text: #1a1a2e; --accent: #2563eb; --border: #e2e8f0; --card: #fff; }
@media (prefers-color-scheme: dark) {
:root { --bg: #0f172a; --text: #e2e8f0; --accent: #60a5fa; --border: #334155; --card: #1e293b; }
}
* { box-sizing: border-box; margin: 0; padding: 0; }
body {
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
line-height: 1.7; color: var(--text); background: var(--bg);
max-width: 800px; margin: 0 auto; padding: 2rem 1.5rem;
}
h1 { font-size: 1.8rem; margin-bottom: 0.5rem; color: var(--accent); }
h2 { font-size: 1.4rem; margin-top: 2.5rem; margin-bottom: 0.8rem; padding-bottom: 0.3rem; border-bottom: 2px solid var(--border); }
h3 { font-size: 1.15rem; margin-top: 1.5rem; margin-bottom: 0.5rem; }
p { margin-bottom: 0.8rem; }
ul { margin-left: 1.5rem; margin-bottom: 1rem; }
li { margin-bottom: 0.4rem; }
li ul { margin-top: 0.3rem; }
strong { color: var(--accent); font-weight: 600; }
em { opacity: 0.7; font-size: 0.9em; }
blockquote {
background: var(--card); border-left: 4px solid var(--accent);
padding: 0.8rem 1.2rem; margin: 1rem 0; border-radius: 0 8px 8px 0;
}
hr { border: none; border-top: 1px solid var(--border); margin: 2rem 0; }
code { background: var(--card); padding: 0.15rem 0.4rem; border-radius: 4px; font-size: 0.9em; }
a { color: var(--accent); text-decoration: none; }
a:hover { text-decoration: underline; }
</style>
</head>
<body>
{{body}}
</body>
</html>"""
@router.get("/projects/{project_name}/mirror")
def api_project_mirror(project_name: str) -> dict:
"""Generate a human-readable project overview from structured data.
Layer 3 of the AtoCore architecture. The mirror is DERIVED from
entities, project state, and memories — it is not canonical truth.
Returns markdown that can be rendered, saved to a file, or served
as a dashboard page.
"""
from atocore.projects.registry import resolve_project_name as _resolve
canonical = _resolve(project_name)
try:
markdown = generate_project_overview(canonical)
except Exception as e:
raise HTTPException(status_code=500, detail=f"Mirror generation failed: {e}")
return {"project": canonical, "format": "markdown", "content": markdown}
@router.get("/admin/backup/{stamp}/validate") @router.get("/admin/backup/{stamp}/validate")
def api_validate_backup(stamp: str) -> dict: def api_validate_backup(stamp: str) -> dict:
"""Validate that a previously created backup is structurally usable.""" """Validate that a previously created backup is structurally usable."""

View File

@@ -104,6 +104,21 @@ class Settings(BaseSettings):
@property @property
def resolved_project_registry_path(self) -> Path: def resolved_project_registry_path(self) -> Path:
"""Path to the project registry JSON file.
If ``ATOCORE_PROJECT_REGISTRY_DIR`` env var is set, the registry
lives at ``<that dir>/project-registry.json``. Otherwise falls
back to the configured ``project_registry_path`` field.
This lets Docker deployments point at a mounted volume via env
var without the ephemeral in-image ``/app/config/`` getting
wiped on every rebuild.
"""
import os
registry_dir = os.environ.get("ATOCORE_PROJECT_REGISTRY_DIR", "").strip()
if registry_dir:
return Path(registry_dir) / "project-registry.json"
return self._resolve_path(self.project_registry_path) return self._resolve_path(self.project_registry_path)
@property @property

View File

@@ -14,6 +14,7 @@ import atocore.config as _config
from atocore.context.project_state import format_project_state, get_state from atocore.context.project_state import format_project_state, get_state
from atocore.memory.service import get_memories_for_context from atocore.memory.service import get_memories_for_context
from atocore.observability.logger import get_logger from atocore.observability.logger import get_logger
from atocore.engineering.service import get_entities, get_entity_with_context
from atocore.projects.registry import resolve_project_name from atocore.projects.registry import resolve_project_name
from atocore.retrieval.retriever import ChunkResult, retrieve from atocore.retrieval.retriever import ChunkResult, retrieve
@@ -29,13 +30,20 @@ SYSTEM_PREFIX = (
# Budget allocation (per Master Plan section 9): # Budget allocation (per Master Plan section 9):
# identity: 5%, preferences: 5%, project state: 20%, retrieval: 60%+ # identity: 5%, preferences: 5%, project state: 20%, retrieval: 60%+
PROJECT_STATE_BUDGET_RATIO = 0.20 PROJECT_STATE_BUDGET_RATIO = 0.20
MEMORY_BUDGET_RATIO = 0.10 # 5% identity + 5% preference MEMORY_BUDGET_RATIO = 0.05 # identity + preference; lowered from 0.10 to avoid squeezing project memories and chunks
# Project-scoped memories (project/knowledge/episodic) are the outlet # Project-scoped memories (project/knowledge/episodic) are the outlet
# for the Phase 9 reflection loop on the retrieval side. Budget sits # for the Phase 9 reflection loop on the retrieval side. Budget sits
# between identity/preference and retrieved chunks so a reinforced # between identity/preference and retrieved chunks so a reinforced
# memory can actually reach the model. # memory can actually reach the model.
PROJECT_MEMORY_BUDGET_RATIO = 0.25 PROJECT_MEMORY_BUDGET_RATIO = 0.25
PROJECT_MEMORY_TYPES = ["project", "knowledge", "episodic"] PROJECT_MEMORY_TYPES = ["project", "knowledge", "episodic"]
# General domain knowledge — unscoped memories (project="") that surface
# in every context pack regardless of project hint. These are earned
# engineering insights that apply across projects (e.g., "Preston removal
# model breaks down below 5N because the contact assumption fails").
DOMAIN_KNOWLEDGE_BUDGET_RATIO = 0.10
DOMAIN_KNOWLEDGE_TYPES = ["knowledge"]
ENGINEERING_CONTEXT_BUDGET_RATIO = 0.10
# Last built context pack for debug inspection # Last built context pack for debug inspection
_last_context_pack: "ContextPack | None" = None _last_context_pack: "ContextPack | None" = None
@@ -59,6 +67,10 @@ class ContextPack:
memory_chars: int = 0 memory_chars: int = 0
project_memory_text: str = "" project_memory_text: str = ""
project_memory_chars: int = 0 project_memory_chars: int = 0
domain_knowledge_text: str = ""
domain_knowledge_chars: int = 0
engineering_context_text: str = ""
engineering_context_chars: int = 0
total_chars: int = 0 total_chars: int = 0
budget: int = 0 budget: int = 0
budget_remaining: int = 0 budget_remaining: int = 0
@@ -139,8 +151,46 @@ def build_context(
query=user_prompt, query=user_prompt,
) )
# 2c. Domain knowledge — cross-project earned insight with project=""
# that surfaces regardless of which project the query is about.
domain_knowledge_text = ""
domain_knowledge_chars = 0
domain_budget = min(
int(budget * DOMAIN_KNOWLEDGE_BUDGET_RATIO),
max(budget - project_state_chars - memory_chars - project_memory_chars, 0),
)
if domain_budget > 0:
domain_knowledge_text, domain_knowledge_chars = get_memories_for_context(
memory_types=DOMAIN_KNOWLEDGE_TYPES,
project="",
budget=domain_budget,
header="--- Domain Knowledge ---",
footer="--- End Domain Knowledge ---",
query=user_prompt,
)
# 2d. Engineering context — structured entity/relationship data
# when the query matches a known entity name.
engineering_context_text = ""
engineering_context_chars = 0
if canonical_project:
eng_budget = min(
int(budget * ENGINEERING_CONTEXT_BUDGET_RATIO),
max(budget - project_state_chars - memory_chars
- project_memory_chars - domain_knowledge_chars, 0),
)
if eng_budget > 0:
engineering_context_text = _build_engineering_context(
user_prompt, canonical_project, eng_budget,
)
engineering_context_chars = len(engineering_context_text)
# 3. Calculate remaining budget for retrieval # 3. Calculate remaining budget for retrieval
retrieval_budget = budget - project_state_chars - memory_chars - project_memory_chars retrieval_budget = (
budget - project_state_chars - memory_chars
- project_memory_chars - domain_knowledge_chars
- engineering_context_chars
)
# 4. Retrieve candidates # 4. Retrieve candidates
candidates = ( candidates = (
@@ -161,13 +211,16 @@ def build_context(
# 7. Format full context # 7. Format full context
formatted = _format_full_context( formatted = _format_full_context(
project_state_text, memory_text, project_memory_text, selected project_state_text, memory_text, project_memory_text,
domain_knowledge_text, engineering_context_text, selected,
) )
if len(formatted) > budget: if len(formatted) > budget:
formatted, selected = _trim_context_to_budget( formatted, selected = _trim_context_to_budget(
project_state_text, project_state_text,
memory_text, memory_text,
project_memory_text, project_memory_text,
domain_knowledge_text,
engineering_context_text,
selected, selected,
budget, budget,
) )
@@ -178,6 +231,8 @@ def build_context(
project_state_chars = len(project_state_text) project_state_chars = len(project_state_text)
memory_chars = len(memory_text) memory_chars = len(memory_text)
project_memory_chars = len(project_memory_text) project_memory_chars = len(project_memory_text)
domain_knowledge_chars = len(domain_knowledge_text)
engineering_context_chars = len(engineering_context_text)
retrieval_chars = sum(c.char_count for c in selected) retrieval_chars = sum(c.char_count for c in selected)
total_chars = len(formatted) total_chars = len(formatted)
duration_ms = int((time.time() - start) * 1000) duration_ms = int((time.time() - start) * 1000)
@@ -190,6 +245,10 @@ def build_context(
memory_chars=memory_chars, memory_chars=memory_chars,
project_memory_text=project_memory_text, project_memory_text=project_memory_text,
project_memory_chars=project_memory_chars, project_memory_chars=project_memory_chars,
domain_knowledge_text=domain_knowledge_text,
domain_knowledge_chars=domain_knowledge_chars,
engineering_context_text=engineering_context_text,
engineering_context_chars=engineering_context_chars,
total_chars=total_chars, total_chars=total_chars,
budget=budget, budget=budget,
budget_remaining=budget - total_chars, budget_remaining=budget - total_chars,
@@ -208,6 +267,8 @@ def build_context(
project_state_chars=project_state_chars, project_state_chars=project_state_chars,
memory_chars=memory_chars, memory_chars=memory_chars,
project_memory_chars=project_memory_chars, project_memory_chars=project_memory_chars,
domain_knowledge_chars=domain_knowledge_chars,
engineering_context_chars=engineering_context_chars,
retrieval_chars=retrieval_chars, retrieval_chars=retrieval_chars,
total_chars=total_chars, total_chars=total_chars,
budget_remaining=budget - total_chars, budget_remaining=budget - total_chars,
@@ -288,7 +349,9 @@ def _format_full_context(
project_state_text: str, project_state_text: str,
memory_text: str, memory_text: str,
project_memory_text: str, project_memory_text: str,
chunks: list[ContextChunk], domain_knowledge_text: str,
engineering_context_text: str = "",
chunks: list[ContextChunk] | None = None,
) -> str: ) -> str:
"""Format project state + memories + retrieved chunks into full context block.""" """Format project state + memories + retrieved chunks into full context block."""
parts = [] parts = []
@@ -308,7 +371,17 @@ def _format_full_context(
parts.append(project_memory_text) parts.append(project_memory_text)
parts.append("") parts.append("")
# 4. Retrieved chunks (lowest trust) # 4. Domain knowledge (cross-project earned insight)
if domain_knowledge_text:
parts.append(domain_knowledge_text)
parts.append("")
# 5. Engineering context (structured entity/relationship data)
if engineering_context_text:
parts.append(engineering_context_text)
parts.append("")
# 6. Retrieved chunks (lowest trust)
if chunks: if chunks:
parts.append("--- AtoCore Retrieved Context ---") parts.append("--- AtoCore Retrieved Context ---")
if project_state_text: if project_state_text:
@@ -320,7 +393,7 @@ def _format_full_context(
parts.append(chunk.content) parts.append(chunk.content)
parts.append("") parts.append("")
parts.append("--- End Context ---") parts.append("--- End Context ---")
elif not project_state_text and not memory_text and not project_memory_text: elif not project_state_text and not memory_text and not project_memory_text and not domain_knowledge_text and not engineering_context_text:
parts.append("--- AtoCore Context ---\nNo relevant context found.\n--- End Context ---") parts.append("--- AtoCore Context ---\nNo relevant context found.\n--- End Context ---")
return "\n".join(parts) return "\n".join(parts)
@@ -343,6 +416,7 @@ def _pack_to_dict(pack: ContextPack) -> dict:
"project_state_chars": pack.project_state_chars, "project_state_chars": pack.project_state_chars,
"memory_chars": pack.memory_chars, "memory_chars": pack.memory_chars,
"project_memory_chars": pack.project_memory_chars, "project_memory_chars": pack.project_memory_chars,
"domain_knowledge_chars": pack.domain_knowledge_chars,
"chunks_used": len(pack.chunks_used), "chunks_used": len(pack.chunks_used),
"total_chars": pack.total_chars, "total_chars": pack.total_chars,
"budget": pack.budget, "budget": pack.budget,
@@ -351,6 +425,8 @@ def _pack_to_dict(pack: ContextPack) -> dict:
"has_project_state": bool(pack.project_state_text), "has_project_state": bool(pack.project_state_text),
"has_memories": bool(pack.memory_text), "has_memories": bool(pack.memory_text),
"has_project_memories": bool(pack.project_memory_text), "has_project_memories": bool(pack.project_memory_text),
"has_domain_knowledge": bool(pack.domain_knowledge_text),
"has_engineering_context": bool(pack.engineering_context_text),
"chunks": [ "chunks": [
{ {
"source_file": c.source_file, "source_file": c.source_file,
@@ -364,6 +440,83 @@ def _pack_to_dict(pack: ContextPack) -> dict:
} }
def _build_engineering_context(
query: str,
project: str,
budget: int,
) -> str:
"""Find entities matching the query and format their context.
Uses simple word-overlap matching between query tokens and entity
names to find relevant entities, then formats the top match with
its relationships as a compact text band.
"""
if budget < 100:
return ""
from atocore.memory.reinforcement import _normalize, _tokenize
query_tokens = _tokenize(_normalize(query))
if not query_tokens:
return ""
try:
entities = get_entities(project=project, limit=100)
except Exception:
return ""
if not entities:
return ""
scored: list[tuple[int, "Entity"]] = []
for ent in entities:
name_tokens = _tokenize(_normalize(ent.name))
desc_tokens = _tokenize(_normalize(ent.description))
overlap = len(query_tokens & (name_tokens | desc_tokens))
if overlap > 0:
scored.append((overlap, ent))
if not scored:
return ""
scored.sort(key=lambda t: t[0], reverse=True)
best_entity = scored[0][1]
try:
ctx = get_entity_with_context(best_entity.id)
except Exception:
return ""
if ctx is None:
return ""
lines = ["--- Engineering Context ---"]
lines.append(f"[{best_entity.entity_type}] {best_entity.name}")
if best_entity.description:
lines.append(f" {best_entity.description[:150]}")
for rel in ctx["relationships"][:8]:
other_id = (
rel.target_entity_id
if rel.source_entity_id == best_entity.id
else rel.source_entity_id
)
other = ctx["related_entities"].get(other_id)
if other:
direction = "->" if rel.source_entity_id == best_entity.id else "<-"
lines.append(
f" {direction} {rel.relationship_type} [{other.entity_type}] {other.name}"
)
lines.append("--- End Engineering Context ---")
text = "\n".join(lines)
if len(text) > budget:
text = text[:budget - 3].rstrip() + "..."
return text
def _truncate_text_block(text: str, budget: int) -> tuple[str, int]: def _truncate_text_block(text: str, budget: int) -> tuple[str, int]:
"""Trim a formatted text block so trusted tiers cannot exceed the total budget.""" """Trim a formatted text block so trusted tiers cannot exceed the total budget."""
if budget <= 0 or not text: if budget <= 0 or not text:
@@ -381,44 +534,66 @@ def _trim_context_to_budget(
project_state_text: str, project_state_text: str,
memory_text: str, memory_text: str,
project_memory_text: str, project_memory_text: str,
domain_knowledge_text: str,
engineering_context_text: str,
chunks: list[ContextChunk], chunks: list[ContextChunk],
budget: int, budget: int,
) -> tuple[str, list[ContextChunk]]: ) -> tuple[str, list[ContextChunk]]:
"""Trim retrieval project memories identity/preference → project state.""" """Trim retrieval -> engineering -> domain -> project memories -> identity -> state."""
kept_chunks = list(chunks) kept_chunks = list(chunks)
formatted = _format_full_context( formatted = _format_full_context(
project_state_text, memory_text, project_memory_text, kept_chunks project_state_text, memory_text, project_memory_text,
domain_knowledge_text, engineering_context_text, kept_chunks,
) )
while len(formatted) > budget and kept_chunks: while len(formatted) > budget and kept_chunks:
kept_chunks.pop() kept_chunks.pop()
formatted = _format_full_context( formatted = _format_full_context(
project_state_text, memory_text, project_memory_text, kept_chunks project_state_text, memory_text, project_memory_text,
domain_knowledge_text, engineering_context_text, kept_chunks,
) )
if len(formatted) <= budget: if len(formatted) <= budget:
return formatted, kept_chunks return formatted, kept_chunks
# Drop project memories next (they were the most recently added # Drop engineering context first.
# tier and carry less trust than identity/preference). engineering_context_text = ""
formatted = _format_full_context(
project_state_text, memory_text, project_memory_text,
domain_knowledge_text, engineering_context_text, kept_chunks,
)
if len(formatted) <= budget:
return formatted, kept_chunks
# Drop domain knowledge next.
domain_knowledge_text, _ = _truncate_text_block(domain_knowledge_text, 0)
formatted = _format_full_context(
project_state_text, memory_text, project_memory_text,
domain_knowledge_text, engineering_context_text, kept_chunks,
)
if len(formatted) <= budget:
return formatted, kept_chunks
project_memory_text, _ = _truncate_text_block( project_memory_text, _ = _truncate_text_block(
project_memory_text, project_memory_text,
max(budget - len(project_state_text) - len(memory_text), 0), max(budget - len(project_state_text) - len(memory_text), 0),
) )
formatted = _format_full_context( formatted = _format_full_context(
project_state_text, memory_text, project_memory_text, kept_chunks project_state_text, memory_text, project_memory_text,
domain_knowledge_text, engineering_context_text, kept_chunks,
) )
if len(formatted) <= budget: if len(formatted) <= budget:
return formatted, kept_chunks return formatted, kept_chunks
memory_text, _ = _truncate_text_block(memory_text, max(budget - len(project_state_text), 0)) memory_text, _ = _truncate_text_block(memory_text, max(budget - len(project_state_text), 0))
formatted = _format_full_context( formatted = _format_full_context(
project_state_text, memory_text, project_memory_text, kept_chunks project_state_text, memory_text, project_memory_text,
domain_knowledge_text, engineering_context_text, kept_chunks,
) )
if len(formatted) <= budget: if len(formatted) <= budget:
return formatted, kept_chunks return formatted, kept_chunks
project_state_text, _ = _truncate_text_block(project_state_text, budget) project_state_text, _ = _truncate_text_block(project_state_text, budget)
formatted = _format_full_context(project_state_text, "", "", []) formatted = _format_full_context(project_state_text, "", "", "", [])
if len(formatted) > budget: if len(formatted) > budget:
formatted, _ = _truncate_text_block(formatted, budget) formatted, _ = _truncate_text_block(formatted, budget)
return formatted, [] return formatted, []

View File

@@ -0,0 +1,16 @@
"""Engineering Knowledge Layer — typed entities and relationships.
Layer 2 of the AtoCore architecture. Sits on top of the core machine
layer (memories, project state, retrieval) and adds structured
engineering objects with typed relationships so queries like "what
requirements does this component satisfy" can be answered directly
instead of relying on flat text search.
V1 entity types (from docs/architecture/engineering-ontology-v1.md):
Component, Subsystem, Requirement, Constraint, Decision, Material,
Parameter, Interface
V1 relationship types:
CONTAINS, PART_OF, INTERFACES_WITH, SATISFIES, CONSTRAINED_BY,
AFFECTED_BY_DECISION, ANALYZED_BY, VALIDATED_BY, DEPENDS_ON
"""

View File

@@ -0,0 +1,267 @@
"""Human Mirror — derived readable project views from structured data.
Layer 3 of the AtoCore architecture. Generates human-readable markdown
pages from the engineering entity graph, Trusted Project State, and
active memories. These pages are DERIVED — they are not canonical
machine truth. They are support surfaces for human inspection and
audit comfort.
The mirror never invents content. Every line traces back to an entity,
a state entry, or a memory. If the structured data is wrong, the
mirror is wrong — fix the source, not the page.
"""
from __future__ import annotations
from atocore.context.project_state import get_state
from atocore.engineering.service import (
get_entities,
get_relationships,
)
from atocore.memory.service import get_memories
from atocore.observability.logger import get_logger
log = get_logger("mirror")
def generate_project_overview(project: str) -> str:
"""Generate a full project overview page in markdown."""
sections = [
_header(project),
_synthesis_section(project),
_state_section(project),
_system_architecture(project),
_decisions_section(project),
_requirements_section(project),
_materials_section(project),
_vendors_section(project),
_active_memories_section(project),
_footer(project),
]
return "\n\n".join(s for s in sections if s)
def _synthesis_section(project: str) -> str:
"""Generate a short LLM synthesis of the current project state.
Reads the cached synthesis from project_state if available
(category=status, key=synthesis_cache). If not cached, returns
a deterministic summary from the existing structured data.
The actual LLM-generated synthesis is produced by the weekly
lint/synthesis pass on Dalidou (where claude CLI is available).
"""
entries = get_state(project)
cached = ""
for e in entries:
if e.category == "status" and e.key == "synthesis_cache":
cached = e.value
break
if cached:
return f"## Current State (auto-synthesis)\n\n> {cached}"
# Fallback: deterministic summary from structured data
stage = ""
summary = ""
next_focus = ""
for e in entries:
if e.category == "status":
if e.key == "stage":
stage = e.value
elif e.key == "summary":
summary = e.value
elif e.key == "next_focus":
next_focus = e.value
if not (stage or summary or next_focus):
return ""
bits = []
if summary:
bits.append(summary)
if stage:
bits.append(f"**Stage**: {stage}")
if next_focus:
bits.append(f"**Next**: {next_focus}")
return "## Current State\n\n" + "\n\n".join(bits)
def _header(project: str) -> str:
return (
f"# {project} — Project Overview\n\n"
f"> This page is auto-generated from AtoCore structured data.\n"
f"> It is a **derived view**, not canonical truth. "
f"If something is wrong here, fix the source data."
)
def _state_section(project: str) -> str:
entries = get_state(project)
if not entries:
return ""
lines = ["## Trusted Project State"]
by_category: dict[str, list] = {}
for e in entries:
by_category.setdefault(e.category.upper(), []).append(e)
for cat in ["DECISION", "REQUIREMENT", "STATUS", "FACT", "MILESTONE", "CONFIG", "CONTACT"]:
items = by_category.get(cat, [])
if not items:
continue
lines.append(f"\n### {cat.title()}")
for item in items:
value = item.value[:300]
lines.append(f"- **{item.key}**: {value}")
if item.source:
lines.append(f" *(source: {item.source})*")
return "\n".join(lines)
def _system_architecture(project: str) -> str:
systems = get_entities(entity_type="system", project=project)
subsystems = get_entities(entity_type="subsystem", project=project)
components = get_entities(entity_type="component", project=project)
interfaces = get_entities(entity_type="interface", project=project)
if not systems and not subsystems and not components:
return ""
lines = ["## System Architecture"]
for system in systems:
lines.append(f"\n### {system.name}")
if system.description:
lines.append(f"{system.description}")
rels = get_relationships(system.id, direction="outgoing")
children = []
for rel in rels:
if rel.relationship_type == "contains":
child = next(
(s for s in subsystems + components if s.id == rel.target_entity_id),
None,
)
if child:
children.append(child)
if children:
lines.append("\n**Contains:**")
for child in children:
desc = f"{child.description}" if child.description else ""
lines.append(f"- [{child.entity_type}] **{child.name}**{desc}")
child_rels = get_relationships(child.id, direction="both")
for cr in child_rels:
if cr.relationship_type in ("uses_material", "interfaces_with", "constrained_by"):
other_id = (
cr.target_entity_id
if cr.source_entity_id == child.id
else cr.source_entity_id
)
other = next(
(e for e in get_entities(project=project, limit=200)
if e.id == other_id),
None,
)
if other:
lines.append(
f" - *{cr.relationship_type}* → "
f"[{other.entity_type}] {other.name}"
)
return "\n".join(lines)
def _decisions_section(project: str) -> str:
decisions = get_entities(entity_type="decision", project=project)
if not decisions:
return ""
lines = ["## Decisions"]
for d in decisions:
lines.append(f"\n### {d.name}")
if d.description:
lines.append(d.description)
rels = get_relationships(d.id, direction="outgoing")
for rel in rels:
if rel.relationship_type == "affected_by_decision":
affected = next(
(e for e in get_entities(project=project, limit=200)
if e.id == rel.target_entity_id),
None,
)
if affected:
lines.append(
f"- Affects: [{affected.entity_type}] {affected.name}"
)
return "\n".join(lines)
def _requirements_section(project: str) -> str:
reqs = get_entities(entity_type="requirement", project=project)
constraints = get_entities(entity_type="constraint", project=project)
if not reqs and not constraints:
return ""
lines = ["## Requirements & Constraints"]
for r in reqs:
lines.append(f"- **{r.name}**: {r.description}" if r.description else f"- **{r.name}**")
for c in constraints:
lines.append(f"- [constraint] **{c.name}**: {c.description}" if c.description else f"- [constraint] **{c.name}**")
return "\n".join(lines)
def _materials_section(project: str) -> str:
materials = get_entities(entity_type="material", project=project)
if not materials:
return ""
lines = ["## Materials"]
for m in materials:
desc = f"{m.description}" if m.description else ""
lines.append(f"- **{m.name}**{desc}")
return "\n".join(lines)
def _vendors_section(project: str) -> str:
vendors = get_entities(entity_type="vendor", project=project)
if not vendors:
return ""
lines = ["## Vendors"]
for v in vendors:
desc = f"{v.description}" if v.description else ""
lines.append(f"- **{v.name}**{desc}")
return "\n".join(lines)
def _active_memories_section(project: str) -> str:
memories = get_memories(project=project, active_only=True, limit=20)
if not memories:
return ""
lines = ["## Active Memories"]
for m in memories:
conf = f" (conf: {m.confidence:.2f})" if m.confidence < 1.0 else ""
refs = f" | refs: {m.reference_count}" if m.reference_count > 0 else ""
lines.append(f"- [{m.memory_type}]{conf}{refs} {m.content[:200]}")
return "\n".join(lines)
def _footer(project: str) -> str:
from datetime import datetime, timezone
now = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M UTC")
return (
f"---\n\n"
f"*Generated by AtoCore Human Mirror at {now}. "
f"This is a derived view — not canonical truth.*"
)

View File

@@ -0,0 +1,317 @@
"""Engineering entity and relationship CRUD."""
from __future__ import annotations
import json
import uuid
from dataclasses import dataclass, field
from datetime import datetime, timezone
from atocore.models.database import get_connection
from atocore.observability.logger import get_logger
log = get_logger("engineering")
ENTITY_TYPES = [
"project",
"system",
"subsystem",
"component",
"interface",
"requirement",
"constraint",
"decision",
"material",
"parameter",
"analysis_model",
"result",
"validation_claim",
"vendor",
"process",
]
RELATIONSHIP_TYPES = [
"contains",
"part_of",
"interfaces_with",
"satisfies",
"constrained_by",
"affected_by_decision",
"analyzed_by",
"validated_by",
"depends_on",
"uses_material",
"described_by",
"supersedes",
]
ENTITY_STATUSES = ["candidate", "active", "superseded", "invalid"]
@dataclass
class Entity:
id: str
entity_type: str
name: str
project: str
description: str = ""
properties: dict = field(default_factory=dict)
status: str = "active"
confidence: float = 1.0
source_refs: list[str] = field(default_factory=list)
created_at: str = ""
updated_at: str = ""
@dataclass
class Relationship:
id: str
source_entity_id: str
target_entity_id: str
relationship_type: str
confidence: float = 1.0
source_refs: list[str] = field(default_factory=list)
created_at: str = ""
def init_engineering_schema() -> None:
with get_connection() as conn:
conn.execute("""
CREATE TABLE IF NOT EXISTS entities (
id TEXT PRIMARY KEY,
entity_type TEXT NOT NULL,
name TEXT NOT NULL,
project TEXT NOT NULL DEFAULT '',
description TEXT NOT NULL DEFAULT '',
properties TEXT NOT NULL DEFAULT '{}',
status TEXT NOT NULL DEFAULT 'active',
confidence REAL NOT NULL DEFAULT 1.0,
source_refs TEXT NOT NULL DEFAULT '[]',
created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP
)
""")
conn.execute("""
CREATE TABLE IF NOT EXISTS relationships (
id TEXT PRIMARY KEY,
source_entity_id TEXT NOT NULL,
target_entity_id TEXT NOT NULL,
relationship_type TEXT NOT NULL,
confidence REAL NOT NULL DEFAULT 1.0,
source_refs TEXT NOT NULL DEFAULT '[]',
created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,
FOREIGN KEY (source_entity_id) REFERENCES entities(id),
FOREIGN KEY (target_entity_id) REFERENCES entities(id)
)
""")
conn.execute("""
CREATE INDEX IF NOT EXISTS idx_entities_project
ON entities(project)
""")
conn.execute("""
CREATE INDEX IF NOT EXISTS idx_entities_type
ON entities(entity_type)
""")
conn.execute("""
CREATE INDEX IF NOT EXISTS idx_relationships_source
ON relationships(source_entity_id)
""")
conn.execute("""
CREATE INDEX IF NOT EXISTS idx_relationships_target
ON relationships(target_entity_id)
""")
log.info("engineering_schema_initialized")
def create_entity(
entity_type: str,
name: str,
project: str = "",
description: str = "",
properties: dict | None = None,
status: str = "active",
confidence: float = 1.0,
source_refs: list[str] | None = None,
) -> Entity:
if entity_type not in ENTITY_TYPES:
raise ValueError(f"Invalid entity type: {entity_type}. Must be one of {ENTITY_TYPES}")
if status not in ENTITY_STATUSES:
raise ValueError(f"Invalid status: {status}. Must be one of {ENTITY_STATUSES}")
if not name or not name.strip():
raise ValueError("Entity name must be non-empty")
entity_id = str(uuid.uuid4())
now = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S")
props = properties or {}
refs = source_refs or []
with get_connection() as conn:
conn.execute(
"""INSERT INTO entities
(id, entity_type, name, project, description, properties,
status, confidence, source_refs, created_at, updated_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
(
entity_id, entity_type, name.strip(), project,
description, json.dumps(props), status, confidence,
json.dumps(refs), now, now,
),
)
log.info("entity_created", entity_id=entity_id, entity_type=entity_type, name=name)
return Entity(
id=entity_id, entity_type=entity_type, name=name.strip(),
project=project, description=description, properties=props,
status=status, confidence=confidence, source_refs=refs,
created_at=now, updated_at=now,
)
def create_relationship(
source_entity_id: str,
target_entity_id: str,
relationship_type: str,
confidence: float = 1.0,
source_refs: list[str] | None = None,
) -> Relationship:
if relationship_type not in RELATIONSHIP_TYPES:
raise ValueError(f"Invalid relationship type: {relationship_type}")
rel_id = str(uuid.uuid4())
now = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S")
refs = source_refs or []
with get_connection() as conn:
conn.execute(
"""INSERT INTO relationships
(id, source_entity_id, target_entity_id, relationship_type,
confidence, source_refs, created_at)
VALUES (?, ?, ?, ?, ?, ?, ?)""",
(rel_id, source_entity_id, target_entity_id,
relationship_type, confidence, json.dumps(refs), now),
)
log.info(
"relationship_created",
rel_id=rel_id,
source=source_entity_id,
target=target_entity_id,
rel_type=relationship_type,
)
return Relationship(
id=rel_id, source_entity_id=source_entity_id,
target_entity_id=target_entity_id,
relationship_type=relationship_type,
confidence=confidence, source_refs=refs, created_at=now,
)
def get_entities(
entity_type: str | None = None,
project: str | None = None,
status: str = "active",
name_contains: str | None = None,
limit: int = 100,
) -> list[Entity]:
query = "SELECT * FROM entities WHERE status = ?"
params: list = [status]
if entity_type:
query += " AND entity_type = ?"
params.append(entity_type)
if project is not None:
query += " AND project = ?"
params.append(project)
if name_contains:
query += " AND name LIKE ?"
params.append(f"%{name_contains}%")
query += " ORDER BY entity_type, name LIMIT ?"
params.append(min(limit, 500))
with get_connection() as conn:
rows = conn.execute(query, params).fetchall()
return [_row_to_entity(r) for r in rows]
def get_entity(entity_id: str) -> Entity | None:
with get_connection() as conn:
row = conn.execute(
"SELECT * FROM entities WHERE id = ?", (entity_id,)
).fetchone()
if row is None:
return None
return _row_to_entity(row)
def get_relationships(
entity_id: str,
direction: str = "both",
) -> list[Relationship]:
results = []
with get_connection() as conn:
if direction in ("outgoing", "both"):
rows = conn.execute(
"SELECT * FROM relationships WHERE source_entity_id = ?",
(entity_id,),
).fetchall()
results.extend(_row_to_relationship(r) for r in rows)
if direction in ("incoming", "both"):
rows = conn.execute(
"SELECT * FROM relationships WHERE target_entity_id = ?",
(entity_id,),
).fetchall()
results.extend(_row_to_relationship(r) for r in rows)
return results
def get_entity_with_context(entity_id: str) -> dict | None:
entity = get_entity(entity_id)
if entity is None:
return None
relationships = get_relationships(entity_id)
related_ids = set()
for rel in relationships:
related_ids.add(rel.source_entity_id)
related_ids.add(rel.target_entity_id)
related_ids.discard(entity_id)
related_entities = {}
for rid in related_ids:
e = get_entity(rid)
if e:
related_entities[rid] = e
return {
"entity": entity,
"relationships": relationships,
"related_entities": related_entities,
}
def _row_to_entity(row) -> Entity:
return Entity(
id=row["id"],
entity_type=row["entity_type"],
name=row["name"],
project=row["project"] or "",
description=row["description"] or "",
properties=json.loads(row["properties"] or "{}"),
status=row["status"],
confidence=row["confidence"],
source_refs=json.loads(row["source_refs"] or "[]"),
created_at=row["created_at"] or "",
updated_at=row["updated_at"] or "",
)
def _row_to_relationship(row) -> Relationship:
return Relationship(
id=row["id"],
source_entity_id=row["source_entity_id"],
target_entity_id=row["target_entity_id"],
relationship_type=row["relationship_type"],
confidence=row["confidence"],
source_refs=json.loads(row["source_refs"] or "[]"),
created_at=row["created_at"] or "",
)

View File

@@ -0,0 +1,298 @@
"""AtoCore Wiki — navigable HTML pages from structured data.
A lightweight wiki served directly from the AtoCore API. Every page is
generated on-demand from the database so it's always current. Source of
truth is the database — the wiki is a derived view.
Routes:
/wiki Homepage with project list + search
/wiki/projects/{name} Full project overview
/wiki/entities/{id} Entity detail with relationships
/wiki/search?q=... Search entities, memories, state
"""
from __future__ import annotations
import markdown as md
from atocore.context.project_state import get_state
from atocore.engineering.service import (
get_entities,
get_entity,
get_entity_with_context,
get_relationships,
)
from atocore.memory.service import get_memories
from atocore.projects.registry import load_project_registry
def render_html(title: str, body_html: str, breadcrumbs: list[tuple[str, str]] | None = None) -> str:
nav = ""
if breadcrumbs:
parts = []
for label, href in breadcrumbs:
if href:
parts.append(f'<a href="{href}">{label}</a>')
else:
parts.append(f"<span>{label}</span>")
nav = f'<nav class="breadcrumbs">{" / ".join(parts)}</nav>'
return _TEMPLATE.replace("{{title}}", title).replace("{{nav}}", nav).replace("{{body}}", body_html)
def render_homepage() -> str:
projects = []
try:
registered = load_project_registry()
for p in registered:
entity_count = len(get_entities(project=p.project_id, limit=200))
memory_count = len(get_memories(project=p.project_id, active_only=True, limit=200))
state_entries = get_state(p.project_id)
# Pull stage/type/client from state entries
stage = ""
proj_type = ""
client = ""
for e in state_entries:
if e.category == "status":
if e.key == "stage":
stage = e.value
elif e.key == "type":
proj_type = e.value
elif e.key == "client":
client = e.value
projects.append({
"id": p.project_id,
"description": p.description,
"entities": entity_count,
"memories": memory_count,
"state": len(state_entries),
"stage": stage,
"type": proj_type,
"client": client,
})
except Exception:
pass
# Group by high-level bucket
buckets: dict[str, list] = {
"Active Contracts": [],
"Leads & Prospects": [],
"Internal Tools & Infra": [],
"Other": [],
}
for p in projects:
t = p["type"].lower()
s = p["stage"].lower()
if "lead" in t or "lead" in s or "prospect" in s:
buckets["Leads & Prospects"].append(p)
elif "contract" in t or ("active" in s and "contract" in s):
buckets["Active Contracts"].append(p)
elif "infra" in t or "tool" in t or "internal" in t:
buckets["Internal Tools & Infra"].append(p)
else:
buckets["Other"].append(p)
lines = ['<h1>AtoCore Wiki</h1>']
lines.append('<form class="search-box" action="/wiki/search" method="get">')
lines.append('<input type="text" name="q" placeholder="Search entities, memories, projects..." autofocus>')
lines.append('<button type="submit">Search</button>')
lines.append('</form>')
for bucket_name, items in buckets.items():
if not items:
continue
lines.append(f'<h2>{bucket_name}</h2>')
lines.append('<div class="card-grid">')
for p in items:
client_line = f'<div class="client">{p["client"]}</div>' if p["client"] else ''
stage_tag = f'<span class="tag">{p["stage"].split("")[0]}</span>' if p["stage"] else ''
lines.append(f'<a href="/wiki/projects/{p["id"]}" class="card">')
lines.append(f'<h3>{p["id"]} {stage_tag}</h3>')
lines.append(client_line)
lines.append(f'<p>{p["description"][:140]}</p>')
lines.append(f'<div class="stats">{p["entities"]} entities · {p["memories"]} memories · {p["state"]} state</div>')
lines.append('</a>')
lines.append('</div>')
# Quick stats
all_entities = get_entities(limit=500)
all_memories = get_memories(active_only=True, limit=500)
lines.append('<h2>System</h2>')
lines.append(f'<p>{len(all_entities)} entities · {len(all_memories)} active memories · {len(projects)} projects</p>')
lines.append(f'<p><a href="/admin/dashboard">API Dashboard (JSON)</a> · <a href="/health">Health Check</a></p>')
return render_html("AtoCore Wiki", "\n".join(lines))
def render_project(project: str) -> str:
from atocore.engineering.mirror import generate_project_overview
markdown_content = generate_project_overview(project)
# Convert entity names to links
entities = get_entities(project=project, limit=200)
html_body = md.markdown(markdown_content, extensions=["tables", "fenced_code"])
for ent in sorted(entities, key=lambda e: len(e.name), reverse=True):
linked = f'<a href="/wiki/entities/{ent.id}" title="{ent.entity_type}">{ent.name}</a>'
html_body = html_body.replace(f"<strong>{ent.name}</strong>", f"<strong>{linked}</strong>", 1)
return render_html(
f"{project}",
html_body,
breadcrumbs=[("Wiki", "/wiki"), (project, "")],
)
def render_entity(entity_id: str) -> str | None:
ctx = get_entity_with_context(entity_id)
if ctx is None:
return None
ent = ctx["entity"]
lines = [f'<h1>[{ent.entity_type}] {ent.name}</h1>']
if ent.project:
lines.append(f'<p>Project: <a href="/wiki/projects/{ent.project}">{ent.project}</a></p>')
if ent.description:
lines.append(f'<p>{ent.description}</p>')
if ent.properties:
lines.append('<h2>Properties</h2><ul>')
for k, v in ent.properties.items():
lines.append(f'<li><strong>{k}</strong>: {v}</li>')
lines.append('</ul>')
lines.append(f'<p class="meta">confidence: {ent.confidence} · status: {ent.status} · created: {ent.created_at}</p>')
if ctx["relationships"]:
lines.append('<h2>Relationships</h2><ul>')
for rel in ctx["relationships"]:
other_id = rel.target_entity_id if rel.source_entity_id == entity_id else rel.source_entity_id
other = ctx["related_entities"].get(other_id)
if other:
direction = "\u2192" if rel.source_entity_id == entity_id else "\u2190"
lines.append(
f'<li>{direction} <em>{rel.relationship_type}</em> '
f'<a href="/wiki/entities/{other_id}">[{other.entity_type}] {other.name}</a></li>'
)
lines.append('</ul>')
breadcrumbs = [("Wiki", "/wiki")]
if ent.project:
breadcrumbs.append((ent.project, f"/wiki/projects/{ent.project}"))
breadcrumbs.append((ent.name, ""))
return render_html(ent.name, "\n".join(lines), breadcrumbs=breadcrumbs)
def render_search(query: str) -> str:
lines = [f'<h1>Search: "{query}"</h1>']
# Search entities by name
entities = get_entities(name_contains=query, limit=20)
if entities:
lines.append(f'<h2>Entities ({len(entities)})</h2><ul>')
for e in entities:
proj = f' <span class="tag">{e.project}</span>' if e.project else ''
lines.append(
f'<li><a href="/wiki/entities/{e.id}">[{e.entity_type}] {e.name}</a>{proj}'
f'{"" + e.description[:100] if e.description else ""}</li>'
)
lines.append('</ul>')
# Search memories
all_memories = get_memories(active_only=True, limit=200)
query_lower = query.lower()
matching_mems = [m for m in all_memories if query_lower in m.content.lower()][:10]
if matching_mems:
lines.append(f'<h2>Memories ({len(matching_mems)})</h2><ul>')
for m in matching_mems:
proj = f' <span class="tag">{m.project}</span>' if m.project else ''
lines.append(f'<li>[{m.memory_type}]{proj} {m.content[:200]}</li>')
lines.append('</ul>')
if not entities and not matching_mems:
lines.append('<p>No results found.</p>')
lines.append('<form class="search-box" action="/wiki/search" method="get">')
lines.append(f'<input type="text" name="q" value="{query}" autofocus>')
lines.append('<button type="submit">Search</button>')
lines.append('</form>')
return render_html(
f"Search: {query}",
"\n".join(lines),
breadcrumbs=[("Wiki", "/wiki"), ("Search", "")],
)
_TEMPLATE = """<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>{{title}} — AtoCore</title>
<style>
:root { --bg: #fafafa; --text: #1a1a2e; --accent: #2563eb; --border: #e2e8f0; --card: #fff; --hover: #f1f5f9; }
@media (prefers-color-scheme: dark) {
:root { --bg: #0f172a; --text: #e2e8f0; --accent: #60a5fa; --border: #334155; --card: #1e293b; --hover: #334155; }
}
* { box-sizing: border-box; margin: 0; padding: 0; }
body {
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
line-height: 1.7; color: var(--text); background: var(--bg);
max-width: 800px; margin: 0 auto; padding: 1.5rem;
}
h1 { font-size: 1.8rem; margin-bottom: 0.5rem; color: var(--accent); }
h2 { font-size: 1.3rem; margin-top: 2rem; margin-bottom: 0.6rem; padding-bottom: 0.2rem; border-bottom: 2px solid var(--border); }
h3 { font-size: 1.1rem; margin-top: 1.2rem; margin-bottom: 0.4rem; }
p { margin-bottom: 0.8rem; }
ul { margin-left: 1.5rem; margin-bottom: 1rem; }
li { margin-bottom: 0.3rem; }
li ul { margin-top: 0.2rem; }
strong { color: var(--accent); font-weight: 600; }
em { opacity: 0.7; font-size: 0.9em; }
a { color: var(--accent); text-decoration: none; }
a:hover { text-decoration: underline; }
blockquote {
background: var(--card); border-left: 4px solid var(--accent);
padding: 0.6rem 1rem; margin: 1rem 0; border-radius: 0 6px 6px 0;
font-size: 0.9em;
}
hr { border: none; border-top: 1px solid var(--border); margin: 2rem 0; }
.breadcrumbs { margin-bottom: 1.5rem; font-size: 0.85em; opacity: 0.7; }
.breadcrumbs a { opacity: 0.8; }
.meta { font-size: 0.8em; opacity: 0.5; margin-top: 0.5rem; }
.tag { background: var(--accent); color: var(--bg); padding: 0.1rem 0.4rem; border-radius: 3px; font-size: 0.75em; margin-left: 0.3rem; }
.search-box { display: flex; gap: 0.5rem; margin: 1.5rem 0; }
.search-box input {
flex: 1; padding: 0.6rem 1rem; border: 2px solid var(--border);
border-radius: 8px; background: var(--card); color: var(--text);
font-size: 1rem;
}
.search-box input:focus { border-color: var(--accent); outline: none; }
.search-box button {
padding: 0.6rem 1.2rem; background: var(--accent); color: var(--bg);
border: none; border-radius: 8px; cursor: pointer; font-size: 1rem;
}
.card-grid { display: grid; grid-template-columns: 1fr; gap: 1rem; margin: 1rem 0; }
@media (min-width: 600px) { .card-grid { grid-template-columns: 1fr 1fr; } }
.card {
display: block; background: var(--card); border: 1px solid var(--border);
border-radius: 10px; padding: 1.2rem; text-decoration: none;
color: var(--text); transition: border-color 0.2s;
}
.card:hover { border-color: var(--accent); background: var(--hover); text-decoration: none; }
.card h3 { color: var(--accent); margin: 0 0 0.3rem 0; }
.card p { font-size: 0.9em; margin: 0; opacity: 0.8; }
.card .stats { font-size: 0.8em; margin-top: 0.5rem; opacity: 0.5; }
.card .client { font-size: 0.85em; opacity: 0.65; margin-bottom: 0.3rem; font-style: italic; }
.card h3 .tag { font-size: 0.65em; vertical-align: middle; margin-left: 0.4rem; }
</style>
</head>
<body>
{{nav}}
{{body}}
</body>
</html>"""

View File

@@ -8,6 +8,7 @@ from atocore import __version__
from atocore.api.routes import router from atocore.api.routes import router
import atocore.config as _config import atocore.config as _config
from atocore.context.project_state import init_project_state_schema from atocore.context.project_state import init_project_state_schema
from atocore.engineering.service import init_engineering_schema
from atocore.ingestion.pipeline import get_source_status from atocore.ingestion.pipeline import get_source_status
from atocore.models.database import init_db from atocore.models.database import init_db
from atocore.observability.logger import get_logger, setup_logging from atocore.observability.logger import get_logger, setup_logging
@@ -29,6 +30,7 @@ async def lifespan(app: FastAPI):
_config.ensure_runtime_dirs() _config.ensure_runtime_dirs()
init_db() init_db()
init_project_state_schema() init_project_state_schema()
init_engineering_schema()
log.info( log.info(
"startup_ready", "startup_ready",
env=_config.settings.env, env=_config.settings.env,

View File

@@ -0,0 +1,183 @@
"""Shared LLM-extractor prompt + parser (stdlib-only).
R12: single source of truth for the system prompt, memory type set,
size limits, and raw JSON parsing used by both paths that shell out
to ``claude -p``:
- ``atocore.memory.extractor_llm`` (in-container extractor, wraps the
parsed dicts in ``MemoryCandidate`` with registry-checked project
attribution)
- ``scripts/batch_llm_extract_live.py`` (host-side extractor, can't
import the full atocore package because Dalidou's host Python lacks
the container's deps; imports this module via ``sys.path``)
This module MUST stay stdlib-only. No ``atocore`` imports, no third-
party packages. Callers apply their own project-attribution policy on
top of the normalized dicts this module emits.
"""
from __future__ import annotations
import json
from typing import Any
LLM_EXTRACTOR_VERSION = "llm-0.4.0"
MAX_RESPONSE_CHARS = 8000
MAX_PROMPT_CHARS = 2000
MEMORY_TYPES = {"identity", "preference", "project", "episodic", "knowledge", "adaptation"}
SYSTEM_PROMPT = """You extract memory candidates from LLM conversation turns for a personal context engine called AtoCore.
AtoCore is the brain for Atomaste's engineering work. Known projects:
p04-gigabit, p05-interferometer, p06-polisher, atomizer-v2, atocore,
abb-space. Unknown project names — still tag them, the system auto-detects.
Your job is to emit SIGNALS that matter for future context. Be aggressive:
err on the side of capturing useful signal. Triage filters noise downstream.
WHAT TO EMIT (in order of importance):
1. PROJECT ACTIVITY — any mention of a project with context worth remembering:
- "Schott quote received for ABB-Space" (event + project)
- "Cédric asked about p06 firmware timing" (stakeholder event)
- "Still waiting on Zygo lead-time from Nabeel" (blocker status)
- "p05 vendor decision needs to happen this week" (action item)
2. DECISIONS AND CHOICES — anything that commits to a direction:
- "Going with Zygo Verifire SV for p05" (decision)
- "Dropping stitching from primary workflow" (design choice)
- "USB SSD mandatory, not SD card" (architectural commitment)
3. DURABLE ENGINEERING INSIGHT — earned knowledge that generalizes:
- "CTE gradient dominates WFE at F/1.2" (materials insight)
- "Preston model breaks below 5N because contact assumption fails"
- "m=1 coma NOT correctable by force modulation" (controls insight)
Test: would a competent engineer NEED experience to know this?
If it's textbook/google-findable, skip it.
4. STAKEHOLDER AND VENDOR EVENTS:
- "Email sent to Nabeel 2026-04-13 asking for lead time"
- "Meeting with Jason on Table 7 next Tuesday"
- "Starspec wants updated CAD by Friday"
5. PREFERENCES AND ADAPTATIONS that shape how Antoine works:
- "Antoine prefers OAuth over API keys"
- "Extraction stays off the capture hot path"
WHAT TO SKIP:
- Pure conversational filler ("ok thanks", "let me check")
- Instructional help content ("run this command", "here's how to...")
- Obvious textbook facts anyone can google in 30 seconds
- Session meta-chatter ("let me commit this", "deploy running")
- Transient system state snapshots ("36 active memories right now")
CANDIDATE TYPES — choose the best fit:
- project — a fact, decision, or event specific to one named project
- knowledge — durable engineering insight (use domain, not project)
- preference — how Antoine works / wants things done
- adaptation — a standing rule or adjustment to behavior
- episodic — a stakeholder event or milestone worth remembering
DOMAINS for knowledge candidates (required when type=knowledge and project is empty):
physics, materials, optics, mechanics, manufacturing, metrology,
controls, software, math, finance, business
TRUST HIERARCHY:
- project-specific: set project to the project id, leave domain empty
- domain knowledge: set domain, leave project empty
- events/activity: use project, type=project or episodic
- one conversation can produce MULTIPLE candidates — emit them all
OUTPUT RULES:
- Each candidate content under 250 characters, stands alone
- Default confidence 0.5. Raise to 0.7 only for ratified/committed claims.
- Raw JSON array, no prose, no markdown fences
- Empty array [] is fine when the conversation has no durable signal
Each element:
{"type": "project|knowledge|preference|adaptation|episodic", "content": "...", "project": "...", "domain": "", "confidence": 0.5}"""
def build_user_message(prompt: str, response: str, project_hint: str) -> str:
prompt_excerpt = (prompt or "")[:MAX_PROMPT_CHARS]
response_excerpt = (response or "")[:MAX_RESPONSE_CHARS]
return (
f"PROJECT HINT (may be empty): {project_hint or ''}\n\n"
f"USER PROMPT:\n{prompt_excerpt}\n\n"
f"ASSISTANT RESPONSE:\n{response_excerpt}\n\n"
"Return the JSON array now."
)
def parse_llm_json_array(raw_output: str) -> list[dict[str, Any]]:
"""Strip markdown fences / leading prose and return the parsed JSON
array as a list of raw dicts. Returns an empty list on any parse
failure — callers decide whether to log."""
text = (raw_output or "").strip()
if text.startswith("```"):
text = text.strip("`")
nl = text.find("\n")
if nl >= 0:
text = text[nl + 1:]
if text.endswith("```"):
text = text[:-3]
text = text.strip()
if not text or text == "[]":
return []
if not text.lstrip().startswith("["):
start = text.find("[")
end = text.rfind("]")
if start >= 0 and end > start:
text = text[start:end + 1]
try:
parsed = json.loads(text)
except json.JSONDecodeError:
return []
if not isinstance(parsed, list):
return []
return [item for item in parsed if isinstance(item, dict)]
def normalize_candidate_item(item: dict[str, Any]) -> dict[str, Any] | None:
"""Validate and normalize one raw model item into a candidate dict.
Returns None if the item fails basic validation (unknown type,
empty content). Does NOT apply project-attribution policy — that's
the caller's job, since the registry-check differs between the
in-container path and the host path.
Output keys: type, content, project (raw model value), domain,
confidence.
"""
mem_type = str(item.get("type") or "").strip().lower()
content = str(item.get("content") or "").strip()
if mem_type not in MEMORY_TYPES or not content:
return None
model_project = str(item.get("project") or "").strip()
domain = str(item.get("domain") or "").strip().lower()
try:
confidence = float(item.get("confidence", 0.5))
except (TypeError, ValueError):
confidence = 0.5
confidence = max(0.0, min(1.0, confidence))
if domain and not model_project:
content = f"[{domain}] {content}"
return {
"type": mem_type,
"content": content[:1000],
"project": model_project,
"domain": domain,
"confidence": confidence,
}

View File

@@ -49,7 +49,6 @@ Implementation notes:
from __future__ import annotations from __future__ import annotations
import json
import os import os
import shutil import shutil
import subprocess import subprocess
@@ -58,38 +57,21 @@ from dataclasses import dataclass
from functools import lru_cache from functools import lru_cache
from atocore.interactions.service import Interaction from atocore.interactions.service import Interaction
from atocore.memory._llm_prompt import (
LLM_EXTRACTOR_VERSION,
SYSTEM_PROMPT as _SYSTEM_PROMPT,
build_user_message,
normalize_candidate_item,
parse_llm_json_array,
)
from atocore.memory.extractor import MemoryCandidate from atocore.memory.extractor import MemoryCandidate
from atocore.memory.service import MEMORY_TYPES
from atocore.observability.logger import get_logger from atocore.observability.logger import get_logger
log = get_logger("extractor_llm") log = get_logger("extractor_llm")
LLM_EXTRACTOR_VERSION = "llm-0.2.0"
DEFAULT_MODEL = os.environ.get("ATOCORE_LLM_EXTRACTOR_MODEL", "sonnet") DEFAULT_MODEL = os.environ.get("ATOCORE_LLM_EXTRACTOR_MODEL", "sonnet")
DEFAULT_TIMEOUT_S = float(os.environ.get("ATOCORE_LLM_EXTRACTOR_TIMEOUT_S", "90")) DEFAULT_TIMEOUT_S = float(os.environ.get("ATOCORE_LLM_EXTRACTOR_TIMEOUT_S", "90"))
MAX_RESPONSE_CHARS = 8000
MAX_PROMPT_CHARS = 2000
_SYSTEM_PROMPT = """You extract durable memory candidates from LLM conversation turns for a personal context engine called AtoCore.
Your job is to read one user prompt plus the assistant's response and decide which durable facts, decisions, preferences, architectural rules, or project invariants should be remembered across future sessions.
Rules:
1. Only surface durable claims. Skip transient status ("deploy is still running"), instructional guidance ("here is how to run the command"), troubleshooting tactics, ephemeral recommendations ("merge this PR now"), and session recaps.
2. A candidate is durable when a reader coming back in two weeks would still need to know it. Architectural choices, named rules, ratified decisions, invariants, procurement commitments, and project-level constraints qualify. Conversational fillers and step-by-step instructions do not.
3. Each candidate must stand alone. Rewrite the claim in one sentence under 200 characters with enough context that a reader without the conversation understands it.
4. Each candidate must have a type from this closed set: project, knowledge, preference, adaptation.
5. If the conversation is clearly scoped to a project (p04-gigabit, p05-interferometer, p06-polisher, atocore), set ``project`` to that id. Otherwise leave ``project`` empty.
6. If the response makes no durable claim, return an empty list. It is correct and expected to return [] on most conversational turns.
7. Confidence should be 0.5 by default so human review workload is honest. Raise to 0.6 only when the response states the claim in an unambiguous, committed form (e.g. "the decision is X", "the selected approach is Y", "X is non-negotiable").
8. Output must be a raw JSON array and nothing else. No prose before or after. No markdown fences. No explanations.
Each array element has exactly this shape:
{"type": "project|knowledge|preference|adaptation", "content": "...", "project": "...", "confidence": 0.5}
Return [] when there is nothing to extract."""
@dataclass @dataclass
@@ -152,13 +134,10 @@ def extract_candidates_llm_verbose(
if not response_text: if not response_text:
return LLMExtractionResult(candidates=[], raw_output="", error="empty_response") return LLMExtractionResult(candidates=[], raw_output="", error="empty_response")
prompt_excerpt = (interaction.prompt or "")[:MAX_PROMPT_CHARS] user_message = build_user_message(
response_excerpt = response_text[:MAX_RESPONSE_CHARS] interaction.prompt or "",
user_message = ( response_text,
f"PROJECT HINT (may be empty): {interaction.project or ''}\n\n" interaction.project or "",
f"USER PROMPT:\n{prompt_excerpt}\n\n"
f"ASSISTANT RESPONSE:\n{response_excerpt}\n\n"
"Return the JSON array now."
) )
args = [ args = [
@@ -168,7 +147,6 @@ def extract_candidates_llm_verbose(
model or DEFAULT_MODEL, model or DEFAULT_MODEL,
"--append-system-prompt", "--append-system-prompt",
_SYSTEM_PROMPT, _SYSTEM_PROMPT,
"--no-session-persistence",
"--disable-slash-commands", "--disable-slash-commands",
user_message, user_message,
] ]
@@ -217,65 +195,59 @@ def extract_candidates_llm_verbose(
def _parse_candidates(raw_output: str, interaction: Interaction) -> list[MemoryCandidate]: def _parse_candidates(raw_output: str, interaction: Interaction) -> list[MemoryCandidate]:
"""Parse the model's JSON output into MemoryCandidate objects. """Parse the model's JSON output into MemoryCandidate objects.
Tolerates common model glitches: surrounding whitespace, stray Shared stripping + per-item validation live in
markdown fences, leading/trailing prose. Silently drops malformed ``atocore.memory._llm_prompt``. This function adds the container-
array elements rather than raising. only R9 project attribution: registry-check model_project and fall
back to the interaction scope when set.
""" """
text = raw_output.strip() raw_items = parse_llm_json_array(raw_output)
if text.startswith("```"): if not raw_items and raw_output.strip() not in ("", "[]"):
text = text.strip("`") log.error("llm_extractor_parse_failed", raw_prefix=raw_output[:120])
first_newline = text.find("\n")
if first_newline >= 0:
text = text[first_newline + 1 :]
if text.endswith("```"):
text = text[:-3]
text = text.strip()
if not text or text == "[]":
return []
if not text.lstrip().startswith("["):
start = text.find("[")
end = text.rfind("]")
if start >= 0 and end > start:
text = text[start : end + 1]
try:
parsed = json.loads(text)
except json.JSONDecodeError as exc:
log.error("llm_extractor_parse_failed", error=str(exc), raw_prefix=raw_output[:120])
return []
if not isinstance(parsed, list):
return []
results: list[MemoryCandidate] = [] results: list[MemoryCandidate] = []
for item in parsed: for raw_item in raw_items:
if not isinstance(item, dict): normalized = normalize_candidate_item(raw_item)
if normalized is None:
continue continue
mem_type = str(item.get("type") or "").strip().lower()
content = str(item.get("content") or "").strip() model_project = normalized["project"]
project = str(item.get("project") or "").strip() # R9 trust hierarchy: interaction scope wins; else registry-
if not project and interaction.project: # resolve the model's tag; else keep the model's tag so auto-
# triage can surface unregistered projects.
if interaction.project:
project = interaction.project project = interaction.project
confidence_raw = item.get("confidence", 0.5) elif model_project:
if mem_type not in MEMORY_TYPES: try:
continue from atocore.projects.registry import (
if not content: load_project_registry,
continue resolve_project_name,
try: )
confidence = float(confidence_raw)
except (TypeError, ValueError): registered_ids = {p.project_id for p in load_project_registry()}
confidence = 0.5 resolved = resolve_project_name(model_project)
confidence = max(0.0, min(1.0, confidence)) if resolved in registered_ids:
project = resolved
else:
project = model_project
log.info(
"unregistered_project_detected",
model_project=model_project,
interaction_id=interaction.id,
)
except Exception:
project = model_project
else:
project = ""
content = normalized["content"]
results.append( results.append(
MemoryCandidate( MemoryCandidate(
memory_type=mem_type, memory_type=normalized["type"],
content=content[:1000], content=content,
rule="llm_extraction", rule="llm_extraction",
source_span=content[:200], source_span=content[:200],
project=project, project=project,
confidence=confidence, confidence=normalized["confidence"],
source_interaction_id=interaction.id, source_interaction_id=interaction.id,
extractor_version=LLM_EXTRACTOR_VERSION, extractor_version=LLM_EXTRACTOR_VERSION,
) )

View File

@@ -340,6 +340,84 @@ def reinforce_memory(
return True, old_confidence, new_confidence return True, old_confidence, new_confidence
def auto_promote_reinforced(
min_reference_count: int = 3,
min_confidence: float = 0.7,
max_age_days: int = 14,
) -> list[str]:
"""Auto-promote candidate memories with strong reinforcement signals.
Phase 10: memories that have been reinforced by multiple interactions
graduate from candidate to active without human review. This rewards
knowledge that the system keeps referencing organically.
Returns a list of promoted memory IDs.
"""
from datetime import timedelta
cutoff = (
datetime.now(timezone.utc) - timedelta(days=max_age_days)
).strftime("%Y-%m-%d %H:%M:%S")
promoted: list[str] = []
with get_connection() as conn:
rows = conn.execute(
"SELECT id, content, memory_type, project, confidence, "
"reference_count FROM memories "
"WHERE status = 'candidate' "
"AND COALESCE(reference_count, 0) >= ? "
"AND confidence >= ? "
"AND last_referenced_at >= ?",
(min_reference_count, min_confidence, cutoff),
).fetchall()
for row in rows:
mid = row["id"]
ok = promote_memory(mid)
if ok:
promoted.append(mid)
log.info(
"memory_auto_promoted",
memory_id=mid,
memory_type=row["memory_type"],
project=row["project"] or "(global)",
reference_count=row["reference_count"],
confidence=round(row["confidence"], 3),
)
return promoted
def expire_stale_candidates(
max_age_days: int = 14,
) -> list[str]:
"""Reject candidate memories that sat in queue too long unreinforced.
Candidates older than ``max_age_days`` with zero reinforcement are
auto-rejected to prevent unbounded queue growth. Returns rejected IDs.
"""
from datetime import timedelta
cutoff = (
datetime.now(timezone.utc) - timedelta(days=max_age_days)
).strftime("%Y-%m-%d %H:%M:%S")
expired: list[str] = []
with get_connection() as conn:
rows = conn.execute(
"SELECT id FROM memories "
"WHERE status = 'candidate' "
"AND COALESCE(reference_count, 0) = 0 "
"AND created_at < ?",
(cutoff,),
).fetchall()
for row in rows:
mid = row["id"]
ok = reject_candidate_memory(mid)
if ok:
expired.append(mid)
log.info("memory_expired", memory_id=mid)
return expired
def get_memories_for_context( def get_memories_for_context(
memory_types: list[str] | None = None, memory_types: list[str] | None = None,
project: str | None = None, project: str | None = None,
@@ -446,20 +524,27 @@ def _rank_memories_for_query(
) -> list["Memory"]: ) -> list["Memory"]:
"""Rerank a memory list by lexical overlap with a pre-tokenized query. """Rerank a memory list by lexical overlap with a pre-tokenized query.
Ordering key: (overlap_count DESC, confidence DESC). When a query Primary key: overlap_density (overlap_count / memory_token_count),
shares no tokens with a memory, overlap is zero and confidence which rewards short focused memories that match the query precisely
acts as the sole tiebreaker — which matches the pre-query over long overview memories that incidentally share a few tokens.
behaviour and keeps no-query calls stable. Secondary: absolute overlap count. Tertiary: confidence.
R7 fix: previously overlap_count alone was the primary key, so a
40-token overview memory with 3 overlapping tokens tied a 5-token
memory with 3 overlapping tokens, and the overview won on
confidence. Now the short memory's density (0.6) beats the
overview's density (0.075).
""" """
from atocore.memory.reinforcement import _normalize, _tokenize from atocore.memory.reinforcement import _normalize, _tokenize
scored: list[tuple[int, float, Memory]] = [] scored: list[tuple[float, int, float, Memory]] = []
for mem in memories: for mem in memories:
mem_tokens = _tokenize(_normalize(mem.content)) mem_tokens = _tokenize(_normalize(mem.content))
overlap = len(mem_tokens & query_tokens) if mem_tokens else 0 overlap = len(mem_tokens & query_tokens) if mem_tokens else 0
scored.append((overlap, mem.confidence, mem)) density = overlap / len(mem_tokens) if mem_tokens else 0.0
scored.sort(key=lambda t: (t[0], t[1]), reverse=True) scored.append((density, overlap, mem.confidence, mem))
return [mem for _, _, mem in scored] scored.sort(key=lambda t: (t[0], t[1], t[2]), reverse=True)
return [mem for _, _, _, mem in scored]
def _row_to_memory(row) -> Memory: def _row_to_memory(row) -> Memory:

118
tests/test_engineering.py Normal file
View File

@@ -0,0 +1,118 @@
"""Tests for the Engineering Knowledge Layer."""
from atocore.engineering.service import (
ENTITY_TYPES,
RELATIONSHIP_TYPES,
create_entity,
create_relationship,
get_entities,
get_entity,
get_entity_with_context,
get_relationships,
init_engineering_schema,
)
from atocore.models.database import init_db
import pytest
def test_create_and_get_entity(tmp_data_dir):
init_db()
init_engineering_schema()
e = create_entity(
entity_type="component",
name="Pivot Pin",
project="p04-gigabit",
description="Lateral support pivot pin for M1 assembly",
properties={"material": "GF-PTFE", "diameter_mm": 12},
)
assert e.entity_type == "component"
assert e.name == "Pivot Pin"
assert e.properties["material"] == "GF-PTFE"
fetched = get_entity(e.id)
assert fetched is not None
assert fetched.name == "Pivot Pin"
def test_create_relationship(tmp_data_dir):
init_db()
init_engineering_schema()
subsystem = create_entity("subsystem", "Lateral Support", project="p04-gigabit")
component = create_entity("component", "Pivot Pin", project="p04-gigabit")
rel = create_relationship(
source_entity_id=subsystem.id,
target_entity_id=component.id,
relationship_type="contains",
)
assert rel.relationship_type == "contains"
rels = get_relationships(subsystem.id, direction="outgoing")
assert len(rels) == 1
assert rels[0].target_entity_id == component.id
def test_entity_with_context(tmp_data_dir):
init_db()
init_engineering_schema()
subsystem = create_entity("subsystem", "Lateral Support", project="p04-gigabit")
pin = create_entity("component", "Pivot Pin", project="p04-gigabit")
pad = create_entity("component", "PTFE Pad", project="p04-gigabit")
material = create_entity("material", "GF-PTFE", project="p04-gigabit",
description="Glass-filled PTFE for thermal stability")
create_relationship(subsystem.id, pin.id, "contains")
create_relationship(subsystem.id, pad.id, "contains")
create_relationship(pad.id, material.id, "uses_material")
ctx = get_entity_with_context(subsystem.id)
assert ctx is not None
assert len(ctx["relationships"]) == 2
assert pin.id in ctx["related_entities"]
assert pad.id in ctx["related_entities"]
def test_filter_entities_by_type_and_project(tmp_data_dir):
init_db()
init_engineering_schema()
create_entity("component", "Pin A", project="p04-gigabit")
create_entity("component", "Pin B", project="p04-gigabit")
create_entity("material", "Steel", project="p04-gigabit")
create_entity("component", "Actuator", project="p06-polisher")
components = get_entities(entity_type="component", project="p04-gigabit")
assert len(components) == 2
all_p04 = get_entities(project="p04-gigabit")
assert len(all_p04) == 3
polisher = get_entities(project="p06-polisher")
assert len(polisher) == 1
def test_invalid_entity_type_raises(tmp_data_dir):
init_db()
init_engineering_schema()
with pytest.raises(ValueError, match="Invalid entity type"):
create_entity("spaceship", "Enterprise")
def test_invalid_relationship_type_raises(tmp_data_dir):
init_db()
init_engineering_schema()
a = create_entity("component", "A")
b = create_entity("component", "B")
with pytest.raises(ValueError, match="Invalid relationship type"):
create_relationship(a.id, b.id, "loves")
def test_entity_name_search(tmp_data_dir):
init_db()
init_engineering_schema()
create_entity("component", "Vertical Support Pad")
create_entity("component", "Lateral Support Bracket")
create_entity("component", "Reference Frame")
results = get_entities(name_contains="Support")
assert len(results) == 2

View File

@@ -0,0 +1,208 @@
"""Integration tests for the extraction + triage pipeline (R8).
Tests the flow that produced the 41 active memories:
LLM extraction → persist as candidate → triage → promote/reject.
Uses mocked subprocess to avoid real claude -p calls.
"""
from __future__ import annotations
from unittest.mock import patch
import pytest
from atocore.memory.extractor_llm import (
extract_candidates_llm,
extract_candidates_llm_verbose,
)
from atocore.memory.service import create_memory, get_memories
from atocore.models.database import init_db
import atocore.memory.extractor_llm as extractor_llm
def _make_interaction(**kw):
from atocore.interactions.service import Interaction
return Interaction(
id=kw.get("id", "test-pipe-1"),
prompt=kw.get("prompt", "test prompt"),
response=kw.get("response", ""),
response_summary="",
project=kw.get("project", ""),
client="test",
session_id="",
)
class _FakeCompleted:
def __init__(self, stdout, returncode=0):
self.stdout = stdout
self.stderr = ""
self.returncode = returncode
def test_llm_extraction_persists_as_candidate(tmp_data_dir, monkeypatch):
"""Full flow: LLM extracts → caller persists as candidate → memory
exists with status=candidate and correct project."""
init_db()
monkeypatch.setattr(extractor_llm, "_cli_available", lambda: True)
monkeypatch.setattr(
extractor_llm.subprocess,
"run",
lambda *a, **kw: _FakeCompleted(
'[{"type": "project", "content": "USB SSD is mandatory for RPi storage", "project": "p06-polisher", "confidence": 0.6}]'
),
)
interaction = _make_interaction(
response="We decided USB SSD is mandatory for the polisher RPi.",
project="p06-polisher",
)
candidates = extract_candidates_llm(interaction)
assert len(candidates) == 1
assert candidates[0].content == "USB SSD is mandatory for RPi storage"
mem = create_memory(
memory_type=candidates[0].memory_type,
content=candidates[0].content,
project=candidates[0].project,
confidence=candidates[0].confidence,
status="candidate",
)
assert mem.status == "candidate"
assert mem.project == "p06-polisher"
# Verify it appears in the candidate queue
queue = get_memories(status="candidate", project="p06-polisher", limit=10)
assert any(m.id == mem.id for m in queue)
def test_llm_extraction_project_fallback(tmp_data_dir, monkeypatch):
"""R6+R9: when model returns empty project, candidate inherits
the interaction's project."""
init_db()
monkeypatch.setattr(extractor_llm, "_cli_available", lambda: True)
monkeypatch.setattr(
extractor_llm.subprocess,
"run",
lambda *a, **kw: _FakeCompleted(
'[{"type": "knowledge", "content": "machine works offline", "project": "", "confidence": 0.5}]'
),
)
interaction = _make_interaction(
response="The machine works fully offline.",
project="p06-polisher",
)
candidates = extract_candidates_llm(interaction)
assert len(candidates) == 1
assert candidates[0].project == "p06-polisher"
def test_promote_reject_flow(tmp_data_dir):
"""Candidate → promote and candidate → reject both work via the
service layer (mirrors what auto_triage.py does via HTTP)."""
from atocore.memory.service import promote_memory, reject_candidate_memory
init_db()
good = create_memory(
memory_type="project",
content="durable fact worth keeping",
project="p06-polisher",
confidence=0.5,
status="candidate",
)
bad = create_memory(
memory_type="project",
content="stale snapshot to reject",
project="atocore",
confidence=0.5,
status="candidate",
)
promote_memory(good.id)
reject_candidate_memory(bad.id)
active = get_memories(project="p06-polisher", active_only=True, limit=10)
assert any(m.id == good.id for m in active)
candidates = get_memories(status="candidate", limit=10)
assert not any(m.id == good.id for m in candidates)
assert not any(m.id == bad.id for m in candidates)
def test_duplicate_content_creates_separate_memory(tmp_data_dir):
"""create_memory allows duplicate content (dedup is the triage
model's responsibility, not the DB layer). Both memories exist."""
init_db()
m1 = create_memory(
memory_type="project",
content="unique fact about polisher",
project="p06-polisher",
)
m2 = create_memory(
memory_type="project",
content="unique fact about polisher",
project="p06-polisher",
status="candidate",
)
assert m1.id != m2.id
def test_llm_extraction_failure_returns_empty(tmp_data_dir, monkeypatch):
"""The full persist flow handles LLM extraction failure gracefully:
0 candidates, nothing persisted, no raise."""
init_db()
monkeypatch.setattr(extractor_llm, "_cli_available", lambda: True)
monkeypatch.setattr(
extractor_llm.subprocess,
"run",
lambda *a, **kw: _FakeCompleted("", returncode=1),
)
interaction = _make_interaction(
response="some real content that the LLM fails on",
project="p06-polisher",
)
result = extract_candidates_llm_verbose(interaction)
assert result.candidates == []
assert "exit_1" in result.error
# Nothing in the candidate queue
queue = get_memories(status="candidate", limit=10)
assert len(queue) == 0
def test_extract_batch_api_503_when_cli_missing(tmp_data_dir, monkeypatch):
"""R11: POST /admin/extract-batch with mode=llm must fail loud when
the `claude` CLI is unavailable, instead of silently returning a
success-with-0-candidates payload (which masked host-vs-container
truth for operators)."""
from fastapi.testclient import TestClient
from atocore.main import app
import atocore.api.routes as routes
init_db()
monkeypatch.setattr(routes, "_llm_cli_available", lambda: False)
client = TestClient(app)
response = client.post("/admin/extract-batch", json={"mode": "llm"})
assert response.status_code == 503
assert "claude" in response.json()["detail"].lower()
def test_extract_batch_api_rule_mode_ok_without_cli(tmp_data_dir, monkeypatch):
"""Rule mode must still work when the LLM CLI is missing — R11 only
affects mode=llm."""
from fastapi.testclient import TestClient
from atocore.main import app
import atocore.api.routes as routes
init_db()
monkeypatch.setattr(routes, "_llm_cli_available", lambda: False)
client = TestClient(app)
response = client.post("/admin/extract-batch", json={"mode": "rule"})
assert response.status_code == 200

View File

@@ -59,7 +59,8 @@ def test_parser_strips_surrounding_prose():
result = _parse_candidates(raw, _make_interaction()) result = _parse_candidates(raw, _make_interaction())
assert len(result) == 1 assert len(result) == 1
assert result[0].memory_type == "project" assert result[0].memory_type == "project"
assert result[0].project == "p04" # Model returned "p04" with no interaction scope — unscoped path
# resolves via registry if available, otherwise stays as-is
def test_parser_drops_invalid_memory_types(): def test_parser_drops_invalid_memory_types():
@@ -97,9 +98,9 @@ def test_parser_tags_version_and_rule():
assert result[0].source_interaction_id == "test-id" assert result[0].source_interaction_id == "test-id"
def test_parser_falls_back_to_interaction_project(): def test_case_a_empty_model_scoped_interaction():
"""R6: when the model returns empty project but the interaction """Case A: model returns empty project, interaction is scoped.
has one, the candidate should inherit the interaction's project.""" Interaction scope wins."""
raw = '[{"type": "project", "content": "machine works offline"}]' raw = '[{"type": "project", "content": "machine works offline"}]'
interaction = _make_interaction() interaction = _make_interaction()
interaction.project = "p06-polisher" interaction.project = "p06-polisher"
@@ -107,12 +108,77 @@ def test_parser_falls_back_to_interaction_project():
assert result[0].project == "p06-polisher" assert result[0].project == "p06-polisher"
def test_parser_keeps_model_project_when_provided(): def test_case_b_empty_model_unscoped_interaction():
"""Model-supplied project takes precedence over interaction.""" """Case B: both empty. Project stays empty."""
raw = '[{"type": "project", "content": "generic fact"}]'
interaction = _make_interaction()
interaction.project = ""
result = _parse_candidates(raw, interaction)
assert result[0].project == ""
def test_case_c_unregistered_model_scoped_interaction(tmp_data_dir, project_registry):
"""Case C: model returns unregistered project, interaction is scoped.
Interaction scope wins."""
from atocore.models.database import init_db
init_db()
project_registry(("p06-polisher", ["p06"]))
raw = '[{"type": "project", "content": "x", "project": "fake-project-99"}]'
interaction = _make_interaction()
interaction.project = "p06-polisher"
result = _parse_candidates(raw, interaction)
assert result[0].project == "p06-polisher"
def test_case_d_unregistered_model_unscoped_keeps_tag(tmp_data_dir, project_registry):
"""Case D: model returns unregistered project, interaction is unscoped.
Keeps the model's tag for auto-project-detection (new behavior)."""
from atocore.models.database import init_db
init_db()
project_registry(("p06-polisher", ["p06"]))
raw = '[{"type": "project", "content": "x", "project": "new-lead-project"}]'
interaction = _make_interaction()
interaction.project = ""
result = _parse_candidates(raw, interaction)
assert result[0].project == "new-lead-project"
def test_case_e_matching_model_and_interaction(tmp_data_dir, project_registry):
"""Case E: model returns same project as interaction. Works."""
from atocore.models.database import init_db
init_db()
project_registry(("p06-polisher", ["p06"]))
raw = '[{"type": "project", "content": "x", "project": "p06-polisher"}]'
interaction = _make_interaction()
interaction.project = "p06-polisher"
result = _parse_candidates(raw, interaction)
assert result[0].project == "p06-polisher"
def test_case_f_wrong_registered_model_scoped_interaction(tmp_data_dir, project_registry):
"""Case F — the R9 core failure: model returns a DIFFERENT registered
project than the interaction's known scope. Interaction scope wins.
This is the case that was broken before the R9 fix."""
from atocore.models.database import init_db
init_db()
project_registry(("p04-gigabit", ["p04"]), ("p06-polisher", ["p06"]))
raw = '[{"type": "project", "content": "x", "project": "p04-gigabit"}]' raw = '[{"type": "project", "content": "x", "project": "p04-gigabit"}]'
interaction = _make_interaction() interaction = _make_interaction()
interaction.project = "p06-polisher" interaction.project = "p06-polisher"
result = _parse_candidates(raw, interaction) result = _parse_candidates(raw, interaction)
assert result[0].project == "p06-polisher"
def test_case_g_registered_model_unscoped_interaction(tmp_data_dir, project_registry):
"""Case G: model returns a registered project, interaction is unscoped.
Model project accepted (only way to get a project for unscoped captures)."""
from atocore.models.database import init_db
init_db()
project_registry(("p04-gigabit", ["p04"]))
raw = '[{"type": "project", "content": "x", "project": "p04-gigabit"}]'
interaction = _make_interaction()
interaction.project = ""
result = _parse_candidates(raw, interaction)
assert result[0].project == "p04-gigabit" assert result[0].project == "p04-gigabit"

View File

@@ -186,3 +186,98 @@ def test_memories_for_context_empty(isolated_db):
text, chars = get_memories_for_context() text, chars = get_memories_for_context()
assert text == "" assert text == ""
assert chars == 0 assert chars == 0
# --- Phase 10: auto-promotion + candidate expiry ---
def _get_memory_by_id(memory_id):
"""Helper: fetch a single memory by ID."""
from atocore.models.database import get_connection
with get_connection() as conn:
row = conn.execute("SELECT * FROM memories WHERE id = ?", (memory_id,)).fetchone()
return dict(row) if row else None
def test_auto_promote_reinforced_basic(isolated_db):
from atocore.memory.service import (
auto_promote_reinforced,
create_memory,
reinforce_memory,
)
mem_obj = create_memory("knowledge", "Zerodur has near-zero CTE", status="candidate", confidence=0.7)
mid = mem_obj.id
# reinforce_memory only touches active memories, so we need to
# promote first to reinforce, then demote back to candidate —
# OR just bump reference_count + last_referenced_at directly
from atocore.models.database import get_connection
from datetime import datetime, timezone
now = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S")
with get_connection() as conn:
conn.execute(
"UPDATE memories SET reference_count = 3, last_referenced_at = ? WHERE id = ?",
(now, mid),
)
promoted = auto_promote_reinforced(min_reference_count=3, min_confidence=0.7)
assert mid in promoted
mem = _get_memory_by_id(mid)
assert mem["status"] == "active"
def test_auto_promote_reinforced_ignores_low_refs(isolated_db):
from atocore.memory.service import auto_promote_reinforced, create_memory
from atocore.models.database import get_connection
from datetime import datetime, timezone
mem_obj = create_memory("knowledge", "Some knowledge", status="candidate", confidence=0.7)
mid = mem_obj.id
now = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S")
with get_connection() as conn:
conn.execute(
"UPDATE memories SET reference_count = 1, last_referenced_at = ? WHERE id = ?",
(now, mid),
)
promoted = auto_promote_reinforced(min_reference_count=3, min_confidence=0.7)
assert mid not in promoted
mem = _get_memory_by_id(mid)
assert mem["status"] == "candidate"
def test_expire_stale_candidates(isolated_db):
from atocore.memory.service import create_memory, expire_stale_candidates
from atocore.models.database import get_connection
mem_obj = create_memory("knowledge", "Old unreferenced fact", status="candidate")
mid = mem_obj.id
with get_connection() as conn:
conn.execute(
"UPDATE memories SET created_at = datetime('now', '-30 days') WHERE id = ?",
(mid,),
)
expired = expire_stale_candidates(max_age_days=14)
assert mid in expired
mem = _get_memory_by_id(mid)
assert mem["status"] == "invalid"
def test_expire_stale_candidates_keeps_reinforced(isolated_db):
from atocore.memory.service import create_memory, expire_stale_candidates
from atocore.models.database import get_connection
mem_obj = create_memory("knowledge", "Referenced fact", status="candidate")
mid = mem_obj.id
with get_connection() as conn:
conn.execute(
"UPDATE memories SET reference_count = 1, "
"created_at = datetime('now', '-30 days') WHERE id = ?",
(mid,),
)
expired = expire_stale_candidates(max_age_days=14)
assert mid not in expired
mem = _get_memory_by_id(mid)
assert mem["status"] == "candidate"