Compare commits
4 Commits
main
...
codex/open
| Author | SHA1 | Date | |
|---|---|---|---|
| 500a29aeba | |||
| 0371739877 | |||
| f2ec5d43de | |||
| 72ca823206 |
@@ -6,22 +6,15 @@
|
||||
|
||||
## Orientation
|
||||
|
||||
- **live_sha** (Dalidou `/health` build_sha): `c2e7064` (verified 2026-04-15 via /health, build_time 2026-04-15T15:08:51Z)
|
||||
- **last_updated**: 2026-04-15 by Claude (deploy caught up; R10/R13 closed)
|
||||
- **main_tip**: `c2e7064` (plus one pending doc/ledger commit for this session)
|
||||
- **test_count**: 299 collected via `pytest --collect-only -q` on a clean checkout, 2026-04-15 (reproduction recipe in Quick Commands)
|
||||
- **harness**: `18/18 PASS`
|
||||
- **vectors**: 33,253
|
||||
- **active_memories**: 84 (31 project, 23 knowledge, 10 episodic, 8 adaptation, 7 preference, 5 identity)
|
||||
- **candidate_memories**: 2
|
||||
- **registered_projects**: atocore, p04-gigabit, p05-interferometer, p06-polisher, atomizer-v2, abb-space (aliased p08)
|
||||
- **project_state_entries**: 78 total (p04=9, p05=13, p06=13, atocore=43)
|
||||
- **entities**: 35 (engineering knowledge graph, Layer 2)
|
||||
- **off_host_backup**: `papa@192.168.86.39:/home/papa/atocore-backups/` via cron, verified
|
||||
- **nightly_pipeline**: backup → cleanup → rsync → **OpenClaw import** (NEW) → vault refresh (NEW) → extract → auto-triage → weekly synth/lint Sundays
|
||||
- **capture_clients**: claude-code (Stop hook), openclaw (plugin + file importer)
|
||||
- **wiki**: http://dalidou:8100/wiki (browse), /wiki/projects/{id}, /wiki/entities/{id}, /wiki/search
|
||||
- **dashboard**: http://dalidou:8100/admin/dashboard
|
||||
- **live_sha** (Dalidou `/health` build_sha): `8951c62` (R9 fix at e5e9a99 not yet deployed)
|
||||
- **last_updated**: 2026-04-12 by Codex (branch `codex/openclaw-capture-plugin`)
|
||||
- **main_tip**: `4f8bec7`
|
||||
- **test_count**: 290 passing (local dev shell)
|
||||
- **harness**: `17/18 PASS` (only p06-tailscale still failing)
|
||||
- **active_memories**: 41
|
||||
- **candidate_memories**: 0
|
||||
- **project_state_entries**: p04=5, p05=6, p06=6 (Wave 2 entries still present on live Dalidou; 17 total visible)
|
||||
- **off_host_backup**: `papa@192.168.86.39:/home/papa/atocore-backups/` via cron env `ATOCORE_BACKUP_RSYNC`, verified
|
||||
|
||||
## Active Plan
|
||||
|
||||
@@ -131,17 +124,17 @@ One branch `codex/extractor-eval-loop` for Day 1-5, a second `codex/retrieval-ha
|
||||
|-----|--------|----------|------------------------------------|-------------------------------------------------------------------------|--------------|--------|------------|-------------|
|
||||
| R1 | Codex | P1 | deploy/hooks/capture_stop.py:76-85 | Live Claude capture still omits `extract`, so "loop closed both sides" remains overstated in practice even though the API supports it | fixed | Claude | 2026-04-11 | c67bec0 |
|
||||
| R2 | Codex | P1 | src/atocore/context/builder.py | Project memories excluded from pack | fixed | Claude | 2026-04-11 | 8ea53f4 |
|
||||
| R3 | Claude | P2 | src/atocore/memory/extractor.py | Rule cues (`## Decision:`) never fire on conversational LLM text | declined | Claude | 2026-04-11 | see 2026-04-14 session log |
|
||||
| R3 | Claude | P2 | src/atocore/memory/extractor.py | Rule cues (`## Decision:`) never fire on conversational LLM text | open | Claude | 2026-04-11 | |
|
||||
| R4 | Codex | P2 | DEV-LEDGER.md:11 | Orientation `main_tip` was stale versus `HEAD` / `origin/main` | fixed | Codex | 2026-04-11 | 81307ce |
|
||||
| R5 | Codex | P1 | src/atocore/interactions/service.py:157-174 | The deployed extraction path still calls only the rule extractor; the new LLM extractor is eval/script-only, so Day 4 "gate cleared" is true as a benchmark result but not as an operational extraction path | fixed | Claude | 2026-04-12 | c67bec0 |
|
||||
| R6 | Codex | P1 | src/atocore/memory/extractor_llm.py:258-276 | LLM extraction accepts model-supplied `project` verbatim with no fallback to `interaction.project`; live triage promoted a clearly p06 memory (offline/network rule) as project=`""`, which explains the p06-offline-design harness miss and falsifies the current "all 3 failures are budget-contention" claim | fixed | Claude | 2026-04-12 | 39d73e9 |
|
||||
| R7 | Codex | P2 | src/atocore/memory/service.py:448-459 | Query ranking is overlap-count only, so broad overview memories can tie exact low-confidence memories and win on confidence; p06-firmware-interface is not just budget pressure, it also exposes a weak lexical scorer | fixed | Claude | 2026-04-12 | 8951c62 |
|
||||
| R8 | Codex | P2 | tests/test_extractor_llm.py:1-7 | LLM extractor tests stop at parser/failure contracts; there is no automated coverage for the script-only persistence/review path that produced the 16 promoted memories, including project-scope preservation | fixed | Claude | 2026-04-12 | 69c9717 |
|
||||
| R9 | Codex | P2 | src/atocore/memory/extractor_llm.py:258-259 | The R6 fallback only repairs empty project output. A wrong non-empty model project still overrides the interaction's known scope, so project attribution is improved but not yet trust-preserving. | fixed | Claude | 2026-04-12 | e5e9a99 |
|
||||
| R10 | Codex | P2 | docs/master-plan-status.md:31-33 | "Phase 8 - OpenClaw Integration" is fair as a baseline milestone, but not as a "primary" integration claim. `t420-openclaw/atocore.py` currently covers a narrow read-oriented subset (13 request shapes vs 32 API routes) plus fail-open health, while memory/interactions/admin write paths remain out of surface. | fixed | Claude | 2026-04-12 | (pending) |
|
||||
| R11 | Codex | P2 | src/atocore/api/routes.py:773-845 | `POST /admin/extract-batch` still accepts `mode="llm"` inside the container and returns a successful 0-candidate result instead of surfacing that host-only LLM extraction is unavailable from this runtime. That is a misleading API contract for operators. | fixed | Claude | 2026-04-12 | (pending) |
|
||||
| R12 | Codex | P2 | scripts/batch_llm_extract_live.py:39-190 | The host-side extractor duplicates the LLM system prompt and JSON parsing logic from `src/atocore/memory/extractor_llm.py`. It works today, but this is now a prompt/parser drift risk across the container and host implementations. | fixed | Claude | 2026-04-12 | (pending) |
|
||||
| R13 | Codex | P2 | DEV-LEDGER.md:12 | The new `286 passing` test-count claim is not reproducibly auditable from the current audit environments: neither Dalidou nor the clean worktree has `pytest` available. The claim may be true in Claude's dev shell, but it remains unverified in this audit. | fixed | Claude | 2026-04-12 | (pending) |
|
||||
| R10 | Codex | P2 | docs/master-plan-status.md:31-33 | "Phase 8 - OpenClaw Integration" is fair as a baseline milestone, but not as a "primary" integration claim. `t420-openclaw/atocore.py` currently covers a narrow read-oriented subset (13 request shapes vs 32 API routes) plus fail-open health, while memory/interactions/admin write paths remain out of surface. | open | Claude | 2026-04-12 | |
|
||||
| R11 | Codex | P2 | src/atocore/api/routes.py:773-845 | `POST /admin/extract-batch` still accepts `mode="llm"` inside the container and returns a successful 0-candidate result instead of surfacing that host-only LLM extraction is unavailable from this runtime. That is a misleading API contract for operators. | open | Claude | 2026-04-12 | |
|
||||
| R12 | Codex | P2 | scripts/batch_llm_extract_live.py:39-190 | The host-side extractor duplicates the LLM system prompt and JSON parsing logic from `src/atocore/memory/extractor_llm.py`. It works today, but this is now a prompt/parser drift risk across the container and host implementations. | open | Claude | 2026-04-12 | |
|
||||
| R13 | Codex | P2 | DEV-LEDGER.md:12 | The new `286 passing` test-count claim is not reproducibly auditable from the current audit environments: neither Dalidou nor the clean worktree has `pytest` available. The claim may be true in Claude's dev shell, but it remains unverified in this audit. | open | Claude | 2026-04-12 | |
|
||||
|
||||
## Recent Decisions
|
||||
|
||||
@@ -159,16 +152,10 @@ One branch `codex/extractor-eval-loop` for Day 1-5, a second `codex/retrieval-ha
|
||||
|
||||
## Session Log
|
||||
|
||||
- **2026-04-15 Claude (pm)** Closed the last harness failure honestly. **p06-tailscale fixed: 18/18 PASS.** Root-caused: not a retrieval bug — the p06 `ARCHITECTURE.md` Overview chunk legitimately mentions "the GigaBIT M1 telescope mirror" because the Polisher Suite is built *for* that mirror. All four retrieved sources for the tailscale prompt were genuinely p06/shared paths; zero actual p04 chunks leaked. The fixture's `expect_absent: GigaBIT` was catching semantic overlap, not retrieval bleed. Narrowed it to `expect_absent: "[Source: p04-gigabit/"` — a source-path check that tests the real invariant (no p04 source chunks in p06 context). Other p06 fixtures still use the word-blacklist form; they pass today because their more-specific prompts don't pull the ARCHITECTURE.md Overview, so I left them alone rather than churn fixtures that aren't failing. Did NOT change retrieval/ranking — no code change, fixture-only fix. Tests unchanged at 299.
|
||||
|
||||
- **2026-04-15 Claude** Deploy + doc debt sweep. Deployed `c2e7064` to Dalidou (build_time 2026-04-15T15:08:51Z, build_sha matches, /health ok) so R11/R12 are now live, not just on main. **R11 verified on live**: `POST /admin/extract-batch {"mode":"llm"}` against http://127.0.0.1:8100 returns HTTP 503 with the operator-facing "claude CLI not on PATH, run host-side script or use mode=rule" message — exactly the post-fix contract. **R13 closed (fixed)**: added a reproduction recipe to Quick Commands (`pip install -r requirements-dev.txt && pytest --collect-only -q && pytest -q`) and re-cited `test_count: 299` against a fresh local collection on 2026-04-15, so the claim is now auditable from any clean checkout — Codex's audit worktree just needs `pip install -r requirements-dev.txt`. **R10 closed (fixed)**: rewrote the `docs/master-plan-status.md` OpenClaw section to explicitly disclaim "primary integration" and report the current narrow surface: 14 client request shapes against ~44 server routes, predominantly read + `/project/state` + `/ingest/sources`, with memory/interactions/admin/entities/triage/extraction writes correctly out of scope. Open findings now: none blocking. Next natural move: the last harness failure `p06-tailscale` (chunk bleed).
|
||||
|
||||
- **2026-04-14 Claude (pm)** Closed R11+R12, declined R3. **R11 (fixed):** `POST /admin/extract-batch` with `mode="llm"` now returns 503 when the `claude` CLI is not on PATH, with a message pointing at the host-side script. Previously it silently returned a success-0 payload, masking host-vs-container truth. 2 new tests in `test_extraction_pipeline.py` cover the 503 path and the rule-mode-still-works path. **R12 (fixed):** extracted shared `SYSTEM_PROMPT` + `parse_llm_json_array` + `normalize_candidate_item` + `build_user_message` into stdlib-only `src/atocore/memory/_llm_prompt.py`. Both `src/atocore/memory/extractor_llm.py` (container) and `scripts/batch_llm_extract_live.py` (host) now import from it. The host script uses `sys.path` to reach the stdlib-only module without needing the full atocore package. Project-attribution policy stays path-specific (container uses registry-check; host defers to server). **R3 (declined):** rule cues not firing on conversational LLM text is by design now — the LLM extractor (llm-0.4.0) is the production path for conversational content as of the Day 4 gate (2026-04-12). Expanding rules to match conversational prose risks the FP blowup Day 2 already showed. Rule extractor stays narrow for structural PKM text. Tests 297 → 299. Live `/health` still `58ea21d`; this session's changes need deploy.
|
||||
|
||||
- **2026-04-14 Claude** MAJOR session: Engineering knowledge layer V1 (Layer 2) built — entity + relationship tables, 15 types, 12 relationship kinds, 35 bootstrapped entities across p04/p05/p06. Human Mirror (Layer 3) — GET /projects/{name}/mirror.html + navigable wiki at /wiki with search. Karpathy-inspired upgrades: contradiction detection in triage, weekly lint pass, weekly synthesis pass producing "current state" paragraphs at top of project pages. Auto-detection of new projects from extraction. Registry persistence fix (ATOCORE_PROJECT_REGISTRY_DIR env var). abb-space/p08 aliases added, atomizer-v2 ingested (568 docs, +12,472 vectors). Identity/preference seed (6 new), signal-aggressive extractor rewrite (llm-0.4.0), auto vault refresh in cron. **OpenClaw one-way pull importer** built per codex proposal — reads /home/papa/clawd SOUL.md, USER.md, MEMORY.md, MODEL-ROUTING.md, memory/*.md via SSH, hash-delta import, pipeline triages. First import: 10 candidates → 10 promoted with lenient triage rule. Active memories 47→84. State entries 61→78. Tests 290→297. Dashboard at /admin/dashboard. Wiki at /wiki.
|
||||
|
||||
|
||||
- **2026-04-12 Claude** `4f8bec7..4ac4e5c` Session close. Merged OpenClaw capture plugin, ingested atomizer-v2 (568 docs, 12,472 new vectors → 33,253 total), seeded Phase 4 identity/preference memories (6 new, 47 total active), added deeper Wave 2 state entries (p05 +3, p06 +3), fixed R9 project trust hierarchy (7 case tests), built auto-triage pipeline, observability dashboard at /admin/dashboard. Updated master-plan-status.md and DEV-LEDGER.md to reflect full current state. 7/14 phases baseline complete. All P1s closed. Nightly pipeline runs unattended with both Claude Code and OpenClaw feeding the reflection loop.
|
||||
- **2026-04-12 Codex (branch `codex/openclaw-capture-plugin`, verification close)** verified the final capture-plugin behavior on Dalidou after the `message_sending` reliability fix. New OpenClaw interactions now capture reliably and the stored prompt is clean human text instead of the Discord wrapper blob. Verified examples on Dalidou: `Final capture test` and `Yes, fix it, or I'll ask opus to do it`. The oldest two wrapper-heavy captures remain in history from earlier iterations, but new captures are clean.
|
||||
- **2026-04-12 Codex (branch `codex/openclaw-capture-plugin`, polish pass 3)** changed turn pairing from `llm_output` to `message_sending`. The plugin now caches the human prompt at `before_dispatch` and posts to AtoCore only when OpenClaw emits the real outbound assistant message. This should restore reliability while keeping prompt cleanliness. Awaiting one more post-restart validation turn.
|
||||
- **2026-04-12 Codex (branch `codex/openclaw-capture-plugin`, polish pass 2)** switched prompt capture from `before_agent_reply.cleanedBody` to `before_dispatch.body` / `content`, because the earlier path still stored Discord wrapper metadata. This should bind capture to the dispatch-stage human message instead of the prompt-builder artifact. Awaiting one more post-restart turn to verify on Dalidou.
|
||||
- **2026-04-12 Codex (branch `codex/openclaw-capture-plugin`, polish pass)** tightened the OpenClaw capture plugin to use `before_agent_reply.cleanedBody` instead of the raw prompt-build input, which should prevent Discord wrapper metadata from being stored as the interaction prompt. Added `agent_end` cleanup and updated plugin docs. A fresh post-restart user turn is still needed to verify prompt cleanliness on Dalidou.
|
||||
- **2026-04-12 Codex (branch `codex/openclaw-capture-plugin`)** added a minimal external OpenClaw plugin at `openclaw-plugins/atocore-capture/` that mirrors Claude Code capture semantics: user-triggered assistant turns are POSTed to AtoCore `/interactions` with `client="openclaw"` and `reinforce=true`, fail-open, no extraction in-path. For live verification, temporarily added the local plugin load path to OpenClaw config and restarted the gateway so the plugin can load. Branch truth is ready; end-to-end verification still needs one fresh post-restart OpenClaw user turn to confirm new `client=openclaw` interactions appear on Dalidou.
|
||||
- **2026-04-12 Claude** Batch 3 (R9 fix): `144dbbd..e5e9a99`. Trust hierarchy for project attribution — interaction scope always wins when set, model project only used for unscoped interactions + registered check. 7 case tests (A-G) cover every combination. Harness 17/18 (no regression). Tests 286->290. Before: wrong registered project could silently override interaction scope. After: interaction.project is the strongest signal; model project is only a fallback for unscoped captures. Not yet guaranteed: nothing prevents the *same* project's model output from being semantically wrong within that project. R9 marked fixed.
|
||||
|
||||
@@ -213,9 +200,4 @@ git push origin main && ssh papa@dalidou "bash /srv/storage/atocore/app/deploy/d
|
||||
python scripts/atocore_client.py batch-extract '' '' 200 false # preview
|
||||
python scripts/atocore_client.py batch-extract '' '' 200 true # persist
|
||||
python scripts/atocore_client.py triage
|
||||
|
||||
# Reproduce the ledger's test_count claim from a clean checkout
|
||||
pip install -r requirements-dev.txt
|
||||
pytest --collect-only -q | tail -1 # -> "N tests collected"
|
||||
pytest -q # -> "N passed"
|
||||
```
|
||||
|
||||
@@ -51,19 +51,4 @@ python3 "$APP_DIR/scripts/auto_triage.py" \
|
||||
log "WARN: auto-triage failed (non-blocking)"
|
||||
}
|
||||
|
||||
# Step C: Weekly synthesis (Sundays only)
|
||||
if [[ "$(date -u +%u)" == "7" ]]; then
|
||||
log "Step C: weekly project synthesis"
|
||||
python3 "$APP_DIR/scripts/synthesize_projects.py" \
|
||||
--base-url "$ATOCORE_URL" \
|
||||
2>&1 || {
|
||||
log "WARN: synthesis failed (non-blocking)"
|
||||
}
|
||||
|
||||
log "Step D: weekly lint pass"
|
||||
python3 "$APP_DIR/scripts/lint_knowledge_base.py" \
|
||||
--base-url "$ATOCORE_URL" \
|
||||
2>&1 || true
|
||||
fi
|
||||
|
||||
log "=== AtoCore batch extraction + triage complete ==="
|
||||
|
||||
@@ -82,32 +82,6 @@ else
|
||||
log "Step 3: ATOCORE_BACKUP_RSYNC not set, skipping off-host copy"
|
||||
fi
|
||||
|
||||
# Step 3a: Pull OpenClaw state from clawdbot (one-way import of
|
||||
# SOUL.md, USER.md, MODEL-ROUTING.md, MEMORY.md, recent memory/*.md).
|
||||
# Loose coupling: OpenClaw's internals don't need to change.
|
||||
# Fail-open: importer failure never blocks the pipeline.
|
||||
log "Step 3a: pull OpenClaw state"
|
||||
OPENCLAW_IMPORT="${ATOCORE_OPENCLAW_IMPORT:-true}"
|
||||
if [[ "$OPENCLAW_IMPORT" == "true" ]]; then
|
||||
python3 "$SCRIPT_DIR/../../scripts/import_openclaw_state.py" \
|
||||
--base-url "$ATOCORE_URL" \
|
||||
2>&1 | while IFS= read -r line; do log " $line"; done || {
|
||||
log " WARN: OpenClaw import failed (non-blocking)"
|
||||
}
|
||||
else
|
||||
log " skipped (ATOCORE_OPENCLAW_IMPORT != true)"
|
||||
fi
|
||||
|
||||
# Step 3b: Auto-refresh vault sources so new PKM files flow in
|
||||
# automatically. Fail-open: never blocks the rest of the pipeline.
|
||||
log "Step 3b: auto-refresh vault sources"
|
||||
REFRESH_RESULT=$(curl -sf -X POST --max-time 600 \
|
||||
"$ATOCORE_URL/ingest/sources" 2>&1) && {
|
||||
log "Sources refresh complete"
|
||||
} || {
|
||||
log "WARN: sources refresh failed (non-blocking): $REFRESH_RESULT"
|
||||
}
|
||||
|
||||
# Step 4: Batch LLM extraction on recent interactions (optional).
|
||||
# Runs HOST-SIDE because claude CLI is on the host, not inside the
|
||||
# Docker container. The script fetches interactions from the API,
|
||||
|
||||
@@ -1,206 +0,0 @@
|
||||
# AtoCore Knowledge Architecture
|
||||
|
||||
## The Problem
|
||||
|
||||
Engineering work produces two kinds of knowledge simultaneously:
|
||||
|
||||
1. **Applied knowledge** — specific to the project being worked on
|
||||
("the p04 support pad layout is driven by CTE gradient analysis")
|
||||
2. **Domain knowledge** — generalizable insight earned through that work
|
||||
("Zerodur CTE gradient dominates WFE at fast focal ratios")
|
||||
|
||||
A system that only stores applied knowledge loses the general insight.
|
||||
A system that mixes them pollutes project context with cross-project
|
||||
noise. AtoCore needs both — separated, but both growing organically
|
||||
from the same conversations.
|
||||
|
||||
## The Quality Bar
|
||||
|
||||
**AtoCore stores earned insight, not information.**
|
||||
|
||||
The test: "Would a competent engineer need experience to know this,
|
||||
or could they find it in 30 seconds?"
|
||||
|
||||
| Store | Don't store |
|
||||
|-------|-------------|
|
||||
| "Preston removal model breaks down below 5N because the contact assumption fails" | "Preston's equation relates removal rate to pressure and velocity" |
|
||||
| "m=1 (coma) is NOT correctable by force modulation (score 0.09)" | "Zernike polynomials describe wavefront aberrations" |
|
||||
| "At F/1.2, CTE gradient costs ~3nm WFE and drives pad placement" | "Zerodur CTE is 0.05 ppm/K" |
|
||||
| "Quilting limit for 16-inch tool is 234N" | "Quilting is a mid-spatial-frequency artifact in polishing" |
|
||||
|
||||
The bar is enforced in the LLM extraction system prompt
|
||||
(`src/atocore/memory/extractor_llm.py`) and the auto-triage prompt
|
||||
(`scripts/auto_triage.py`). Both explicitly list examples of what
|
||||
qualifies and what doesn't.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Five-tier context assembly
|
||||
|
||||
When AtoCore builds a context pack for any LLM query, it assembles
|
||||
five tiers in strict trust order:
|
||||
|
||||
```
|
||||
Tier 1: Trusted Project State [project-specific, highest trust]
|
||||
Curated key-value entries from the project state API.
|
||||
Example: "decision/vendor_path: Twyman-Green preferred, 4D
|
||||
technical lead but cost-challenged"
|
||||
|
||||
Tier 2: Identity / Preferences [global, always included]
|
||||
Who the user is and how they work.
|
||||
Example: "Antoine Letarte, mechanical/optical engineer at
|
||||
Atomaste" / "No API keys — uses OAuth exclusively"
|
||||
|
||||
Tier 3: Project Memories [project-specific]
|
||||
Reinforced memories from the reflection loop, scoped to the
|
||||
queried project. Example: "Firmware interface contract is
|
||||
invariant: controller-job.v1 in, run-log.v1 out"
|
||||
|
||||
Tier 4: Domain Knowledge [cross-project]
|
||||
Earned engineering insight with project="" and a domain tag.
|
||||
Surfaces in ALL project packs when query-relevant.
|
||||
Example: "[materials] Zerodur CTE gradient dominates WFE at
|
||||
fast focal ratios — costs ~3nm at F/1.2"
|
||||
|
||||
Tier 5: Retrieved Chunks [project-boosted, lowest trust]
|
||||
Vector-similarity search over the ingested document corpus.
|
||||
Project-hinted but not filtered — cross-project docs can
|
||||
appear at lower rank.
|
||||
```
|
||||
|
||||
### Budget allocation (at default 3000 chars)
|
||||
|
||||
| Tier | Budget ratio | Approx chars | Entries |
|
||||
|------|-------------|-------------|---------|
|
||||
| Project State | 20% | 600 | all curated entries |
|
||||
| Identity/Preferences | 5% | 150 | 1 memory |
|
||||
| Project Memories | 25% | 750 | 2-3 memories |
|
||||
| Domain Knowledge | 10% | 300 | 1-2 memories |
|
||||
| Retrieved Chunks | 40% | 1200 | 2-4 chunks |
|
||||
|
||||
Trim order when budget is tight: chunks first, then domain knowledge,
|
||||
then project memories, then identity, then project state last.
|
||||
|
||||
### Knowledge domains
|
||||
|
||||
The LLM extractor tags domain knowledge with one of these domains:
|
||||
|
||||
| Domain | What qualifies |
|
||||
|--------|---------------|
|
||||
| `physics` | Optical physics, wave propagation, diffraction, thermal effects |
|
||||
| `materials` | Material properties in context, CTE behavior, stress limits |
|
||||
| `optics` | Lens/mirror design, aberration analysis, metrology techniques |
|
||||
| `mechanics` | Structural FEA insights, support system design, kinematics |
|
||||
| `manufacturing` | Polishing, grinding, machining, process control |
|
||||
| `metrology` | Measurement systems, interferometry, calibration techniques |
|
||||
| `controls` | PID tuning, force control, servo systems, real-time constraints |
|
||||
| `software` | Architecture patterns, testing strategies, deployment insights |
|
||||
| `math` | Numerical methods, optimization, statistical analysis |
|
||||
| `finance` | Cost modeling, procurement strategy, budget optimization |
|
||||
|
||||
New domains can be added by updating the system prompt in
|
||||
`extractor_llm.py` and `batch_llm_extract_live.py`.
|
||||
|
||||
### How domain knowledge is stored
|
||||
|
||||
Domain tags are embedded as a prefix in the memory content:
|
||||
|
||||
```
|
||||
memory_type: knowledge
|
||||
project: "" ← empty = cross-project
|
||||
content: "[materials] Zerodur CTE gradient dominates WFE at F/1.2"
|
||||
```
|
||||
|
||||
The `[domain]` prefix is a lightweight encoding that avoids a schema
|
||||
migration. The context builder's query-relevance ranking matches on
|
||||
domain terms naturally (a query about "materials" or "CTE" will rank
|
||||
a `[materials]` memory higher). A future migration can parse the
|
||||
prefix into a proper `domain` column.
|
||||
|
||||
## How knowledge flows
|
||||
|
||||
### Capture → Extract → Triage → Surface
|
||||
|
||||
```
|
||||
1. CAPTURE
|
||||
Claude Code (Stop hook) or OpenClaw (plugin)
|
||||
→ POST /interactions with reinforce=true
|
||||
→ Interaction stored on Dalidou
|
||||
|
||||
2. EXTRACT (nightly cron, 03:00 UTC)
|
||||
batch_llm_extract_live.py runs claude -p sonnet
|
||||
→ For each interaction, the LLM decides:
|
||||
- Is this project-specific? → candidate with project=X
|
||||
- Is this generalizable insight? → candidate with domain=Y, project=""
|
||||
- Is it both? → TWO candidates emitted
|
||||
- Is it common knowledge? → skip (quality bar)
|
||||
→ Candidates persisted as status=candidate
|
||||
|
||||
3. TRIAGE (nightly, immediately after extraction)
|
||||
auto_triage.py runs claude -p sonnet
|
||||
→ Each candidate classified: promote / reject / needs_human
|
||||
→ Auto-promote at confidence ≥ 0.8 + no duplicate
|
||||
→ Auto-reject stale snapshots, duplicates, common knowledge
|
||||
→ Only needs_human reaches the operator
|
||||
|
||||
4. SURFACE (every context/build query)
|
||||
→ Project-specific memories appear in Tier 3
|
||||
→ Domain knowledge appears in Tier 4 (regardless of project)
|
||||
→ Both are query-ranked by overlap-density
|
||||
```
|
||||
|
||||
### Example: knowledge earned on p04 surfaces on p06
|
||||
|
||||
Working on p04-gigabit, you discover that Zerodur CTE gradient is
|
||||
the dominant WFE contributor at fast focal ratios. The extraction
|
||||
produces:
|
||||
|
||||
```json
|
||||
[
|
||||
{"type": "project", "content": "CTE gradient analysis drove the
|
||||
M1 support pad layout — 2nd largest WFE contributor after gravity",
|
||||
"project": "p04-gigabit", "domain": "", "confidence": 0.6},
|
||||
|
||||
{"type": "knowledge", "content": "Zerodur CTE gradient dominates
|
||||
WFE contribution at fast focal ratios (F/1.2 = ~3nm)",
|
||||
"project": "", "domain": "materials", "confidence": 0.6}
|
||||
]
|
||||
```
|
||||
|
||||
Two weeks later, working on p06-polisher (which also uses Zerodur):
|
||||
|
||||
```
|
||||
Query: "thermal effects on polishing accuracy"
|
||||
Project: p06-polisher
|
||||
|
||||
Tier 3 (Project Memories):
|
||||
[project] Calibration loop adjusts Preston kp from surface measurements...
|
||||
|
||||
Tier 4 (Domain Knowledge):
|
||||
[materials] Zerodur CTE gradient dominates WFE contribution at fast
|
||||
focal ratios — THIS CAME FROM P04 WORK
|
||||
```
|
||||
|
||||
The insight crosses over without any manual curation.
|
||||
|
||||
## Future directions
|
||||
|
||||
### Personal knowledge branch
|
||||
The same architecture supports personal domains (health, finance,
|
||||
personal) by adding new domain tags and a trust boundary so
|
||||
Atomaste project data never leaks into personal packs. The domain
|
||||
system is domain-agnostic — it doesn't care whether the domain is
|
||||
"optics" or "nutrition".
|
||||
|
||||
### Multi-model extraction
|
||||
Different models can specialize: sonnet for extraction, opus or
|
||||
Gemini for triage review. Independent validation reduces correlated
|
||||
blind spots on what qualifies as "earned insight" vs "common
|
||||
knowledge."
|
||||
|
||||
### Reinforcement-based domain promotion
|
||||
A domain-knowledge memory that gets reinforced across multiple
|
||||
projects (its content echoed in p04, p05, and p06 responses)
|
||||
accumulates confidence faster than a project-specific memory.
|
||||
High-confidence domain memories could auto-promote to a "verified
|
||||
knowledge" tier above regular domain knowledge.
|
||||
@@ -33,21 +33,15 @@ read-only additive mode.
|
||||
at 5% budget ratio. Future identity/preference extraction happens
|
||||
organically via the nightly LLM extraction pipeline.
|
||||
|
||||
- Phase 8 - OpenClaw Integration (baseline only, not primary surface).
|
||||
As of 2026-04-15 the T420 OpenClaw helper (`t420-openclaw/atocore.py`)
|
||||
is verified end-to-end against live Dalidou: health check, auto-context
|
||||
with project detection, Trusted Project State surfacing, project-memory
|
||||
band, fail-open on unreachable host. Tested from both the development
|
||||
machine and the T420 via SSH. Scope is narrow: **14 request shapes
|
||||
against ~44 server routes**, predominantly read-oriented plus
|
||||
`POST/DELETE /project/state` and `POST /ingest/sources`. Memory
|
||||
management, interactions capture (covered separately by the OpenClaw
|
||||
capture plugin), admin/backup, entities, triage, and extraction write
|
||||
paths remain out of this client's surface by design — they are scoped
|
||||
to the operator client (`scripts/atocore_client.py`) per the
|
||||
read-heavy additive integration model. "Primary integration" is
|
||||
therefore overclaim; "baseline read + project-state write helper" is
|
||||
the accurate framing.
|
||||
- Phase 8 - OpenClaw Integration. As of 2026-04-12 the T420 OpenClaw
|
||||
helper (`t420-openclaw/atocore.py`) is verified end-to-end against
|
||||
live Dalidou: health check, auto-context with project detection,
|
||||
Trusted Project State surfacing, project-memory band, fail-open on
|
||||
unreachable host. Tested from both the development machine and the
|
||||
T420 via SSH. The helper covers 15 of the 33 API endpoints — the
|
||||
excluded endpoints (memory management, interactions, backup) are
|
||||
correctly scoped to the operator client (`scripts/atocore_client.py`)
|
||||
per the read-only additive integration model.
|
||||
|
||||
### Baseline Complete
|
||||
|
||||
@@ -126,52 +120,59 @@ This sits implicitly between Phase 8 (OpenClaw) and Phase 11
|
||||
(multi-model). Memory-review and engineering-entity commands are
|
||||
deferred from the shared client until their workflows are exercised.
|
||||
|
||||
## What Is Real Today (updated 2026-04-12)
|
||||
## What Is Real Today
|
||||
|
||||
- canonical AtoCore runtime on Dalidou (build_sha tracked, deploy.sh verified)
|
||||
- 33,253 vectors across 5 registered projects
|
||||
- project registry with template, proposal, register, update, refresh
|
||||
- 5 registered projects:
|
||||
- `p04-gigabit` (483 docs, 5 state entries)
|
||||
- `p05-interferometer` (109 docs, 9 state entries)
|
||||
- `p06-polisher` (564 docs, 9 state entries)
|
||||
- `atomizer-v2` (568 docs, newly ingested 2026-04-12)
|
||||
- `atocore` (drive source, 38 state entries)
|
||||
- 47 active memories (16 project, 16 knowledge, 6 adaptation, 3 identity, 3 preference, 3 episodic)
|
||||
- context pack assembly with 4 tiers: Trusted Project State > identity/preference > project memories > retrieved chunks
|
||||
- query-relevance memory ranking with overlap-density scoring
|
||||
- retrieval eval harness: 18 fixtures, 17/18 passing
|
||||
- 290 tests passing
|
||||
- nightly pipeline: backup → cleanup → rsync → LLM extraction (sonnet) → auto-triage
|
||||
- off-host backup to clawdbot (T420) via rsync
|
||||
- both Claude Code and OpenClaw capture interactions to AtoCore
|
||||
- DEV-LEDGER.md as shared operating memory between Claude and Codex
|
||||
- observability dashboard at GET /admin/dashboard
|
||||
- canonical AtoCore runtime on Dalidou
|
||||
- canonical machine DB and vector store on Dalidou
|
||||
- project registry with:
|
||||
- template
|
||||
- proposal preview
|
||||
- register
|
||||
- update
|
||||
- refresh
|
||||
- read-only additive OpenClaw helper on the T420
|
||||
- seeded project corpus for:
|
||||
- `p04-gigabit`
|
||||
- `p05-interferometer`
|
||||
- `p06-polisher`
|
||||
- conservative Trusted Project State for those active projects
|
||||
- first operational backup foundation for SQLite + project registry
|
||||
- implementation-facing architecture notes for future engineering knowledge work
|
||||
- first organic routing layer in OpenClaw via:
|
||||
- `detect-project`
|
||||
- `auto-context`
|
||||
|
||||
## Now
|
||||
|
||||
These are the current practical priorities.
|
||||
|
||||
1. **Observe and stabilize** — let the nightly pipeline run for a week,
|
||||
check the dashboard daily, verify memories accumulate correctly
|
||||
from organic Claude Code and OpenClaw use
|
||||
2. **Multi-model triage** (Phase 11 entry) — switch auto-triage to a
|
||||
different model than the extractor for independent validation
|
||||
3. **Automated eval in cron** (Phase 12 entry) — add retrieval harness
|
||||
to the nightly cron so regressions are caught automatically
|
||||
4. **Atomizer-v2 state entries** — curate Trusted Project State for the
|
||||
newly ingested Atomizer knowledge base
|
||||
1. Finish practical OpenClaw integration
|
||||
- make the helper lifecycle feel natural in daily use
|
||||
- use the new organic routing layer for project-knowledge questions
|
||||
- confirm fail-open behavior remains acceptable
|
||||
- keep AtoCore clearly additive
|
||||
2. Tighten retrieval quality
|
||||
- reduce cross-project competition
|
||||
- improve ranking on short or ambiguous prompts
|
||||
- add only a few anchor docs where retrieval is still weak
|
||||
3. Continue controlled ingestion
|
||||
- deepen active projects selectively
|
||||
- avoid noisy bulk corpus growth
|
||||
4. Strengthen operational boringness
|
||||
- backup and restore procedure
|
||||
- Chroma rebuild / backup policy
|
||||
- retention and restore validation
|
||||
|
||||
## Next
|
||||
|
||||
These are the next major layers after the current stabilization pass.
|
||||
These are the next major layers after the current practical pass.
|
||||
|
||||
1. Phase 10 Write-back — confidence-based auto-promotion from
|
||||
reinforcement signal (a memory reinforced N times auto-promotes)
|
||||
2. Phase 6 AtoDrive — clarify Google Drive as a trusted operational
|
||||
source and ingest from it
|
||||
3. Phase 13 Hardening — Chroma backup policy, monitoring, alerting,
|
||||
failure visibility beyond log files
|
||||
1. Clarify AtoDrive as a real operational truth layer
|
||||
2. Mature identity / preferences handling
|
||||
3. Improve observability for:
|
||||
- retrieval quality
|
||||
- context-pack inspection
|
||||
- comparison of behavior with and without AtoCore
|
||||
|
||||
## Later
|
||||
|
||||
@@ -189,16 +190,11 @@ direction, but not yet ready for immediate implementation.
|
||||
|
||||
These remain intentionally deferred.
|
||||
|
||||
- ~~automatic write-back from OpenClaw into AtoCore~~ — OpenClaw capture
|
||||
plugin now exists (`openclaw-plugins/atocore-capture/`), interactions
|
||||
flow. Write-back of promoted memories back to OpenClaw's own memory
|
||||
system is still deferred.
|
||||
- ~~automatic memory promotion~~ — auto-triage now handles promote/reject
|
||||
for extraction candidates. Reinforcement-based auto-promotion
|
||||
(Phase 10) is the remaining piece.
|
||||
- ~~reflection loop integration~~ — fully operational: capture (both
|
||||
clients) → reinforce (automatic) → extract (nightly cron, sonnet) →
|
||||
auto-triage (nightly, sonnet) → only needs_human reaches the user.
|
||||
- automatic write-back from OpenClaw into AtoCore
|
||||
- automatic memory promotion
|
||||
- ~~reflection loop integration~~ — baseline now in (capture→reinforce
|
||||
auto, extract batch/manual). Extractor tuning and scheduled batch
|
||||
extraction still open.
|
||||
- replacing OpenClaw's own memory system
|
||||
- live machine-DB sync between machines
|
||||
- full ontology / graph expansion before the current baseline is stable
|
||||
|
||||
@@ -1,56 +0,0 @@
|
||||
# OpenClaw -> AtoCore Integration Proposal
|
||||
|
||||
One-way pull is the right pattern.
|
||||
|
||||
**Stable surface to pull**
|
||||
- Durable files in the OpenClaw workspace:
|
||||
- `SOUL.md`
|
||||
- `USER.md`
|
||||
- `MODEL-ROUTING.md`
|
||||
- `MEMORY.md`
|
||||
- `memory/YYYY-MM-DD.md`
|
||||
- `memory/heartbeat-state.json`
|
||||
- `HEARTBEAT.md` only as operational state, not long-term truth
|
||||
- These are explicitly documented in `t420-openclaw/AGENTS.md` as the continuity layer OpenClaw reads every session.
|
||||
|
||||
**Volatile vs durable**
|
||||
- Durable:
|
||||
- `SOUL.md`, `USER.md`, `MODEL-ROUTING.md`, `MEMORY.md`
|
||||
- dated memory notes under `memory/`
|
||||
- explicit JSON state like `memory/heartbeat-state.json`
|
||||
- Volatile:
|
||||
- in-session context
|
||||
- ephemeral heartbeat work
|
||||
- transient orchestration state
|
||||
- platform response buffers
|
||||
- Semi-durable:
|
||||
- `HEARTBEAT.md` and operational notes; useful for importer hints, but not canonical identity/memory truth
|
||||
|
||||
**Formats**
|
||||
- Mostly Markdown
|
||||
- Some JSON (`heartbeat-state.json`)
|
||||
- No stable OpenClaw-local DB or API surface is visible in this snapshot
|
||||
|
||||
**How pull should work**
|
||||
- Start with cron-based filesystem reads, not an OpenClaw HTTP API.
|
||||
- Read the durable files on a schedule, hash them, and import only deltas.
|
||||
- Map them by type:
|
||||
- `SOUL.md` / `USER.md` -> identity/preferences review candidates
|
||||
- `MEMORY.md` -> curated long-term memory candidates
|
||||
- `memory/YYYY-MM-DD.md` -> interaction/episodic import stream
|
||||
- `heartbeat-state.json` -> low-priority ops metadata only if useful
|
||||
|
||||
**Discord**
|
||||
- I do not see a documented durable Discord message store in the OpenClaw workspace snapshot.
|
||||
- `AGENTS.md` references Discord behavior, but not a canonical local log/database.
|
||||
- Treat Discord as transient unless OpenClaw exposes an explicit export/log file later.
|
||||
|
||||
**Biggest risk**
|
||||
- Importing raw OpenClaw files as truth will blur curated memory and noisy session chatter.
|
||||
- Mitigation: importer should classify by source tier, preserve provenance, and default to candidate/episodic ingestion rather than active memory promotion.
|
||||
|
||||
**Recommendation**
|
||||
- Do not build two-way sync.
|
||||
- Do not require OpenClaw to change architecture.
|
||||
- Build one importer against the file continuity layer first.
|
||||
- Add a formal export surface later only if the importer becomes too heuristic.
|
||||
@@ -3,6 +3,7 @@
|
||||
Minimal OpenClaw plugin that mirrors Claude Code's `capture_stop.py` behavior:
|
||||
|
||||
- watches user-triggered assistant turns
|
||||
- uses OpenClaw's dispatch-stage message body (`before_dispatch.body`) for the human prompt, then pairs it with the actual outbound assistant message on `message_sending`
|
||||
- POSTs `prompt` + `response` to `POST /interactions`
|
||||
- sets `client="openclaw"`
|
||||
- sets `reinforce=true`
|
||||
@@ -25,5 +26,7 @@ If `baseUrl` is omitted, the plugin uses `ATOCORE_BASE_URL` or defaults to `http
|
||||
## Notes
|
||||
|
||||
- Project detection is intentionally left empty for now. Unscoped capture is acceptable because AtoCore's extraction pipeline handles unscoped interactions.
|
||||
- Prompt cleaning is done inside the plugin by reading OpenClaw's dispatch-stage message body instead of the raw prompt-build input.
|
||||
- Turn pairing is done by caching the prompt on dispatch and posting only when OpenClaw emits the outbound assistant message, which is more reliable than pairing against raw model output events.
|
||||
- Extraction is **not** part of the capture path. This plugin only records interactions and lets AtoCore reinforcement run automatically.
|
||||
- The plugin captures only user-triggered turns, not heartbeats or system-only runs.
|
||||
|
||||
@@ -20,6 +20,30 @@ function shouldCapturePrompt(prompt, minLength) {
|
||||
return text.length >= minLength;
|
||||
}
|
||||
|
||||
function buildKeys(...values) {
|
||||
return [...new Set(values.map((v) => trimText(v)).filter(Boolean))];
|
||||
}
|
||||
|
||||
function rememberPending(store, keys, payload) {
|
||||
for (const key of keys) store.set(key, payload);
|
||||
}
|
||||
|
||||
function takePending(store, keys) {
|
||||
for (const key of keys) {
|
||||
const value = store.get(key);
|
||||
if (value) {
|
||||
for (const k of keys) store.delete(k);
|
||||
store.delete(key);
|
||||
return value;
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function clearPending(store, keys) {
|
||||
for (const key of keys) store.delete(key);
|
||||
}
|
||||
|
||||
async function postInteraction(baseUrl, payload, logger) {
|
||||
try {
|
||||
const res = await fetch(`${baseUrl.replace(/\/$/, "")}/interactions`, {
|
||||
@@ -46,30 +70,34 @@ export default definePluginEntry({
|
||||
const logger = api.logger;
|
||||
const pendingBySession = new Map();
|
||||
|
||||
api.on("before_agent_start", async (event, ctx) => {
|
||||
if (ctx?.trigger && ctx.trigger !== "user") return;
|
||||
api.on("before_dispatch", async (event, ctx) => {
|
||||
const config = api.getConfig?.() || {};
|
||||
const minPromptLength = Number(config.minPromptLength || DEFAULT_MIN_PROMPT_LENGTH);
|
||||
const prompt = trimText(event?.prompt || "");
|
||||
const prompt = trimText(event?.body || event?.content || "");
|
||||
const keys = buildKeys(ctx?.sessionKey, ctx?.sessionId, event?.sessionKey, event?.sessionId, ctx?.conversationId, event?.conversationId);
|
||||
if (!keys.length) return;
|
||||
if (!shouldCapturePrompt(prompt, minPromptLength)) {
|
||||
pendingBySession.delete(ctx.sessionId);
|
||||
clearPending(pendingBySession, keys);
|
||||
return;
|
||||
}
|
||||
pendingBySession.set(ctx.sessionId, {
|
||||
rememberPending(pendingBySession, keys, {
|
||||
prompt,
|
||||
sessionId: ctx.sessionId,
|
||||
sessionKey: ctx.sessionKey || "",
|
||||
sessionId: trimText(ctx?.sessionId || event?.sessionId || ""),
|
||||
sessionKey: trimText(ctx?.sessionKey || event?.sessionKey || ""),
|
||||
conversationId: trimText(ctx?.conversationId || event?.conversationId || ""),
|
||||
project: ""
|
||||
});
|
||||
});
|
||||
|
||||
api.on("llm_output", async (event, ctx) => {
|
||||
if (ctx?.trigger && ctx.trigger !== "user") return;
|
||||
const pending = pendingBySession.get(ctx.sessionId);
|
||||
api.on("message_sending", async (event, ctx) => {
|
||||
const keys = buildKeys(ctx?.sessionKey, ctx?.sessionId, ctx?.conversationId);
|
||||
const pending = takePending(pendingBySession, keys);
|
||||
if (!pending) return;
|
||||
|
||||
const assistantTexts = Array.isArray(event?.assistantTexts) ? event.assistantTexts : [];
|
||||
const response = truncateResponse(trimText(assistantTexts.join("\n\n")), Number((api.getConfig?.() || {}).maxResponseLength || DEFAULT_MAX_RESPONSE_LENGTH));
|
||||
const response = truncateResponse(
|
||||
trimText(event?.content || ""),
|
||||
Number((api.getConfig?.() || {}).maxResponseLength || DEFAULT_MAX_RESPONSE_LENGTH)
|
||||
);
|
||||
if (!response) return;
|
||||
|
||||
const config = api.getConfig?.() || {};
|
||||
@@ -78,17 +106,20 @@ export default definePluginEntry({
|
||||
prompt: pending.prompt,
|
||||
response,
|
||||
client: "openclaw",
|
||||
session_id: pending.sessionKey || pending.sessionId,
|
||||
session_id: pending.sessionKey || pending.sessionId || pending.conversationId,
|
||||
project: pending.project || "",
|
||||
reinforce: true
|
||||
};
|
||||
|
||||
await postInteraction(baseUrl, payload, logger);
|
||||
pendingBySession.delete(ctx.sessionId);
|
||||
});
|
||||
|
||||
api.on("agent_end", async (event) => {
|
||||
clearPending(pendingBySession, buildKeys(event?.sessionKey, event?.sessionId));
|
||||
});
|
||||
|
||||
api.on("session_end", async (event) => {
|
||||
if (event?.sessionId) pendingBySession.delete(event.sessionId);
|
||||
clearPending(pendingBySession, buildKeys(event?.sessionKey, event?.sessionId));
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
@@ -16,7 +16,6 @@ dependencies = [
|
||||
"pydantic>=2.6.0",
|
||||
"pydantic-settings>=2.1.0",
|
||||
"structlog>=24.1.0",
|
||||
"markdown>=3.5.0",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
|
||||
@@ -6,4 +6,3 @@ sentence-transformers>=2.5.0
|
||||
pydantic>=2.6.0
|
||||
pydantic-settings>=2.1.0
|
||||
structlog>=24.1.0
|
||||
markdown>=3.5.0
|
||||
|
||||
@@ -47,7 +47,7 @@ You will receive:
|
||||
|
||||
For each candidate, output exactly one JSON object:
|
||||
|
||||
{"verdict": "promote|reject|needs_human|contradicts", "confidence": 0.0-1.0, "reason": "one sentence", "conflicts_with": "id of existing memory if contradicts"}
|
||||
{"verdict": "promote|reject|needs_human", "confidence": 0.0-1.0, "reason": "one sentence"}
|
||||
|
||||
Rules:
|
||||
|
||||
@@ -61,13 +61,9 @@ Rules:
|
||||
- A session observation or conversational filler
|
||||
- A process rule that belongs in DEV-LEDGER.md or AGENTS.md, not memory
|
||||
|
||||
3. CONTRADICTS when the candidate *conflicts* with an existing active memory (not a duplicate, but states something that can't both be true). Set `conflicts_with` to the existing memory id. This flags the tension for human review instead of silently rejecting or double-storing. Examples: "Option A selected" vs "Option B selected" for the same decision; "uses material X" vs "uses material Y" for the same component.
|
||||
3. NEEDS_HUMAN when you're genuinely unsure — the candidate might be valuable but you can't tell without domain knowledge. This should be rare (< 20% of candidates).
|
||||
|
||||
4. OPENCLAW-CURATED content (candidate content starts with "From OpenClaw/"): apply a MUCH LOWER bar. OpenClaw's SOUL.md, USER.md, MEMORY.md, MODEL-ROUTING.md, and dated memory/*.md files are ALREADY curated by OpenClaw as canonical continuity. Promote unless clearly wrong or a genuine duplicate. Do NOT reject OpenClaw content as "process rule belongs elsewhere" or "session log" — that's exactly what AtoCore wants to absorb. Session events, project updates, stakeholder notes, and decisions from OpenClaw daily memory files ARE valuable context and should promote.
|
||||
|
||||
5. NEEDS_HUMAN when you're genuinely unsure — the candidate might be valuable but you can't tell without domain knowledge. This should be rare (< 20% of candidates).
|
||||
|
||||
6. Output ONLY the JSON object. No prose, no markdown, no explanation outside the reason field."""
|
||||
4. Output ONLY the JSON object. No prose, no markdown, no explanation outside the reason field."""
|
||||
|
||||
_sandbox_cwd = None
|
||||
|
||||
@@ -173,7 +169,7 @@ def parse_verdict(raw):
|
||||
return {"verdict": "needs_human", "confidence": 0.0, "reason": "failed to parse triage output"}
|
||||
|
||||
verdict = str(parsed.get("verdict", "needs_human")).strip().lower()
|
||||
if verdict not in {"promote", "reject", "needs_human", "contradicts"}:
|
||||
if verdict not in {"promote", "reject", "needs_human"}:
|
||||
verdict = "needs_human"
|
||||
|
||||
confidence = parsed.get("confidence", 0.5)
|
||||
@@ -183,13 +179,7 @@ def parse_verdict(raw):
|
||||
confidence = 0.5
|
||||
|
||||
reason = str(parsed.get("reason", "")).strip()[:200]
|
||||
conflicts_with = str(parsed.get("conflicts_with", "")).strip()
|
||||
return {
|
||||
"verdict": verdict,
|
||||
"confidence": confidence,
|
||||
"reason": reason,
|
||||
"conflicts_with": conflicts_with,
|
||||
}
|
||||
return {"verdict": verdict, "confidence": confidence, "reason": reason}
|
||||
|
||||
|
||||
def main():
|
||||
@@ -221,7 +211,6 @@ def main():
|
||||
verdict = verdict_obj["verdict"]
|
||||
conf = verdict_obj["confidence"]
|
||||
reason = verdict_obj["reason"]
|
||||
conflicts_with = verdict_obj.get("conflicts_with", "")
|
||||
|
||||
mid = cand["id"]
|
||||
label = f"[{i:2d}/{len(candidates)}] {mid[:8]} [{cand['memory_type']}]"
|
||||
@@ -247,13 +236,6 @@ def main():
|
||||
except Exception:
|
||||
errors += 1
|
||||
rejected += 1
|
||||
elif verdict == "contradicts":
|
||||
# Leave candidate in queue but flag the conflict in content
|
||||
# so the wiki/triage shows it. This is conservative: we
|
||||
# don't silently merge or reject when sources disagree.
|
||||
print(f" CONTRADICTS {label} vs {conflicts_with[:8] if conflicts_with else '?'} {reason}")
|
||||
contradicts_count = locals().get('contradicts_count', 0) + 1
|
||||
needs_human += 1
|
||||
else:
|
||||
print(f" NEEDS_HUMAN {label} conf={conf:.2f} {reason}")
|
||||
needs_human += 1
|
||||
|
||||
@@ -1,15 +1,12 @@
|
||||
"""Host-side LLM batch extraction — HTTP client + shared prompt module.
|
||||
"""Host-side LLM batch extraction — pure HTTP client, no atocore imports.
|
||||
|
||||
Fetches interactions from the AtoCore API, runs ``claude -p`` locally
|
||||
for each, and POSTs candidates back. Uses stdlib + the ``claude`` CLI
|
||||
on PATH, plus the stdlib-only shared prompt/parser module at
|
||||
``atocore.memory._llm_prompt`` to eliminate prompt/parser drift
|
||||
against the in-container extractor (R12).
|
||||
for each, and POSTs candidates back. Zero dependency on atocore source
|
||||
or Python packages — only uses stdlib + the ``claude`` CLI on PATH.
|
||||
|
||||
This is necessary because the ``claude`` CLI is on the Dalidou HOST
|
||||
but not inside the Docker container, and the host's Python doesn't
|
||||
have the container's dependencies (pydantic_settings, etc.) — so we
|
||||
only import the one stdlib-only module, not the full atocore package.
|
||||
have the container's dependencies (pydantic_settings, etc.).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -26,26 +23,34 @@ import urllib.parse
|
||||
import urllib.request
|
||||
from datetime import datetime, timezone
|
||||
|
||||
# R12: share the prompt + parser with the in-container extractor so
|
||||
# the two paths can't drift. The imported module is stdlib-only by
|
||||
# design; see src/atocore/memory/_llm_prompt.py.
|
||||
_SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
|
||||
_SRC_DIR = os.path.abspath(os.path.join(_SCRIPT_DIR, "..", "src"))
|
||||
if _SRC_DIR not in sys.path:
|
||||
sys.path.insert(0, _SRC_DIR)
|
||||
|
||||
from atocore.memory._llm_prompt import ( # noqa: E402
|
||||
MEMORY_TYPES,
|
||||
SYSTEM_PROMPT,
|
||||
build_user_message,
|
||||
normalize_candidate_item,
|
||||
parse_llm_json_array,
|
||||
)
|
||||
|
||||
DEFAULT_BASE_URL = os.environ.get("ATOCORE_BASE_URL", "http://localhost:8100")
|
||||
DEFAULT_MODEL = os.environ.get("ATOCORE_LLM_EXTRACTOR_MODEL", "sonnet")
|
||||
DEFAULT_TIMEOUT_S = float(os.environ.get("ATOCORE_LLM_EXTRACTOR_TIMEOUT_S", "90"))
|
||||
MAX_RESPONSE_CHARS = 8000
|
||||
MAX_PROMPT_CHARS = 2000
|
||||
|
||||
MEMORY_TYPES = {"identity", "preference", "project", "episodic", "knowledge", "adaptation"}
|
||||
|
||||
SYSTEM_PROMPT = """You extract durable memory candidates from LLM conversation turns for a personal context engine called AtoCore.
|
||||
|
||||
Your job is to read one user prompt plus the assistant's response and decide which durable facts, decisions, preferences, architectural rules, or project invariants should be remembered across future sessions.
|
||||
|
||||
Rules:
|
||||
|
||||
1. Only surface durable claims. Skip transient status ("deploy is still running"), instructional guidance ("here is how to run the command"), troubleshooting tactics, ephemeral recommendations ("merge this PR now"), and session recaps.
|
||||
2. A candidate is durable when a reader coming back in two weeks would still need to know it. Architectural choices, named rules, ratified decisions, invariants, procurement commitments, and project-level constraints qualify. Conversational fillers and step-by-step instructions do not.
|
||||
3. Each candidate must stand alone. Rewrite the claim in one sentence under 200 characters with enough context that a reader without the conversation understands it.
|
||||
4. Each candidate must have a type from this closed set: project, knowledge, preference, adaptation.
|
||||
5. If the conversation is clearly scoped to a project (p04-gigabit, p05-interferometer, p06-polisher, atocore), set ``project`` to that id. Otherwise leave ``project`` empty.
|
||||
6. If the response makes no durable claim, return an empty list. It is correct and expected to return [] on most conversational turns.
|
||||
7. Confidence should be 0.5 by default so human review workload is honest. Raise to 0.6 only when the response states the claim in an unambiguous, committed form (e.g. "the decision is X", "the selected approach is Y", "X is non-negotiable").
|
||||
8. Output must be a raw JSON array and nothing else. No prose before or after. No markdown fences. No explanations.
|
||||
|
||||
Each array element has exactly this shape:
|
||||
|
||||
{"type": "project|knowledge|preference|adaptation", "content": "...", "project": "...", "confidence": 0.5}
|
||||
|
||||
Return [] when there is nothing to extract."""
|
||||
|
||||
_sandbox_cwd = None
|
||||
|
||||
@@ -116,7 +121,14 @@ def extract_one(prompt, response, project, model, timeout_s):
|
||||
if not shutil.which("claude"):
|
||||
return [], "claude_cli_missing"
|
||||
|
||||
user_message = build_user_message(prompt, response, project)
|
||||
prompt_excerpt = prompt[:MAX_PROMPT_CHARS]
|
||||
response_excerpt = response[:MAX_RESPONSE_CHARS]
|
||||
user_message = (
|
||||
f"PROJECT HINT (may be empty): {project}\n\n"
|
||||
f"USER PROMPT:\n{prompt_excerpt}\n\n"
|
||||
f"ASSISTANT RESPONSE:\n{response_excerpt}\n\n"
|
||||
"Return the JSON array now."
|
||||
)
|
||||
|
||||
args = [
|
||||
"claude", "-p",
|
||||
@@ -145,25 +157,61 @@ def extract_one(prompt, response, project, model, timeout_s):
|
||||
|
||||
|
||||
def parse_candidates(raw, interaction_project):
|
||||
"""Parse model JSON output into candidate dicts.
|
||||
"""Parse model JSON output into candidate dicts."""
|
||||
text = raw.strip()
|
||||
if text.startswith("```"):
|
||||
text = text.strip("`")
|
||||
nl = text.find("\n")
|
||||
if nl >= 0:
|
||||
text = text[nl + 1:]
|
||||
if text.endswith("```"):
|
||||
text = text[:-3]
|
||||
text = text.strip()
|
||||
|
||||
if not text or text == "[]":
|
||||
return []
|
||||
|
||||
if not text.lstrip().startswith("["):
|
||||
start = text.find("[")
|
||||
end = text.rfind("]")
|
||||
if start >= 0 and end > start:
|
||||
text = text[start:end + 1]
|
||||
|
||||
try:
|
||||
parsed = json.loads(text)
|
||||
except json.JSONDecodeError:
|
||||
return []
|
||||
|
||||
if not isinstance(parsed, list):
|
||||
return []
|
||||
|
||||
Stripping + per-item normalization come from the shared
|
||||
``_llm_prompt`` module. Host-side project attribution: interaction
|
||||
scope wins, otherwise keep the model's tag (the API's own R9
|
||||
registry-check will happen server-side in the container on write;
|
||||
here we preserve the signal instead of dropping it).
|
||||
"""
|
||||
results = []
|
||||
for item in parse_llm_json_array(raw):
|
||||
normalized = normalize_candidate_item(item)
|
||||
if normalized is None:
|
||||
for item in parsed:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
project = interaction_project or normalized["project"] or ""
|
||||
mem_type = str(item.get("type") or "").strip().lower()
|
||||
content = str(item.get("content") or "").strip()
|
||||
model_project = str(item.get("project") or "").strip()
|
||||
# R9 trust hierarchy: interaction scope always wins when set.
|
||||
# Model project only used for unscoped interactions + registered check.
|
||||
if interaction_project:
|
||||
project = interaction_project
|
||||
elif model_project and model_project in _known_projects:
|
||||
project = model_project
|
||||
else:
|
||||
project = ""
|
||||
conf = item.get("confidence", 0.5)
|
||||
if mem_type not in MEMORY_TYPES or not content:
|
||||
continue
|
||||
try:
|
||||
conf = max(0.0, min(1.0, float(conf)))
|
||||
except (TypeError, ValueError):
|
||||
conf = 0.5
|
||||
results.append({
|
||||
"memory_type": normalized["type"],
|
||||
"content": normalized["content"],
|
||||
"memory_type": mem_type,
|
||||
"content": content[:1000],
|
||||
"project": project,
|
||||
"confidence": normalized["confidence"],
|
||||
"confidence": conf,
|
||||
})
|
||||
return results
|
||||
|
||||
|
||||
@@ -1,188 +0,0 @@
|
||||
"""Bootstrap engineering entities from existing project knowledge.
|
||||
|
||||
One-shot script that seeds the entity/relationship graph from what
|
||||
AtoCore already knows via memories, project state, and vault docs.
|
||||
Safe to re-run — uses name+project dedup.
|
||||
|
||||
Usage:
|
||||
|
||||
python3 scripts/bootstrap_entities.py --base-url http://localhost:8100
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import urllib.request
|
||||
|
||||
DEFAULT_BASE_URL = os.environ.get("ATOCORE_BASE_URL", "http://dalidou:8100")
|
||||
|
||||
|
||||
def post(base_url, path, body):
|
||||
data = json.dumps(body).encode("utf-8")
|
||||
req = urllib.request.Request(
|
||||
f"{base_url}{path}", method="POST",
|
||||
headers={"Content-Type": "application/json"}, data=data,
|
||||
)
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=10) as resp:
|
||||
return json.loads(resp.read().decode("utf-8"))
|
||||
except Exception as e:
|
||||
return {"error": str(e)}
|
||||
|
||||
|
||||
def entity(base_url, etype, name, project="", desc="", props=None):
|
||||
result = post(base_url, "/entities", {
|
||||
"entity_type": etype, "name": name, "project": project,
|
||||
"description": desc, "properties": props or {},
|
||||
})
|
||||
eid = result.get("id", "")
|
||||
status = "+" if eid else "skip"
|
||||
print(f" {status} [{etype}] {name}")
|
||||
return eid
|
||||
|
||||
|
||||
def rel(base_url, src, tgt, rtype):
|
||||
if not src or not tgt:
|
||||
return
|
||||
result = post(base_url, "/relationships", {
|
||||
"source_entity_id": src, "target_entity_id": tgt,
|
||||
"relationship_type": rtype,
|
||||
})
|
||||
print(f" -> {rtype}")
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--base-url", default=DEFAULT_BASE_URL)
|
||||
args = parser.parse_args()
|
||||
b = args.base_url
|
||||
|
||||
print("=== P04 GigaBIT M1 ===")
|
||||
p04 = entity(b, "project", "GigaBIT M1", "p04-gigabit",
|
||||
"1.2m primary mirror for stratospheric balloon telescope")
|
||||
|
||||
p04_m1 = entity(b, "system", "M1 Mirror Assembly", "p04-gigabit",
|
||||
"Primary mirror blank + support system + reference frame")
|
||||
rel(b, p04, p04_m1, "contains")
|
||||
|
||||
p04_vs = entity(b, "subsystem", "Vertical Support", "p04-gigabit",
|
||||
"18-point whiffletree axial support from below")
|
||||
p04_ls = entity(b, "subsystem", "Lateral Support", "p04-gigabit",
|
||||
"Circumferential constraint system with GF-PTFE pads")
|
||||
p04_rf = entity(b, "subsystem", "Reference Frame", "p04-gigabit",
|
||||
"Structural mounting interface between mirror and OTA")
|
||||
p04_blank = entity(b, "component", "M1 Blank", "p04-gigabit",
|
||||
"1.2m Zerodur aspheric blank from Schott",
|
||||
{"material": "Zerodur", "diameter_m": 1.2, "focal_ratio": "F/1.2"})
|
||||
rel(b, p04_m1, p04_vs, "contains")
|
||||
rel(b, p04_m1, p04_ls, "contains")
|
||||
rel(b, p04_m1, p04_rf, "contains")
|
||||
rel(b, p04_m1, p04_blank, "contains")
|
||||
|
||||
p04_zerodur = entity(b, "material", "Zerodur", "p04-gigabit",
|
||||
"Glass-ceramic with near-zero CTE for mirror blanks")
|
||||
p04_ptfe = entity(b, "material", "GF-PTFE", "p04-gigabit",
|
||||
"Glass-filled PTFE for thermal stability on lateral pads")
|
||||
rel(b, p04_blank, p04_zerodur, "uses_material")
|
||||
rel(b, p04_ls, p04_ptfe, "uses_material")
|
||||
|
||||
p04_optb = entity(b, "decision", "Option B Conical Back", "p04-gigabit",
|
||||
"Selected mirror architecture: conical-back lightweighting")
|
||||
rel(b, p04_optb, p04_blank, "affected_by_decision")
|
||||
|
||||
p04_wfe = entity(b, "requirement", "WFE < 15nm RMS filtered", "p04-gigabit",
|
||||
"Filtered mechanical wavefront error below 15 nm across 20-60 deg elevation")
|
||||
p04_mass = entity(b, "requirement", "Mass < 103.5 kg", "p04-gigabit",
|
||||
"Total mirror assembly mass constraint")
|
||||
rel(b, p04_m1, p04_wfe, "constrained_by")
|
||||
rel(b, p04_m1, p04_mass, "constrained_by")
|
||||
|
||||
print("\n=== P05 Interferometer ===")
|
||||
p05 = entity(b, "project", "Interferometer System", "p05-interferometer",
|
||||
"Metrology system for GigaBIT M1 figuring")
|
||||
|
||||
p05_rig = entity(b, "system", "Test Rig", "p05-interferometer",
|
||||
"Folded-beam interferometric test setup for M1 measurement")
|
||||
rel(b, p05, p05_rig, "contains")
|
||||
|
||||
p05_ifm = entity(b, "component", "Interferometer", "p05-interferometer",
|
||||
"Fixed horizontal Twyman-Green dynamic interferometer")
|
||||
p05_fold = entity(b, "component", "Fold Mirror", "p05-interferometer",
|
||||
"45-degree beam redirect, <= lambda/20 surface quality")
|
||||
p05_cgh = entity(b, "component", "CGH Null Corrector", "p05-interferometer",
|
||||
"6-inch transmission CGH for F/1.2 asphere null test",
|
||||
{"diameter": "6 inch", "substrate": "fused silica", "error_budget_nm": 5.5})
|
||||
p05_tilt = entity(b, "subsystem", "Tilting Platform", "p05-interferometer",
|
||||
"Mirror tilting platform, co-tilts with interferometer")
|
||||
rel(b, p05_rig, p05_ifm, "contains")
|
||||
rel(b, p05_rig, p05_fold, "contains")
|
||||
rel(b, p05_rig, p05_cgh, "contains")
|
||||
rel(b, p05_rig, p05_tilt, "contains")
|
||||
rel(b, p05_ifm, p05_fold, "interfaces_with")
|
||||
rel(b, p05_cgh, p05_tilt, "interfaces_with")
|
||||
|
||||
p05_vendor_dec = entity(b, "decision", "Vendor Path: Twyman-Green preferred", "p05-interferometer",
|
||||
"4D technical lead but cost-challenged; Zygo Verifire SV at 55K is value path")
|
||||
p05_vendor_zygo = entity(b, "vendor", "Zygo / AMETEK", "p05-interferometer",
|
||||
"Certified used Verifire SV, 55K, Nabeel Sufi contact")
|
||||
p05_vendor_4d = entity(b, "vendor", "4D Technology", "p05-interferometer",
|
||||
"PC6110/PC4030, above budget but strongest technical option")
|
||||
p05_vendor_aom = entity(b, "vendor", "AOM (CGH)", "p05-interferometer",
|
||||
"CGH design and fabrication, 28-30K package")
|
||||
rel(b, p05_vendor_dec, p05_ifm, "affected_by_decision")
|
||||
|
||||
print("\n=== P06 Polisher ===")
|
||||
p06 = entity(b, "project", "Polisher System", "p06-polisher",
|
||||
"Machine overhaul + software suite for optical polishing")
|
||||
|
||||
p06_machine = entity(b, "system", "Polisher Machine", "p06-polisher",
|
||||
"Swing-arm polishing machine with force modulation")
|
||||
p06_sw = entity(b, "system", "Software Suite", "p06-polisher",
|
||||
"Three-layer software: polisher-sim, polisher-post, polisher-control")
|
||||
rel(b, p06, p06_machine, "contains")
|
||||
rel(b, p06, p06_sw, "contains")
|
||||
|
||||
p06_sim = entity(b, "subsystem", "polisher-sim", "p06-polisher",
|
||||
"Digital twin: surface assimilation, removal simulation, planning")
|
||||
p06_post = entity(b, "subsystem", "polisher-post", "p06-polisher",
|
||||
"Bridge: validation, translation, packaging for machine")
|
||||
p06_ctrl = entity(b, "subsystem", "polisher-control", "p06-polisher",
|
||||
"Executor: state machine, interlocks, telemetry, run logs")
|
||||
rel(b, p06_sw, p06_sim, "contains")
|
||||
rel(b, p06_sw, p06_post, "contains")
|
||||
rel(b, p06_sw, p06_ctrl, "contains")
|
||||
rel(b, p06_sim, p06_post, "interfaces_with")
|
||||
rel(b, p06_post, p06_ctrl, "interfaces_with")
|
||||
|
||||
p06_fc = entity(b, "subsystem", "Force Control", "p06-polisher",
|
||||
"Frame-grounded counterweight actuator with cable tension modulation",
|
||||
{"actuator_capacity_N": "150-200", "compliance_spring_Nmm": "3-5"})
|
||||
p06_zaxis = entity(b, "component", "Z-Axis", "p06-polisher",
|
||||
"Binary engage/retract mechanism, not continuous position")
|
||||
p06_cam = entity(b, "component", "Cam Mechanism", "p06-polisher",
|
||||
"Mechanically set by operator, read by encoders, not actuated")
|
||||
rel(b, p06_machine, p06_fc, "contains")
|
||||
rel(b, p06_machine, p06_zaxis, "contains")
|
||||
rel(b, p06_machine, p06_cam, "contains")
|
||||
|
||||
p06_fw = entity(b, "decision", "Firmware Interface Contract", "p06-polisher",
|
||||
"controller-job.v1 in, run-log.v1 + telemetry out — invariant")
|
||||
p06_offline = entity(b, "decision", "Offline-First Design", "p06-polisher",
|
||||
"Machine works fully offline; network is for remote access only")
|
||||
p06_usb = entity(b, "decision", "USB SSD Storage", "p06-polisher",
|
||||
"USB SSD mandatory on RPi, not SD card")
|
||||
|
||||
p06_contracts = entity(b, "constraint", "Shared Contracts", "p06-polisher",
|
||||
"Stable IDs, explicit versions, hashable artifacts, planned-vs-executed separation")
|
||||
rel(b, p06_sw, p06_contracts, "constrained_by")
|
||||
|
||||
p06_preston = entity(b, "parameter", "Preston Coefficient kp", "p06-polisher",
|
||||
"Calibrated from before/after surface measurements, multi-run inverse-variance weighting")
|
||||
|
||||
print(f"\nDone.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,254 +0,0 @@
|
||||
"""OpenClaw state importer — one-way pull from clawdbot into AtoCore.
|
||||
|
||||
Reads OpenClaw's file continuity layer (SOUL.md, USER.md, MODEL-ROUTING.md,
|
||||
MEMORY.md, memory/YYYY-MM-DD.md) from the T420 via SSH and imports them
|
||||
into AtoCore as candidate memories. Hash-based delta detection — only
|
||||
re-imports files that changed since the last run.
|
||||
|
||||
Classification per codex's integration proposal:
|
||||
- SOUL.md -> identity candidates
|
||||
- USER.md -> identity + preference candidates
|
||||
- MODEL-ROUTING.md -> adaptation candidates (routing rules)
|
||||
- MEMORY.md -> long-term memory candidates (type varies)
|
||||
- memory/YYYY-MM-DD.md -> episodic memory candidates (daily logs)
|
||||
- heartbeat-state.json -> skipped (ops metadata only)
|
||||
|
||||
All candidates land as status=candidate. Auto-triage filters noise.
|
||||
This importer is conservative: it doesn't promote directly, it just
|
||||
feeds signal. The triage pipeline decides what graduates to active.
|
||||
|
||||
Usage:
|
||||
python3 scripts/import_openclaw_state.py \
|
||||
--base-url http://localhost:8100 \
|
||||
--openclaw-host papa@192.168.86.39 \
|
||||
--openclaw-path /home/papa/openclaw-workspace
|
||||
|
||||
Runs nightly via cron (added as Step 2c in cron-backup.sh).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
|
||||
DEFAULT_BASE_URL = os.environ.get("ATOCORE_BASE_URL", "http://localhost:8100")
|
||||
DEFAULT_OPENCLAW_HOST = os.environ.get("ATOCORE_OPENCLAW_HOST", "papa@192.168.86.39")
|
||||
DEFAULT_OPENCLAW_PATH = os.environ.get("ATOCORE_OPENCLAW_PATH", "/home/papa/clawd")
|
||||
|
||||
# Files to pull and how to classify them
|
||||
DURABLE_FILES = [
|
||||
("SOUL.md", "identity"),
|
||||
("USER.md", "identity"),
|
||||
("MODEL-ROUTING.md", "adaptation"),
|
||||
("MEMORY.md", "memory"), # type parsed from entries
|
||||
]
|
||||
DAILY_MEMORY_GLOB = "memory/*.md"
|
||||
HASH_STATE_KEY = "openclaw_import_hashes"
|
||||
|
||||
|
||||
def api_get(base_url, path):
|
||||
try:
|
||||
with urllib.request.urlopen(f"{base_url}{path}", timeout=15) as r:
|
||||
return json.loads(r.read())
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def api_post(base_url, path, body):
|
||||
data = json.dumps(body).encode("utf-8")
|
||||
req = urllib.request.Request(
|
||||
f"{base_url}{path}", method="POST",
|
||||
headers={"Content-Type": "application/json"}, data=data,
|
||||
)
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=15) as r:
|
||||
return json.loads(r.read())
|
||||
except urllib.error.HTTPError as exc:
|
||||
if exc.code == 400:
|
||||
return {"skipped": True}
|
||||
raise
|
||||
|
||||
|
||||
def ssh_cat(host, remote_path):
|
||||
"""Cat a remote file via SSH. Returns content or None if missing."""
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["ssh", "-o", "ConnectTimeout=5", "-o", "BatchMode=yes",
|
||||
host, f"cat {remote_path}"],
|
||||
capture_output=True, text=True, timeout=30,
|
||||
encoding="utf-8", errors="replace",
|
||||
)
|
||||
if result.returncode == 0:
|
||||
return result.stdout
|
||||
except Exception:
|
||||
pass
|
||||
return None
|
||||
|
||||
|
||||
def ssh_ls(host, remote_glob):
|
||||
"""List files matching a glob on the remote host."""
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["ssh", "-o", "ConnectTimeout=5", "-o", "BatchMode=yes",
|
||||
host, f"ls -1 {remote_glob} 2>/dev/null"],
|
||||
capture_output=True, text=True, timeout=10,
|
||||
encoding="utf-8", errors="replace",
|
||||
)
|
||||
if result.returncode == 0:
|
||||
return [line.strip() for line in result.stdout.splitlines() if line.strip()]
|
||||
except Exception:
|
||||
pass
|
||||
return []
|
||||
|
||||
|
||||
def content_hash(text):
|
||||
return hashlib.sha256(text.encode("utf-8")).hexdigest()[:16]
|
||||
|
||||
|
||||
def load_hash_state(base_url):
|
||||
"""Load the hash state from project_state so we know what's changed."""
|
||||
state = api_get(base_url, "/project/state/atocore?category=status")
|
||||
if not state:
|
||||
return {}
|
||||
for entry in state.get("entries", []):
|
||||
if entry.get("key") == HASH_STATE_KEY:
|
||||
try:
|
||||
return json.loads(entry["value"])
|
||||
except Exception:
|
||||
return {}
|
||||
return {}
|
||||
|
||||
|
||||
def save_hash_state(base_url, hashes):
|
||||
api_post(base_url, "/project/state", {
|
||||
"project": "atocore",
|
||||
"category": "status",
|
||||
"key": HASH_STATE_KEY,
|
||||
"value": json.dumps(hashes),
|
||||
"source": "import_openclaw_state.py",
|
||||
})
|
||||
|
||||
|
||||
def import_file_as_memory(base_url, filename, content, memory_type, source_tag):
|
||||
"""Import a file's content as a single candidate memory for triage."""
|
||||
# Trim to reasonable size — auto-triage can handle long content but
|
||||
# we don't want single mega-memories dominating the queue
|
||||
trimmed = content[:2000]
|
||||
if len(content) > 2000:
|
||||
trimmed += f"\n\n[...truncated from {len(content)} chars]"
|
||||
|
||||
body = {
|
||||
"memory_type": memory_type,
|
||||
"content": f"From OpenClaw/{filename}: {trimmed}",
|
||||
"project": "", # global/identity, not project-scoped
|
||||
"confidence": 0.5,
|
||||
"status": "candidate",
|
||||
}
|
||||
return api_post(base_url, "/memory", body)
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--base-url", default=DEFAULT_BASE_URL)
|
||||
parser.add_argument("--openclaw-host", default=DEFAULT_OPENCLAW_HOST)
|
||||
parser.add_argument("--openclaw-path", default=DEFAULT_OPENCLAW_PATH)
|
||||
parser.add_argument("--dry-run", action="store_true")
|
||||
args = parser.parse_args()
|
||||
|
||||
print(f"openclaw_host={args.openclaw_host} openclaw_path={args.openclaw_path}")
|
||||
print(f"dry_run={args.dry_run}")
|
||||
|
||||
# Check SSH connectivity first
|
||||
test = ssh_cat(args.openclaw_host, f"{args.openclaw_path}/SOUL.md")
|
||||
if test is None:
|
||||
print("ERROR: cannot reach OpenClaw workspace via SSH or SOUL.md not found")
|
||||
print("Check: ssh key installed? path correct? workspace exists?")
|
||||
return 1
|
||||
|
||||
hashes = load_hash_state(args.base_url)
|
||||
imported = skipped = errors = 0
|
||||
|
||||
# 1. Durable files
|
||||
for filename, mem_type in DURABLE_FILES:
|
||||
remote = f"{args.openclaw_path}/{filename}"
|
||||
content = ssh_cat(args.openclaw_host, remote)
|
||||
if content is None or not content.strip():
|
||||
print(f" - {filename}: not found or empty")
|
||||
continue
|
||||
|
||||
h = content_hash(content)
|
||||
if hashes.get(filename) == h:
|
||||
print(f" = {filename}: unchanged (hash {h})")
|
||||
skipped += 1
|
||||
continue
|
||||
|
||||
print(f" + {filename}: changed (hash {h}, {len(content)}ch)")
|
||||
if not args.dry_run:
|
||||
try:
|
||||
result = import_file_as_memory(
|
||||
args.base_url, filename, content, mem_type,
|
||||
source_tag="openclaw-durable",
|
||||
)
|
||||
if result.get("skipped"):
|
||||
print(f" (duplicate content, skipped)")
|
||||
else:
|
||||
print(f" -> candidate {result.get('id', '?')[:8]}")
|
||||
imported += 1
|
||||
hashes[filename] = h
|
||||
except Exception as e:
|
||||
print(f" ! error: {e}")
|
||||
errors += 1
|
||||
|
||||
# 2. Daily memory logs (memory/YYYY-MM-DD.md)
|
||||
daily_glob = f"{args.openclaw_path}/{DAILY_MEMORY_GLOB}"
|
||||
daily_files = ssh_ls(args.openclaw_host, daily_glob)
|
||||
print(f"\ndaily memory files: {len(daily_files)}")
|
||||
|
||||
# Only process the most recent 7 daily files to avoid flooding
|
||||
for remote_path in sorted(daily_files)[-7:]:
|
||||
filename = Path(remote_path).name
|
||||
content = ssh_cat(args.openclaw_host, remote_path)
|
||||
if content is None or not content.strip():
|
||||
continue
|
||||
|
||||
h = content_hash(content)
|
||||
key = f"daily/{filename}"
|
||||
if hashes.get(key) == h:
|
||||
print(f" = {filename}: unchanged")
|
||||
skipped += 1
|
||||
continue
|
||||
|
||||
print(f" + {filename}: changed ({len(content)}ch)")
|
||||
if not args.dry_run:
|
||||
try:
|
||||
result = import_file_as_memory(
|
||||
args.base_url, filename, content, "episodic",
|
||||
source_tag="openclaw-daily",
|
||||
)
|
||||
if not result.get("skipped"):
|
||||
print(f" -> candidate {result.get('id', '?')[:8]}")
|
||||
imported += 1
|
||||
hashes[key] = h
|
||||
except Exception as e:
|
||||
print(f" ! error: {e}")
|
||||
errors += 1
|
||||
|
||||
# Save hash state
|
||||
if not args.dry_run and imported > 0:
|
||||
save_hash_state(args.base_url, hashes)
|
||||
|
||||
print(f"\nimported={imported} skipped={skipped} errors={errors}")
|
||||
print("Candidates queued — auto-triage will filter them on next run.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main() or 0)
|
||||
@@ -1,170 +0,0 @@
|
||||
"""Weekly lint pass — health check for the AtoCore knowledge base.
|
||||
|
||||
Inspired by Karpathy's LLM Wiki pattern (the 'lint' operation).
|
||||
Checks for orphans, stale claims, contradictions, and gaps.
|
||||
Outputs a report that can be posted to the wiki as needs_review.
|
||||
|
||||
Usage:
|
||||
python3 scripts/lint_knowledge_base.py --base-url http://dalidou:8100
|
||||
|
||||
Run weekly via cron, or on-demand when the knowledge base feels stale.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import urllib.request
|
||||
from datetime import datetime, timezone, timedelta
|
||||
|
||||
DEFAULT_BASE_URL = os.environ.get("ATOCORE_BASE_URL", "http://localhost:8100")
|
||||
ORPHAN_AGE_DAYS = 14
|
||||
|
||||
|
||||
def api_get(base_url: str, path: str):
|
||||
with urllib.request.urlopen(f"{base_url}{path}", timeout=15) as r:
|
||||
return json.loads(r.read())
|
||||
|
||||
|
||||
def parse_ts(ts: str) -> datetime | None:
|
||||
if not ts:
|
||||
return None
|
||||
try:
|
||||
return datetime.strptime(ts[:19], "%Y-%m-%d %H:%M:%S").replace(tzinfo=timezone.utc)
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--base-url", default=DEFAULT_BASE_URL)
|
||||
args = parser.parse_args()
|
||||
b = args.base_url
|
||||
now = datetime.now(timezone.utc)
|
||||
orphan_threshold = now - timedelta(days=ORPHAN_AGE_DAYS)
|
||||
|
||||
print(f"=== AtoCore Lint — {now.strftime('%Y-%m-%d %H:%M UTC')} ===\n")
|
||||
|
||||
findings = {
|
||||
"orphan_memories": [],
|
||||
"stale_candidates": [],
|
||||
"unused_entities": [],
|
||||
"empty_state_projects": [],
|
||||
"unregistered_projects": [],
|
||||
}
|
||||
|
||||
# 1. Orphan memories: active but never reinforced after N days
|
||||
memories = api_get(b, "/memory?active_only=true&limit=500").get("memories", [])
|
||||
for m in memories:
|
||||
updated = parse_ts(m.get("updated_at", ""))
|
||||
if m.get("reference_count", 0) == 0 and updated and updated < orphan_threshold:
|
||||
findings["orphan_memories"].append({
|
||||
"id": m["id"],
|
||||
"type": m["memory_type"],
|
||||
"project": m.get("project") or "(none)",
|
||||
"age_days": (now - updated).days,
|
||||
"content": m["content"][:120],
|
||||
})
|
||||
|
||||
# 2. Stale candidates: been in queue > 7 days without triage
|
||||
candidates = api_get(b, "/memory?status=candidate&limit=500").get("memories", [])
|
||||
stale_threshold = now - timedelta(days=7)
|
||||
for c in candidates:
|
||||
updated = parse_ts(c.get("updated_at", ""))
|
||||
if updated and updated < stale_threshold:
|
||||
findings["stale_candidates"].append({
|
||||
"id": c["id"],
|
||||
"age_days": (now - updated).days,
|
||||
"content": c["content"][:120],
|
||||
})
|
||||
|
||||
# 3. Unused entities: no relationships in either direction
|
||||
entities = api_get(b, "/entities?limit=500").get("entities", [])
|
||||
for e in entities:
|
||||
try:
|
||||
detail = api_get(b, f"/entities/{e['id']}")
|
||||
if not detail.get("relationships"):
|
||||
findings["unused_entities"].append({
|
||||
"id": e["id"],
|
||||
"type": e["entity_type"],
|
||||
"name": e["name"],
|
||||
"project": e.get("project") or "(none)",
|
||||
})
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# 4. Registered projects with no state entries
|
||||
try:
|
||||
projects = api_get(b, "/projects").get("projects", [])
|
||||
for p in projects:
|
||||
state = api_get(b, f"/project/state/{p['id']}").get("entries", [])
|
||||
if not state:
|
||||
findings["empty_state_projects"].append(p["id"])
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# 5. Memories tagged to unregistered projects (auto-detection candidates)
|
||||
registered_ids = {p["id"] for p in projects} | {
|
||||
a for p in projects for a in p.get("aliases", [])
|
||||
}
|
||||
all_mems = api_get(b, "/memory?limit=500").get("memories", [])
|
||||
for m in all_mems:
|
||||
proj = m.get("project", "")
|
||||
if proj and proj not in registered_ids and proj != "(none)":
|
||||
if proj not in findings["unregistered_projects"]:
|
||||
findings["unregistered_projects"].append(proj)
|
||||
|
||||
# Print report
|
||||
print(f"## Orphan memories (active, no reinforcement, >{ORPHAN_AGE_DAYS} days old)")
|
||||
if findings["orphan_memories"]:
|
||||
print(f" Found: {len(findings['orphan_memories'])}")
|
||||
for o in findings["orphan_memories"][:10]:
|
||||
print(f" - [{o['type']}] {o['project']} ({o['age_days']}d): {o['content']}")
|
||||
else:
|
||||
print(" (none)")
|
||||
|
||||
print(f"\n## Stale candidates (>7 days in queue)")
|
||||
if findings["stale_candidates"]:
|
||||
print(f" Found: {len(findings['stale_candidates'])}")
|
||||
for s in findings["stale_candidates"][:10]:
|
||||
print(f" - ({s['age_days']}d): {s['content']}")
|
||||
else:
|
||||
print(" (none)")
|
||||
|
||||
print(f"\n## Unused entities (no relationships)")
|
||||
if findings["unused_entities"]:
|
||||
print(f" Found: {len(findings['unused_entities'])}")
|
||||
for u in findings["unused_entities"][:10]:
|
||||
print(f" - [{u['type']}] {u['project']}: {u['name']}")
|
||||
else:
|
||||
print(" (none)")
|
||||
|
||||
print(f"\n## Empty-state projects")
|
||||
if findings["empty_state_projects"]:
|
||||
print(f" Found: {len(findings['empty_state_projects'])}")
|
||||
for p in findings["empty_state_projects"]:
|
||||
print(f" - {p}")
|
||||
else:
|
||||
print(" (none)")
|
||||
|
||||
print(f"\n## Unregistered projects detected in memories")
|
||||
if findings["unregistered_projects"]:
|
||||
print(f" Found: {len(findings['unregistered_projects'])}")
|
||||
print(" These were auto-detected by extraction — consider registering them:")
|
||||
for p in findings["unregistered_projects"]:
|
||||
print(f" - {p}")
|
||||
else:
|
||||
print(" (none)")
|
||||
|
||||
total_findings = sum(
|
||||
len(v) if isinstance(v, list) else 0 for v in findings.values()
|
||||
)
|
||||
print(f"\n=== Total findings: {total_findings} ===")
|
||||
|
||||
# Return exit code based on findings count (for CI)
|
||||
return 0 if total_findings == 0 else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -218,8 +218,8 @@
|
||||
"Tailscale"
|
||||
],
|
||||
"expect_absent": [
|
||||
"[Source: p04-gigabit/"
|
||||
"GigaBIT"
|
||||
],
|
||||
"notes": "New p06 memory: Tailscale mesh for RPi remote access. Cross-project guard is a source-path check, not a word blacklist: the polisher ARCHITECTURE.md legitimately mentions the GigaBIT M1 mirror (it is what the polisher is built for), so testing for absence of that word produces false positives. The real invariant is that no p04 source chunks are retrieved into p06 context."
|
||||
"notes": "New p06 memory: Tailscale mesh for RPi remote access"
|
||||
}
|
||||
]
|
||||
|
||||
@@ -1,168 +0,0 @@
|
||||
"""Weekly project synthesis — LLM-generated 'current state' paragraph per project.
|
||||
|
||||
Reads each registered project's state entries, memories, and entities,
|
||||
asks sonnet for a 3-5 sentence synthesis, and caches it under
|
||||
project_state/status/synthesis_cache. The wiki's project page reads
|
||||
this cached synthesis as the top band.
|
||||
|
||||
Runs weekly via cron (or manually). Cheap — one LLM call per project.
|
||||
|
||||
Usage:
|
||||
python3 scripts/synthesize_projects.py --base-url http://localhost:8100
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import tempfile
|
||||
import urllib.request
|
||||
|
||||
DEFAULT_BASE_URL = os.environ.get("ATOCORE_BASE_URL", "http://localhost:8100")
|
||||
DEFAULT_MODEL = os.environ.get("ATOCORE_SYNTHESIS_MODEL", "sonnet")
|
||||
TIMEOUT_S = 60
|
||||
|
||||
SYSTEM_PROMPT = """You are summarizing the current state of an engineering project for a personal context engine called AtoCore.
|
||||
|
||||
You will receive:
|
||||
- Project state entries (decisions, requirements, status)
|
||||
- Active memories tagged to this project
|
||||
- Entity graph (subsystems, components, materials, decisions)
|
||||
|
||||
Write a 3-5 sentence synthesis covering:
|
||||
1. What the project is and its current stage
|
||||
2. The key locked-in decisions and architecture
|
||||
3. What the next focus is
|
||||
|
||||
Rules:
|
||||
- Plain prose, no bullet lists
|
||||
- Factual, grounded in what the data says — don't invent or speculate
|
||||
- Present tense
|
||||
- Under 500 characters total
|
||||
- No markdown formatting, just prose
|
||||
- If the data is sparse, say so honestly ("limited project data available")
|
||||
|
||||
Output ONLY the synthesis paragraph. No preamble, no JSON, no markdown headers."""
|
||||
|
||||
|
||||
_cwd = None
|
||||
|
||||
|
||||
def get_cwd():
|
||||
global _cwd
|
||||
if _cwd is None:
|
||||
_cwd = tempfile.mkdtemp(prefix="ato-synth-")
|
||||
return _cwd
|
||||
|
||||
|
||||
def api_get(base_url, path):
|
||||
with urllib.request.urlopen(f"{base_url}{path}", timeout=15) as r:
|
||||
return json.loads(r.read())
|
||||
|
||||
|
||||
def api_post(base_url, path, body):
|
||||
data = json.dumps(body).encode("utf-8")
|
||||
req = urllib.request.Request(
|
||||
f"{base_url}{path}", method="POST",
|
||||
headers={"Content-Type": "application/json"}, data=data,
|
||||
)
|
||||
with urllib.request.urlopen(req, timeout=15) as r:
|
||||
return json.loads(r.read())
|
||||
|
||||
|
||||
def synthesize_project(base_url, project_id, model):
|
||||
# Gather context
|
||||
state = api_get(base_url, f"/project/state/{project_id}").get("entries", [])
|
||||
memories = api_get(base_url, f"/memory?project={project_id}&active_only=true&limit=20").get("memories", [])
|
||||
entities = api_get(base_url, f"/entities?project={project_id}&limit=50").get("entities", [])
|
||||
|
||||
if not (state or memories or entities):
|
||||
return None
|
||||
|
||||
lines = [f"PROJECT: {project_id}\n"]
|
||||
if state:
|
||||
lines.append("STATE ENTRIES:")
|
||||
for e in state[:15]:
|
||||
if e.get("key") == "synthesis_cache":
|
||||
continue
|
||||
lines.append(f" [{e['category']}] {e['key']}: {e['value'][:200]}")
|
||||
|
||||
if memories:
|
||||
lines.append("\nACTIVE MEMORIES:")
|
||||
for m in memories[:10]:
|
||||
lines.append(f" [{m['memory_type']}] {m['content'][:200]}")
|
||||
|
||||
if entities:
|
||||
lines.append("\nENTITIES:")
|
||||
by_type = {}
|
||||
for e in entities:
|
||||
by_type.setdefault(e["entity_type"], []).append(e["name"])
|
||||
for t, names in by_type.items():
|
||||
lines.append(f" {t}: {', '.join(names[:8])}")
|
||||
|
||||
user_msg = "\n".join(lines) + "\n\nWrite the synthesis paragraph now."
|
||||
|
||||
if not shutil.which("claude"):
|
||||
print(f" ! claude CLI not available, skipping {project_id}")
|
||||
return None
|
||||
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["claude", "-p", "--model", model,
|
||||
"--append-system-prompt", SYSTEM_PROMPT,
|
||||
"--disable-slash-commands",
|
||||
user_msg],
|
||||
capture_output=True, text=True, timeout=TIMEOUT_S,
|
||||
cwd=get_cwd(), encoding="utf-8", errors="replace",
|
||||
)
|
||||
except Exception as e:
|
||||
print(f" ! subprocess failed for {project_id}: {e}")
|
||||
return None
|
||||
|
||||
if result.returncode != 0:
|
||||
print(f" ! claude exit {result.returncode} for {project_id}")
|
||||
return None
|
||||
|
||||
synthesis = (result.stdout or "").strip()
|
||||
if not synthesis or len(synthesis) < 50:
|
||||
return None
|
||||
return synthesis[:1000]
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--base-url", default=DEFAULT_BASE_URL)
|
||||
parser.add_argument("--model", default=DEFAULT_MODEL)
|
||||
parser.add_argument("--project", default=None, help="single project to synthesize")
|
||||
args = parser.parse_args()
|
||||
|
||||
projects = api_get(args.base_url, "/projects").get("projects", [])
|
||||
if args.project:
|
||||
projects = [p for p in projects if p["id"] == args.project]
|
||||
|
||||
print(f"Synthesizing {len(projects)} project(s) with {args.model}...")
|
||||
|
||||
for p in projects:
|
||||
pid = p["id"]
|
||||
print(f"\n- {pid}")
|
||||
synthesis = synthesize_project(args.base_url, pid, args.model)
|
||||
if synthesis:
|
||||
print(f" {synthesis[:200]}...")
|
||||
try:
|
||||
api_post(args.base_url, "/project/state", {
|
||||
"project": pid,
|
||||
"category": "status",
|
||||
"key": "synthesis_cache",
|
||||
"value": synthesis,
|
||||
"source": "weekly synthesis pass",
|
||||
})
|
||||
print(f" + cached")
|
||||
except Exception as e:
|
||||
print(f" ! save failed: {e}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -3,7 +3,6 @@
|
||||
from pathlib import Path
|
||||
|
||||
from fastapi import APIRouter, HTTPException
|
||||
from fastapi.responses import HTMLResponse
|
||||
from pydantic import BaseModel
|
||||
|
||||
import atocore.config as _config
|
||||
@@ -31,23 +30,6 @@ from atocore.interactions.service import (
|
||||
list_interactions,
|
||||
record_interaction,
|
||||
)
|
||||
from atocore.engineering.mirror import generate_project_overview
|
||||
from atocore.engineering.wiki import (
|
||||
render_entity,
|
||||
render_homepage,
|
||||
render_project,
|
||||
render_search,
|
||||
)
|
||||
from atocore.engineering.service import (
|
||||
ENTITY_TYPES,
|
||||
RELATIONSHIP_TYPES,
|
||||
create_entity,
|
||||
create_relationship,
|
||||
get_entities,
|
||||
get_entity,
|
||||
get_entity_with_context,
|
||||
get_relationships,
|
||||
)
|
||||
from atocore.memory.extractor import (
|
||||
EXTRACTOR_VERSION,
|
||||
MemoryCandidate,
|
||||
@@ -55,7 +37,6 @@ from atocore.memory.extractor import (
|
||||
)
|
||||
from atocore.memory.extractor_llm import (
|
||||
LLM_EXTRACTOR_VERSION,
|
||||
_cli_available as _llm_cli_available,
|
||||
extract_candidates_llm,
|
||||
)
|
||||
from atocore.memory.reinforcement import reinforce_from_interaction
|
||||
@@ -92,33 +73,6 @@ router = APIRouter()
|
||||
log = get_logger("api")
|
||||
|
||||
|
||||
# --- Wiki routes (HTML, served first for clean URLs) ---
|
||||
|
||||
|
||||
@router.get("/wiki", response_class=HTMLResponse)
|
||||
def wiki_home() -> HTMLResponse:
|
||||
return HTMLResponse(content=render_homepage())
|
||||
|
||||
|
||||
@router.get("/wiki/projects/{project_name}", response_class=HTMLResponse)
|
||||
def wiki_project(project_name: str) -> HTMLResponse:
|
||||
from atocore.projects.registry import resolve_project_name as _resolve
|
||||
return HTMLResponse(content=render_project(_resolve(project_name)))
|
||||
|
||||
|
||||
@router.get("/wiki/entities/{entity_id}", response_class=HTMLResponse)
|
||||
def wiki_entity(entity_id: str) -> HTMLResponse:
|
||||
html = render_entity(entity_id)
|
||||
if html is None:
|
||||
raise HTTPException(status_code=404, detail="Entity not found")
|
||||
return HTMLResponse(content=html)
|
||||
|
||||
|
||||
@router.get("/wiki/search", response_class=HTMLResponse)
|
||||
def wiki_search(q: str = "") -> HTMLResponse:
|
||||
return HTMLResponse(content=render_search(q))
|
||||
|
||||
|
||||
# --- Request/Response models ---
|
||||
|
||||
|
||||
@@ -833,18 +787,6 @@ def api_extract_batch(req: ExtractBatchRequest | None = None) -> dict:
|
||||
invoke this endpoint explicitly (cron, manual curl, CLI).
|
||||
"""
|
||||
payload = req or ExtractBatchRequest()
|
||||
|
||||
if payload.mode == "llm" and not _llm_cli_available():
|
||||
raise HTTPException(
|
||||
status_code=503,
|
||||
detail=(
|
||||
"LLM extraction unavailable in this runtime: the `claude` CLI "
|
||||
"is not on PATH. Run host-side via "
|
||||
"`scripts/batch_llm_extract_live.py` instead, or call this "
|
||||
"endpoint with mode=\"rule\"."
|
||||
),
|
||||
)
|
||||
|
||||
since = payload.since
|
||||
|
||||
if not since:
|
||||
@@ -984,229 +926,6 @@ def api_dashboard() -> dict:
|
||||
}
|
||||
|
||||
|
||||
# --- Engineering Knowledge Layer (Layer 2) ---
|
||||
|
||||
|
||||
class EntityCreateRequest(BaseModel):
|
||||
entity_type: str
|
||||
name: str
|
||||
project: str = ""
|
||||
description: str = ""
|
||||
properties: dict | None = None
|
||||
status: str = "active"
|
||||
confidence: float = 1.0
|
||||
source_refs: list[str] | None = None
|
||||
|
||||
|
||||
class RelationshipCreateRequest(BaseModel):
|
||||
source_entity_id: str
|
||||
target_entity_id: str
|
||||
relationship_type: str
|
||||
confidence: float = 1.0
|
||||
source_refs: list[str] | None = None
|
||||
|
||||
|
||||
@router.post("/entities")
|
||||
def api_create_entity(req: EntityCreateRequest) -> dict:
|
||||
"""Create a new engineering entity."""
|
||||
try:
|
||||
entity = create_entity(
|
||||
entity_type=req.entity_type,
|
||||
name=req.name,
|
||||
project=req.project,
|
||||
description=req.description,
|
||||
properties=req.properties,
|
||||
status=req.status,
|
||||
confidence=req.confidence,
|
||||
source_refs=req.source_refs,
|
||||
)
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
return {"status": "ok", "id": entity.id, "entity_type": entity.entity_type, "name": entity.name}
|
||||
|
||||
|
||||
@router.get("/entities")
|
||||
def api_list_entities(
|
||||
entity_type: str | None = None,
|
||||
project: str | None = None,
|
||||
status: str = "active",
|
||||
name_contains: str | None = None,
|
||||
limit: int = 100,
|
||||
) -> dict:
|
||||
"""List engineering entities with optional filters."""
|
||||
entities = get_entities(
|
||||
entity_type=entity_type,
|
||||
project=project,
|
||||
status=status,
|
||||
name_contains=name_contains,
|
||||
limit=limit,
|
||||
)
|
||||
return {
|
||||
"entities": [
|
||||
{
|
||||
"id": e.id,
|
||||
"entity_type": e.entity_type,
|
||||
"name": e.name,
|
||||
"project": e.project,
|
||||
"description": e.description,
|
||||
"properties": e.properties,
|
||||
"status": e.status,
|
||||
"confidence": e.confidence,
|
||||
}
|
||||
for e in entities
|
||||
],
|
||||
"count": len(entities),
|
||||
}
|
||||
|
||||
|
||||
@router.get("/entities/{entity_id}")
|
||||
def api_get_entity(entity_id: str) -> dict:
|
||||
"""Get an entity with its relationships and related entities."""
|
||||
result = get_entity_with_context(entity_id)
|
||||
if result is None:
|
||||
raise HTTPException(status_code=404, detail=f"Entity not found: {entity_id}")
|
||||
entity = result["entity"]
|
||||
return {
|
||||
"entity": {
|
||||
"id": entity.id,
|
||||
"entity_type": entity.entity_type,
|
||||
"name": entity.name,
|
||||
"project": entity.project,
|
||||
"description": entity.description,
|
||||
"properties": entity.properties,
|
||||
"status": entity.status,
|
||||
"confidence": entity.confidence,
|
||||
"source_refs": entity.source_refs,
|
||||
"created_at": entity.created_at,
|
||||
"updated_at": entity.updated_at,
|
||||
},
|
||||
"relationships": [
|
||||
{
|
||||
"id": r.id,
|
||||
"source_entity_id": r.source_entity_id,
|
||||
"target_entity_id": r.target_entity_id,
|
||||
"relationship_type": r.relationship_type,
|
||||
"confidence": r.confidence,
|
||||
}
|
||||
for r in result["relationships"]
|
||||
],
|
||||
"related_entities": {
|
||||
eid: {
|
||||
"entity_type": e.entity_type,
|
||||
"name": e.name,
|
||||
"project": e.project,
|
||||
"description": e.description[:200],
|
||||
}
|
||||
for eid, e in result["related_entities"].items()
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
@router.post("/relationships")
|
||||
def api_create_relationship(req: RelationshipCreateRequest) -> dict:
|
||||
"""Create a relationship between two entities."""
|
||||
try:
|
||||
rel = create_relationship(
|
||||
source_entity_id=req.source_entity_id,
|
||||
target_entity_id=req.target_entity_id,
|
||||
relationship_type=req.relationship_type,
|
||||
confidence=req.confidence,
|
||||
source_refs=req.source_refs,
|
||||
)
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
return {
|
||||
"status": "ok",
|
||||
"id": rel.id,
|
||||
"relationship_type": rel.relationship_type,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/projects/{project_name}/mirror.html", response_class=HTMLResponse)
|
||||
def api_project_mirror_html(project_name: str) -> HTMLResponse:
|
||||
"""Serve a readable HTML project overview page.
|
||||
|
||||
Open in a browser for a clean, styled project dashboard derived
|
||||
from AtoCore's structured data. Source of truth is the database —
|
||||
this page is a derived view.
|
||||
"""
|
||||
from atocore.projects.registry import resolve_project_name as _resolve
|
||||
|
||||
canonical = _resolve(project_name)
|
||||
try:
|
||||
md_content = generate_project_overview(canonical)
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=f"Mirror generation failed: {e}")
|
||||
|
||||
import markdown
|
||||
|
||||
html_body = markdown.markdown(md_content, extensions=["tables", "fenced_code"])
|
||||
html = _MIRROR_HTML_TEMPLATE.replace("{{title}}", f"{canonical} — AtoCore Mirror")
|
||||
html = html.replace("{{body}}", html_body)
|
||||
return HTMLResponse(content=html)
|
||||
|
||||
|
||||
_MIRROR_HTML_TEMPLATE = """<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>{{title}}</title>
|
||||
<style>
|
||||
:root { --bg: #fafafa; --text: #1a1a2e; --accent: #2563eb; --border: #e2e8f0; --card: #fff; }
|
||||
@media (prefers-color-scheme: dark) {
|
||||
:root { --bg: #0f172a; --text: #e2e8f0; --accent: #60a5fa; --border: #334155; --card: #1e293b; }
|
||||
}
|
||||
* { box-sizing: border-box; margin: 0; padding: 0; }
|
||||
body {
|
||||
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
|
||||
line-height: 1.7; color: var(--text); background: var(--bg);
|
||||
max-width: 800px; margin: 0 auto; padding: 2rem 1.5rem;
|
||||
}
|
||||
h1 { font-size: 1.8rem; margin-bottom: 0.5rem; color: var(--accent); }
|
||||
h2 { font-size: 1.4rem; margin-top: 2.5rem; margin-bottom: 0.8rem; padding-bottom: 0.3rem; border-bottom: 2px solid var(--border); }
|
||||
h3 { font-size: 1.15rem; margin-top: 1.5rem; margin-bottom: 0.5rem; }
|
||||
p { margin-bottom: 0.8rem; }
|
||||
ul { margin-left: 1.5rem; margin-bottom: 1rem; }
|
||||
li { margin-bottom: 0.4rem; }
|
||||
li ul { margin-top: 0.3rem; }
|
||||
strong { color: var(--accent); font-weight: 600; }
|
||||
em { opacity: 0.7; font-size: 0.9em; }
|
||||
blockquote {
|
||||
background: var(--card); border-left: 4px solid var(--accent);
|
||||
padding: 0.8rem 1.2rem; margin: 1rem 0; border-radius: 0 8px 8px 0;
|
||||
}
|
||||
hr { border: none; border-top: 1px solid var(--border); margin: 2rem 0; }
|
||||
code { background: var(--card); padding: 0.15rem 0.4rem; border-radius: 4px; font-size: 0.9em; }
|
||||
a { color: var(--accent); text-decoration: none; }
|
||||
a:hover { text-decoration: underline; }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
{{body}}
|
||||
</body>
|
||||
</html>"""
|
||||
|
||||
|
||||
@router.get("/projects/{project_name}/mirror")
|
||||
def api_project_mirror(project_name: str) -> dict:
|
||||
"""Generate a human-readable project overview from structured data.
|
||||
|
||||
Layer 3 of the AtoCore architecture. The mirror is DERIVED from
|
||||
entities, project state, and memories — it is not canonical truth.
|
||||
Returns markdown that can be rendered, saved to a file, or served
|
||||
as a dashboard page.
|
||||
"""
|
||||
from atocore.projects.registry import resolve_project_name as _resolve
|
||||
|
||||
canonical = _resolve(project_name)
|
||||
try:
|
||||
markdown = generate_project_overview(canonical)
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=f"Mirror generation failed: {e}")
|
||||
return {"project": canonical, "format": "markdown", "content": markdown}
|
||||
|
||||
|
||||
@router.get("/admin/backup/{stamp}/validate")
|
||||
def api_validate_backup(stamp: str) -> dict:
|
||||
"""Validate that a previously created backup is structurally usable."""
|
||||
|
||||
@@ -104,21 +104,6 @@ class Settings(BaseSettings):
|
||||
|
||||
@property
|
||||
def resolved_project_registry_path(self) -> Path:
|
||||
"""Path to the project registry JSON file.
|
||||
|
||||
If ``ATOCORE_PROJECT_REGISTRY_DIR`` env var is set, the registry
|
||||
lives at ``<that dir>/project-registry.json``. Otherwise falls
|
||||
back to the configured ``project_registry_path`` field.
|
||||
|
||||
This lets Docker deployments point at a mounted volume via env
|
||||
var without the ephemeral in-image ``/app/config/`` getting
|
||||
wiped on every rebuild.
|
||||
"""
|
||||
import os
|
||||
|
||||
registry_dir = os.environ.get("ATOCORE_PROJECT_REGISTRY_DIR", "").strip()
|
||||
if registry_dir:
|
||||
return Path(registry_dir) / "project-registry.json"
|
||||
return self._resolve_path(self.project_registry_path)
|
||||
|
||||
@property
|
||||
|
||||
@@ -14,7 +14,6 @@ import atocore.config as _config
|
||||
from atocore.context.project_state import format_project_state, get_state
|
||||
from atocore.memory.service import get_memories_for_context
|
||||
from atocore.observability.logger import get_logger
|
||||
from atocore.engineering.service import get_entities, get_entity_with_context
|
||||
from atocore.projects.registry import resolve_project_name
|
||||
from atocore.retrieval.retriever import ChunkResult, retrieve
|
||||
|
||||
@@ -37,13 +36,6 @@ MEMORY_BUDGET_RATIO = 0.05 # identity + preference; lowered from 0.10 to avoid
|
||||
# memory can actually reach the model.
|
||||
PROJECT_MEMORY_BUDGET_RATIO = 0.25
|
||||
PROJECT_MEMORY_TYPES = ["project", "knowledge", "episodic"]
|
||||
# General domain knowledge — unscoped memories (project="") that surface
|
||||
# in every context pack regardless of project hint. These are earned
|
||||
# engineering insights that apply across projects (e.g., "Preston removal
|
||||
# model breaks down below 5N because the contact assumption fails").
|
||||
DOMAIN_KNOWLEDGE_BUDGET_RATIO = 0.10
|
||||
DOMAIN_KNOWLEDGE_TYPES = ["knowledge"]
|
||||
ENGINEERING_CONTEXT_BUDGET_RATIO = 0.10
|
||||
|
||||
# Last built context pack for debug inspection
|
||||
_last_context_pack: "ContextPack | None" = None
|
||||
@@ -67,10 +59,6 @@ class ContextPack:
|
||||
memory_chars: int = 0
|
||||
project_memory_text: str = ""
|
||||
project_memory_chars: int = 0
|
||||
domain_knowledge_text: str = ""
|
||||
domain_knowledge_chars: int = 0
|
||||
engineering_context_text: str = ""
|
||||
engineering_context_chars: int = 0
|
||||
total_chars: int = 0
|
||||
budget: int = 0
|
||||
budget_remaining: int = 0
|
||||
@@ -151,46 +139,8 @@ def build_context(
|
||||
query=user_prompt,
|
||||
)
|
||||
|
||||
# 2c. Domain knowledge — cross-project earned insight with project=""
|
||||
# that surfaces regardless of which project the query is about.
|
||||
domain_knowledge_text = ""
|
||||
domain_knowledge_chars = 0
|
||||
domain_budget = min(
|
||||
int(budget * DOMAIN_KNOWLEDGE_BUDGET_RATIO),
|
||||
max(budget - project_state_chars - memory_chars - project_memory_chars, 0),
|
||||
)
|
||||
if domain_budget > 0:
|
||||
domain_knowledge_text, domain_knowledge_chars = get_memories_for_context(
|
||||
memory_types=DOMAIN_KNOWLEDGE_TYPES,
|
||||
project="",
|
||||
budget=domain_budget,
|
||||
header="--- Domain Knowledge ---",
|
||||
footer="--- End Domain Knowledge ---",
|
||||
query=user_prompt,
|
||||
)
|
||||
|
||||
# 2d. Engineering context — structured entity/relationship data
|
||||
# when the query matches a known entity name.
|
||||
engineering_context_text = ""
|
||||
engineering_context_chars = 0
|
||||
if canonical_project:
|
||||
eng_budget = min(
|
||||
int(budget * ENGINEERING_CONTEXT_BUDGET_RATIO),
|
||||
max(budget - project_state_chars - memory_chars
|
||||
- project_memory_chars - domain_knowledge_chars, 0),
|
||||
)
|
||||
if eng_budget > 0:
|
||||
engineering_context_text = _build_engineering_context(
|
||||
user_prompt, canonical_project, eng_budget,
|
||||
)
|
||||
engineering_context_chars = len(engineering_context_text)
|
||||
|
||||
# 3. Calculate remaining budget for retrieval
|
||||
retrieval_budget = (
|
||||
budget - project_state_chars - memory_chars
|
||||
- project_memory_chars - domain_knowledge_chars
|
||||
- engineering_context_chars
|
||||
)
|
||||
retrieval_budget = budget - project_state_chars - memory_chars - project_memory_chars
|
||||
|
||||
# 4. Retrieve candidates
|
||||
candidates = (
|
||||
@@ -211,16 +161,13 @@ def build_context(
|
||||
|
||||
# 7. Format full context
|
||||
formatted = _format_full_context(
|
||||
project_state_text, memory_text, project_memory_text,
|
||||
domain_knowledge_text, engineering_context_text, selected,
|
||||
project_state_text, memory_text, project_memory_text, selected
|
||||
)
|
||||
if len(formatted) > budget:
|
||||
formatted, selected = _trim_context_to_budget(
|
||||
project_state_text,
|
||||
memory_text,
|
||||
project_memory_text,
|
||||
domain_knowledge_text,
|
||||
engineering_context_text,
|
||||
selected,
|
||||
budget,
|
||||
)
|
||||
@@ -231,8 +178,6 @@ def build_context(
|
||||
project_state_chars = len(project_state_text)
|
||||
memory_chars = len(memory_text)
|
||||
project_memory_chars = len(project_memory_text)
|
||||
domain_knowledge_chars = len(domain_knowledge_text)
|
||||
engineering_context_chars = len(engineering_context_text)
|
||||
retrieval_chars = sum(c.char_count for c in selected)
|
||||
total_chars = len(formatted)
|
||||
duration_ms = int((time.time() - start) * 1000)
|
||||
@@ -245,10 +190,6 @@ def build_context(
|
||||
memory_chars=memory_chars,
|
||||
project_memory_text=project_memory_text,
|
||||
project_memory_chars=project_memory_chars,
|
||||
domain_knowledge_text=domain_knowledge_text,
|
||||
domain_knowledge_chars=domain_knowledge_chars,
|
||||
engineering_context_text=engineering_context_text,
|
||||
engineering_context_chars=engineering_context_chars,
|
||||
total_chars=total_chars,
|
||||
budget=budget,
|
||||
budget_remaining=budget - total_chars,
|
||||
@@ -267,8 +208,6 @@ def build_context(
|
||||
project_state_chars=project_state_chars,
|
||||
memory_chars=memory_chars,
|
||||
project_memory_chars=project_memory_chars,
|
||||
domain_knowledge_chars=domain_knowledge_chars,
|
||||
engineering_context_chars=engineering_context_chars,
|
||||
retrieval_chars=retrieval_chars,
|
||||
total_chars=total_chars,
|
||||
budget_remaining=budget - total_chars,
|
||||
@@ -349,9 +288,7 @@ def _format_full_context(
|
||||
project_state_text: str,
|
||||
memory_text: str,
|
||||
project_memory_text: str,
|
||||
domain_knowledge_text: str,
|
||||
engineering_context_text: str = "",
|
||||
chunks: list[ContextChunk] | None = None,
|
||||
chunks: list[ContextChunk],
|
||||
) -> str:
|
||||
"""Format project state + memories + retrieved chunks into full context block."""
|
||||
parts = []
|
||||
@@ -371,17 +308,7 @@ def _format_full_context(
|
||||
parts.append(project_memory_text)
|
||||
parts.append("")
|
||||
|
||||
# 4. Domain knowledge (cross-project earned insight)
|
||||
if domain_knowledge_text:
|
||||
parts.append(domain_knowledge_text)
|
||||
parts.append("")
|
||||
|
||||
# 5. Engineering context (structured entity/relationship data)
|
||||
if engineering_context_text:
|
||||
parts.append(engineering_context_text)
|
||||
parts.append("")
|
||||
|
||||
# 6. Retrieved chunks (lowest trust)
|
||||
# 4. Retrieved chunks (lowest trust)
|
||||
if chunks:
|
||||
parts.append("--- AtoCore Retrieved Context ---")
|
||||
if project_state_text:
|
||||
@@ -393,7 +320,7 @@ def _format_full_context(
|
||||
parts.append(chunk.content)
|
||||
parts.append("")
|
||||
parts.append("--- End Context ---")
|
||||
elif not project_state_text and not memory_text and not project_memory_text and not domain_knowledge_text and not engineering_context_text:
|
||||
elif not project_state_text and not memory_text and not project_memory_text:
|
||||
parts.append("--- AtoCore Context ---\nNo relevant context found.\n--- End Context ---")
|
||||
|
||||
return "\n".join(parts)
|
||||
@@ -416,7 +343,6 @@ def _pack_to_dict(pack: ContextPack) -> dict:
|
||||
"project_state_chars": pack.project_state_chars,
|
||||
"memory_chars": pack.memory_chars,
|
||||
"project_memory_chars": pack.project_memory_chars,
|
||||
"domain_knowledge_chars": pack.domain_knowledge_chars,
|
||||
"chunks_used": len(pack.chunks_used),
|
||||
"total_chars": pack.total_chars,
|
||||
"budget": pack.budget,
|
||||
@@ -425,8 +351,6 @@ def _pack_to_dict(pack: ContextPack) -> dict:
|
||||
"has_project_state": bool(pack.project_state_text),
|
||||
"has_memories": bool(pack.memory_text),
|
||||
"has_project_memories": bool(pack.project_memory_text),
|
||||
"has_domain_knowledge": bool(pack.domain_knowledge_text),
|
||||
"has_engineering_context": bool(pack.engineering_context_text),
|
||||
"chunks": [
|
||||
{
|
||||
"source_file": c.source_file,
|
||||
@@ -440,83 +364,6 @@ def _pack_to_dict(pack: ContextPack) -> dict:
|
||||
}
|
||||
|
||||
|
||||
def _build_engineering_context(
|
||||
query: str,
|
||||
project: str,
|
||||
budget: int,
|
||||
) -> str:
|
||||
"""Find entities matching the query and format their context.
|
||||
|
||||
Uses simple word-overlap matching between query tokens and entity
|
||||
names to find relevant entities, then formats the top match with
|
||||
its relationships as a compact text band.
|
||||
"""
|
||||
if budget < 100:
|
||||
return ""
|
||||
|
||||
from atocore.memory.reinforcement import _normalize, _tokenize
|
||||
|
||||
query_tokens = _tokenize(_normalize(query))
|
||||
if not query_tokens:
|
||||
return ""
|
||||
|
||||
try:
|
||||
entities = get_entities(project=project, limit=100)
|
||||
except Exception:
|
||||
return ""
|
||||
|
||||
if not entities:
|
||||
return ""
|
||||
|
||||
scored: list[tuple[int, "Entity"]] = []
|
||||
for ent in entities:
|
||||
name_tokens = _tokenize(_normalize(ent.name))
|
||||
desc_tokens = _tokenize(_normalize(ent.description))
|
||||
overlap = len(query_tokens & (name_tokens | desc_tokens))
|
||||
if overlap > 0:
|
||||
scored.append((overlap, ent))
|
||||
|
||||
if not scored:
|
||||
return ""
|
||||
|
||||
scored.sort(key=lambda t: t[0], reverse=True)
|
||||
best_entity = scored[0][1]
|
||||
|
||||
try:
|
||||
ctx = get_entity_with_context(best_entity.id)
|
||||
except Exception:
|
||||
return ""
|
||||
|
||||
if ctx is None:
|
||||
return ""
|
||||
|
||||
lines = ["--- Engineering Context ---"]
|
||||
lines.append(f"[{best_entity.entity_type}] {best_entity.name}")
|
||||
if best_entity.description:
|
||||
lines.append(f" {best_entity.description[:150]}")
|
||||
|
||||
for rel in ctx["relationships"][:8]:
|
||||
other_id = (
|
||||
rel.target_entity_id
|
||||
if rel.source_entity_id == best_entity.id
|
||||
else rel.source_entity_id
|
||||
)
|
||||
other = ctx["related_entities"].get(other_id)
|
||||
if other:
|
||||
direction = "->" if rel.source_entity_id == best_entity.id else "<-"
|
||||
lines.append(
|
||||
f" {direction} {rel.relationship_type} [{other.entity_type}] {other.name}"
|
||||
)
|
||||
|
||||
lines.append("--- End Engineering Context ---")
|
||||
text = "\n".join(lines)
|
||||
|
||||
if len(text) > budget:
|
||||
text = text[:budget - 3].rstrip() + "..."
|
||||
|
||||
return text
|
||||
|
||||
|
||||
def _truncate_text_block(text: str, budget: int) -> tuple[str, int]:
|
||||
"""Trim a formatted text block so trusted tiers cannot exceed the total budget."""
|
||||
if budget <= 0 or not text:
|
||||
@@ -534,66 +381,44 @@ def _trim_context_to_budget(
|
||||
project_state_text: str,
|
||||
memory_text: str,
|
||||
project_memory_text: str,
|
||||
domain_knowledge_text: str,
|
||||
engineering_context_text: str,
|
||||
chunks: list[ContextChunk],
|
||||
budget: int,
|
||||
) -> tuple[str, list[ContextChunk]]:
|
||||
"""Trim retrieval -> engineering -> domain -> project memories -> identity -> state."""
|
||||
"""Trim retrieval → project memories → identity/preference → project state."""
|
||||
kept_chunks = list(chunks)
|
||||
formatted = _format_full_context(
|
||||
project_state_text, memory_text, project_memory_text,
|
||||
domain_knowledge_text, engineering_context_text, kept_chunks,
|
||||
project_state_text, memory_text, project_memory_text, kept_chunks
|
||||
)
|
||||
while len(formatted) > budget and kept_chunks:
|
||||
kept_chunks.pop()
|
||||
formatted = _format_full_context(
|
||||
project_state_text, memory_text, project_memory_text,
|
||||
domain_knowledge_text, engineering_context_text, kept_chunks,
|
||||
project_state_text, memory_text, project_memory_text, kept_chunks
|
||||
)
|
||||
|
||||
if len(formatted) <= budget:
|
||||
return formatted, kept_chunks
|
||||
|
||||
# Drop engineering context first.
|
||||
engineering_context_text = ""
|
||||
formatted = _format_full_context(
|
||||
project_state_text, memory_text, project_memory_text,
|
||||
domain_knowledge_text, engineering_context_text, kept_chunks,
|
||||
)
|
||||
if len(formatted) <= budget:
|
||||
return formatted, kept_chunks
|
||||
|
||||
# Drop domain knowledge next.
|
||||
domain_knowledge_text, _ = _truncate_text_block(domain_knowledge_text, 0)
|
||||
formatted = _format_full_context(
|
||||
project_state_text, memory_text, project_memory_text,
|
||||
domain_knowledge_text, engineering_context_text, kept_chunks,
|
||||
)
|
||||
if len(formatted) <= budget:
|
||||
return formatted, kept_chunks
|
||||
|
||||
# Drop project memories next (they were the most recently added
|
||||
# tier and carry less trust than identity/preference).
|
||||
project_memory_text, _ = _truncate_text_block(
|
||||
project_memory_text,
|
||||
max(budget - len(project_state_text) - len(memory_text), 0),
|
||||
)
|
||||
formatted = _format_full_context(
|
||||
project_state_text, memory_text, project_memory_text,
|
||||
domain_knowledge_text, engineering_context_text, kept_chunks,
|
||||
project_state_text, memory_text, project_memory_text, kept_chunks
|
||||
)
|
||||
if len(formatted) <= budget:
|
||||
return formatted, kept_chunks
|
||||
|
||||
memory_text, _ = _truncate_text_block(memory_text, max(budget - len(project_state_text), 0))
|
||||
formatted = _format_full_context(
|
||||
project_state_text, memory_text, project_memory_text,
|
||||
domain_knowledge_text, engineering_context_text, kept_chunks,
|
||||
project_state_text, memory_text, project_memory_text, kept_chunks
|
||||
)
|
||||
if len(formatted) <= budget:
|
||||
return formatted, kept_chunks
|
||||
|
||||
project_state_text, _ = _truncate_text_block(project_state_text, budget)
|
||||
formatted = _format_full_context(project_state_text, "", "", "", [])
|
||||
formatted = _format_full_context(project_state_text, "", "", [])
|
||||
if len(formatted) > budget:
|
||||
formatted, _ = _truncate_text_block(formatted, budget)
|
||||
return formatted, []
|
||||
|
||||
@@ -1,16 +0,0 @@
|
||||
"""Engineering Knowledge Layer — typed entities and relationships.
|
||||
|
||||
Layer 2 of the AtoCore architecture. Sits on top of the core machine
|
||||
layer (memories, project state, retrieval) and adds structured
|
||||
engineering objects with typed relationships so queries like "what
|
||||
requirements does this component satisfy" can be answered directly
|
||||
instead of relying on flat text search.
|
||||
|
||||
V1 entity types (from docs/architecture/engineering-ontology-v1.md):
|
||||
Component, Subsystem, Requirement, Constraint, Decision, Material,
|
||||
Parameter, Interface
|
||||
|
||||
V1 relationship types:
|
||||
CONTAINS, PART_OF, INTERFACES_WITH, SATISFIES, CONSTRAINED_BY,
|
||||
AFFECTED_BY_DECISION, ANALYZED_BY, VALIDATED_BY, DEPENDS_ON
|
||||
"""
|
||||
@@ -1,267 +0,0 @@
|
||||
"""Human Mirror — derived readable project views from structured data.
|
||||
|
||||
Layer 3 of the AtoCore architecture. Generates human-readable markdown
|
||||
pages from the engineering entity graph, Trusted Project State, and
|
||||
active memories. These pages are DERIVED — they are not canonical
|
||||
machine truth. They are support surfaces for human inspection and
|
||||
audit comfort.
|
||||
|
||||
The mirror never invents content. Every line traces back to an entity,
|
||||
a state entry, or a memory. If the structured data is wrong, the
|
||||
mirror is wrong — fix the source, not the page.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from atocore.context.project_state import get_state
|
||||
from atocore.engineering.service import (
|
||||
get_entities,
|
||||
get_relationships,
|
||||
)
|
||||
from atocore.memory.service import get_memories
|
||||
from atocore.observability.logger import get_logger
|
||||
|
||||
log = get_logger("mirror")
|
||||
|
||||
|
||||
def generate_project_overview(project: str) -> str:
|
||||
"""Generate a full project overview page in markdown."""
|
||||
sections = [
|
||||
_header(project),
|
||||
_synthesis_section(project),
|
||||
_state_section(project),
|
||||
_system_architecture(project),
|
||||
_decisions_section(project),
|
||||
_requirements_section(project),
|
||||
_materials_section(project),
|
||||
_vendors_section(project),
|
||||
_active_memories_section(project),
|
||||
_footer(project),
|
||||
]
|
||||
return "\n\n".join(s for s in sections if s)
|
||||
|
||||
|
||||
def _synthesis_section(project: str) -> str:
|
||||
"""Generate a short LLM synthesis of the current project state.
|
||||
|
||||
Reads the cached synthesis from project_state if available
|
||||
(category=status, key=synthesis_cache). If not cached, returns
|
||||
a deterministic summary from the existing structured data.
|
||||
The actual LLM-generated synthesis is produced by the weekly
|
||||
lint/synthesis pass on Dalidou (where claude CLI is available).
|
||||
"""
|
||||
entries = get_state(project)
|
||||
cached = ""
|
||||
for e in entries:
|
||||
if e.category == "status" and e.key == "synthesis_cache":
|
||||
cached = e.value
|
||||
break
|
||||
|
||||
if cached:
|
||||
return f"## Current State (auto-synthesis)\n\n> {cached}"
|
||||
|
||||
# Fallback: deterministic summary from structured data
|
||||
stage = ""
|
||||
summary = ""
|
||||
next_focus = ""
|
||||
for e in entries:
|
||||
if e.category == "status":
|
||||
if e.key == "stage":
|
||||
stage = e.value
|
||||
elif e.key == "summary":
|
||||
summary = e.value
|
||||
elif e.key == "next_focus":
|
||||
next_focus = e.value
|
||||
|
||||
if not (stage or summary or next_focus):
|
||||
return ""
|
||||
|
||||
bits = []
|
||||
if summary:
|
||||
bits.append(summary)
|
||||
if stage:
|
||||
bits.append(f"**Stage**: {stage}")
|
||||
if next_focus:
|
||||
bits.append(f"**Next**: {next_focus}")
|
||||
|
||||
return "## Current State\n\n" + "\n\n".join(bits)
|
||||
|
||||
|
||||
def _header(project: str) -> str:
|
||||
return (
|
||||
f"# {project} — Project Overview\n\n"
|
||||
f"> This page is auto-generated from AtoCore structured data.\n"
|
||||
f"> It is a **derived view**, not canonical truth. "
|
||||
f"If something is wrong here, fix the source data."
|
||||
)
|
||||
|
||||
|
||||
def _state_section(project: str) -> str:
|
||||
entries = get_state(project)
|
||||
if not entries:
|
||||
return ""
|
||||
|
||||
lines = ["## Trusted Project State"]
|
||||
by_category: dict[str, list] = {}
|
||||
for e in entries:
|
||||
by_category.setdefault(e.category.upper(), []).append(e)
|
||||
|
||||
for cat in ["DECISION", "REQUIREMENT", "STATUS", "FACT", "MILESTONE", "CONFIG", "CONTACT"]:
|
||||
items = by_category.get(cat, [])
|
||||
if not items:
|
||||
continue
|
||||
lines.append(f"\n### {cat.title()}")
|
||||
for item in items:
|
||||
value = item.value[:300]
|
||||
lines.append(f"- **{item.key}**: {value}")
|
||||
if item.source:
|
||||
lines.append(f" *(source: {item.source})*")
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def _system_architecture(project: str) -> str:
|
||||
systems = get_entities(entity_type="system", project=project)
|
||||
subsystems = get_entities(entity_type="subsystem", project=project)
|
||||
components = get_entities(entity_type="component", project=project)
|
||||
interfaces = get_entities(entity_type="interface", project=project)
|
||||
|
||||
if not systems and not subsystems and not components:
|
||||
return ""
|
||||
|
||||
lines = ["## System Architecture"]
|
||||
|
||||
for system in systems:
|
||||
lines.append(f"\n### {system.name}")
|
||||
if system.description:
|
||||
lines.append(f"{system.description}")
|
||||
|
||||
rels = get_relationships(system.id, direction="outgoing")
|
||||
children = []
|
||||
for rel in rels:
|
||||
if rel.relationship_type == "contains":
|
||||
child = next(
|
||||
(s for s in subsystems + components if s.id == rel.target_entity_id),
|
||||
None,
|
||||
)
|
||||
if child:
|
||||
children.append(child)
|
||||
|
||||
if children:
|
||||
lines.append("\n**Contains:**")
|
||||
for child in children:
|
||||
desc = f" — {child.description}" if child.description else ""
|
||||
lines.append(f"- [{child.entity_type}] **{child.name}**{desc}")
|
||||
|
||||
child_rels = get_relationships(child.id, direction="both")
|
||||
for cr in child_rels:
|
||||
if cr.relationship_type in ("uses_material", "interfaces_with", "constrained_by"):
|
||||
other_id = (
|
||||
cr.target_entity_id
|
||||
if cr.source_entity_id == child.id
|
||||
else cr.source_entity_id
|
||||
)
|
||||
other = next(
|
||||
(e for e in get_entities(project=project, limit=200)
|
||||
if e.id == other_id),
|
||||
None,
|
||||
)
|
||||
if other:
|
||||
lines.append(
|
||||
f" - *{cr.relationship_type}* → "
|
||||
f"[{other.entity_type}] {other.name}"
|
||||
)
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def _decisions_section(project: str) -> str:
|
||||
decisions = get_entities(entity_type="decision", project=project)
|
||||
if not decisions:
|
||||
return ""
|
||||
|
||||
lines = ["## Decisions"]
|
||||
for d in decisions:
|
||||
lines.append(f"\n### {d.name}")
|
||||
if d.description:
|
||||
lines.append(d.description)
|
||||
rels = get_relationships(d.id, direction="outgoing")
|
||||
for rel in rels:
|
||||
if rel.relationship_type == "affected_by_decision":
|
||||
affected = next(
|
||||
(e for e in get_entities(project=project, limit=200)
|
||||
if e.id == rel.target_entity_id),
|
||||
None,
|
||||
)
|
||||
if affected:
|
||||
lines.append(
|
||||
f"- Affects: [{affected.entity_type}] {affected.name}"
|
||||
)
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def _requirements_section(project: str) -> str:
|
||||
reqs = get_entities(entity_type="requirement", project=project)
|
||||
constraints = get_entities(entity_type="constraint", project=project)
|
||||
if not reqs and not constraints:
|
||||
return ""
|
||||
|
||||
lines = ["## Requirements & Constraints"]
|
||||
for r in reqs:
|
||||
lines.append(f"- **{r.name}**: {r.description}" if r.description else f"- **{r.name}**")
|
||||
for c in constraints:
|
||||
lines.append(f"- [constraint] **{c.name}**: {c.description}" if c.description else f"- [constraint] **{c.name}**")
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def _materials_section(project: str) -> str:
|
||||
materials = get_entities(entity_type="material", project=project)
|
||||
if not materials:
|
||||
return ""
|
||||
|
||||
lines = ["## Materials"]
|
||||
for m in materials:
|
||||
desc = f" — {m.description}" if m.description else ""
|
||||
lines.append(f"- **{m.name}**{desc}")
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def _vendors_section(project: str) -> str:
|
||||
vendors = get_entities(entity_type="vendor", project=project)
|
||||
if not vendors:
|
||||
return ""
|
||||
|
||||
lines = ["## Vendors"]
|
||||
for v in vendors:
|
||||
desc = f" — {v.description}" if v.description else ""
|
||||
lines.append(f"- **{v.name}**{desc}")
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def _active_memories_section(project: str) -> str:
|
||||
memories = get_memories(project=project, active_only=True, limit=20)
|
||||
if not memories:
|
||||
return ""
|
||||
|
||||
lines = ["## Active Memories"]
|
||||
for m in memories:
|
||||
conf = f" (conf: {m.confidence:.2f})" if m.confidence < 1.0 else ""
|
||||
refs = f" | refs: {m.reference_count}" if m.reference_count > 0 else ""
|
||||
lines.append(f"- [{m.memory_type}]{conf}{refs} {m.content[:200]}")
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def _footer(project: str) -> str:
|
||||
from datetime import datetime, timezone
|
||||
|
||||
now = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M UTC")
|
||||
return (
|
||||
f"---\n\n"
|
||||
f"*Generated by AtoCore Human Mirror at {now}. "
|
||||
f"This is a derived view — not canonical truth.*"
|
||||
)
|
||||
@@ -1,317 +0,0 @@
|
||||
"""Engineering entity and relationship CRUD."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import uuid
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from atocore.models.database import get_connection
|
||||
from atocore.observability.logger import get_logger
|
||||
|
||||
log = get_logger("engineering")
|
||||
|
||||
ENTITY_TYPES = [
|
||||
"project",
|
||||
"system",
|
||||
"subsystem",
|
||||
"component",
|
||||
"interface",
|
||||
"requirement",
|
||||
"constraint",
|
||||
"decision",
|
||||
"material",
|
||||
"parameter",
|
||||
"analysis_model",
|
||||
"result",
|
||||
"validation_claim",
|
||||
"vendor",
|
||||
"process",
|
||||
]
|
||||
|
||||
RELATIONSHIP_TYPES = [
|
||||
"contains",
|
||||
"part_of",
|
||||
"interfaces_with",
|
||||
"satisfies",
|
||||
"constrained_by",
|
||||
"affected_by_decision",
|
||||
"analyzed_by",
|
||||
"validated_by",
|
||||
"depends_on",
|
||||
"uses_material",
|
||||
"described_by",
|
||||
"supersedes",
|
||||
]
|
||||
|
||||
ENTITY_STATUSES = ["candidate", "active", "superseded", "invalid"]
|
||||
|
||||
|
||||
@dataclass
|
||||
class Entity:
|
||||
id: str
|
||||
entity_type: str
|
||||
name: str
|
||||
project: str
|
||||
description: str = ""
|
||||
properties: dict = field(default_factory=dict)
|
||||
status: str = "active"
|
||||
confidence: float = 1.0
|
||||
source_refs: list[str] = field(default_factory=list)
|
||||
created_at: str = ""
|
||||
updated_at: str = ""
|
||||
|
||||
|
||||
@dataclass
|
||||
class Relationship:
|
||||
id: str
|
||||
source_entity_id: str
|
||||
target_entity_id: str
|
||||
relationship_type: str
|
||||
confidence: float = 1.0
|
||||
source_refs: list[str] = field(default_factory=list)
|
||||
created_at: str = ""
|
||||
|
||||
|
||||
def init_engineering_schema() -> None:
|
||||
with get_connection() as conn:
|
||||
conn.execute("""
|
||||
CREATE TABLE IF NOT EXISTS entities (
|
||||
id TEXT PRIMARY KEY,
|
||||
entity_type TEXT NOT NULL,
|
||||
name TEXT NOT NULL,
|
||||
project TEXT NOT NULL DEFAULT '',
|
||||
description TEXT NOT NULL DEFAULT '',
|
||||
properties TEXT NOT NULL DEFAULT '{}',
|
||||
status TEXT NOT NULL DEFAULT 'active',
|
||||
confidence REAL NOT NULL DEFAULT 1.0,
|
||||
source_refs TEXT NOT NULL DEFAULT '[]',
|
||||
created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
updated_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP
|
||||
)
|
||||
""")
|
||||
conn.execute("""
|
||||
CREATE TABLE IF NOT EXISTS relationships (
|
||||
id TEXT PRIMARY KEY,
|
||||
source_entity_id TEXT NOT NULL,
|
||||
target_entity_id TEXT NOT NULL,
|
||||
relationship_type TEXT NOT NULL,
|
||||
confidence REAL NOT NULL DEFAULT 1.0,
|
||||
source_refs TEXT NOT NULL DEFAULT '[]',
|
||||
created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
FOREIGN KEY (source_entity_id) REFERENCES entities(id),
|
||||
FOREIGN KEY (target_entity_id) REFERENCES entities(id)
|
||||
)
|
||||
""")
|
||||
conn.execute("""
|
||||
CREATE INDEX IF NOT EXISTS idx_entities_project
|
||||
ON entities(project)
|
||||
""")
|
||||
conn.execute("""
|
||||
CREATE INDEX IF NOT EXISTS idx_entities_type
|
||||
ON entities(entity_type)
|
||||
""")
|
||||
conn.execute("""
|
||||
CREATE INDEX IF NOT EXISTS idx_relationships_source
|
||||
ON relationships(source_entity_id)
|
||||
""")
|
||||
conn.execute("""
|
||||
CREATE INDEX IF NOT EXISTS idx_relationships_target
|
||||
ON relationships(target_entity_id)
|
||||
""")
|
||||
log.info("engineering_schema_initialized")
|
||||
|
||||
|
||||
def create_entity(
|
||||
entity_type: str,
|
||||
name: str,
|
||||
project: str = "",
|
||||
description: str = "",
|
||||
properties: dict | None = None,
|
||||
status: str = "active",
|
||||
confidence: float = 1.0,
|
||||
source_refs: list[str] | None = None,
|
||||
) -> Entity:
|
||||
if entity_type not in ENTITY_TYPES:
|
||||
raise ValueError(f"Invalid entity type: {entity_type}. Must be one of {ENTITY_TYPES}")
|
||||
if status not in ENTITY_STATUSES:
|
||||
raise ValueError(f"Invalid status: {status}. Must be one of {ENTITY_STATUSES}")
|
||||
if not name or not name.strip():
|
||||
raise ValueError("Entity name must be non-empty")
|
||||
|
||||
entity_id = str(uuid.uuid4())
|
||||
now = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S")
|
||||
props = properties or {}
|
||||
refs = source_refs or []
|
||||
|
||||
with get_connection() as conn:
|
||||
conn.execute(
|
||||
"""INSERT INTO entities
|
||||
(id, entity_type, name, project, description, properties,
|
||||
status, confidence, source_refs, created_at, updated_at)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
|
||||
(
|
||||
entity_id, entity_type, name.strip(), project,
|
||||
description, json.dumps(props), status, confidence,
|
||||
json.dumps(refs), now, now,
|
||||
),
|
||||
)
|
||||
|
||||
log.info("entity_created", entity_id=entity_id, entity_type=entity_type, name=name)
|
||||
return Entity(
|
||||
id=entity_id, entity_type=entity_type, name=name.strip(),
|
||||
project=project, description=description, properties=props,
|
||||
status=status, confidence=confidence, source_refs=refs,
|
||||
created_at=now, updated_at=now,
|
||||
)
|
||||
|
||||
|
||||
def create_relationship(
|
||||
source_entity_id: str,
|
||||
target_entity_id: str,
|
||||
relationship_type: str,
|
||||
confidence: float = 1.0,
|
||||
source_refs: list[str] | None = None,
|
||||
) -> Relationship:
|
||||
if relationship_type not in RELATIONSHIP_TYPES:
|
||||
raise ValueError(f"Invalid relationship type: {relationship_type}")
|
||||
|
||||
rel_id = str(uuid.uuid4())
|
||||
now = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S")
|
||||
refs = source_refs or []
|
||||
|
||||
with get_connection() as conn:
|
||||
conn.execute(
|
||||
"""INSERT INTO relationships
|
||||
(id, source_entity_id, target_entity_id, relationship_type,
|
||||
confidence, source_refs, created_at)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?)""",
|
||||
(rel_id, source_entity_id, target_entity_id,
|
||||
relationship_type, confidence, json.dumps(refs), now),
|
||||
)
|
||||
|
||||
log.info(
|
||||
"relationship_created",
|
||||
rel_id=rel_id,
|
||||
source=source_entity_id,
|
||||
target=target_entity_id,
|
||||
rel_type=relationship_type,
|
||||
)
|
||||
return Relationship(
|
||||
id=rel_id, source_entity_id=source_entity_id,
|
||||
target_entity_id=target_entity_id,
|
||||
relationship_type=relationship_type,
|
||||
confidence=confidence, source_refs=refs, created_at=now,
|
||||
)
|
||||
|
||||
|
||||
def get_entities(
|
||||
entity_type: str | None = None,
|
||||
project: str | None = None,
|
||||
status: str = "active",
|
||||
name_contains: str | None = None,
|
||||
limit: int = 100,
|
||||
) -> list[Entity]:
|
||||
query = "SELECT * FROM entities WHERE status = ?"
|
||||
params: list = [status]
|
||||
|
||||
if entity_type:
|
||||
query += " AND entity_type = ?"
|
||||
params.append(entity_type)
|
||||
if project is not None:
|
||||
query += " AND project = ?"
|
||||
params.append(project)
|
||||
if name_contains:
|
||||
query += " AND name LIKE ?"
|
||||
params.append(f"%{name_contains}%")
|
||||
|
||||
query += " ORDER BY entity_type, name LIMIT ?"
|
||||
params.append(min(limit, 500))
|
||||
|
||||
with get_connection() as conn:
|
||||
rows = conn.execute(query, params).fetchall()
|
||||
return [_row_to_entity(r) for r in rows]
|
||||
|
||||
|
||||
def get_entity(entity_id: str) -> Entity | None:
|
||||
with get_connection() as conn:
|
||||
row = conn.execute(
|
||||
"SELECT * FROM entities WHERE id = ?", (entity_id,)
|
||||
).fetchone()
|
||||
if row is None:
|
||||
return None
|
||||
return _row_to_entity(row)
|
||||
|
||||
|
||||
def get_relationships(
|
||||
entity_id: str,
|
||||
direction: str = "both",
|
||||
) -> list[Relationship]:
|
||||
results = []
|
||||
with get_connection() as conn:
|
||||
if direction in ("outgoing", "both"):
|
||||
rows = conn.execute(
|
||||
"SELECT * FROM relationships WHERE source_entity_id = ?",
|
||||
(entity_id,),
|
||||
).fetchall()
|
||||
results.extend(_row_to_relationship(r) for r in rows)
|
||||
if direction in ("incoming", "both"):
|
||||
rows = conn.execute(
|
||||
"SELECT * FROM relationships WHERE target_entity_id = ?",
|
||||
(entity_id,),
|
||||
).fetchall()
|
||||
results.extend(_row_to_relationship(r) for r in rows)
|
||||
return results
|
||||
|
||||
|
||||
def get_entity_with_context(entity_id: str) -> dict | None:
|
||||
entity = get_entity(entity_id)
|
||||
if entity is None:
|
||||
return None
|
||||
relationships = get_relationships(entity_id)
|
||||
related_ids = set()
|
||||
for rel in relationships:
|
||||
related_ids.add(rel.source_entity_id)
|
||||
related_ids.add(rel.target_entity_id)
|
||||
related_ids.discard(entity_id)
|
||||
|
||||
related_entities = {}
|
||||
for rid in related_ids:
|
||||
e = get_entity(rid)
|
||||
if e:
|
||||
related_entities[rid] = e
|
||||
|
||||
return {
|
||||
"entity": entity,
|
||||
"relationships": relationships,
|
||||
"related_entities": related_entities,
|
||||
}
|
||||
|
||||
|
||||
def _row_to_entity(row) -> Entity:
|
||||
return Entity(
|
||||
id=row["id"],
|
||||
entity_type=row["entity_type"],
|
||||
name=row["name"],
|
||||
project=row["project"] or "",
|
||||
description=row["description"] or "",
|
||||
properties=json.loads(row["properties"] or "{}"),
|
||||
status=row["status"],
|
||||
confidence=row["confidence"],
|
||||
source_refs=json.loads(row["source_refs"] or "[]"),
|
||||
created_at=row["created_at"] or "",
|
||||
updated_at=row["updated_at"] or "",
|
||||
)
|
||||
|
||||
|
||||
def _row_to_relationship(row) -> Relationship:
|
||||
return Relationship(
|
||||
id=row["id"],
|
||||
source_entity_id=row["source_entity_id"],
|
||||
target_entity_id=row["target_entity_id"],
|
||||
relationship_type=row["relationship_type"],
|
||||
confidence=row["confidence"],
|
||||
source_refs=json.loads(row["source_refs"] or "[]"),
|
||||
created_at=row["created_at"] or "",
|
||||
)
|
||||
@@ -1,298 +0,0 @@
|
||||
"""AtoCore Wiki — navigable HTML pages from structured data.
|
||||
|
||||
A lightweight wiki served directly from the AtoCore API. Every page is
|
||||
generated on-demand from the database so it's always current. Source of
|
||||
truth is the database — the wiki is a derived view.
|
||||
|
||||
Routes:
|
||||
/wiki Homepage with project list + search
|
||||
/wiki/projects/{name} Full project overview
|
||||
/wiki/entities/{id} Entity detail with relationships
|
||||
/wiki/search?q=... Search entities, memories, state
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import markdown as md
|
||||
|
||||
from atocore.context.project_state import get_state
|
||||
from atocore.engineering.service import (
|
||||
get_entities,
|
||||
get_entity,
|
||||
get_entity_with_context,
|
||||
get_relationships,
|
||||
)
|
||||
from atocore.memory.service import get_memories
|
||||
from atocore.projects.registry import load_project_registry
|
||||
|
||||
|
||||
def render_html(title: str, body_html: str, breadcrumbs: list[tuple[str, str]] | None = None) -> str:
|
||||
nav = ""
|
||||
if breadcrumbs:
|
||||
parts = []
|
||||
for label, href in breadcrumbs:
|
||||
if href:
|
||||
parts.append(f'<a href="{href}">{label}</a>')
|
||||
else:
|
||||
parts.append(f"<span>{label}</span>")
|
||||
nav = f'<nav class="breadcrumbs">{" / ".join(parts)}</nav>'
|
||||
|
||||
return _TEMPLATE.replace("{{title}}", title).replace("{{nav}}", nav).replace("{{body}}", body_html)
|
||||
|
||||
|
||||
def render_homepage() -> str:
|
||||
projects = []
|
||||
try:
|
||||
registered = load_project_registry()
|
||||
for p in registered:
|
||||
entity_count = len(get_entities(project=p.project_id, limit=200))
|
||||
memory_count = len(get_memories(project=p.project_id, active_only=True, limit=200))
|
||||
state_entries = get_state(p.project_id)
|
||||
|
||||
# Pull stage/type/client from state entries
|
||||
stage = ""
|
||||
proj_type = ""
|
||||
client = ""
|
||||
for e in state_entries:
|
||||
if e.category == "status":
|
||||
if e.key == "stage":
|
||||
stage = e.value
|
||||
elif e.key == "type":
|
||||
proj_type = e.value
|
||||
elif e.key == "client":
|
||||
client = e.value
|
||||
|
||||
projects.append({
|
||||
"id": p.project_id,
|
||||
"description": p.description,
|
||||
"entities": entity_count,
|
||||
"memories": memory_count,
|
||||
"state": len(state_entries),
|
||||
"stage": stage,
|
||||
"type": proj_type,
|
||||
"client": client,
|
||||
})
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# Group by high-level bucket
|
||||
buckets: dict[str, list] = {
|
||||
"Active Contracts": [],
|
||||
"Leads & Prospects": [],
|
||||
"Internal Tools & Infra": [],
|
||||
"Other": [],
|
||||
}
|
||||
for p in projects:
|
||||
t = p["type"].lower()
|
||||
s = p["stage"].lower()
|
||||
if "lead" in t or "lead" in s or "prospect" in s:
|
||||
buckets["Leads & Prospects"].append(p)
|
||||
elif "contract" in t or ("active" in s and "contract" in s):
|
||||
buckets["Active Contracts"].append(p)
|
||||
elif "infra" in t or "tool" in t or "internal" in t:
|
||||
buckets["Internal Tools & Infra"].append(p)
|
||||
else:
|
||||
buckets["Other"].append(p)
|
||||
|
||||
lines = ['<h1>AtoCore Wiki</h1>']
|
||||
lines.append('<form class="search-box" action="/wiki/search" method="get">')
|
||||
lines.append('<input type="text" name="q" placeholder="Search entities, memories, projects..." autofocus>')
|
||||
lines.append('<button type="submit">Search</button>')
|
||||
lines.append('</form>')
|
||||
|
||||
for bucket_name, items in buckets.items():
|
||||
if not items:
|
||||
continue
|
||||
lines.append(f'<h2>{bucket_name}</h2>')
|
||||
lines.append('<div class="card-grid">')
|
||||
for p in items:
|
||||
client_line = f'<div class="client">{p["client"]}</div>' if p["client"] else ''
|
||||
stage_tag = f'<span class="tag">{p["stage"].split(" — ")[0]}</span>' if p["stage"] else ''
|
||||
lines.append(f'<a href="/wiki/projects/{p["id"]}" class="card">')
|
||||
lines.append(f'<h3>{p["id"]} {stage_tag}</h3>')
|
||||
lines.append(client_line)
|
||||
lines.append(f'<p>{p["description"][:140]}</p>')
|
||||
lines.append(f'<div class="stats">{p["entities"]} entities · {p["memories"]} memories · {p["state"]} state</div>')
|
||||
lines.append('</a>')
|
||||
lines.append('</div>')
|
||||
|
||||
# Quick stats
|
||||
all_entities = get_entities(limit=500)
|
||||
all_memories = get_memories(active_only=True, limit=500)
|
||||
lines.append('<h2>System</h2>')
|
||||
lines.append(f'<p>{len(all_entities)} entities · {len(all_memories)} active memories · {len(projects)} projects</p>')
|
||||
lines.append(f'<p><a href="/admin/dashboard">API Dashboard (JSON)</a> · <a href="/health">Health Check</a></p>')
|
||||
|
||||
return render_html("AtoCore Wiki", "\n".join(lines))
|
||||
|
||||
|
||||
def render_project(project: str) -> str:
|
||||
from atocore.engineering.mirror import generate_project_overview
|
||||
|
||||
markdown_content = generate_project_overview(project)
|
||||
# Convert entity names to links
|
||||
entities = get_entities(project=project, limit=200)
|
||||
html_body = md.markdown(markdown_content, extensions=["tables", "fenced_code"])
|
||||
|
||||
for ent in sorted(entities, key=lambda e: len(e.name), reverse=True):
|
||||
linked = f'<a href="/wiki/entities/{ent.id}" title="{ent.entity_type}">{ent.name}</a>'
|
||||
html_body = html_body.replace(f"<strong>{ent.name}</strong>", f"<strong>{linked}</strong>", 1)
|
||||
|
||||
return render_html(
|
||||
f"{project}",
|
||||
html_body,
|
||||
breadcrumbs=[("Wiki", "/wiki"), (project, "")],
|
||||
)
|
||||
|
||||
|
||||
def render_entity(entity_id: str) -> str | None:
|
||||
ctx = get_entity_with_context(entity_id)
|
||||
if ctx is None:
|
||||
return None
|
||||
|
||||
ent = ctx["entity"]
|
||||
lines = [f'<h1>[{ent.entity_type}] {ent.name}</h1>']
|
||||
|
||||
if ent.project:
|
||||
lines.append(f'<p>Project: <a href="/wiki/projects/{ent.project}">{ent.project}</a></p>')
|
||||
if ent.description:
|
||||
lines.append(f'<p>{ent.description}</p>')
|
||||
if ent.properties:
|
||||
lines.append('<h2>Properties</h2><ul>')
|
||||
for k, v in ent.properties.items():
|
||||
lines.append(f'<li><strong>{k}</strong>: {v}</li>')
|
||||
lines.append('</ul>')
|
||||
|
||||
lines.append(f'<p class="meta">confidence: {ent.confidence} · status: {ent.status} · created: {ent.created_at}</p>')
|
||||
|
||||
if ctx["relationships"]:
|
||||
lines.append('<h2>Relationships</h2><ul>')
|
||||
for rel in ctx["relationships"]:
|
||||
other_id = rel.target_entity_id if rel.source_entity_id == entity_id else rel.source_entity_id
|
||||
other = ctx["related_entities"].get(other_id)
|
||||
if other:
|
||||
direction = "\u2192" if rel.source_entity_id == entity_id else "\u2190"
|
||||
lines.append(
|
||||
f'<li>{direction} <em>{rel.relationship_type}</em> '
|
||||
f'<a href="/wiki/entities/{other_id}">[{other.entity_type}] {other.name}</a></li>'
|
||||
)
|
||||
lines.append('</ul>')
|
||||
|
||||
breadcrumbs = [("Wiki", "/wiki")]
|
||||
if ent.project:
|
||||
breadcrumbs.append((ent.project, f"/wiki/projects/{ent.project}"))
|
||||
breadcrumbs.append((ent.name, ""))
|
||||
|
||||
return render_html(ent.name, "\n".join(lines), breadcrumbs=breadcrumbs)
|
||||
|
||||
|
||||
def render_search(query: str) -> str:
|
||||
lines = [f'<h1>Search: "{query}"</h1>']
|
||||
|
||||
# Search entities by name
|
||||
entities = get_entities(name_contains=query, limit=20)
|
||||
if entities:
|
||||
lines.append(f'<h2>Entities ({len(entities)})</h2><ul>')
|
||||
for e in entities:
|
||||
proj = f' <span class="tag">{e.project}</span>' if e.project else ''
|
||||
lines.append(
|
||||
f'<li><a href="/wiki/entities/{e.id}">[{e.entity_type}] {e.name}</a>{proj}'
|
||||
f'{" — " + e.description[:100] if e.description else ""}</li>'
|
||||
)
|
||||
lines.append('</ul>')
|
||||
|
||||
# Search memories
|
||||
all_memories = get_memories(active_only=True, limit=200)
|
||||
query_lower = query.lower()
|
||||
matching_mems = [m for m in all_memories if query_lower in m.content.lower()][:10]
|
||||
if matching_mems:
|
||||
lines.append(f'<h2>Memories ({len(matching_mems)})</h2><ul>')
|
||||
for m in matching_mems:
|
||||
proj = f' <span class="tag">{m.project}</span>' if m.project else ''
|
||||
lines.append(f'<li>[{m.memory_type}]{proj} {m.content[:200]}</li>')
|
||||
lines.append('</ul>')
|
||||
|
||||
if not entities and not matching_mems:
|
||||
lines.append('<p>No results found.</p>')
|
||||
|
||||
lines.append('<form class="search-box" action="/wiki/search" method="get">')
|
||||
lines.append(f'<input type="text" name="q" value="{query}" autofocus>')
|
||||
lines.append('<button type="submit">Search</button>')
|
||||
lines.append('</form>')
|
||||
|
||||
return render_html(
|
||||
f"Search: {query}",
|
||||
"\n".join(lines),
|
||||
breadcrumbs=[("Wiki", "/wiki"), ("Search", "")],
|
||||
)
|
||||
|
||||
|
||||
_TEMPLATE = """<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>{{title}} — AtoCore</title>
|
||||
<style>
|
||||
:root { --bg: #fafafa; --text: #1a1a2e; --accent: #2563eb; --border: #e2e8f0; --card: #fff; --hover: #f1f5f9; }
|
||||
@media (prefers-color-scheme: dark) {
|
||||
:root { --bg: #0f172a; --text: #e2e8f0; --accent: #60a5fa; --border: #334155; --card: #1e293b; --hover: #334155; }
|
||||
}
|
||||
* { box-sizing: border-box; margin: 0; padding: 0; }
|
||||
body {
|
||||
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
|
||||
line-height: 1.7; color: var(--text); background: var(--bg);
|
||||
max-width: 800px; margin: 0 auto; padding: 1.5rem;
|
||||
}
|
||||
h1 { font-size: 1.8rem; margin-bottom: 0.5rem; color: var(--accent); }
|
||||
h2 { font-size: 1.3rem; margin-top: 2rem; margin-bottom: 0.6rem; padding-bottom: 0.2rem; border-bottom: 2px solid var(--border); }
|
||||
h3 { font-size: 1.1rem; margin-top: 1.2rem; margin-bottom: 0.4rem; }
|
||||
p { margin-bottom: 0.8rem; }
|
||||
ul { margin-left: 1.5rem; margin-bottom: 1rem; }
|
||||
li { margin-bottom: 0.3rem; }
|
||||
li ul { margin-top: 0.2rem; }
|
||||
strong { color: var(--accent); font-weight: 600; }
|
||||
em { opacity: 0.7; font-size: 0.9em; }
|
||||
a { color: var(--accent); text-decoration: none; }
|
||||
a:hover { text-decoration: underline; }
|
||||
blockquote {
|
||||
background: var(--card); border-left: 4px solid var(--accent);
|
||||
padding: 0.6rem 1rem; margin: 1rem 0; border-radius: 0 6px 6px 0;
|
||||
font-size: 0.9em;
|
||||
}
|
||||
hr { border: none; border-top: 1px solid var(--border); margin: 2rem 0; }
|
||||
.breadcrumbs { margin-bottom: 1.5rem; font-size: 0.85em; opacity: 0.7; }
|
||||
.breadcrumbs a { opacity: 0.8; }
|
||||
.meta { font-size: 0.8em; opacity: 0.5; margin-top: 0.5rem; }
|
||||
.tag { background: var(--accent); color: var(--bg); padding: 0.1rem 0.4rem; border-radius: 3px; font-size: 0.75em; margin-left: 0.3rem; }
|
||||
.search-box { display: flex; gap: 0.5rem; margin: 1.5rem 0; }
|
||||
.search-box input {
|
||||
flex: 1; padding: 0.6rem 1rem; border: 2px solid var(--border);
|
||||
border-radius: 8px; background: var(--card); color: var(--text);
|
||||
font-size: 1rem;
|
||||
}
|
||||
.search-box input:focus { border-color: var(--accent); outline: none; }
|
||||
.search-box button {
|
||||
padding: 0.6rem 1.2rem; background: var(--accent); color: var(--bg);
|
||||
border: none; border-radius: 8px; cursor: pointer; font-size: 1rem;
|
||||
}
|
||||
.card-grid { display: grid; grid-template-columns: 1fr; gap: 1rem; margin: 1rem 0; }
|
||||
@media (min-width: 600px) { .card-grid { grid-template-columns: 1fr 1fr; } }
|
||||
.card {
|
||||
display: block; background: var(--card); border: 1px solid var(--border);
|
||||
border-radius: 10px; padding: 1.2rem; text-decoration: none;
|
||||
color: var(--text); transition: border-color 0.2s;
|
||||
}
|
||||
.card:hover { border-color: var(--accent); background: var(--hover); text-decoration: none; }
|
||||
.card h3 { color: var(--accent); margin: 0 0 0.3rem 0; }
|
||||
.card p { font-size: 0.9em; margin: 0; opacity: 0.8; }
|
||||
.card .stats { font-size: 0.8em; margin-top: 0.5rem; opacity: 0.5; }
|
||||
.card .client { font-size: 0.85em; opacity: 0.65; margin-bottom: 0.3rem; font-style: italic; }
|
||||
.card h3 .tag { font-size: 0.65em; vertical-align: middle; margin-left: 0.4rem; }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
{{nav}}
|
||||
{{body}}
|
||||
</body>
|
||||
</html>"""
|
||||
@@ -8,7 +8,6 @@ from atocore import __version__
|
||||
from atocore.api.routes import router
|
||||
import atocore.config as _config
|
||||
from atocore.context.project_state import init_project_state_schema
|
||||
from atocore.engineering.service import init_engineering_schema
|
||||
from atocore.ingestion.pipeline import get_source_status
|
||||
from atocore.models.database import init_db
|
||||
from atocore.observability.logger import get_logger, setup_logging
|
||||
@@ -30,7 +29,6 @@ async def lifespan(app: FastAPI):
|
||||
_config.ensure_runtime_dirs()
|
||||
init_db()
|
||||
init_project_state_schema()
|
||||
init_engineering_schema()
|
||||
log.info(
|
||||
"startup_ready",
|
||||
env=_config.settings.env,
|
||||
|
||||
@@ -1,183 +0,0 @@
|
||||
"""Shared LLM-extractor prompt + parser (stdlib-only).
|
||||
|
||||
R12: single source of truth for the system prompt, memory type set,
|
||||
size limits, and raw JSON parsing used by both paths that shell out
|
||||
to ``claude -p``:
|
||||
|
||||
- ``atocore.memory.extractor_llm`` (in-container extractor, wraps the
|
||||
parsed dicts in ``MemoryCandidate`` with registry-checked project
|
||||
attribution)
|
||||
- ``scripts/batch_llm_extract_live.py`` (host-side extractor, can't
|
||||
import the full atocore package because Dalidou's host Python lacks
|
||||
the container's deps; imports this module via ``sys.path``)
|
||||
|
||||
This module MUST stay stdlib-only. No ``atocore`` imports, no third-
|
||||
party packages. Callers apply their own project-attribution policy on
|
||||
top of the normalized dicts this module emits.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from typing import Any
|
||||
|
||||
LLM_EXTRACTOR_VERSION = "llm-0.4.0"
|
||||
MAX_RESPONSE_CHARS = 8000
|
||||
MAX_PROMPT_CHARS = 2000
|
||||
MEMORY_TYPES = {"identity", "preference", "project", "episodic", "knowledge", "adaptation"}
|
||||
|
||||
SYSTEM_PROMPT = """You extract memory candidates from LLM conversation turns for a personal context engine called AtoCore.
|
||||
|
||||
AtoCore is the brain for Atomaste's engineering work. Known projects:
|
||||
p04-gigabit, p05-interferometer, p06-polisher, atomizer-v2, atocore,
|
||||
abb-space. Unknown project names — still tag them, the system auto-detects.
|
||||
|
||||
Your job is to emit SIGNALS that matter for future context. Be aggressive:
|
||||
err on the side of capturing useful signal. Triage filters noise downstream.
|
||||
|
||||
WHAT TO EMIT (in order of importance):
|
||||
|
||||
1. PROJECT ACTIVITY — any mention of a project with context worth remembering:
|
||||
- "Schott quote received for ABB-Space" (event + project)
|
||||
- "Cédric asked about p06 firmware timing" (stakeholder event)
|
||||
- "Still waiting on Zygo lead-time from Nabeel" (blocker status)
|
||||
- "p05 vendor decision needs to happen this week" (action item)
|
||||
|
||||
2. DECISIONS AND CHOICES — anything that commits to a direction:
|
||||
- "Going with Zygo Verifire SV for p05" (decision)
|
||||
- "Dropping stitching from primary workflow" (design choice)
|
||||
- "USB SSD mandatory, not SD card" (architectural commitment)
|
||||
|
||||
3. DURABLE ENGINEERING INSIGHT — earned knowledge that generalizes:
|
||||
- "CTE gradient dominates WFE at F/1.2" (materials insight)
|
||||
- "Preston model breaks below 5N because contact assumption fails"
|
||||
- "m=1 coma NOT correctable by force modulation" (controls insight)
|
||||
Test: would a competent engineer NEED experience to know this?
|
||||
If it's textbook/google-findable, skip it.
|
||||
|
||||
4. STAKEHOLDER AND VENDOR EVENTS:
|
||||
- "Email sent to Nabeel 2026-04-13 asking for lead time"
|
||||
- "Meeting with Jason on Table 7 next Tuesday"
|
||||
- "Starspec wants updated CAD by Friday"
|
||||
|
||||
5. PREFERENCES AND ADAPTATIONS that shape how Antoine works:
|
||||
- "Antoine prefers OAuth over API keys"
|
||||
- "Extraction stays off the capture hot path"
|
||||
|
||||
WHAT TO SKIP:
|
||||
|
||||
- Pure conversational filler ("ok thanks", "let me check")
|
||||
- Instructional help content ("run this command", "here's how to...")
|
||||
- Obvious textbook facts anyone can google in 30 seconds
|
||||
- Session meta-chatter ("let me commit this", "deploy running")
|
||||
- Transient system state snapshots ("36 active memories right now")
|
||||
|
||||
CANDIDATE TYPES — choose the best fit:
|
||||
|
||||
- project — a fact, decision, or event specific to one named project
|
||||
- knowledge — durable engineering insight (use domain, not project)
|
||||
- preference — how Antoine works / wants things done
|
||||
- adaptation — a standing rule or adjustment to behavior
|
||||
- episodic — a stakeholder event or milestone worth remembering
|
||||
|
||||
DOMAINS for knowledge candidates (required when type=knowledge and project is empty):
|
||||
physics, materials, optics, mechanics, manufacturing, metrology,
|
||||
controls, software, math, finance, business
|
||||
|
||||
TRUST HIERARCHY:
|
||||
|
||||
- project-specific: set project to the project id, leave domain empty
|
||||
- domain knowledge: set domain, leave project empty
|
||||
- events/activity: use project, type=project or episodic
|
||||
- one conversation can produce MULTIPLE candidates — emit them all
|
||||
|
||||
OUTPUT RULES:
|
||||
|
||||
- Each candidate content under 250 characters, stands alone
|
||||
- Default confidence 0.5. Raise to 0.7 only for ratified/committed claims.
|
||||
- Raw JSON array, no prose, no markdown fences
|
||||
- Empty array [] is fine when the conversation has no durable signal
|
||||
|
||||
Each element:
|
||||
{"type": "project|knowledge|preference|adaptation|episodic", "content": "...", "project": "...", "domain": "", "confidence": 0.5}"""
|
||||
|
||||
|
||||
def build_user_message(prompt: str, response: str, project_hint: str) -> str:
|
||||
prompt_excerpt = (prompt or "")[:MAX_PROMPT_CHARS]
|
||||
response_excerpt = (response or "")[:MAX_RESPONSE_CHARS]
|
||||
return (
|
||||
f"PROJECT HINT (may be empty): {project_hint or ''}\n\n"
|
||||
f"USER PROMPT:\n{prompt_excerpt}\n\n"
|
||||
f"ASSISTANT RESPONSE:\n{response_excerpt}\n\n"
|
||||
"Return the JSON array now."
|
||||
)
|
||||
|
||||
|
||||
def parse_llm_json_array(raw_output: str) -> list[dict[str, Any]]:
|
||||
"""Strip markdown fences / leading prose and return the parsed JSON
|
||||
array as a list of raw dicts. Returns an empty list on any parse
|
||||
failure — callers decide whether to log."""
|
||||
text = (raw_output or "").strip()
|
||||
if text.startswith("```"):
|
||||
text = text.strip("`")
|
||||
nl = text.find("\n")
|
||||
if nl >= 0:
|
||||
text = text[nl + 1:]
|
||||
if text.endswith("```"):
|
||||
text = text[:-3]
|
||||
text = text.strip()
|
||||
|
||||
if not text or text == "[]":
|
||||
return []
|
||||
|
||||
if not text.lstrip().startswith("["):
|
||||
start = text.find("[")
|
||||
end = text.rfind("]")
|
||||
if start >= 0 and end > start:
|
||||
text = text[start:end + 1]
|
||||
|
||||
try:
|
||||
parsed = json.loads(text)
|
||||
except json.JSONDecodeError:
|
||||
return []
|
||||
|
||||
if not isinstance(parsed, list):
|
||||
return []
|
||||
return [item for item in parsed if isinstance(item, dict)]
|
||||
|
||||
|
||||
def normalize_candidate_item(item: dict[str, Any]) -> dict[str, Any] | None:
|
||||
"""Validate and normalize one raw model item into a candidate dict.
|
||||
|
||||
Returns None if the item fails basic validation (unknown type,
|
||||
empty content). Does NOT apply project-attribution policy — that's
|
||||
the caller's job, since the registry-check differs between the
|
||||
in-container path and the host path.
|
||||
|
||||
Output keys: type, content, project (raw model value), domain,
|
||||
confidence.
|
||||
"""
|
||||
mem_type = str(item.get("type") or "").strip().lower()
|
||||
content = str(item.get("content") or "").strip()
|
||||
if mem_type not in MEMORY_TYPES or not content:
|
||||
return None
|
||||
|
||||
model_project = str(item.get("project") or "").strip()
|
||||
domain = str(item.get("domain") or "").strip().lower()
|
||||
|
||||
try:
|
||||
confidence = float(item.get("confidence", 0.5))
|
||||
except (TypeError, ValueError):
|
||||
confidence = 0.5
|
||||
confidence = max(0.0, min(1.0, confidence))
|
||||
|
||||
if domain and not model_project:
|
||||
content = f"[{domain}] {content}"
|
||||
|
||||
return {
|
||||
"type": mem_type,
|
||||
"content": content[:1000],
|
||||
"project": model_project,
|
||||
"domain": domain,
|
||||
"confidence": confidence,
|
||||
}
|
||||
@@ -49,6 +49,7 @@ Implementation notes:
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
@@ -57,21 +58,38 @@ from dataclasses import dataclass
|
||||
from functools import lru_cache
|
||||
|
||||
from atocore.interactions.service import Interaction
|
||||
from atocore.memory._llm_prompt import (
|
||||
LLM_EXTRACTOR_VERSION,
|
||||
SYSTEM_PROMPT as _SYSTEM_PROMPT,
|
||||
build_user_message,
|
||||
normalize_candidate_item,
|
||||
parse_llm_json_array,
|
||||
)
|
||||
from atocore.memory.extractor import MemoryCandidate
|
||||
from atocore.memory.service import MEMORY_TYPES
|
||||
from atocore.observability.logger import get_logger
|
||||
|
||||
log = get_logger("extractor_llm")
|
||||
|
||||
LLM_EXTRACTOR_VERSION = "llm-0.2.0"
|
||||
DEFAULT_MODEL = os.environ.get("ATOCORE_LLM_EXTRACTOR_MODEL", "sonnet")
|
||||
DEFAULT_TIMEOUT_S = float(os.environ.get("ATOCORE_LLM_EXTRACTOR_TIMEOUT_S", "90"))
|
||||
MAX_RESPONSE_CHARS = 8000
|
||||
MAX_PROMPT_CHARS = 2000
|
||||
|
||||
_SYSTEM_PROMPT = """You extract durable memory candidates from LLM conversation turns for a personal context engine called AtoCore.
|
||||
|
||||
Your job is to read one user prompt plus the assistant's response and decide which durable facts, decisions, preferences, architectural rules, or project invariants should be remembered across future sessions.
|
||||
|
||||
Rules:
|
||||
|
||||
1. Only surface durable claims. Skip transient status ("deploy is still running"), instructional guidance ("here is how to run the command"), troubleshooting tactics, ephemeral recommendations ("merge this PR now"), and session recaps.
|
||||
2. A candidate is durable when a reader coming back in two weeks would still need to know it. Architectural choices, named rules, ratified decisions, invariants, procurement commitments, and project-level constraints qualify. Conversational fillers and step-by-step instructions do not.
|
||||
3. Each candidate must stand alone. Rewrite the claim in one sentence under 200 characters with enough context that a reader without the conversation understands it.
|
||||
4. Each candidate must have a type from this closed set: project, knowledge, preference, adaptation.
|
||||
5. If the conversation is clearly scoped to a project (p04-gigabit, p05-interferometer, p06-polisher, atocore), set ``project`` to that id. Otherwise leave ``project`` empty.
|
||||
6. If the response makes no durable claim, return an empty list. It is correct and expected to return [] on most conversational turns.
|
||||
7. Confidence should be 0.5 by default so human review workload is honest. Raise to 0.6 only when the response states the claim in an unambiguous, committed form (e.g. "the decision is X", "the selected approach is Y", "X is non-negotiable").
|
||||
8. Output must be a raw JSON array and nothing else. No prose before or after. No markdown fences. No explanations.
|
||||
|
||||
Each array element has exactly this shape:
|
||||
|
||||
{"type": "project|knowledge|preference|adaptation", "content": "...", "project": "...", "confidence": 0.5}
|
||||
|
||||
Return [] when there is nothing to extract."""
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -134,10 +152,13 @@ def extract_candidates_llm_verbose(
|
||||
if not response_text:
|
||||
return LLMExtractionResult(candidates=[], raw_output="", error="empty_response")
|
||||
|
||||
user_message = build_user_message(
|
||||
interaction.prompt or "",
|
||||
response_text,
|
||||
interaction.project or "",
|
||||
prompt_excerpt = (interaction.prompt or "")[:MAX_PROMPT_CHARS]
|
||||
response_excerpt = response_text[:MAX_RESPONSE_CHARS]
|
||||
user_message = (
|
||||
f"PROJECT HINT (may be empty): {interaction.project or ''}\n\n"
|
||||
f"USER PROMPT:\n{prompt_excerpt}\n\n"
|
||||
f"ASSISTANT RESPONSE:\n{response_excerpt}\n\n"
|
||||
"Return the JSON array now."
|
||||
)
|
||||
|
||||
args = [
|
||||
@@ -195,25 +216,50 @@ def extract_candidates_llm_verbose(
|
||||
def _parse_candidates(raw_output: str, interaction: Interaction) -> list[MemoryCandidate]:
|
||||
"""Parse the model's JSON output into MemoryCandidate objects.
|
||||
|
||||
Shared stripping + per-item validation live in
|
||||
``atocore.memory._llm_prompt``. This function adds the container-
|
||||
only R9 project attribution: registry-check model_project and fall
|
||||
back to the interaction scope when set.
|
||||
Tolerates common model glitches: surrounding whitespace, stray
|
||||
markdown fences, leading/trailing prose. Silently drops malformed
|
||||
array elements rather than raising.
|
||||
"""
|
||||
raw_items = parse_llm_json_array(raw_output)
|
||||
if not raw_items and raw_output.strip() not in ("", "[]"):
|
||||
log.error("llm_extractor_parse_failed", raw_prefix=raw_output[:120])
|
||||
text = raw_output.strip()
|
||||
if text.startswith("```"):
|
||||
text = text.strip("`")
|
||||
first_newline = text.find("\n")
|
||||
if first_newline >= 0:
|
||||
text = text[first_newline + 1 :]
|
||||
if text.endswith("```"):
|
||||
text = text[:-3]
|
||||
text = text.strip()
|
||||
|
||||
if not text or text == "[]":
|
||||
return []
|
||||
|
||||
if not text.lstrip().startswith("["):
|
||||
start = text.find("[")
|
||||
end = text.rfind("]")
|
||||
if start >= 0 and end > start:
|
||||
text = text[start : end + 1]
|
||||
|
||||
try:
|
||||
parsed = json.loads(text)
|
||||
except json.JSONDecodeError as exc:
|
||||
log.error("llm_extractor_parse_failed", error=str(exc), raw_prefix=raw_output[:120])
|
||||
return []
|
||||
|
||||
if not isinstance(parsed, list):
|
||||
return []
|
||||
|
||||
results: list[MemoryCandidate] = []
|
||||
for raw_item in raw_items:
|
||||
normalized = normalize_candidate_item(raw_item)
|
||||
if normalized is None:
|
||||
for item in parsed:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
|
||||
model_project = normalized["project"]
|
||||
# R9 trust hierarchy: interaction scope wins; else registry-
|
||||
# resolve the model's tag; else keep the model's tag so auto-
|
||||
# triage can surface unregistered projects.
|
||||
mem_type = str(item.get("type") or "").strip().lower()
|
||||
content = str(item.get("content") or "").strip()
|
||||
model_project = str(item.get("project") or "").strip()
|
||||
# R9 trust hierarchy for project attribution:
|
||||
# 1. Interaction scope always wins when set (strongest signal)
|
||||
# 2. Model project used only when interaction is unscoped
|
||||
# AND model project resolves to a registered project
|
||||
# 3. Empty string when both are empty/unregistered
|
||||
if interaction.project:
|
||||
project = interaction.project
|
||||
elif model_project:
|
||||
@@ -225,29 +271,29 @@ def _parse_candidates(raw_output: str, interaction: Interaction) -> list[MemoryC
|
||||
|
||||
registered_ids = {p.project_id for p in load_project_registry()}
|
||||
resolved = resolve_project_name(model_project)
|
||||
if resolved in registered_ids:
|
||||
project = resolved
|
||||
else:
|
||||
project = model_project
|
||||
log.info(
|
||||
"unregistered_project_detected",
|
||||
model_project=model_project,
|
||||
interaction_id=interaction.id,
|
||||
)
|
||||
project = resolved if resolved in registered_ids else ""
|
||||
except Exception:
|
||||
project = model_project
|
||||
project = ""
|
||||
else:
|
||||
project = ""
|
||||
|
||||
content = normalized["content"]
|
||||
confidence_raw = item.get("confidence", 0.5)
|
||||
if mem_type not in MEMORY_TYPES:
|
||||
continue
|
||||
if not content:
|
||||
continue
|
||||
try:
|
||||
confidence = float(confidence_raw)
|
||||
except (TypeError, ValueError):
|
||||
confidence = 0.5
|
||||
confidence = max(0.0, min(1.0, confidence))
|
||||
results.append(
|
||||
MemoryCandidate(
|
||||
memory_type=normalized["type"],
|
||||
content=content,
|
||||
memory_type=mem_type,
|
||||
content=content[:1000],
|
||||
rule="llm_extraction",
|
||||
source_span=content[:200],
|
||||
project=project,
|
||||
confidence=normalized["confidence"],
|
||||
confidence=confidence,
|
||||
source_interaction_id=interaction.id,
|
||||
extractor_version=LLM_EXTRACTOR_VERSION,
|
||||
)
|
||||
|
||||
@@ -1,118 +0,0 @@
|
||||
"""Tests for the Engineering Knowledge Layer."""
|
||||
|
||||
from atocore.engineering.service import (
|
||||
ENTITY_TYPES,
|
||||
RELATIONSHIP_TYPES,
|
||||
create_entity,
|
||||
create_relationship,
|
||||
get_entities,
|
||||
get_entity,
|
||||
get_entity_with_context,
|
||||
get_relationships,
|
||||
init_engineering_schema,
|
||||
)
|
||||
from atocore.models.database import init_db
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
def test_create_and_get_entity(tmp_data_dir):
|
||||
init_db()
|
||||
init_engineering_schema()
|
||||
e = create_entity(
|
||||
entity_type="component",
|
||||
name="Pivot Pin",
|
||||
project="p04-gigabit",
|
||||
description="Lateral support pivot pin for M1 assembly",
|
||||
properties={"material": "GF-PTFE", "diameter_mm": 12},
|
||||
)
|
||||
assert e.entity_type == "component"
|
||||
assert e.name == "Pivot Pin"
|
||||
assert e.properties["material"] == "GF-PTFE"
|
||||
|
||||
fetched = get_entity(e.id)
|
||||
assert fetched is not None
|
||||
assert fetched.name == "Pivot Pin"
|
||||
|
||||
|
||||
def test_create_relationship(tmp_data_dir):
|
||||
init_db()
|
||||
init_engineering_schema()
|
||||
subsystem = create_entity("subsystem", "Lateral Support", project="p04-gigabit")
|
||||
component = create_entity("component", "Pivot Pin", project="p04-gigabit")
|
||||
|
||||
rel = create_relationship(
|
||||
source_entity_id=subsystem.id,
|
||||
target_entity_id=component.id,
|
||||
relationship_type="contains",
|
||||
)
|
||||
assert rel.relationship_type == "contains"
|
||||
|
||||
rels = get_relationships(subsystem.id, direction="outgoing")
|
||||
assert len(rels) == 1
|
||||
assert rels[0].target_entity_id == component.id
|
||||
|
||||
|
||||
def test_entity_with_context(tmp_data_dir):
|
||||
init_db()
|
||||
init_engineering_schema()
|
||||
subsystem = create_entity("subsystem", "Lateral Support", project="p04-gigabit")
|
||||
pin = create_entity("component", "Pivot Pin", project="p04-gigabit")
|
||||
pad = create_entity("component", "PTFE Pad", project="p04-gigabit")
|
||||
material = create_entity("material", "GF-PTFE", project="p04-gigabit",
|
||||
description="Glass-filled PTFE for thermal stability")
|
||||
|
||||
create_relationship(subsystem.id, pin.id, "contains")
|
||||
create_relationship(subsystem.id, pad.id, "contains")
|
||||
create_relationship(pad.id, material.id, "uses_material")
|
||||
|
||||
ctx = get_entity_with_context(subsystem.id)
|
||||
assert ctx is not None
|
||||
assert len(ctx["relationships"]) == 2
|
||||
assert pin.id in ctx["related_entities"]
|
||||
assert pad.id in ctx["related_entities"]
|
||||
|
||||
|
||||
def test_filter_entities_by_type_and_project(tmp_data_dir):
|
||||
init_db()
|
||||
init_engineering_schema()
|
||||
create_entity("component", "Pin A", project="p04-gigabit")
|
||||
create_entity("component", "Pin B", project="p04-gigabit")
|
||||
create_entity("material", "Steel", project="p04-gigabit")
|
||||
create_entity("component", "Actuator", project="p06-polisher")
|
||||
|
||||
components = get_entities(entity_type="component", project="p04-gigabit")
|
||||
assert len(components) == 2
|
||||
|
||||
all_p04 = get_entities(project="p04-gigabit")
|
||||
assert len(all_p04) == 3
|
||||
|
||||
polisher = get_entities(project="p06-polisher")
|
||||
assert len(polisher) == 1
|
||||
|
||||
|
||||
def test_invalid_entity_type_raises(tmp_data_dir):
|
||||
init_db()
|
||||
init_engineering_schema()
|
||||
with pytest.raises(ValueError, match="Invalid entity type"):
|
||||
create_entity("spaceship", "Enterprise")
|
||||
|
||||
|
||||
def test_invalid_relationship_type_raises(tmp_data_dir):
|
||||
init_db()
|
||||
init_engineering_schema()
|
||||
a = create_entity("component", "A")
|
||||
b = create_entity("component", "B")
|
||||
with pytest.raises(ValueError, match="Invalid relationship type"):
|
||||
create_relationship(a.id, b.id, "loves")
|
||||
|
||||
|
||||
def test_entity_name_search(tmp_data_dir):
|
||||
init_db()
|
||||
init_engineering_schema()
|
||||
create_entity("component", "Vertical Support Pad")
|
||||
create_entity("component", "Lateral Support Bracket")
|
||||
create_entity("component", "Reference Frame")
|
||||
|
||||
results = get_entities(name_contains="Support")
|
||||
assert len(results) == 2
|
||||
@@ -171,38 +171,3 @@ def test_llm_extraction_failure_returns_empty(tmp_data_dir, monkeypatch):
|
||||
# Nothing in the candidate queue
|
||||
queue = get_memories(status="candidate", limit=10)
|
||||
assert len(queue) == 0
|
||||
|
||||
|
||||
def test_extract_batch_api_503_when_cli_missing(tmp_data_dir, monkeypatch):
|
||||
"""R11: POST /admin/extract-batch with mode=llm must fail loud when
|
||||
the `claude` CLI is unavailable, instead of silently returning a
|
||||
success-with-0-candidates payload (which masked host-vs-container
|
||||
truth for operators)."""
|
||||
from fastapi.testclient import TestClient
|
||||
from atocore.main import app
|
||||
import atocore.api.routes as routes
|
||||
|
||||
init_db()
|
||||
monkeypatch.setattr(routes, "_llm_cli_available", lambda: False)
|
||||
|
||||
client = TestClient(app)
|
||||
response = client.post("/admin/extract-batch", json={"mode": "llm"})
|
||||
|
||||
assert response.status_code == 503
|
||||
assert "claude" in response.json()["detail"].lower()
|
||||
|
||||
|
||||
def test_extract_batch_api_rule_mode_ok_without_cli(tmp_data_dir, monkeypatch):
|
||||
"""Rule mode must still work when the LLM CLI is missing — R11 only
|
||||
affects mode=llm."""
|
||||
from fastapi.testclient import TestClient
|
||||
from atocore.main import app
|
||||
import atocore.api.routes as routes
|
||||
|
||||
init_db()
|
||||
monkeypatch.setattr(routes, "_llm_cli_available", lambda: False)
|
||||
|
||||
client = TestClient(app)
|
||||
response = client.post("/admin/extract-batch", json={"mode": "rule"})
|
||||
|
||||
assert response.status_code == 200
|
||||
|
||||
@@ -130,17 +130,17 @@ def test_case_c_unregistered_model_scoped_interaction(tmp_data_dir, project_regi
|
||||
assert result[0].project == "p06-polisher"
|
||||
|
||||
|
||||
def test_case_d_unregistered_model_unscoped_keeps_tag(tmp_data_dir, project_registry):
|
||||
def test_case_d_unregistered_model_unscoped_interaction(tmp_data_dir, project_registry):
|
||||
"""Case D: model returns unregistered project, interaction is unscoped.
|
||||
Keeps the model's tag for auto-project-detection (new behavior)."""
|
||||
Falls to empty (not the hallucinated name)."""
|
||||
from atocore.models.database import init_db
|
||||
init_db()
|
||||
project_registry(("p06-polisher", ["p06"]))
|
||||
raw = '[{"type": "project", "content": "x", "project": "new-lead-project"}]'
|
||||
raw = '[{"type": "project", "content": "x", "project": "fake-project-99"}]'
|
||||
interaction = _make_interaction()
|
||||
interaction.project = ""
|
||||
result = _parse_candidates(raw, interaction)
|
||||
assert result[0].project == "new-lead-project"
|
||||
assert result[0].project == ""
|
||||
|
||||
|
||||
def test_case_e_matching_model_and_interaction(tmp_data_dir, project_registry):
|
||||
|
||||
Reference in New Issue
Block a user