6 Commits

Author SHA1 Message Date
b790e7eb30 audit: record final 2026-04-12 findings 2026-04-12 13:03:10 +00:00
e2895b5d2b feat: Phase 8 OpenClaw integration verified end-to-end
Verified t420-openclaw/atocore.py against live Dalidou from both
the development machine and the T420 (clawdbot @ 192.168.86.39):

- health: returns 0.2.0 + build_sha + vector count
- auto-context: project detection + context/build produces full
  packs with Trusted Project State, Project Memories band, and
  retrieved chunks (tested p05 vendor query and p06 firmware query)
- fail-open: unreachable host returns {status: unavailable,
  fail_open: true} without crashing or blocking the session

API surface coverage: atocore.py hits 15/33 endpoints (core
retrieval + project state + context build). Memory management,
interactions, and backup endpoints are correctly excluded — those
belong to the operator client (scripts/atocore_client.py) per the
read-only additive integration model.

No code changes needed — the April 6 atocore.py already matches
the current API surface. Wave 2 state entries and project-memory
band changes are transparent to the client (they enrich
formatted_context without requiring client-side updates).

Cloned repo to T420 at /home/papa/ATOCore for future OpenClaw use.
Updated master-plan-status.md: Phase 8 moved from Partial to
Baseline Complete.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 08:50:51 -04:00
2b79680167 chore(ledger): Wave 2 ingestion + codex audit response session log
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 07:57:32 -04:00
39d73e91b4 fix(R6): fall back to interaction.project when LLM returns empty
Codex R6: the LLM extractor accepted the model's project field
verbatim. When the model returned empty string, clearly p06 memories
got promoted as project='', making them invisible to the p06
project-memory band and explaining the p06-offline-design harness
failure.

Fix: if model returns empty project but interaction.project is set,
inherit the interaction's project. Model-supplied project still takes
precedence when non-empty.

Two new tests lock the fallback and precedence behaviors.
R5 acknowledged (LLM extractor not yet wired into API — next task).

Test count: 278 -> 280. Harness re-run pending after deploy.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 07:37:14 -04:00
7ddf0e38ee Merge codex/audit-2026-04-12 — R5-R8 findings
Codex correctly identified:
- R5 (P1): LLM extractor is script-only, not wired into the API
- R6 (P1): LLM extractor drops interaction.project when model
  returns empty — caused the p06-offline-design harness failure
- R7 (P2): lexical scorer ties on overlap count, broad memories
  win on confidence tiebreaker
- R8 (P2): no integration test for the persist/triage flow

Also corrected the harness-failure narrative: not all 3 are budget
contention. One is a ranking tie, one is a project-scope miss,
one is chunk bleed.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 07:35:09 -04:00
b0fde3ee60 config: default LLM extractor model haiku -> sonnet
Haiku was producing noisy candidates (31% accept rate on first
triage). Sonnet should give tighter extraction with fewer false
positives while still catching the same durable-fact patterns.
Override: ATOCORE_LLM_EXTRACTOR_MODEL=haiku to revert.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 07:31:34 -04:00
4 changed files with 49 additions and 11 deletions

View File

@@ -6,12 +6,13 @@
## Orientation
- **live_sha** (Dalidou `/health` build_sha): `5c69f77`
- **last_updated**: 2026-04-12 by Codex (audit branch `codex/audit-2026-04-12`)
- **main_tip**: `146f2e4`
- **test_count**: 278 passing
- **harness**: `15/18 PASS` (remaining failures are mixed: p06-firmware-interface exposes a lexical-ranking tie, p06-offline-design is a live triage scoping miss, p06-tailscale still has retrieved-chunk bleed)
- **active_memories**: 36 (was 20 before mini-phase; p06-polisher 2->16, atocore 0->5)
- **live_sha** (Dalidou `/health` build_sha): `39d73e9`
- **last_updated**: 2026-04-12 by Codex (audit branch `codex/audit-2026-04-12-final`)
- **main_tip**: `e2895b5`
- **test_count**: 280 passing
- **harness**: `16/18 PASS` (p06-firmware-interface = R7 ranking tie; p06-tailscale = chunk bleed)
- **active_memories**: 36 (p06-polisher 16, p05-interferometer 6, p04-gigabit 5, atocore 5, other 4)
- **project_state_entries**: p04=5, p05=6, p06=6 (Wave 2 entries present on live Dalidou; 17 total visible)
- **off_host_backup**: `papa@192.168.86.39:/home/papa/atocore-backups/` via cron env `ATOCORE_BACKUP_RSYNC`, verified
## Active Plan
@@ -124,10 +125,12 @@ One branch `codex/extractor-eval-loop` for Day 1-5, a second `codex/retrieval-ha
| R2 | Codex | P1 | src/atocore/context/builder.py | Project memories excluded from pack | fixed | Claude | 2026-04-11 | 8ea53f4 |
| R3 | Claude | P2 | src/atocore/memory/extractor.py | Rule cues (`## Decision:`) never fire on conversational LLM text | open | Claude | 2026-04-11 | |
| R4 | Codex | P2 | DEV-LEDGER.md:11 | Orientation `main_tip` was stale versus `HEAD` / `origin/main` | fixed | Codex | 2026-04-11 | 81307ce |
| R5 | Codex | P1 | src/atocore/interactions/service.py:157-174 | The deployed extraction path still calls only the rule extractor; the new LLM extractor is eval/script-only, so Day 4 "gate cleared" is true as a benchmark result but not as an operational extraction path | open | Claude | 2026-04-12 | |
| R6 | Codex | P1 | src/atocore/memory/extractor_llm.py:258-276 | LLM extraction accepts model-supplied `project` verbatim with no fallback to `interaction.project`; live triage promoted a clearly p06 memory (offline/network rule) as project=`""`, which explains the p06-offline-design harness miss and falsifies the current "all 3 failures are budget-contention" claim | open | Claude | 2026-04-12 | |
| R5 | Codex | P1 | src/atocore/interactions/service.py:157-174 | The deployed extraction path still calls only the rule extractor; the new LLM extractor is eval/script-only, so Day 4 "gate cleared" is true as a benchmark result but not as an operational extraction path | acknowledged | Claude | 2026-04-12 | |
| R6 | Codex | P1 | src/atocore/memory/extractor_llm.py:258-276 | LLM extraction accepts model-supplied `project` verbatim with no fallback to `interaction.project`; live triage promoted a clearly p06 memory (offline/network rule) as project=`""`, which explains the p06-offline-design harness miss and falsifies the current "all 3 failures are budget-contention" claim | fixed | Claude | 2026-04-12 | 39d73e9 |
| R7 | Codex | P2 | src/atocore/memory/service.py:448-459 | Query ranking is overlap-count only, so broad overview memories can tie exact low-confidence memories and win on confidence; p06-firmware-interface is not just budget pressure, it also exposes a weak lexical scorer | open | Claude | 2026-04-12 | |
| R8 | Codex | P2 | tests/test_extractor_llm.py:1-7 | LLM extractor tests stop at parser/failure contracts; there is no automated coverage for the script-only persistence/review path that produced the 16 promoted memories, including project-scope preservation | open | Claude | 2026-04-12 | |
| R9 | Codex | P2 | src/atocore/memory/extractor_llm.py:258-259 | The R6 fallback only repairs empty project output. A wrong non-empty model project still overrides the interaction's known scope, so project attribution is improved but not yet trust-preserving. | open | Claude | 2026-04-12 | |
| R10 | Codex | P2 | docs/master-plan-status.md:31-33 | "Phase 8 - OpenClaw Integration" is fair as a baseline milestone, but not as a "primary" integration claim. `t420-openclaw/atocore.py` currently covers a narrow read-oriented subset (13 request shapes vs 32 API routes) plus fail-open health, while memory/interactions/admin write paths remain out of surface. | open | Claude | 2026-04-12 | |
## Recent Decisions
@@ -145,6 +148,9 @@ One branch `codex/extractor-eval-loop` for Day 1-5, a second `codex/retrieval-ha
## Session Log
- **2026-04-12 Codex (audit branch `codex/audit-2026-04-12-final`)** audited `c5bad99..e2895b5` against origin/main, live Dalidou, and the OpenClaw client script. Live state checked: build `39d73e9`, harness reproducible at **16/18 PASS**, active memories **36**, and `t420-openclaw/atocore.py health` fails open correctly with `fail_open=true`. Spot-checks of Wave 2 project-state entries matched their cited vault docs. Updated R5-R8 status reality (R6 fixed by `39d73e9`), added R9-R10, and corrected Orientation `main_tip` to `e2895b5` because the ledger had drifted behind origin/main. Note: live Dalidou is still on `39d73e9`, so branch-truth and deploy-truth are not the same yet.
- **2026-04-12 Claude** Wave 2 trusted operational ingestion + codex audit response. Read 6 vault docs, created 8 new Trusted Project State entries (p04 +2, p05 +3, p06 +3). Fixed R6 (project fallback in LLM extractor) per codex audit. Fixed misscoped p06 offline memory on live Dalidou. Merged codex/audit-2026-04-12. Switched default LLM model from haiku to sonnet. Harness 15/18 -> 16/18. Tests 278 -> 280. main_tip 146f2e4 -> 39d73e9.
- **2026-04-12 Codex (audit branch `codex/audit-2026-04-12`)** audited `c5bad99..146f2e4` against code, live Dalidou, and the 36 active memories. Confirmed: `claude -p` invocation is not shell-injection-prone (`subprocess.run(args)` with no shell), off-host backup wiring matches the ledger, and R1 remains unresolved in practice. Added R5-R8. Corrected Orientation `main_tip` (`146f2e4`, not `5c69f77`) and tightened the harness note: p06-firmware-interface is a ranking-tie issue, p06-offline-design comes from a project-scope miss in live triage, and p06-tailscale is retrieved-chunk bleed rather than memory-band budget contention.
- **2026-04-12 Claude** `06792d8..5c69f77` Day 5-8 close. Documented extractor scope (5 in-scope, 6 out-of-scope categories). Expanded harness from 6 to 18 fixtures (p04 +1, p05 +1, p06 +7, adversarial +2). Per-entry memory cap at 250 chars fixed 1 of 4 budget-contention failures. Final harness: 15/18 PASS. Mini-phase complete. Before/after: rule extractor 0% recall -> LLM 100%; harness 6/6 -> 15/18; active memories 20 -> 36.
- **2026-04-12 Claude** `330ecfb..06792d8` (merged eval-loop branch + triage). Day 1-4 of the mini-phase completed in one session. Day 2 baseline: rule extractor 0% recall, 5 distinct miss classes. Day 4 gate cleared: LLM extractor (claude -p haiku, OAuth) hit 100% recall, 2.55 yield/interaction. Refactored from anthropic SDK to subprocess after "no API key" rule. First live triage: 51 candidates -> 16 promoted, 35 rejected. Active memories 20->36. p06-polisher went from 2 to 16 memories (firmware/telemetry architecture set). POST /memory now accepts status field. Test count 264->278.

View File

@@ -27,7 +27,18 @@ read-only additive mode.
### Partial
- Phase 4 - Identity / Preferences
- Phase 8 - OpenClaw Integration
### Baseline Complete
- Phase 8 - OpenClaw Integration. As of 2026-04-12 the T420 OpenClaw
helper (`t420-openclaw/atocore.py`) is verified end-to-end against
live Dalidou: health check, auto-context with project detection,
Trusted Project State surfacing, project-memory band, fail-open on
unreachable host. Tested from both the development machine and the
T420 via SSH. The helper covers 15 of the 33 API endpoints — the
excluded endpoints (memory management, interactions, backup) are
correctly scoped to the operator client (`scripts/atocore_client.py`)
per the read-only additive integration model.
### Baseline Complete

View File

@@ -27,7 +27,7 @@ Configuration:
- Requires the ``claude`` CLI on PATH (``claude --version`` should work).
- ``ATOCORE_LLM_EXTRACTOR_MODEL`` overrides the model alias (default
``haiku``).
``sonnet``).
- ``ATOCORE_LLM_EXTRACTOR_TIMEOUT_S`` overrides the per-call timeout
(default 90 seconds — first invocation is slow because Node.js
startup plus OAuth check is non-trivial).
@@ -65,7 +65,7 @@ from atocore.observability.logger import get_logger
log = get_logger("extractor_llm")
LLM_EXTRACTOR_VERSION = "llm-0.2.0"
DEFAULT_MODEL = os.environ.get("ATOCORE_LLM_EXTRACTOR_MODEL", "haiku")
DEFAULT_MODEL = os.environ.get("ATOCORE_LLM_EXTRACTOR_MODEL", "sonnet")
DEFAULT_TIMEOUT_S = float(os.environ.get("ATOCORE_LLM_EXTRACTOR_TIMEOUT_S", "90"))
MAX_RESPONSE_CHARS = 8000
MAX_PROMPT_CHARS = 2000
@@ -256,6 +256,8 @@ def _parse_candidates(raw_output: str, interaction: Interaction) -> list[MemoryC
mem_type = str(item.get("type") or "").strip().lower()
content = str(item.get("content") or "").strip()
project = str(item.get("project") or "").strip()
if not project and interaction.project:
project = interaction.project
confidence_raw = item.get("confidence", 0.5)
if mem_type not in MEMORY_TYPES:
continue

View File

@@ -97,6 +97,25 @@ def test_parser_tags_version_and_rule():
assert result[0].source_interaction_id == "test-id"
def test_parser_falls_back_to_interaction_project():
"""R6: when the model returns empty project but the interaction
has one, the candidate should inherit the interaction's project."""
raw = '[{"type": "project", "content": "machine works offline"}]'
interaction = _make_interaction()
interaction.project = "p06-polisher"
result = _parse_candidates(raw, interaction)
assert result[0].project == "p06-polisher"
def test_parser_keeps_model_project_when_provided():
"""Model-supplied project takes precedence over interaction."""
raw = '[{"type": "project", "content": "x", "project": "p04-gigabit"}]'
interaction = _make_interaction()
interaction.project = "p06-polisher"
result = _parse_candidates(raw, interaction)
assert result[0].project == "p04-gigabit"
def test_missing_cli_returns_empty(monkeypatch):
"""If ``claude`` is not on PATH the extractor returns empty, never raises."""
monkeypatch.setattr(extractor_llm, "_cli_available", lambda: False)