Commit Graph

188 Commits

Author SHA1 Message Date
22a37a7241 docs(ledger): record V1-0 closure + prod backfill + R14 residual
- Orientation bumped: live_sha 775960c -> 2712c5d, test_count
  533 -> 547, main_tip 999788b -> 2712c5d
- Recent Decisions: V1-0 approved, merged, deployed, 31-row prod
  backfill with zero-remainder follow-up dry-run
- Open Review Findings: new R14 (P2) — POST /entities/{id}/promote
  returns 500 instead of 400 on the new ValueError from legacy
  no-provenance candidate promotion. Non-blocking, follow-up tidy
- Session Log: full V1-0 closure narrative + V1-A gate status

V1-A (Q-001 subsystem-scoped variant + Q-6 killer-correctness
integration) holds until the soak window ends ~2026-04-26 and the
100-active-memory density target is hit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-22 15:17:06 -04:00
2712c5d2d0 feat(engineering): enforce V1-0 write invariants 2026-04-22 14:59:17 -04:00
9ab5b3c9d8 docs(planning): V1 Completion Plan — Codex sign-off (third round)
Codex's third-round audit closed the remaining five open questions
with concrete file:line resolutions, patched inline in the plan:

- F-7 (P1): graduation stack is partially built — graduated_to_entity_id
  at database.py:143-146, graduated memory status, promote preserves
  original at service.py:354-356, tests at test_engineering_v1_phase5.py.
  Gaps: missing direct POST /memory/{id}/graduate route; spec's
  knowledge -> Fact mismatches ontology (no fact type). Reconcile to
  parameter or similar. V1-E 2 days -> 3-4 days.

- Q-5 / V1-D (P2): renderer reads wall-clock in _footer at mirror.py:320.
  Fix is injecting regenerated timestamp + checksum as renderer inputs,
  sorting DB iteration, removing dict ordering deps. Render code must
  not call wall-clock directly.

- project vs project_id (P3): doc note only, no storage rename.

- Total estimate: 17.5-19.5 focused days (calendar buffer on top).

- Release notes must NOT canonize "Minions" as a V2 name. Use neutral
  "queued background processing / async workers" wording.

Sign-off from Codex: "with those edits, I'd sign off on the five
questions. The only non-architectural uncertainty left in the plan is
scheduling discipline against the current Now list; that does not
block V1-0 once the soak window and memory-density gate clear."

Plan frozen. V1-0 starts after pipeline soak (~2026-04-26) and the
100-active-memory density gate clear.

Co-Authored-By: Codex <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-22 14:24:43 -04:00
44724c81ab docs(planning): V1 Completion Plan revised per Codex file-level audit
Three findings folded in, all with exact file:line refs from Codex:

- F-1 downgraded from done to partial. Entity dataclass at
  service.py:67 and entities table missing extractor_version and
  canonical_home fields per engineering-v1-acceptance.md:45. V1-0
  scope now adds both via additive migration + doc note that
  project is the project_id per "fields equivalent to" wording.

- F-2 replaced guesses with ground truth per-query status:
  9 of 20 v1-required queries done, 1 partial (Q-001 needs
  subsystem-scoped variant), 10 missing. V1-A scope shrank to
  Q-001 shape fix + Q-6 integration. V1-C closes the 8 net-new
  queries; Q-020 deferred to V1-D (mirror).

- F-5 reframed. Generic conflicts + conflict_members schema
  already present at database.py:190, no migration needed.
  Divergence is detector body (per-type dispatch needs
  generalization) + routes (/admin/conflicts/* needs
  /conflicts/* alias). V1-F scope is detector + routes only.

Totals revised: 16.5-17.5 days, ~60 tests.

Three of Codex's eight open questions now resolved. Remaining:
F-7 graduation depth, mirror determinism, project naming,
velocity calibration, minions-as-V2 naming.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-22 14:09:50 -04:00
ce3a87857e docs(planning): V1 Completion Plan + gbrain-plan rejection record
- docs/decisions/2026-04-22-gbrain-plan-rejection.md: record of
  gbrain-inspired "Phase 8 Minions + typed edges" plan rejection.
  Three high findings from Codex verified against cited architecture
  docs (ontology V1 predicate set, canonical entity contract,
  master-plan-status Now list sequencing).

- docs/plans/engineering-v1-completion-plan.md: seven-phase plan
  for finishing Engineering V1 against engineering-v1-acceptance.md.
  V1-0 (write-time invariants: F-8 provenance + F-5 hooks + F-1
  audit) as hard prerequisite per Codex first-round review. Per-
  criterion gap audit against each F/Q/O/D acceptance item with
  code:line references. Explicit collision points with the Now
  list; schedule shifted ~4 weeks to avoid pipeline-soak window.
  Awaiting Codex file-level audit.

- DEV-LEDGER.md: Recent Decisions + Session Log entries covering
  both the rejection and the revised plan.

No code changes. Docs + ledger only.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-22 13:58:10 -04:00
e147ab2abd feat(wiki): [[wikilinks]] with redlinks + cross-project resolver (Issue B)
Last P2 from Antoine's "daily-usable" sprint. Entities referenced via
[[Name]] in descriptions or mirror markdown now render as:

- live wikilink if the name matches an entity in the same project
- live cross-project link with "(in project X)" scope indicator if the
  only match is in another project
- red italic redlink pointing at /wiki/new?name=... otherwise

Clicking a redlink opens a pre-filled "create this entity" form that
POSTs to /v1/entities and redirects to the new entity's page.

- engineering/wiki.py: _wikilink_transform + _resolve_wikilink,
  applied in render_project (pre-markdown) and render_entity
  (description body). render_new_entity_form for the create page.
  CSS for .wikilink / .wikilink-cross / .redlink / .new-entity-form
- api/routes.py: GET /wiki/new?name&project
- tests/test_wikilinks.py: 12 tests including the spec regression
  (A references [[B]] -> redlink; create B -> link becomes live)
- DEV-LEDGER.md: session log + test_count 521 -> 533

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-22 09:15:14 -04:00
b94f9dff56 feat(api): PATCH /entities/{id} + /v1/engineering/* aliases
PATCH lets users edit an active entity's description, properties,
confidence, and source_refs without cloning — closes the duplicate-trap
half-fixed by /invalidate + /supersede. Issue D just adds the
/engineering/* query surface to the /v1 allowlist.

- engineering/service.py: update_entity supports description replace,
  properties shallow merge with null-delete semantics, confidence
  0..1 bounds check, source_refs dedup-append. Writes audit row
- api/routes.py: PATCH /entities/{id} with EntityPatchRequest
- main.py: engineering/* query endpoints aliased under /v1 (Issue D)
- tests/test_patch_entity.py: 12 tests (merge, null-delete, bounds,
  dedup, 404, audit, v1 alias)
- DEV-LEDGER.md: session log + test_count 509 -> 521

Forbidden fields via PATCH (by design): entity_type, project, name,
status. Use supersede+create or the dedicated status endpoints.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-22 09:02:13 -04:00
081c058f77 feat(api): invalidate + supersede for active entities and memories (Issue E)
Public retraction path so mistakes can be corrected without SQL. Unblocks
the correction workflows that the live AKC p05 session exposed.

- engineering/service.py: invalidate_active_entity returns (ok, code)
  with codes invalidated/already_invalid/not_active/not_found for clean
  HTTP mapping. supersede_entity gains superseded_by + auto-creates the
  supersedes relationship (new SUPERSEDES old), rejects self-supersede
- memory/service.py: invalidate_memory/supersede_memory accept reason
  string that lands in audit note
- api/routes.py: POST /entities/{id}/invalidate, /supersede;
  POST /memory/{id}/invalidate, /supersede (all 4 behind /v1 aliases)
- tests/test_invalidate_supersede.py: 15 tests (idempotency, 404/409,
  supersede relationship auto-creation, self-supersede rejection,
  missing-replacement rejection, v1 alias presence)
- DEV-LEDGER.md: session log + test_count 494 -> 509

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-21 21:56:24 -04:00
069d155585 feat(assets): binary asset store + artifact entity + wiki evidence (Issue F)
Wires visual evidence into the knowledge graph. Images, PDFs, and CAD
exports can now be uploaded, deduped by SHA-256, thumbnailed, linked to
entities via EVIDENCED_BY, and rendered inline on wiki pages. Unblocks
AKC uploading voice-session screenshots alongside extracted entities.

- assets/ module: store_asset (hash dedup + MIME allowlist + 20 MB cap),
  get_asset_binary, get_thumbnail (Pillow, on-disk cache under
  .thumbnails/<size>/), list_orphan_assets, invalidate_asset
- models/database.py: new `assets` table + indexes
- engineering/service.py: `artifact` added to ENTITY_TYPES
- api/routes.py: POST /assets (multipart), GET /assets/{id},
  /assets/{id}/thumbnail, /assets/{id}/meta, /admin/assets/orphans,
  DELETE /assets/{id} (409 if still referenced),
  GET /entities/{id}/evidence (EVIDENCED_BY artifacts with asset meta)
- main.py: all new paths aliased under /v1
- engineering/wiki.py: entity pages render EVIDENCED_BY → artifact as a
  "Visual evidence" thumbnail strip; artifact pages render the full
  image + caption + capture_context
- deploy/dalidou/docker-compose.yml: bind-mount ${ATOCORE_ASSETS_DIR}
- config.py: assets_dir + assets_max_upload_bytes settings
- requirements.txt + pyproject.toml: python-multipart, Pillow>=10.0.0
- tests/test_assets.py: 16 tests (dedup, cap, thumbnail cache, orphan
  detection, invalidate gating, API upload/fetch, evidence, v1 aliases,
  wiki rendering)
- DEV-LEDGER.md: session log + cleanup note + test_count 478 -> 494

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-21 21:46:52 -04:00
b1a3dd071e feat(entities): inbox + cross-project (project="") support (Issue C)
Makes `inbox` a reserved pseudo-project and `project=""` a first-class
cross-project bucket. Unblocks AKC capturing pre-project leads/quotes
and cross-project facts (materials, vendors) that don't fit a single
registered project.

- projects/registry.py: INBOX_PROJECT/GLOBAL_PROJECT constants,
  is_reserved_project(), register/update guards, resolve_project_name
  passthrough for "inbox"
- engineering/service.py: get_entities scoping rules (inbox-only,
  global-only, real+global default, scope_only=true strict).
  promote_entity accepts target_project to retarget on promote
- api/routes.py: GET /entities gains scope_only; POST /entities accepts
  project=null as ""; POST /entities/{id}/promote accepts
  {target_project, note}
- engineering/wiki.py: homepage shows "Inbox & Global" cards with live
  counts linking to scoped lists
- tests/test_inbox_crossproject.py: 15 tests (reserved enforcement,
  scoping rules, API shape, promote retargeting)
- DEV-LEDGER.md: session log, test_count 463 -> 478

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-21 20:17:32 -04:00
5fbd7e6094 feat(api): /v1 alias router for stable external contract (Issue A)
Mounts an explicit allowlist of public handlers under /v1 alongside the
existing unversioned paths. External clients (AKC, OpenClaw, future
tools) should target /v1; internal callers (hooks, wiki, admin UI) keep
working unchanged. Breaking schema changes will bump the prefix to /v2.

- src/atocore/main.py: _V1_PUBLIC_PATHS allowlist + second router
- tests/test_v1_aliases.py: parity + OpenAPI presence (5 tests)
- README.md: API versioning section
- DEV-LEDGER.md: session log, test_count 459 -> 463

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-21 20:04:46 -04:00
90001c1956 fix(7A): host-side memory_dedup.py must stay stdlib-only
Broke the dedup-watcher cron when I wrote memory_dedup.py in session
7A: imported atocore.memory.similarity, which transitively pulls
sentence-transformers + pydantic_settings onto host Python that
intentionally doesn't have them. Every UI-triggered + cron dedup scan
since 7A deployed was silently crashing with ModuleNotFoundError
(visible only in /home/papa/atocore-logs/dedup-ondemand-*.log).

I even documented this architecture rule in atocore.memory._llm_prompt
('This module MUST stay stdlib-only') then violated it one session
later. Shame.

Real fix — matches the extractor pattern:
- New endpoint POST /admin/memory/dedup-cluster on the server: takes
  {project, similarity_threshold, max_clusters}, runs the embedding +
  transitive-clustering inside the container where
  sentence-transformers lives, returns cluster shape.
- scripts/memory_dedup.py now pure stdlib: pulls clusters via HTTP,
  LLM-drafts merges via claude CLI, POSTs proposals back. No atocore
  imports beyond the stdlib-only _dedup_prompt shared module.
- Regression test pins the rule: test_memory_dedup_script_is_stdlib_only
  snapshots sys.modules before/after importing the script and asserts
  no non-allowed atocore modules were pulled.

Also: similarity.py + cluster_by_threshold stay server-side, still
covered by the same tests that used to live in the host tier-helper
section.

Tests 459 → 458 (-1 via rewrite of obsolete host-tier helper tests,
+2 for the new stdlib-only regression + endpoint shape tests).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-21 16:18:00 -04:00
6a2471d509 fix: persist +x bit on deploy scripts + hook scripts
Git on Windows was stripping the executable bit every time a script
got edited, which broke the dedup-watcher cron (~100s of 'Permission
denied' entries in dedup-watcher.log since 7A deploy) and silently
disabled the auto-triage-watcher, batch-extract, graduation-watcher,
and hourly-extract cadences whenever they were touched from Windows.

Used `git update-index --chmod=+x` to store the bits in the index so
subsequent deploys preserve them regardless of the editor platform.

No functional changes; the scripts themselves are unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-21 15:39:26 -04:00
83b4d78cb7 docs(ledger): session log for Phase 7A.1/7C/7D/7I + UI refresh + capture surface scope 2026-04-19 12:14:14 -04:00
9c91d778d9 feat: Claude Code context injection (UserPromptSubmit hook)
Closes the asymmetry the user surfaced: before this, Claude Code
captured every turn (Stop hook) but retrieval only happened when
Claude chose to call atocore_context (opt-in MCP tool). OpenClaw had
both sides covered after 7I; Claude Code did not.

Now symmetric. Every Claude Code prompt is auto-sent to
/context/build and the returned pack is prepended via
hookSpecificOutput.additionalContext — same as what OpenClaw's
before_agent_start hook now does.

- deploy/hooks/inject_context.py — UserPromptSubmit hook. Fail-open
  (always exit 0). Skips short/XML prompts. 5s timeout. Project
  inference mirrors capture_stop.py cwd→slug table. Kill switch:
  ATOCORE_CONTEXT_DISABLED=1.
- ~/.claude/settings.json registered the hook (local config, not
  committed; copy-paste snippet in docs/capture-surfaces.md).
- Removed /wiki/capture from topnav. Endpoint still exists but the
  page is now labeled "fallback only" with a warning banner. The
  sanctioned surfaces are Claude Code + OpenClaw; manual paste is
  explicitly not the design.
- docs/capture-surfaces.md — scope statement: two surfaces, nothing
  else. Anthropic API polling explicitly prohibited.

Tests: +8 for inject_context.py (exit 0 on all failure modes, kill
switch, short prompt filter, XML filter, bad stdin, mock-server
success shape, project inference from cwd). Updated 2 wiki tests
for the topnav change. 450 → 459.

Verified live with real AtoCore: injected 2979 chars of atocore
project context on a cwd-matched prompt.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-19 12:01:41 -04:00
6e43cc7383 feat: Phase 7I + UI refresh (capture form, memory/domain/activity pages, topnav)
Closes three gaps the user surfaced: (1) OpenClaw agents run blind
without AtoCore context, (2) mobile/desktop chats can't be captured
at all, (3) wiki UI hadn't kept up with backend capabilities.

Phase 7I — OpenClaw two-way bridge
- Plugin now calls /context/build on before_agent_start and prepends
  the context pack to event.prompt, so whatever LLM runs underneath
  (sonnet, opus, codex, local model) answers grounded in AtoCore
  knowledge. Captured prompt stays the user's original text; fail-open
  with a 5s timeout. Config-gated via injectContext flag.
- Plugin version 0.0.0 → 0.2.0; README rewritten.

UI refresh
- /wiki/capture — paste-to-ingest form for Claude Desktop / web / mobile
  / ChatGPT / other. Goes through normal /interactions pipeline with
  client="claude-desktop|claude-web|claude-mobile|chatgpt|other".
  Fixes the rotovap/mushroom-on-phone gap.
- /wiki/memories/{id} (Phase 7E) — full memory detail: content, status,
  confidence, refs, valid_until, domain_tags (clickable to domain
  pages), project link, source chunk, graduated-to-entity link, full
  audit trail, related-by-tag neighbors.
- /wiki/domains/{tag} (Phase 7F) — cross-project view: all active
  memories with the given tag grouped by project, sorted by count.
  Case-insensitive, whitespace-tolerant. Also surfaces graduated
  entities carrying the tag.
- /wiki/activity — autonomous-activity timeline feed. Summary chips
  by action (created/promoted/merged/superseded/decayed/canonicalized)
  and by actor (auto-dedup-tier1, auto-dedup-tier2, confidence-decay,
  phase10-auto-promote, transient-to-durable, tag-canon, human-triage).
  Answers "what has the brain been doing while I was away?"
- Home refresh: persistent topnav (Home · Activity · Capture · Triage
  · Dashboard), "What the brain is doing" snippet above project cards
  showing recent autonomous-actor counts, link to full activity.

Tests: +10 (capture page, memory detail + 404, domain cross-project +
empty + tag normalization, activity feed + groupings, home topnav,
superseded-source detail after merge). 440 → 450.

Known next: capture-browser extension for Claude.ai web (bigger
project, deferred); voice/mobile relay (adjacent).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-19 10:14:15 -04:00
877b97ec78 feat: Phase 7C — tag canonicalization (autonomous, weekly)
LLM proposes alias→canonical mappings for domain_tags; confidence >= 0.8
auto-apply, below goes to human triage. Protects project identifiers
(p04, p05, p06, atocore, apm, etc.) from ever being canonicalized
since they're their own namespace, not concepts.

Problem solved: tag drift fragments retrieval. "fw" vs "firmware" vs
"firmware-control" all mean the same thing, but cross-cutting queries
that filter by tag only hit one variant. Weekly canonicalization pass
keeps the tag graph clean.

- Schema: tag_aliases table (pending | approved | rejected)
- atocore.memory._tag_canon_prompt (stdlib-only, protected project tokens)
- service: get_tag_distribution, apply_tag_alias (atomic per-memory,
  dedupes if both alias + canonical present), create / approve / reject
  proposal lifecycle, per-memory audit rows with action="tag_canonicalized"
- scripts/canonicalize_tags.py: host-side detector, autonomous by default,
  --no-auto-approve kill switch
- 6 API endpoints under /admin/tags/* (distribution, list, propose,
  apply, approve/{id}, reject/{id})
- Step B4 in batch-extract.sh (Sundays only — weekly cadence)
- 26 new tests (prompt parser, normalizer protections, distribution
  counting, rewrite atomicity, dedup, audit, lifecycle). 414 → 440.

Design: aggressive protection of project tokens because a false
canonicalization (p04 → p04-gigabit, or vice versa) would scramble
cross-project filtering. Err toward preservation; the alias only
applies if the model is very confident AND both strings appear in
the current distribution.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-19 09:41:02 -04:00
e840ef4be3 feat: Phase 7D — confidence decay on unreferenced cold memories
Daily job multiplies confidence by 0.97 (~2-month half-life) for
active memories with reference_count=0 AND idle > 30 days. Below
0.3 → auto-supersede with audit. Reversible via reinforcement
(which already bumps confidence back up).

Rationale: stale memories currently rank equal to fresh ones in
retrieval. Without decay, the brain accumulates obsolete facts
that compete with fresh knowledge for context-pack slots. With
decay, memories earn their longevity via reference.

- decay_unreferenced_memories() in service.py (stdlib-only, no cron
  infra needed)
- POST /admin/memory/decay-run endpoint
- Nightly Step F4 in batch-extract.sh
- Exempt: reinforced (refcount > 0), graduated, superseded, invalid
- Audit row per supersession ("decayed below floor, no references"),
  actor="confidence-decay". Per-decay rows skipped (chatty, no
  human value — status change is the meaningful signal).
- Configurable via env: ATOCORE_DECAY_* (exposed through endpoint body)

Tests: +13 (basic decay, reinforcement protection, supersede at floor,
audit trail, graduated/superseded exemption, reinforcement reversibility,
threshold tuning, parameter validation, cross-run stacking).
401 → 414.

Next in Phase 7: 7C tag canonicalization (weekly), then 7B contradiction
detection.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-18 16:50:20 -04:00
56d5df0ab4 feat: Phase 7A.1 — autonomous merge tiering (sonnet → opus → human)
Dedup detector now merges high-confidence duplicates silently instead
of piling every proposal into a human triage queue. Matches the 3-tier
escalation pattern that auto_triage already uses.

Tiering decision per cluster:
  TIER-1 auto-approve: sonnet confidence >= 0.8 AND min_pairwise_sim >= 0.92
                       AND all sources share project+type → auto-merge silently
                       (actor="auto-dedup-tier1" in audit log)
  TIER-2 escalation:   sonnet 0.5-0.8 conf OR sim 0.85-0.92 → opus second opinion.
                       Opus confirms with conf >= 0.8 → auto-merge (actor="auto-dedup-tier2").
                       Opus overrides (reject) → skip silently.
                       Opus low conf → human triage with opus's refined draft.
  HUMAN triage:        Only the genuinely ambiguous land in /admin/triage.

Env-tunable thresholds:
  ATOCORE_DEDUP_AUTO_APPROVE_CONF (0.8)
  ATOCORE_DEDUP_AUTO_APPROVE_SIM (0.92)
  ATOCORE_DEDUP_TIER2_MIN_CONF (0.5)
  ATOCORE_DEDUP_TIER2_MIN_SIM (0.85)
  ATOCORE_DEDUP_TIER2_MODEL (opus)

New flag --no-auto-approve for kill-switch testing (everything → human queue).

Tests: +6 (tier-2 prompt content, same_bucket edges, min_pairwise_similarity
on identical + transitive clusters). 395 → 401.

Rationale: user asked for autonomous behavior — "this needs to be intelligent,
I don't want to manually triage stuff". Matches the consolidation principle:
never discard details, but let the brain tidy up on its own for the easy cases.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-18 15:46:26 -04:00
028d4c3594 feat: Phase 7A — semantic memory dedup ("sleep cycle" V1)
New table memory_merge_candidates + service functions to cluster
near-duplicate active memories within (project, memory_type) buckets,
draft a unified content via LLM, and merge on human approval. Source
memories become superseded (never deleted); merged memory carries
union of tags, max of confidence, sum of reference_count.

- schema migration for memory_merge_candidates
- atocore.memory.similarity: cosine + transitive clustering
- atocore.memory._dedup_prompt: stdlib-only LLM prompt preserving every specific
- service: merge_memories / create_merge_candidate / get_merge_candidates / reject_merge_candidate
- scripts/memory_dedup.py: host-side detector (HTTP-only, idempotent)
- 5 API endpoints under /admin/memory/merge-candidates* + /admin/memory/dedup-scan
- triage UI: purple "🔗 Merge Candidates" section + "🔗 Scan for duplicates" bar
- batch-extract.sh Step B3 (0.90 daily, 0.85 Sundays)
- deploy/dalidou/dedup-watcher.sh for UI-triggered scans
- 21 new tests (374 → 395)
- docs/PHASE-7-MEMORY-CONSOLIDATION.md covering 7A-7H roadmap

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-18 10:30:49 -04:00
9f262a21b0 feat: extractor llm-0.6.0 — bolder unknown-project tagging
User observation: APM work was captured + extracted, but candidates got
tagged project=atocore or left blank instead of project=apm. Reason:
the prompt said 'Unknown project names — still tag them' but was too
terse; sonnet hedged toward registered matches rather than proposing
new slugs.

Fix: explicit guidance in the system prompt for when to propose an
unregistered project name vs when to stick with a registered one.

New instructions:
- When a memory is clearly ABOUT a named tool/product/project/system
  not in the known list, use a slugified version as the project tag
  ('apm' for 'Atomaste Part Manager'). The Living Taxonomy detector
  (Phase 6 C.1) scans these and surfaces for one-click registration
  once ≥3 memories accumulate with that tag.
- Exception: if a memory merely USES an unknown tool but is about a
  registered project ('p04 parts missing materials in APM'), tag with
  the registered project and mention the tool in content.

This closes the loop on the Phase 6 detector: extractor now produces
taggable data for it, detector surfaces, user registers with one click.

Version bump: llm-0.5.0 → llm-0.6.0.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-18 08:31:09 -04:00
7863ab3825 feat: hourly extract+triage cron — close the 24h latency gap
User observation that triggered this: 'AtoCore was meant to remember
and triage by its own, not me specifically asking to remember things'.
Correct — the system IS capturing autonomously (Stop hook + OpenClaw
plugin), but extraction was nightly-only. So 'I talked about APM
today' didn't show up in memories until the next 03:00 UTC cron run.

Fix: split the lightweight extraction + triage into a new hourly cron.
The heavy nightly (backup, rsync, OpenClaw import, synthesis, harness,
integrity, emerging detector) stays at 03:00 UTC — no reason to run
those hourly.

hourly-extract.sh does ONLY:
- Step A: batch_llm_extract_live.py (limit 50, ~1h window)
- Step B: auto_triage.py (3-tier, max_batches=3)

Lock file prevents overlap on rate-limit retries.

After this lands: latency from 'you told me X' to 'X is an active
memory' drops from ~24h to ~1h (plus the ~5min it takes for
extraction + triage to complete on a typical <20 interactions/hour).

The 'atocore_remember' MCP tool stays as an escape hatch for
conversations that happen outside captured channels (Claude Desktop
web, phone), NOT as the primary capture path. The primary path is
automatic: Claude Code / OpenClaw captures → hourly extract → 3-tier
triage → active memory.

Install cron entry manually:
  0 * * * * /srv/storage/atocore/app/deploy/dalidou/hourly-extract.sh \
      >> /home/papa/atocore-logs/hourly-extract.log 2>&1

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-18 08:24:49 -04:00
3ba49e92a9 fix: detector uses HTTP-only (host lacks atocore deps)
Same pattern as integrity_check.py — the host-side Python doesn't
have pydantic_settings. Refactor detect_emerging.py to talk to the
container via HTTP instead of importing atocore.memory.service.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-18 08:15:50 -04:00
02055e8db3 feat: Phase 6 — Living Taxonomy + Universal Capture
Closes two real-use gaps:
1. "APM tool" gap: work done outside Claude Code (desktop, web, phone,
   other machine) was invisible to AtoCore.
2. Project discovery gap: manual JSON-file edits required to promote
   an emerging theme to a first-class project.

B — atocore_remember MCP tool (scripts/atocore_mcp.py):
- New MCP tool for universal capture from any MCP-aware client
  (Claude Desktop, Code, Cursor, Zed, Windsurf, etc.)
- Accepts content (required) + memory_type/project/confidence/
  valid_until/domain_tags (all optional with sensible defaults)
- Creates a candidate memory, goes through the existing 3-tier triage
  (no bypass — the quality gate catches noise)
- Detailed tool description guides Claude on when to invoke: "remember
  this", "save that for later", "don't lose this fact"
- Total tools exposed by MCP server: 14 → 15

C.1 Emerging-concepts detector (scripts/detect_emerging.py):
- Nightly scan of active + candidate memories for:
  * Unregistered project names with ≥3 memory occurrences
  * Top 20 domain_tags by frequency (emerging categories)
  * Active memories with reference_count ≥ 5 + valid_until set
    (reinforced transients — candidates for extension)
- Writes findings to atocore/proposals/* project state entries
- Emits "warning" alert via Phase 4 framework the FIRST time a new
  project crosses the 5-memory alert threshold (avoids spam)
- Configurable via env vars: ATOCORE_EMERGING_PROJECT_MIN (default 3),
  ATOCORE_EMERGING_ALERT_THRESHOLD (default 5), TOP_TAGS_LIMIT (20)

C.2 Registration surface (src/atocore/api/routes.py + wiki.py):
- POST /admin/projects/register-emerging — one-click register with
  sensible defaults (ingest_roots auto-filled with
  vault:incoming/projects/<id>/ convention). Clears the proposal
  from the dashboard list on success.
- Dashboard /admin/dashboard: new "proposals" section with
  unregistered_projects + emerging_categories + reinforced_transients.
- Wiki homepage: "📋 Emerging" section rendering each unregistered
  project as a card with count + 2 sample memory previews + inline
  "📌 Register as project" button that calls the endpoint via fetch,
  reloads the page on success.

C.3 Transient-to-durable extension
(src/atocore/memory/service.py + API + cron):
- New extend_reinforced_valid_until() function — scans active memories
  with valid_until in the next 30 days and reference_count ≥ 5.
  Extends expiry by 90 days. If reference_count ≥ 10, clears expiry
  entirely (makes permanent). Writes audit rows via the Phase 4
  memory_audit framework with actor="transient-to-durable".
- POST /admin/memory/extend-reinforced — API wrapper for cron.
- Matches the user's intuition: "something transient becomes important
  if you keep coming back to it".

Nightly cron (deploy/dalidou/batch-extract.sh):
- Step F2: detect_emerging.py (after F pipeline summary)
- Step F3: /admin/memory/extend-reinforced (before integrity check)
- Both fail-open; errors don't break the pipeline.

Tests: 366 → 374 (+8 for Phase 6):
- 6 tests for extend_reinforced_valid_until covering:
  extension path, permanent path, skip far-future, skip low-refs,
  skip permanent memories, audit row write
- 2 smoke tests for the detector (imports cleanly, handles empty DB)
- MCP tool changes don't need new tests — the wrapper is pure passthrough

Design decisions documented in plan file:
- atocore_remember deliberately doesn't bypass triage (quality gate)
- Detector is passive (surfaces proposals) not active (auto-registers)
- Sensible ingest-root defaults ("vault:incoming/projects/<id>/")
  so registration is one-click with no file-path thinking
- Extension adds 90 days rather than clearing expiry (gradual
  permanence earned through sustained reinforcement)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-18 08:08:55 -04:00
cc68839306 fix: OpenClaw plugin filters cron-initiated agent runs
OpenClaw scheduled tasks (DXF email watcher, calendar reminder pings)
fire agent sessions with prompts that begin '[cron:<id> ...]'. These
were all getting captured as AtoCore interactions — 45 out of 50
recent interactions today were cron noise, not real user turns.

Filter at the plugin level: before_agent_start ignores any prompt
starting with '[cron:'. The gateway has been restarted with the
updated plugin.

Impact: graduation, triage, and context pipelines stop seeing noise
from OpenClaw's own internal automation. Only real user turns (via
chat channels) feed the brain going forward.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-17 11:09:44 -04:00
45196f352f fix: force UTF-8 on MCP stdio for Windows compatibility
Windows Python defaults stdout to cp1252. Any non-ASCII char in tool
responses (emojis, ≥, →, etc.) crashes the MCP server with a
UnicodeEncodeError. Explicitly reconfigure stdin/stdout/stderr to
UTF-8 at startup. No-op on Linux/macOS.

Noticed when Claude Code called atocore_search and atocore_memory_list
and both crashed on  / ≥ characters that came back in the response.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-17 11:08:24 -04:00
d456d3c86a fix: local json import in graduation request/status handlers
NameError on /admin/graduation/request — routes.py doesn't import json
at module scope. Added local 'import json as _json' in both graduation
handlers matching the pattern used elsewhere in this file.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-17 09:53:17 -04:00
0dfecb3c14 feat: one-click memory graduation button + host watcher
Closes the graduation UX loop: no more SSH required to populate the
entity graph from memories. Click button → host watcher picks up
→ graduation runs → entity candidates appear in the same triage UI.

New API endpoints (src/atocore/api/routes.py):
- POST /admin/graduation/request: takes {project, limit}, writes flag
  to project_state. Host watcher picks up within 2 min.
- GET /admin/graduation/status: returns requested/running/last_result
  fields for UI polling.

Triage UI (src/atocore/engineering/triage_ui.py):
- Graduation bar with:
  - 🎓 Graduate memories button
  - Project selector populated from registry (or "all projects")
  - Limit number input (default 30, max 200)
  - Status message area
- Poll every 10s until is_running=false, then auto-reload the page to
  show new entity candidates in the Entity section below
- Graduation bar appears on both populated and empty triage page
  states so you can kick off graduation from either

Host watcher (deploy/dalidou/graduation-watcher.sh):
- Mirrors auto-triage-watcher.sh pattern: poll, lock, clear flag,
  run, record result, unlock
- Parses {project, limit} JSON from the flag payload
- Runs graduate_memories.py with those args
- Records graduation_running/started/finished/last_result in project
  state for the UI to display
- Lock file prevents concurrent runs

Install on host (one-time, via cron):
  */2 * * * * /srv/storage/atocore/app/deploy/dalidou/graduation-watcher.sh \
    >> /home/papa/atocore-logs/graduation-watcher.log 2>&1

This completes the Phase 5 self-service loop: queue triage happens
autonomously via the 3-tier escalation (shipped in 3ca1972); entity
graph population happens autonomously via a button click. No shell
required for daily use.

Tests: 366 passing (no new tests — UI + shell are integration-level).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-17 09:45:12 -04:00
3ca19724a5 feat: 3-tier triage escalation + project validation + enriched context
Addresses the triage-quality problems the user observed:
- Candidates getting wrong project/product attribution
- Stale facts promoted as if still true
- "Hard to decide" items reaching human queue without real value

Solution: let sonnet handle the easy 80%, escalate borderline cases
to opus, auto-discard (or flag) what two models can't resolve.
Plus enrich the context the triage model sees so it can catch
misattribution, contradictions, and temporal drift earlier.

THE 3-TIER FLOW (scripts/auto_triage.py):

Tier 1: sonnet (fast, cheap)
  - confidence >= 0.8 + clear verdict → PROMOTE or REJECT (done)
  - otherwise → escalate to tier 2

Tier 2: opus (smarter, sees tier-1 verdict + reasoning)
  - second opinion with explicit "sonnet said X, resolve the uncertainty"
  - confidence >= 0.8 → PROMOTE or REJECT with note="[opus]"
  - still uncertain → tier 3

Tier 3: configurable (default discard)
  - ATOCORE_TRIAGE_TIER3=discard (default): auto-reject with
    "two models couldn't decide" reason
  - ATOCORE_TRIAGE_TIER3=human: leave in queue for /admin/triage

Configuration via env vars:
  ATOCORE_TRIAGE_MODEL_TIER1  (default sonnet)
  ATOCORE_TRIAGE_MODEL_TIER2  (default opus)
  ATOCORE_TRIAGE_TIER3        (default discard)
  ATOCORE_TRIAGE_ESCALATION_THRESHOLD (default 0.75)
  ATOCORE_TRIAGE_TIER2_TIMEOUT_S (default 120 — opus is slower)

ENRICHED CONTEXT shown to the triage model (both tiers):
- List of registered project ids so misattribution is detectable
- Trusted project state entries (ground truth, higher trust than memories)
- Top 30 active memories for the claimed project (was 20)
- Tier 2 additionally sees tier 1's verdict + reason

PROJECT MISATTRIBUTION DETECTION:
- Triage prompt asks the model to output "suggested_project" when it
  detects the claimed project is wrong but the content clearly belongs
  to a registered one
- Main loop auto-applies the fix via PUT /memory/{id} (which canonicalizes
  through the registry)
- Misattribution is the #1 pollution source — this catches it upstream

TEMPORAL AGGRESSIVENESS:
- Prompt upgraded: "be aggressive with valid_until for anything that
  reads like 'current state' or 'this week'. When in doubt, 2-4 week
  expiry rather than null."
- Stale facts decay automatically via Phase 3's expiry filter

CONFIDENCE GRADING (new in prompt):
- 0.9+: crystal clear durable fact or clear noise
- 0.75-0.9: confident but not cryptographic-certain
- 0.6-0.75: borderline — WILL escalate
- <0.6: genuinely ambiguous — human or discard

Tests: 356 → 366 (10 new, all in test_triage_escalation.py):
- High-confidence tier-1 promote/reject → no tier-2 call
- Low-confidence tier-1 → tier-2 escalates → decides
- needs_human always escalates regardless of confidence
- tier-2 uncertain → discard by default
- tier-2 uncertain → human when configured
- dry-run skips all API calls
- suggested_project flag surfaces + gets printed
- parse_verdict captures suggested_project

Runtime behavior unchanged for the clear cases (sonnet still handles
them). The 20-30% of candidates that currently land as needs_human
will now route through opus, and only the genuinely stuck get a human
(or discard) action.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-17 09:09:58 -04:00
3316ff99f9 feat: Phase 5F/5G/5H — graduation, conflicts, MCP engineering tools
The population move + the safety net + the universal consumer hookup,
all shipped together. This is where the engineering graph becomes
genuinely useful against the real 262-memory corpus.

5F: Memory → Entity graduation (THE population move)
- src/atocore/engineering/_graduation_prompt.py: stdlib-only shared
  prompt module mirroring _llm_prompt.py pattern (container + host
  use same system prompt, no drift)
- scripts/graduate_memories.py: host-side batch driver that asks
  claude-p "does this memory describe a typed entity?" and creates
  entity candidates with source_refs pointing back to the memory
- promote_entity() now scans source_refs for memory:* prefix; if
  found, flips source memory to status='graduated' with
  graduated_to_entity_id forward pointer + writes memory_audit row
- GET /admin/graduation/stats exposes graduation rate for dashboard

5G: Sync conflict detection on entity promote
- src/atocore/engineering/conflicts.py: detect_conflicts_for_entity()
  runs on every active promote. V1 checks 3 slot kinds narrowly to
  avoid false positives:
  * component.material (multiple USES_MATERIAL edges)
  * component.part_of (multiple PART_OF edges)
  * requirement.name (duplicate active Requirements in same project)
- Conflicts + members persist via the tables built in 5A
- Fires a "warning" alert via Phase 4 framework
- Deduplicates: same (slot_kind, slot_key) won't get a new row
- resolve_conflict(action="dismiss|supersede_others|no_action"):
  supersede_others marks non-winner members as status='superseded'
- GET /admin/conflicts + POST /admin/conflicts/{id}/resolve

5H: MCP + context pack integration
- scripts/atocore_mcp.py: 7 new engineering tools exposed to every
  MCP-aware client (Claude Desktop, Claude Code, Cursor, Zed):
  * atocore_engineering_map (Q-001/004 system tree)
  * atocore_engineering_gaps (Q-006/009/011 killer queries — THE
    director's question surfaced as a built-in tool)
  * atocore_engineering_requirements_for_component (Q-005)
  * atocore_engineering_decisions (Q-008)
  * atocore_engineering_changes (Q-013 — reads entity audit log)
  * atocore_engineering_impact (Q-016 BFS downstream)
  * atocore_engineering_evidence (Q-017 inbound provenance)
- MCP tools total: 14 (7 memory/state/health + 7 engineering)
- context/builder.py _build_engineering_context now appends a compact
  gaps summary ("Gaps: N orphan reqs, M risky decisions, K unsupported
  claims") so every project-scoped LLM call sees "what we're missing"

Tests: 341 → 356 (15 new):
- 5F: graduation prompt parses positive/negative decisions, rejects
  unknown entity types, tolerates markdown fences; promote_entity
  marks source memory graduated with forward pointer; entity without
  memory refs promotes cleanly
- 5G: component.material + component.part_of + requirement.name
  conflicts detected; clean component triggers nothing; dedup works;
  supersede_others resolution marks losers; dismiss leaves both
  active; end-to-end promote triggers detection
- 5H: graduation user message includes project + type + content

No regressions across the 341 prior tests. The MCP server now answers
"which p05 requirements aren't satisfied?" directly from any Claude
session — no user prompt engineering, no context hacks.

Next to kick off from user: run graduation script on Dalidou to
populate the graph from 262 existing memories:
  ssh papa@dalidou 'cd /srv/storage/atocore/app && PYTHONPATH=src \
    python3 scripts/graduate_memories.py --project p05-interferometer --limit 30 --dry-run'

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-17 07:53:03 -04:00
53b71639ad feat: Phase 5B-5D — 10 canonical engineering queries + triage UI
The graph becomes useful. Before this commit, entities sat in the DB
as data with no narrative. After: the director can ask "what am I
forgetting?" and get a structured answer in milliseconds.

New module (src/atocore/engineering/queries.py, 360 lines):

Structure queries (Q-001/004/005/008/013):
- system_map(project): full subsystem → component tree + orphans +
  materials joined per component
- decisions_affecting(project, subsystem_id?): decisions linked via
  AFFECTED_BY_DECISION, scoped to a subsystem or whole project
- requirements_for(component_id): Q-005 forward trace
- recent_changes(project, since, limit): Q-013 via memory_audit join
  (reuses the Phase 4 audit infrastructure — entity_kind='entity')

The 3 killer queries (the real value):
- orphan_requirements(project): requirements with NO inbound SATISFIES
  edge. "What do I claim the system must do that nothing actually
  claims to handle?" Q-006.
- risky_decisions(project): decisions whose BASED_ON_ASSUMPTION edge
  points to an assumption with status in ('superseded','invalid') OR
  properties.flagged=True. Finds cascading risk from shaky premises. Q-009.
- unsupported_claims(project): ValidationClaim entities with no inbound
  SUPPORTS edge — asserted but no Result to back them. Q-011.
- all_gaps(project): runs all three in one call for dashboards.

History + impact (Q-016/017):
- impact_analysis(entity_id, max_depth=3): BFS over outbound edges.
  "What's downstream of this if I change it?"
- evidence_chain(entity_id): inbound SUPPORTS/EVIDENCED_BY/DESCRIBED_BY/
  VALIDATED_BY/ANALYZED_BY. "How do I know this is true?"

API (src/atocore/api/routes.py) exposes 10 endpoints:
- GET /engineering/projects/{p}/systems
- GET /engineering/decisions?project=&subsystem=
- GET /engineering/components/{id}/requirements
- GET /engineering/changes?project=&since=&limit=
- GET /engineering/gaps/orphan-requirements?project=
- GET /engineering/gaps/risky-decisions?project=
- GET /engineering/gaps/unsupported-claims?project=
- GET /engineering/gaps?project=  (combined)
- GET /engineering/impact?entity=&max_depth=
- GET /engineering/evidence?entity=

Mirror integration (src/atocore/engineering/mirror.py):
- New _gaps_section() renders at top of every project page
- If any gap non-empty: shows up-to-10 per category with names + context
- Clean project: " No gaps detected" — signals everything is traced

Triage UI (src/atocore/engineering/triage_ui.py):
- /admin/triage now shows BOTH memory candidates AND entity candidates
- Entity cards: name, type, project, confidence, source provenance,
  Promote/Reject buttons, link to wiki entity page
- Entity promote/reject via fetch to /entities/{id}/promote|reject
- One triage UI for the whole pipeline — consistent muscle memory

Tests: 326 → 341 (15 new, all in test_engineering_queries.py):
- System map structure + orphan detection + material joins
- Killer queries: positive + negative cases (empty when clean)
- Decisions query: project-wide and subsystem-scoped
- Impact analysis walks outbound BFS
- Evidence chain walks inbound provenance

No regressions. All 10 daily queries from the plan are now live and
answering real questions against the graph.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-17 07:18:46 -04:00
07664bd743 feat: Phase 5A — Engineering V1 foundation
First slice of the Engineering V1 sprint. Lays the schema + lifecycle
plumbing so the 10 canonical queries, memory graduation, and conflict
detection can land cleanly on top.

Schema (src/atocore/models/database.py):
- conflicts + conflict_members tables per conflict-model.md (with 5
  indexes on status/project/slot/members)
- memory_audit.entity_kind discriminator — same audit table serves
  both memories ("memory") and entities ("entity"); unified history
  without duplicating infrastructure
- memories.graduated_to_entity_id forward pointer for graduated
  memories (M → E transition preserves the memory as historical
  pointer)

Memory (src/atocore/memory/service.py):
- MEMORY_STATUSES gains "graduated" — memory-entity graduation flow
  ready to wire in Phase 5F

Engineering service (src/atocore/engineering/service.py):
- RELATIONSHIP_TYPES organized into 4 families per ontology-v1.md:
  + Structural: contains, part_of, interfaces_with
  + Intent: satisfies, constrained_by, affected_by_decision,
    based_on_assumption (new), supersedes
  + Validation: analyzed_by, validated_by, supports (new),
    conflicts_with (new), depends_on
  + Provenance: described_by, updated_by_session (new),
    evidenced_by (new), summarized_in (new)
- create_entity + create_relationship now call resolve_project_name()
  on write (canonicalization contract per doc)
- Both accept actor= parameter for audit provenance
- _audit_entity() helper uses shared memory_audit table with
  entity_kind="entity" — one observability layer for everything
- promote_entity / reject_entity_candidate / supersede_entity —
  mirror the memory lifecycle exactly (same pattern, same naming)
- get_entity_audit() reads from the shared table filtered by
  entity_kind

API (src/atocore/api/routes.py):
- POST /entities/{id}/promote (candidate → active)
- POST /entities/{id}/reject (candidate → invalid)
- GET /entities/{id}/audit (full history for one entity)
- POST /entities passes actor="api-http" through

Tests: 317 → 326 (9 new):
- test_entity_project_canonicalization (p04 → p04-gigabit)
- test_promote_entity_candidate_to_active
- test_reject_entity_candidate
- test_promote_active_entity_noop (only candidates promote)
- test_entity_audit_log_captures_lifecycle (before/after snapshots)
- test_new_relationship_types_available (6 new types present)
- test_conflicts_tables_exist
- test_memory_audit_has_entity_kind
- test_graduated_status_accepted

What's next (5B-5I, deferred): entity triage UI tab, core structure
queries, the 3 killer queries, memory graduation script, conflict
detection, MCP + context pack integration. See plan file.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-17 07:01:28 -04:00
bb46e21c9b fix: integrity check runs in container (host lacks deps)
scripts/integrity_check.py now POSTs to /admin/integrity-check
instead of importing atocore directly. The actual scan lives in
the container where DB access + deps are available. Host-side
cron just triggers and logs the result.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-16 22:01:43 -04:00
88f2f7c4e1 feat: Phase 4 V1 — Robustness Hardening
Adds the observability + safety layer that turns AtoCore from
"works until something silently breaks" into "every mutation is
traceable, drift is detected, failures raise alerts."

1. Audit log (memory_audit table):
   - New table with id, memory_id, action, actor, before/after JSON,
     note, timestamp; 3 indexes for memory_id/timestamp/action
   - _audit_memory() helper called from every mutation:
     create_memory, update_memory, promote_memory,
     reject_candidate_memory, invalidate_memory, supersede_memory,
     reinforce_memory, auto_promote_reinforced, expire_stale_candidates
   - Action verb auto-selected: promoted/rejected/invalidated/
     superseded/updated based on state transition
   - "actor" threaded through: api-http, human-triage, phase10-auto-
     promote, candidate-expiry, reinforcement, etc.
   - Fail-open: audit write failure logs but never breaks the mutation
   - GET /memory/{id}/audit: full history for one memory
   - GET /admin/audit/recent: last 50 mutations across the system

2. Alerts framework (src/atocore/observability/alerts.py):
   - emit_alert(severity, title, message, context) fans out to:
     - structlog logger (always)
     - ~/atocore-logs/alerts.log append (configurable via
       ATOCORE_ALERT_LOG)
     - project_state atocore/alert/last_{severity} (dashboard surface)
     - ATOCORE_ALERT_WEBHOOK POST if set (auto-detects Discord webhook
       format for nice embeds; generic JSON otherwise)
   - Every sink fail-open — one failure doesn't prevent the others
   - Pipeline alert step in nightly cron: harness < 85% → warning;
     candidate queue > 200 → warning

3. Integrity checks (scripts/integrity_check.py):
   - Nightly scan for drift:
     - Memories → missing source_chunk_id references
     - Duplicate active memories (same type+content+project)
     - project_state → missing projects
     - Orphaned source_chunks (no parent document)
   - Results persisted to atocore/status/integrity_check_result
   - Any finding emits a warning alert
   - Added as Step G in deploy/dalidou/batch-extract.sh nightly cron

4. Dashboard surfaces it all:
   - integrity (findings + details)
   - alerts (last info/warning/critical per severity)
   - recent_audit (last 10 mutations with actor + action + preview)

Tests: 308 → 317 (9 new):
  - test_audit_create_logs_entry
  - test_audit_promote_logs_entry
  - test_audit_reject_logs_entry
  - test_audit_update_captures_before_after
  - test_audit_reinforce_logs_entry
  - test_recent_audit_returns_cross_memory_entries
  - test_emit_alert_writes_log_file
  - test_emit_alert_invalid_severity_falls_back_to_info
  - test_emit_alert_fails_open_on_log_write_error

Deferred: formal migration framework with rollback (current additive
pattern is fine for V1); memory detail wiki page with audit view
(quick follow-up).

To enable Discord alerts: set ATOCORE_ALERT_WEBHOOK to a Discord
webhook URL in Dalidou's environment. Default = log-only.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-16 21:54:10 -04:00
bfa7dba4de feat: Phase 3 V1 — Auto-Organization (domain_tags + valid_until)
Adds structural metadata that the LLM triage was already implicitly
reasoning about ("stale snapshot" → reject). Phase 3 captures that
reasoning as fields so it can DRIVE retrieval, not just rejection.

Schema (src/atocore/models/database.py):
- domain_tags TEXT DEFAULT '[]'  JSON array of lowercase topic keywords
- valid_until DATETIME            ISO date; null = permanent
- idx_memories_valid_until index for efficient expiry queries

Memory service (src/atocore/memory/service.py):
- Memory dataclass gains domain_tags + valid_until
- create_memory, update_memory accept/persist both
- _row_to_memory safely reads both (JSON-decode + null handling)
- _normalize_tags helper: lowercase, dedup, strip, cap at 10
- get_memories_for_context filters expired (valid_until < today UTC)
- _rank_memories_for_query adds tag-boost: memories whose domain_tags
  appear as substrings in query text rank higher (tertiary key after
  content-overlap density + absolute overlap, before confidence)

LLM extractor (_llm_prompt.py → llm-0.5.0):
- SYSTEM_PROMPT documents domain_tags (2-5 keywords) + valid_until
  (time-bounded facts get expiry dates; durable facts stay null)
- normalize_candidate_item parses both fields from model output with
  graceful fallback for string/null/missing

LLM triage (scripts/auto_triage.py):
- TRIAGE_SYSTEM_PROMPT documents same two fields
- parse_verdict extracts them from verdict JSON
- On promote: PUT /memory/{id} with tags + valid_until BEFORE
  POST /memory/{id}/promote, so active memories carry them

API (src/atocore/api/routes.py):
- MemoryCreateRequest: adds domain_tags, valid_until
- MemoryUpdateRequest: adds domain_tags, valid_until, memory_type
- GET /memory response exposes domain_tags + valid_until + created_at

Triage UI (src/atocore/engineering/triage_ui.py):
- Renders existing tags as colored badges
- Adds inline text field for tags (comma-separated) + date picker for
  valid_until on every candidate card
- Save&Promote button persists edits via PUT then promotes
- Plain Promote (and Y shortcut) also saves tags/expiry if edited

Wiki (src/atocore/engineering/wiki.py):
- Search now matches memory content OR domain_tags
- Search results render tags as clickable badges linking to
  /wiki/search?q=<tag> for cross-project navigation
- valid_until shown as amber "valid until YYYY-MM-DD" hint

Tests: 303 → 308 (5 new for Phase 3 behavior):
- test_create_memory_with_tags_and_valid_until
- test_create_memory_normalizes_tags
- test_update_memory_sets_tags_and_valid_until
- test_get_memories_for_context_excludes_expired
- test_context_builder_tag_boost_orders_results

Deferred (explicitly): temporal_scope enum, source_refs memory graph,
HDBSCAN clustering, memory detail wiki page, backfill of existing
actives. See docs/MASTER-BRAIN-PLAN.md.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-16 21:37:01 -04:00
271ee25d99 feat: on-demand auto-triage from web UI
Adds an "Auto-process queue" button to /admin/triage that lets the
user kick off a full LLM triage pass without SSH. Bridges the gap
between web UI (in container) and claude CLI (host-only).

Architecture:
- UI button POSTs to /admin/triage/request-drain
- Endpoint writes atocore/config/auto_triage_requested_at flag
- Host-side watcher cron (every 2 min) checks for the flag
- When found: clears flag, acquires lock, runs auto_triage.py,
  records progress via atocore/status/* entries
- UI polls /admin/triage/drain-status every 10s to show progress,
  auto-reloads when done

Safety:
- Lock file prevents concurrent runs on host
- Flag cleared before run so duplicate clicks queue at most one re-run
- Fail-open: watcher errors just log, don't break anything
- Status endpoint stays read-only

Installation on host (one-time):
  */2 * * * * /srv/storage/atocore/app/deploy/dalidou/auto-triage-watcher.sh \
    >> /home/papa/atocore-logs/auto-triage-watcher.log 2>&1

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-16 21:05:30 -04:00
d8b370fd0a feat: /admin/triage web UI + auto-drain loop
Makes human triage sustainable. Before: command-line-only review,
auto-triage stopped after 100 candidates/run. Now:

1. Web UI at /admin/triage
   - Lists all pending candidates with inline promote/reject/edit
   - Edit content in-place before promoting (PUT /memory/{id})
   - Change type via dropdown
   - Keyboard shortcuts: Y=promote, N=reject, E=edit, S=scroll-next
   - Cards fade out after action, queue count updates live
   - Zero JS framework — vanilla fetch + event delegation

2. auto_triage.py drains queue
   - Loops up to 20 batches (default) of 100 candidates each
   - Tracks seen IDs so needs_human items don't reprocess
   - Exits cleanly when queue empty
   - Nightly cron naturally drains everything

3. Dashboard + wiki surface triage queue
   - Dashboard /admin/dashboard: new "triage" section with pending
     count + /admin/triage URL + warning/notice severity levels
   - Wiki homepage: prominent callout "N candidates awaiting triage —
     review now" linking to /admin/triage, styled with triage-warning
     (>50) or triage-notice (>20) CSS

Pattern: human intervenes only when AI can't decide. The UI makes
that intervention fast (20 candidates in 60 seconds). Nightly
auto-triage drains the queue so the human queue stays bounded.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-16 20:28:56 -04:00
86637f8eee feat: universal LLM consumption (Phase 1 complete)
Completes the Phase 1 master brain keystone: every LLM interaction
across the ecosystem now pulls context from AtoCore automatically.

Three adapters, one HTTP backend:

1. OpenClaw plugin pull (handler.js):
   - Added before_prompt_build hook that calls /context/build and
     injects the pack via prependContext
   - Existing capture hooks (before_agent_start + llm_output)
     unchanged
   - 6s context timeout, fail-open on AtoCore unreachable
   - Deployed to T420, gateway restarted, "7 plugins loaded"

2. atocore-proxy (scripts/atocore_proxy.py):
   - Stdlib-only OpenAI-compatible HTTP middleware
   - Drop-in layer for Codex, Ollama, LiteLLM, any OpenAI-compat client
   - Intercepts /chat/completions: extracts query, pulls context,
     injects as system message, forwards to upstream, captures back
   - Fail-open: AtoCore down = passthrough without injection
   - Configurable via env: UPSTREAM, PORT, CLIENT_LABEL, INJECT, CAPTURE

3. (from prior commit c49363f) atocore-mcp:
   - stdio MCP server, stdlib Python, 7 tools exposed
   - Registered in Claude Code: "✓ Connected"

Plus quick win:
- Project synthesis moved from Sunday-only to daily cron so wiki /
  mirror pages stay fresh (Step C in batch-extract.sh). Lint stays
  weekly.

Plus docs:
- docs/universal-consumption.md: configuration guide for all 3 adapters
  with registration/env-var tables and verification checklist

Plus housekeeping:
- .gitignore: add .mypy_cache/

Tests: 303/303 passing.

This closes the consumption gap: the reinforcement feedback loop
can now actually work (memories get injected → get referenced →
reinforcement fires → auto-promotion). Every Claude, OpenClaw,
Codex, or Ollama session is automatically AtoCore-grounded.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-16 20:14:25 -04:00
c49363fccc feat: atocore-mcp server for universal LLM consumption (Phase 1)
Stdlib-only Python stdio MCP server that wraps the AtoCore HTTP
API. Makes AtoCore available as built-in tools to every MCP-aware
client (Claude Desktop, Claude Code, Cursor, Zed, Windsurf).

7 tools exposed:
- atocore_context: full context pack (state + memories + chunks)
- atocore_search: semantic retrieval with scores + sources
- atocore_memory_list: filter active memories by project/type
- atocore_memory_create: propose a candidate memory
- atocore_project_state: query Trusted Project State by category
- atocore_projects: list registered projects + aliases
- atocore_health: service status check

Design choices:
- stdlib only (no mcp SDK dep) — AtoCore philosophy
- Thin HTTP passthrough — zero business logic, zero drift risk
- Fail-open: AtoCore unreachable returns graceful error, not crash
- Protocol MCP 2024-11-05 compatible

Registered in Claude Code: `claude mcp add atocore -- python ...`
Verified: ✓ Connected, all 7 tools exposed, context/search/state
return live data from Dalidou (sha=775960c8, vectors=33253).

This is the keystone for master brain vision: every Claude session
now has AtoCore available as built-in capability without the user
or agent having to remember to invoke it.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-16 20:08:20 -04:00
33a6c61ca6 feat: daily backup to Windows main computer via pull-based scp
Third backup tier (after Dalidou local + T420 off-host): pull-based
backup to the user's Windows main computer.

- scripts/windows/atocore-backup-pull.ps1: PowerShell script using
  built-in OpenSSH scp. Fail-open: exits cleanly if Dalidou
  unreachable (e.g., laptop on the road). Pulls whole snapshots dir
  (~45MB, bounded by Dalidou's retention policy).
- docs/windows-backup-setup.md: Task Scheduler setup (automated +
  manual). Runs daily 10:00 local, catches up missed days via
  StartWhenAvailable, retries 2x on failure.

Verified: pulled 3 snapshots (45MB) to
C:\Users\antoi\Documents\ATOCore_Backups\. Task "AtoCore Backup
Pull" registered in Task Scheduler, State: Ready.

Three independent backup tiers now: Dalidou local, T420 off-host,
user Windows machine. Any two can fail without data loss.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-16 20:04:00 -04:00
33a106732f docs: master brain plan — vision, universal consumption, roadmap
Documents the path from current AtoCore (capture-only, thin
knowledge) to master brain status (universal consumption, dense
knowledge, auto-organized, self-growing, flawless).

Key strategic decisions documented:
- HTTP API is the canonical truth; every client gets a thin adapter
- MCP is for Claude ecosystem; OpenClaw plugin + middleware proxy
  handle Codex/Ollama/others
- Three-tier integration: MCP server, OpenClaw plugin, generic proxy
- Phase 1 (keystone) = universal consumption at prompt time
- 7-phase roadmap over 8-10 weeks

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-16 19:55:19 -04:00
3011aa77da fix: retry + stderr capture + pacing in triage/extractor
Both scripts now:
- Retry up to 3x with 2s/4s exponential backoff on transient
  failures (rate limits, capacity spikes)
- Capture claude CLI stderr in the error message (200 char cap)
  instead of just the exit code — diagnostics actually useful now
- Sleep 0.5s between calls to avoid bursting the backend

Context: last batch run hit 100% failure in triage (every call
exit 1) after 40% failure in extraction. claude CLI worked fine
immediately after, so the failures were capacity/rate-limit
transients. With retry + pacing these batches should complete
cleanly now. 439 candidates are already in the queue waiting
for triage.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-16 16:29:20 -04:00
ba36a28453 docs: sprint documentation — ledger + master-plan sync
Updated DEV-LEDGER orientation with post-sprint state:
- live_sha 775960c, tests 303, harness 17/18 on live
- interactions 234 (192 claude-code + 38 openclaw)
- project_state_entries 110 across 6 projects
- nightly pipeline now includes auto-promote, harness, summary

Updated master-plan-status.md "What Is Real Today" to match
actual 2026-04-16 state. Phase 10 moved from "Next" to
operational. New "Now" priorities: observe pipeline, knowledge
density, multi-model triage, fix p04-constraints.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-16 14:08:19 -04:00
999788b790 chore: OpenClaw capture handler (llm_output) + ledger sync
- openclaw-plugins/atocore-capture/handler.js: simplified version
  using before_agent_start + llm_output hooks (survives gateway
  restarts). The production copy lives on T420 at
  /tmp/atocore-openclaw-capture-plugin/openclaw-plugins/atocore-capture/
- DEV-LEDGER: updated orientation (live_sha b687e7f, capture clients)
  and session log for 2026-04-16

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-16 14:04:40 -04:00
775960c8c8 feat: "Make It Actually Useful" sprint — observability + Phase 10
Pipeline observability:
- Retrieval harness runs nightly (Step E in batch-extract.sh)
- Pipeline summary persisted to project state after each run
  (pipeline_last_run, pipeline_summary, retrieval_harness_result)
- Dashboard enhanced: interaction total + by_client, pipeline health
  (last_run, hours_since, harness results, triage stats), dynamic
  project list from registry

Phase 10 — reinforcement-based auto-promotion:
- auto_promote_reinforced(): candidates with reference_count >= 3 and
  confidence >= 0.7 auto-graduate to active
- expire_stale_candidates(): candidates unreinforced for 14+ days
  auto-rejected to prevent unbounded queue growth
- Both wired into nightly cron (Step B2)
- Batch script: scripts/auto_promote_reinforced.py (--dry-run support)

Knowledge seeding:
- scripts/seed_project_state.py: 26 curated Trusted Project State
  entries across p04-gigabit, p05-interferometer, p06-polisher,
  atomizer-v2, abb-space, atocore (decisions, requirements, facts,
  contacts, milestones)

Tests: 299 → 303 (4 new Phase 10 tests)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-16 13:59:12 -04:00
b687e7fa6f feat(capture): wire project inference from cwd
Populate _PROJECT_PATH_MAP in capture_stop.py so Claude Code
interactions get tagged with the correct project at capture time
instead of relying on the nightly LLM extractor to guess from
content. Covers 6 vault PARA sub-projects (P04, P05, P11/P06,
P08, I01, I02) and 4 local code repos (ATOCore, Polisher-Sim,
Fullum-Interferometer, Atomizer-V2).

Also sync project-registry.json with live Dalidou (adds abb-space,
atomizer-v2, and p11/polisher-fullum aliases to p06-polisher).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-16 09:01:38 -04:00
4d4d5f437a test(harness): fix p06-tailscale false positive, 18/18 PASS
The fixture's expect_absent: "GigaBIT" was catching legitimate
semantic overlap, not retrieval bleed. The p06 ARCHITECTURE.md
Overview describes the Polisher Suite as built for the GigaBIT M1
mirror — it is what the polisher is for, so the word appears
correctly in p06 content. All retrieved sources for this prompt
were genuinely p06/shared paths; zero actual p04 chunks leaked.

Narrowed the assertion to expect_absent: "[Source: p04-gigabit/",
which tests the real invariant (no p04 source chunks retrieved
into p06 context) without the false positive.

No retrieval/ranking code change. Fixture-only fix.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-15 11:23:00 -04:00
5b114baa87 docs(ledger): deploy c2e7064 live; close R10 + R13
- R10 fixed: master-plan-status Phase 8 now disclaims "primary
  integration", reports current narrow surface (14 client shapes vs
  ~44 routes, read-heavy + project-state/ingest writes).
- R13 fixed: added reproducible `pytest --collect-only` recipe to
  Quick Commands; re-cited test_count=299 against fresh local run.
- Orientation bumped: live_sha and main_tip c2e7064.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-15 11:19:55 -04:00
c2e7064238 fix(extraction): R11 container 503 + R12 shared prompt module
R11: POST /admin/extract-batch with mode=llm now returns 503 when the
claude CLI is unavailable (was silently returning success with 0
candidates), with a message pointing at the host-side script. +2 tests.

R12: extracted SYSTEM_PROMPT + parse_llm_json_array +
normalize_candidate_item + build_user_message into stdlib-only
src/atocore/memory/_llm_prompt.py. Both the container extractor and
scripts/batch_llm_extract_live.py now import from it, eliminating the
prompt/parser drift risk.

Tests 297 -> 299.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-15 10:47:01 -04:00
dc9fdd3a38 chore(ledger): end-of-session sync (2026-04-14)
Reflects today's massive work: engineering layer + wiki + Karpathy
upgrades + OpenClaw importer + auto-detection. Active memories
47 -> 84. Ready for next session to pick up cold.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-14 11:24:25 -04:00