scripts/auto_triage.py: fetches candidate memories, asks a triage
model (claude -p, default sonnet) to classify each as promote /
reject / needs_human, and executes the verdict via the API.
Trust model:
- Auto-promote: model says promote AND confidence >= 0.8 AND
dedup-checked against existing active memories for the project
- Auto-reject: model says reject
- needs_human: everything else stays in queue for manual review
The triage model receives both the candidate content AND a summary
of existing active memories for the same project, so it can detect
duplicates and near-duplicates. The system prompt explicitly lists
the rejection categories learned from the first two manual triage
passes (stale snapshots, impl details, planned-not-implemented,
process rules that belong in ledger not memory).
deploy/dalidou/batch-extract.sh now runs extraction (Step A) then
auto-triage (Step B) in sequence. The nightly cron at 03:00 UTC
will run the full pipeline: backup → cleanup → rsync → extract →
triage. Only needs_human candidates reach the human.
Supports --dry-run for preview without executing.
Supports --model override for multi-model triage (e.g. opus for
higher-quality review, or a future Gemini/Ollama backend).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The claude CLI is installed on the Dalidou HOST but not inside
the Docker container. The /admin/extract-batch API endpoint with
mode=llm silently returned 0 candidates because
shutil.which('claude') was None inside the container.
Fix: extraction runs host-side via deploy/dalidou/batch-extract.sh
which calls scripts/batch_llm_extract_live.py with the host's
PYTHONPATH pointing at the repo's src/. The script:
- Fetches interactions from the API (GET /interactions?since=...)
- Runs extract_candidates_llm() locally (host has claude CLI)
- POSTs candidates back to the API (POST /memory, status=candidate)
- Tracks last-run timestamp via project state
The cron now calls the host-side script instead of the container
API endpoint for LLM mode. Rule-mode extraction in the container
still works via /admin/extract-batch.
The API endpoint retains the mode=llm option for environments
where claude IS inside the container (future Docker image with
claude CLI, or a different deployment model).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Step 4 added to the daily cron: POST /admin/extract-batch with
mode=llm, persist=true, limit=50. Runs after backup + cleanup +
rsync. Fail-open: extraction failure never blocks the backup.
Gated on ATOCORE_EXTRACT_BATCH=true (defaults to true). The
endpoint uses the last_extract_batch_run timestamp from project
state to auto-resume, so the cron doesn't need to track state.
curl --max-time 600 gives the LLM extractor up to 10 minutes
for the batch (50 interactions × ~20s each worst case = ~17 min,
but most will be no-ops if already extracted).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
deploy.sh sync-checkout was landing the file without an exec bit,
so the cron run hit 'Permission denied' until chmod +x was applied
manually on Dalidou. Persist the exec bit in the git index so
future deploys don't regress.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Reinforcement matcher now handles paragraph-length memories via a
dual-mode threshold: short memories keep the 70% overlap rule,
long memories (>15 stems) require 12 absolute overlaps AND 35%
fraction so organic paraphrase can still reinforce. Diagnosis:
every active memory stayed at reference_count=0 because 40-token
project summaries never hit 70% overlap on real responses.
- scripts/atocore_client.py gains batch-extract (fan out
/interactions/{id}/extract over recent interactions) and triage
(interactive promote/reject walker for the candidate queue),
matching the Phase 9 reflection-loop review flow without pulling
extraction into the capture hot path.
- deploy/dalidou/cron-backup.sh adds an optional off-host rsync step
gated on ATOCORE_BACKUP_RSYNC, fail-open when the target is offline
so a laptop being off at 03:00 UTC never reds the local backup.
- docs/next-steps.md records the retrieval-quality sweep: project
state surfaces, chunks are on-topic but broad, active memories
never reach the pack (reflection loop has no retrieval outlet yet).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- POST /admin/backup/cleanup — retention cleanup via API (dry-run by default)
- record_interaction() accepts extract=True to auto-extract candidate
memories from response text using the Phase 9C rule-based extractor
- POST /interactions accepts extract field to enable extraction on capture
- deploy/dalidou/cron-backup.sh — daily backup + cleanup for cron
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When deploy.sh itself changes in the commit being pulled, the bash
process is still running the OLD script from memory — git reset --hard
updated the file on disk but the in-memory instructions are stale.
This bit the 2026-04-09 Dalidou deploy: the old pre-build-sha Step 2
ran against fresh source, so the container started with
ATOCORE_BUILD_SHA="unknown" instead of the real commit. Manual
re-run fixed it, but the class of bug will re-emerge every time
deploy.sh itself changes.
Fix (Step 1.5):
- After git reset --hard, sha1 the running script ($0) and the
on-disk copy at $APP_DIR/deploy/dalidou/deploy.sh
- If they differ, export ATOCORE_DEPLOY_REEXECED=1 and exec into
the fresh copy so Step 2 onward runs under the new script
- The sentinel env var prevents recursion
- Skipped in dry-run mode, when $0 isn't readable, or when the
on-disk script doesn't exist yet
Docs (docs/dalidou-deployment.md):
- New "The deploy.sh self-update race" troubleshooting section
explaining the root cause, the Step 1.5 mechanism, what the log
output looks like, and how to opt out
Verified syntax and dry-run. 219/219 tests still passing.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Make /health report the precise git SHA the container was built from,
so 'is the live service current?' can be answered without ambiguity.
0.2.0 was too coarse to trust as a 'live is current' signal — many
commits share the same __version__.
Three layers:
1. /health endpoint (src/atocore/api/routes.py)
- Reads ATOCORE_BUILD_SHA, ATOCORE_BUILD_TIME, ATOCORE_BUILD_BRANCH
from environment, defaults to 'unknown'
- Reports them alongside existing code_version field
2. docker-compose.yml
- Forwards the three env vars from the host into the container
- Defaults to 'unknown' so direct `docker compose up` runs (without
deploy.sh) cleanly signal missing build provenance
3. deploy.sh
- Step 2 captures git SHA + UTC timestamp + branch and exports them
as env vars before `docker compose up -d --build`
- Step 6 reads /health post-deploy and compares the reported
build_sha against the freshly-built one. Mismatch exits non-zero
(exit code 6) with a remediation hint covering cached image,
env propagation, and concurrent restart cases
Tests (tests/test_api_storage.py):
- test_health_endpoint_reports_code_version_from_module
- test_health_endpoint_reports_build_metadata_from_env
- test_health_endpoint_reports_unknown_when_build_env_unset
Docs (docs/dalidou-deployment.md):
- Three-level drift detection table (code_version coarse,
build_sha precise, build_time/branch forensic)
- Canonical drift check script using LIVE_SHA vs EXPECTED_SHA
- Note that running deploy.sh is itself the simplest drift check
219/219 tests passing.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Dalidou Claude's second re-deploy (commit b492f5f) reported one
remaining friction point: the app dir was root-owned from the
previous manual-workaround deploy (when ALTER TABLE was run as
root to work around the schema init bug), so deploy.sh's git
fetch/reset hit a permission wall. They worked around it with
a one-shot docker run chown, but the script itself produced
cryptic git errors before that, so the fix wasn't obvious until
after the fact.
This commit adds a permission pre-flight check that runs BEFORE
any git operations and exits cleanly with an explicit remediation
message instead of letting git produce half-state on partial
failure.
The check:
1. Reads the current owner of the app dir via `stat -c '%U:%G'`
2. Reports the current user via `id -un` / `id -u:id -g`
3. Attempts to create a throwaway marker file in the app dir
4. If the marker write fails, prints three distinct remediation
commands covering the common environments:
a. sudo chown -R 1000:1000 $APP_DIR (if passwordless sudo)
b. sudo bash $0 (if running deploy.sh itself as root works)
c. docker run --rm -v $APP_DIR:/app alpine chown -R ...
(what Dalidou Claude actually did on 2026-04-08)
5. Exits with code 5 so CI / automation can distinguish "no
permission" from other deploy failures
Dry-run mode skips the check (nothing is mutated in dry-run).
A brief WARNING is also printed early if the app dir exists but
doesn't appear writable, before the fatal check — this gives
operators a heads-up even in the happy-path case.
Syntax check: bash -n passes.
Full suite: 216 passing (unchanged; no code changes to the app).
What this commit does NOT do
----------------------------
- Does NOT automatically fix permissions. chown needs root and
we don't want deploy.sh to escalate silently. The operator
runs one of the three remediation commands manually.
- Does NOT check permissions on nested files (like .git/config)
individually. The marker-file test on the app dir root is the
cheapest proxy that catches the common case (root-owned dir
tree after a previous sudo-based operation).
- Does NOT change behavior on first-time deploys where the app
dir doesn't exist yet. The check is gated on `-d $APP_DIR`.
Three issues Dalidou Claude surfaced during the first real deploy
of commit e877e5b to the live service (report from 2026-04-08).
Bug 1 was the critical one — a schema init ordering bug that would
have bitten every future upgrade from a pre-Phase-9 schema — and
the other two were usability traps around hostname resolution.
Bug 1 (CRITICAL): schema init ordering
--------------------------------------
src/atocore/models/database.py
SCHEMA_SQL contained CREATE INDEX statements that referenced
columns added later by _apply_migrations():
CREATE INDEX IF NOT EXISTS idx_memories_project ON memories(project);
CREATE INDEX IF NOT EXISTS idx_interactions_project_name ON interactions(project);
CREATE INDEX IF NOT EXISTS idx_interactions_session ON interactions(session_id);
On a FRESH install, CREATE TABLE IF NOT EXISTS creates the tables
with the Phase 9 shape (columns present), so the CREATE INDEX runs
cleanly and _apply_migrations is effectively a no-op.
On an UPGRADE from a pre-Phase-9 schema, CREATE TABLE IF NOT EXISTS
is a no-op (the tables already exist in the old shape), the columns
are NOT added yet, and the CREATE INDEX fails with
"OperationalError: no such column: project" before
_apply_migrations gets a chance to add the columns.
Dalidou Claude hit this exactly when redeploying from 0.1.0 to
0.2.0 — had to manually ALTER TABLE to add the Phase 9 columns
before the container could start.
The fix is to remove the Phase 9-column indexes from SCHEMA_SQL.
They already exist in _apply_migrations() AFTER the corresponding
ALTER TABLE, so they still get created on both fresh and upgrade
paths — just after the columns exist, not before.
Indexes still in SCHEMA_SQL (all safe — reference columns that
have existed since the first release):
- idx_chunks_document on source_chunks(document_id)
- idx_memories_type on memories(memory_type)
- idx_memories_status on memories(status)
- idx_interactions_project on interactions(project_id)
Indexes moved to _apply_migrations (already there — just no longer
duplicated in SCHEMA_SQL):
- idx_memories_project on memories(project)
- idx_interactions_project_name on interactions(project)
- idx_interactions_session on interactions(session_id)
- idx_interactions_created_at on interactions(created_at)
Regression test: tests/test_database.py
---------------------------------------
New test_init_db_upgrades_pre_phase9_schema_without_failing:
- Seeds the DB with the exact pre-Phase-9 shape (no project /
last_referenced_at / reference_count on memories; no project /
client / session_id / response / memories_used / chunks_used on
interactions)
- Calls init_db() — which used to raise OperationalError before
the fix
- Verifies all Phase 9 columns are present after the call
- Verifies the migration indexes exist
Before the fix this test would have failed with
"OperationalError: no such column: project" on the init_db call.
After the fix it passes. This locks the invariant "init_db is
safe on any legacy schema shape" so the bug can't silently come
back.
Full suite: 216 passing (was 215), 1 warning. The +1 is the new
regression test.
Bug 3 (usability): deploy.sh DNS default
----------------------------------------
deploy/dalidou/deploy.sh
ATOCORE_GIT_REMOTE defaulted to http://dalidou:3000/Antoine/ATOCore.git
which requires the "dalidou" hostname to resolve. On the Dalidou
host itself it didn't (no /etc/hosts entry for localhost alias),
so deploy.sh had to be run with the IP as a manual workaround.
Fix: default ATOCORE_GIT_REMOTE to http://127.0.0.1:3000/Antoine/ATOCore.git.
Loopback always works on the host running the script. Callers
from a remote host (e.g. running deploy.sh from a laptop against
the Dalidou LAN) set ATOCORE_GIT_REMOTE explicitly. The script
header's Environment Variables section documents this with an
explicit reference to the 2026-04-08 Dalidou deploy report so the
rationale isn't lost.
docs/dalidou-deployment.md gets a new "Troubleshooting hostname
resolution" subsection and a new example invocation showing how
to deploy from a remote host with an explicit ATOCORE_GIT_REMOTE
override.
Bug 2 (usability): atocore_client.py ATOCORE_BASE_URL documentation
-------------------------------------------------------------------
scripts/atocore_client.py
Same class of issue as bug 3. BASE_URL defaults to
http://dalidou:8100 which resolves fine from a remote caller
(laptop, T420/OpenClaw over Tailscale) but NOT from the Dalidou
host itself or from inside the atocore container. Dalidou Claude
saw the CLI return
{"status": "unavailable", "fail_open": true}
while direct curl to http://127.0.0.1:8100 worked.
The fix here is NOT to change the default (remote callers are
the common case and would break) but to DOCUMENT the override
clearly so the next operator knows what's happening:
- The script module docstring grew a new "Environment variables"
section covering ATOCORE_BASE_URL, ATOCORE_TIMEOUT_SECONDS,
ATOCORE_REFRESH_TIMEOUT_SECONDS, and ATOCORE_FAIL_OPEN, with
the explicit override example for on-host/in-container use
- It calls out the exact symptom (fail-open envelope when the
base URL doesn't resolve) so the diagnosis is obvious from
the error alone
- docs/dalidou-deployment.md troubleshooting section mirrors
this guidance so there's one place to look regardless of
whether the operator starts with the client help or the
deploy doc
What this commit does NOT do
----------------------------
- Does NOT change the default ATOCORE_BASE_URL. Doing that would
break the T420 OpenClaw helper and every remote caller who
currently relies on the hostname. Documentation is the right
fix for this case.
- Does NOT fix /etc/hosts on Dalidou. That's a host-level
configuration issue that the user can fix if they prefer
having the hostname resolve; the deploy.sh fix makes it
unnecessary regardless.
- Does NOT re-run the validation on Dalidou. The next step is
for the live service to pull this commit via deploy.sh (which
should now work without the IP workaround) and re-run the
Phase 9 loop test to confirm nothing regressed.
Dalidou Claude's validation run against the live service exposed a
structural gap: the deployment at /srv/storage/atocore/app has no
git connection, the running container was built from pre-Phase-9
source, and /health hardcoded 'version: 0.1.0' so drift is
invisible. Weeks of work have been shipping to Gitea but never
reaching the live service.
This commit fixes both the drift-invisibility problem and the
absence of an update workflow, so the next deploy to Dalidou can
go live cleanly and future drifts surface immediately.
Layer 1: deployment drift is now visible via /health
----------------------------------------------------
- src/atocore/__init__.py: __version__ bumped from 0.1.0 to 0.2.0
and documented as the source of truth for the deployed code
version, with a history block explaining when each bump happens
(API surface change, schema change, user-visible behavior change)
- src/atocore/main.py: FastAPI constructor now uses __version__
instead of the hardcoded '0.1.0' string, so the OpenAPI docs
reflect the actual code version
- src/atocore/api/routes.py: /health now reads from __version__
dynamically. Both the existing 'version' field and a new
'code_version' field report the same value for backwards compat.
A new docstring explains that comparing this to the main
branch's __version__ is the fastest way to detect drift.
- pyproject.toml: version bumped to 0.2.0 to stay in sync
The comparison is now:
curl /health -> "code_version": "0.2.0"
grep __version__ src/atocore/__init__.py -> "0.2.0"
If those differ, the deployment is stale. Concrete, unambiguous.
Layer 2: deploy.sh as the canonical update path
-----------------------------------------------
New file: deploy/dalidou/deploy.sh
One-shot bash script that handles both the first-time deploy
(where /srv/storage/atocore/app may not be a git repo yet) and
the ongoing update case. Steps:
1. If app dir is not a git checkout, back it up as
<dir>.pre-git-<utc-stamp> and re-clone from Gitea.
If it IS a checkout, fetch + reset --hard origin/<branch>.
2. Report the deployable commit SHA
3. Check that deploy/dalidou/.env exists (hard fail if missing
with a clear message pointing at .env.example)
4. docker compose up -d --build — rebuilds the image from
current source, restarts the container
5. Poll /health for up to 30 seconds; on failure, print the
last 50 lines of container logs and exit non-zero
6. Parse /health.code_version and compare to the __version__
in the freshly-pulled source. If they differ, exit non-zero
with a message suggesting docker compose down && up
7. On success, report commit + code_version + "health: ok"
Configurable via env vars:
- ATOCORE_APP_DIR (default /srv/storage/atocore/app)
- ATOCORE_GIT_REMOTE (default http://dalidou:3000/Antoine/ATOCore.git)
- ATOCORE_BRANCH (default main)
- ATOCORE_HEALTH_URL (default http://127.0.0.1:8100/health)
- ATOCORE_DEPLOY_DRY_RUN=1 for preview-only mode
Explicit non-goals documented in the script header:
- does not manage secrets (.env is the caller's responsibility)
- does not take a pre-deploy backup (call /admin/backup first
if you want one)
- does not roll back on failure (redeploy a known-good commit
to recover)
- does not touch the DB directly — schema migrations run at
service startup via the lifespan handler, and all existing
_apply_migrations ALTERs are idempotent ADD COLUMN operations
Layer 3: updated docs/dalidou-deployment.md
-------------------------------------------
- First-time deployment steps now explicitly say "git clone", not
"place the repository", so future first-time deploys don't end
up as static snapshots again
- New "Updating a running deployment" section covering deploy.sh
usage with all three modes (normal / branch override / dry-run)
- New "Deployment drift detection" section with the one-liner
comparison between /health code_version and the repo's
__version__
- New "Schema migrations on redeploy" section enumerating the
exact ALTER TABLE statements that run on a pre-0.2.0 -> 0.2.0
upgrade, confirming they are additive-only and safe, and
recommending a backup via /admin/backup before any redeploy
Full suite: 215 passing, 1 warning. No test was hardcoded to the
old version string, so the version bump was safe without test
changes.
What this commit does NOT do
----------------------------
- Does NOT execute the deploy on the live Dalidou instance. That
requires Dalidou access and is the next step. A ready-to-paste
prompt for Dalidou Claude will be provided separately.
- Does NOT add CI/CD, webhook-based auto-deploy, or reverse
proxy. Those remain in the 'deferred' section of the
deployment doc.
- Does NOT change the Dockerfile. The existing 'COPY source at
build time' pattern is what deploy.sh relies on — rebuilding
the image picks up new code.
- Does NOT modify the database schema. The Phase 9 migrations
that Dalidou's DB needs will be applied automatically on next
service startup via the existing _apply_migrations path.
