feat: Phase 7A — semantic memory dedup ("sleep cycle" V1)
New table memory_merge_candidates + service functions to cluster near-duplicate active memories within (project, memory_type) buckets, draft a unified content via LLM, and merge on human approval. Source memories become superseded (never deleted); merged memory carries union of tags, max of confidence, sum of reference_count. - schema migration for memory_merge_candidates - atocore.memory.similarity: cosine + transitive clustering - atocore.memory._dedup_prompt: stdlib-only LLM prompt preserving every specific - service: merge_memories / create_merge_candidate / get_merge_candidates / reject_merge_candidate - scripts/memory_dedup.py: host-side detector (HTTP-only, idempotent) - 5 API endpoints under /admin/memory/merge-candidates* + /admin/memory/dedup-scan - triage UI: purple "🔗 Merge Candidates" section + "🔗 Scan for duplicates" bar - batch-extract.sh Step B3 (0.90 daily, 0.85 Sundays) - deploy/dalidou/dedup-watcher.sh for UI-triggered scans - 21 new tests (374 → 395) - docs/PHASE-7-MEMORY-CONSOLIDATION.md covering 7A-7H roadmap Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -166,6 +166,26 @@ curl -sSf -X POST "$ATOCORE_URL/admin/memory/extend-reinforced" \
|
||||
log "WARN: extend-reinforced failed (non-blocking)"
|
||||
}
|
||||
|
||||
# Step B3: Memory dedup scan (Phase 7A)
|
||||
# Nightly at 0.90 (tight — only near-duplicates). Sundays run a deeper
|
||||
# pass at 0.85 to catch semantically-similar-but-differently-worded memories.
|
||||
if [[ "$(date -u +%u)" == "7" ]]; then
|
||||
DEDUP_THRESHOLD="0.85"
|
||||
DEDUP_BATCH="80"
|
||||
log "Step B3: memory dedup (Sunday deep pass, threshold $DEDUP_THRESHOLD)"
|
||||
else
|
||||
DEDUP_THRESHOLD="0.90"
|
||||
DEDUP_BATCH="50"
|
||||
log "Step B3: memory dedup (daily, threshold $DEDUP_THRESHOLD)"
|
||||
fi
|
||||
python3 "$APP_DIR/scripts/memory_dedup.py" \
|
||||
--base-url "$ATOCORE_URL" \
|
||||
--similarity-threshold "$DEDUP_THRESHOLD" \
|
||||
--max-batch "$DEDUP_BATCH" \
|
||||
2>&1 || {
|
||||
log "WARN: memory dedup failed (non-blocking)"
|
||||
}
|
||||
|
||||
# Step G: Integrity check (Phase 4 V1)
|
||||
log "Step G: integrity check"
|
||||
python3 "$APP_DIR/scripts/integrity_check.py" \
|
||||
|
||||
Reference in New Issue
Block a user