Files
ATOCore/src/atocore/engineering/mirror.py

329 lines
11 KiB
Python
Raw Normal View History

"""Human Mirror — derived readable project views from structured data.
Layer 3 of the AtoCore architecture. Generates human-readable markdown
pages from the engineering entity graph, Trusted Project State, and
active memories. These pages are DERIVED they are not canonical
machine truth. They are support surfaces for human inspection and
audit comfort.
The mirror never invents content. Every line traces back to an entity,
a state entry, or a memory. If the structured data is wrong, the
mirror is wrong fix the source, not the page.
"""
from __future__ import annotations
from atocore.context.project_state import get_state
from atocore.engineering.service import (
get_entities,
get_relationships,
)
from atocore.memory.service import get_memories
from atocore.observability.logger import get_logger
log = get_logger("mirror")
def generate_project_overview(project: str) -> str:
"""Generate a full project overview page in markdown."""
sections = [
_header(project),
feat: Karpathy-inspired upgrades — contradiction, lint, synthesis Three additive upgrades borrowed from Karpathy's LLM Wiki pattern: 1. CONTRADICTION DETECTION: auto-triage now has a fourth verdict — "contradicts". When a candidate conflicts with an existing memory (not duplicates, genuine disagreement like "Option A selected" vs "Option B selected"), the triage model flags it and leaves it in the queue for human review instead of silently rejecting or double-storing. Preserves source tension rather than suppressing it. 2. WEEKLY LINT PASS: scripts/lint_knowledge_base.py checks for: - Orphan memories (active but zero references after 14 days) - Stale candidates (>7 days unreviewed) - Unused entities (no relationships) - Empty-state projects - Unregistered projects auto-detected in memories Runs Sundays via the cron. Outputs a report. 3. WEEKLY SYNTHESIS: scripts/synthesize_projects.py uses sonnet to generate a 3-5 sentence "current state" paragraph per project from state + memories + entities. Cached in project_state under status/synthesis_cache. Wiki project pages now show this at the top under "Current State (auto-synthesis)". Falls back to a deterministic summary if no cache exists. deploy/dalidou/batch-extract.sh: added Step C (synthesis) and Step D (lint) gated to Sundays via date check. All additive — nothing existing changes behavior. The database remains the source of truth; these operations just produce better synthesized views and catch rot. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 21:08:13 -04:00
_synthesis_section(project),
feat: Phase 5B-5D — 10 canonical engineering queries + triage UI The graph becomes useful. Before this commit, entities sat in the DB as data with no narrative. After: the director can ask "what am I forgetting?" and get a structured answer in milliseconds. New module (src/atocore/engineering/queries.py, 360 lines): Structure queries (Q-001/004/005/008/013): - system_map(project): full subsystem → component tree + orphans + materials joined per component - decisions_affecting(project, subsystem_id?): decisions linked via AFFECTED_BY_DECISION, scoped to a subsystem or whole project - requirements_for(component_id): Q-005 forward trace - recent_changes(project, since, limit): Q-013 via memory_audit join (reuses the Phase 4 audit infrastructure — entity_kind='entity') The 3 killer queries (the real value): - orphan_requirements(project): requirements with NO inbound SATISFIES edge. "What do I claim the system must do that nothing actually claims to handle?" Q-006. - risky_decisions(project): decisions whose BASED_ON_ASSUMPTION edge points to an assumption with status in ('superseded','invalid') OR properties.flagged=True. Finds cascading risk from shaky premises. Q-009. - unsupported_claims(project): ValidationClaim entities with no inbound SUPPORTS edge — asserted but no Result to back them. Q-011. - all_gaps(project): runs all three in one call for dashboards. History + impact (Q-016/017): - impact_analysis(entity_id, max_depth=3): BFS over outbound edges. "What's downstream of this if I change it?" - evidence_chain(entity_id): inbound SUPPORTS/EVIDENCED_BY/DESCRIBED_BY/ VALIDATED_BY/ANALYZED_BY. "How do I know this is true?" API (src/atocore/api/routes.py) exposes 10 endpoints: - GET /engineering/projects/{p}/systems - GET /engineering/decisions?project=&subsystem= - GET /engineering/components/{id}/requirements - GET /engineering/changes?project=&since=&limit= - GET /engineering/gaps/orphan-requirements?project= - GET /engineering/gaps/risky-decisions?project= - GET /engineering/gaps/unsupported-claims?project= - GET /engineering/gaps?project= (combined) - GET /engineering/impact?entity=&max_depth= - GET /engineering/evidence?entity= Mirror integration (src/atocore/engineering/mirror.py): - New _gaps_section() renders at top of every project page - If any gap non-empty: shows up-to-10 per category with names + context - Clean project: "✅ No gaps detected" — signals everything is traced Triage UI (src/atocore/engineering/triage_ui.py): - /admin/triage now shows BOTH memory candidates AND entity candidates - Entity cards: name, type, project, confidence, source provenance, Promote/Reject buttons, link to wiki entity page - Entity promote/reject via fetch to /entities/{id}/promote|reject - One triage UI for the whole pipeline — consistent muscle memory Tests: 326 → 341 (15 new, all in test_engineering_queries.py): - System map structure + orphan detection + material joins - Killer queries: positive + negative cases (empty when clean) - Decisions query: project-wide and subsystem-scoped - Impact analysis walks outbound BFS - Evidence chain walks inbound provenance No regressions. All 10 daily queries from the plan are now live and answering real questions against the graph. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-17 07:18:46 -04:00
_gaps_section(project), # Phase 5: killer queries surface here
_state_section(project),
_system_architecture(project),
_decisions_section(project),
_requirements_section(project),
_materials_section(project),
_vendors_section(project),
_active_memories_section(project),
_footer(project),
]
return "\n\n".join(s for s in sections if s)
feat: Phase 5B-5D — 10 canonical engineering queries + triage UI The graph becomes useful. Before this commit, entities sat in the DB as data with no narrative. After: the director can ask "what am I forgetting?" and get a structured answer in milliseconds. New module (src/atocore/engineering/queries.py, 360 lines): Structure queries (Q-001/004/005/008/013): - system_map(project): full subsystem → component tree + orphans + materials joined per component - decisions_affecting(project, subsystem_id?): decisions linked via AFFECTED_BY_DECISION, scoped to a subsystem or whole project - requirements_for(component_id): Q-005 forward trace - recent_changes(project, since, limit): Q-013 via memory_audit join (reuses the Phase 4 audit infrastructure — entity_kind='entity') The 3 killer queries (the real value): - orphan_requirements(project): requirements with NO inbound SATISFIES edge. "What do I claim the system must do that nothing actually claims to handle?" Q-006. - risky_decisions(project): decisions whose BASED_ON_ASSUMPTION edge points to an assumption with status in ('superseded','invalid') OR properties.flagged=True. Finds cascading risk from shaky premises. Q-009. - unsupported_claims(project): ValidationClaim entities with no inbound SUPPORTS edge — asserted but no Result to back them. Q-011. - all_gaps(project): runs all three in one call for dashboards. History + impact (Q-016/017): - impact_analysis(entity_id, max_depth=3): BFS over outbound edges. "What's downstream of this if I change it?" - evidence_chain(entity_id): inbound SUPPORTS/EVIDENCED_BY/DESCRIBED_BY/ VALIDATED_BY/ANALYZED_BY. "How do I know this is true?" API (src/atocore/api/routes.py) exposes 10 endpoints: - GET /engineering/projects/{p}/systems - GET /engineering/decisions?project=&subsystem= - GET /engineering/components/{id}/requirements - GET /engineering/changes?project=&since=&limit= - GET /engineering/gaps/orphan-requirements?project= - GET /engineering/gaps/risky-decisions?project= - GET /engineering/gaps/unsupported-claims?project= - GET /engineering/gaps?project= (combined) - GET /engineering/impact?entity=&max_depth= - GET /engineering/evidence?entity= Mirror integration (src/atocore/engineering/mirror.py): - New _gaps_section() renders at top of every project page - If any gap non-empty: shows up-to-10 per category with names + context - Clean project: "✅ No gaps detected" — signals everything is traced Triage UI (src/atocore/engineering/triage_ui.py): - /admin/triage now shows BOTH memory candidates AND entity candidates - Entity cards: name, type, project, confidence, source provenance, Promote/Reject buttons, link to wiki entity page - Entity promote/reject via fetch to /entities/{id}/promote|reject - One triage UI for the whole pipeline — consistent muscle memory Tests: 326 → 341 (15 new, all in test_engineering_queries.py): - System map structure + orphan detection + material joins - Killer queries: positive + negative cases (empty when clean) - Decisions query: project-wide and subsystem-scoped - Impact analysis walks outbound BFS - Evidence chain walks inbound provenance No regressions. All 10 daily queries from the plan are now live and answering real questions against the graph. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-17 07:18:46 -04:00
def _gaps_section(project: str) -> str:
"""Phase 5: surface the 3 killer-query gaps on every project page.
If any gap is non-empty, it appears near the top so the director
sees "what am I forgetting?" before the rest of the report.
"""
try:
from atocore.engineering.queries import all_gaps
result = all_gaps(project)
except Exception:
return ""
orphan = result["orphan_requirements"]["count"]
risky = result["risky_decisions"]["count"]
unsup = result["unsupported_claims"]["count"]
if orphan == 0 and risky == 0 and unsup == 0:
return (
"## Coverage Gaps\n\n"
"> ✅ No gaps detected: every requirement is satisfied, "
"no decisions rest on flagged assumptions, every claim has evidence.\n"
)
lines = ["## Coverage Gaps", ""]
lines.append(
"> ⚠️ Items below need attention — gaps in the engineering graph.\n"
)
if orphan:
lines.append(f"### {orphan} Orphan Requirement(s)")
lines.append("*Requirements with no component claiming to satisfy them:*")
lines.append("")
for r in result["orphan_requirements"]["gaps"][:10]:
lines.append(f"- **{r['name']}** — {(r['description'] or '')[:120]}")
if orphan > 10:
lines.append(f"- _...and {orphan - 10} more_")
lines.append("")
if risky:
lines.append(f"### {risky} Risky Decision(s)")
lines.append("*Decisions based on assumptions that are flagged, superseded, or invalid:*")
lines.append("")
for d in result["risky_decisions"]["gaps"][:10]:
lines.append(
f"- **{d['decision_name']}** — based on flagged assumption "
f"_{d['assumption_name']}_ ({d['assumption_status']})"
)
lines.append("")
if unsup:
lines.append(f"### {unsup} Unsupported Claim(s)")
lines.append("*Validation claims with no supporting Result entity:*")
lines.append("")
for c in result["unsupported_claims"]["gaps"][:10]:
lines.append(f"- **{c['name']}** — {(c['description'] or '')[:120]}")
lines.append("")
return "\n".join(lines)
feat: Karpathy-inspired upgrades — contradiction, lint, synthesis Three additive upgrades borrowed from Karpathy's LLM Wiki pattern: 1. CONTRADICTION DETECTION: auto-triage now has a fourth verdict — "contradicts". When a candidate conflicts with an existing memory (not duplicates, genuine disagreement like "Option A selected" vs "Option B selected"), the triage model flags it and leaves it in the queue for human review instead of silently rejecting or double-storing. Preserves source tension rather than suppressing it. 2. WEEKLY LINT PASS: scripts/lint_knowledge_base.py checks for: - Orphan memories (active but zero references after 14 days) - Stale candidates (>7 days unreviewed) - Unused entities (no relationships) - Empty-state projects - Unregistered projects auto-detected in memories Runs Sundays via the cron. Outputs a report. 3. WEEKLY SYNTHESIS: scripts/synthesize_projects.py uses sonnet to generate a 3-5 sentence "current state" paragraph per project from state + memories + entities. Cached in project_state under status/synthesis_cache. Wiki project pages now show this at the top under "Current State (auto-synthesis)". Falls back to a deterministic summary if no cache exists. deploy/dalidou/batch-extract.sh: added Step C (synthesis) and Step D (lint) gated to Sundays via date check. All additive — nothing existing changes behavior. The database remains the source of truth; these operations just produce better synthesized views and catch rot. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 21:08:13 -04:00
def _synthesis_section(project: str) -> str:
"""Generate a short LLM synthesis of the current project state.
Reads the cached synthesis from project_state if available
(category=status, key=synthesis_cache). If not cached, returns
a deterministic summary from the existing structured data.
The actual LLM-generated synthesis is produced by the weekly
lint/synthesis pass on Dalidou (where claude CLI is available).
"""
entries = get_state(project)
cached = ""
for e in entries:
if e.category == "status" and e.key == "synthesis_cache":
cached = e.value
break
if cached:
return f"## Current State (auto-synthesis)\n\n> {cached}"
# Fallback: deterministic summary from structured data
stage = ""
summary = ""
next_focus = ""
for e in entries:
if e.category == "status":
if e.key == "stage":
stage = e.value
elif e.key == "summary":
summary = e.value
elif e.key == "next_focus":
next_focus = e.value
if not (stage or summary or next_focus):
return ""
bits = []
if summary:
bits.append(summary)
if stage:
bits.append(f"**Stage**: {stage}")
if next_focus:
bits.append(f"**Next**: {next_focus}")
return "## Current State\n\n" + "\n\n".join(bits)
def _header(project: str) -> str:
return (
f"# {project} — Project Overview\n\n"
f"> This page is auto-generated from AtoCore structured data.\n"
f"> It is a **derived view**, not canonical truth. "
f"If something is wrong here, fix the source data."
)
def _state_section(project: str) -> str:
entries = get_state(project)
if not entries:
return ""
lines = ["## Trusted Project State"]
by_category: dict[str, list] = {}
for e in entries:
by_category.setdefault(e.category.upper(), []).append(e)
for cat in ["DECISION", "REQUIREMENT", "STATUS", "FACT", "MILESTONE", "CONFIG", "CONTACT"]:
items = by_category.get(cat, [])
if not items:
continue
lines.append(f"\n### {cat.title()}")
for item in items:
value = item.value[:300]
lines.append(f"- **{item.key}**: {value}")
if item.source:
lines.append(f" *(source: {item.source})*")
return "\n".join(lines)
def _system_architecture(project: str) -> str:
systems = get_entities(entity_type="system", project=project)
subsystems = get_entities(entity_type="subsystem", project=project)
components = get_entities(entity_type="component", project=project)
interfaces = get_entities(entity_type="interface", project=project)
if not systems and not subsystems and not components:
return ""
lines = ["## System Architecture"]
for system in systems:
lines.append(f"\n### {system.name}")
if system.description:
lines.append(f"{system.description}")
rels = get_relationships(system.id, direction="outgoing")
children = []
for rel in rels:
if rel.relationship_type == "contains":
child = next(
(s for s in subsystems + components if s.id == rel.target_entity_id),
None,
)
if child:
children.append(child)
if children:
lines.append("\n**Contains:**")
for child in children:
desc = f"{child.description}" if child.description else ""
lines.append(f"- [{child.entity_type}] **{child.name}**{desc}")
child_rels = get_relationships(child.id, direction="both")
for cr in child_rels:
if cr.relationship_type in ("uses_material", "interfaces_with", "constrained_by"):
other_id = (
cr.target_entity_id
if cr.source_entity_id == child.id
else cr.source_entity_id
)
other = next(
(e for e in get_entities(project=project, limit=200)
if e.id == other_id),
None,
)
if other:
lines.append(
f" - *{cr.relationship_type}* → "
f"[{other.entity_type}] {other.name}"
)
return "\n".join(lines)
def _decisions_section(project: str) -> str:
decisions = get_entities(entity_type="decision", project=project)
if not decisions:
return ""
lines = ["## Decisions"]
for d in decisions:
lines.append(f"\n### {d.name}")
if d.description:
lines.append(d.description)
rels = get_relationships(d.id, direction="outgoing")
for rel in rels:
if rel.relationship_type == "affected_by_decision":
affected = next(
(e for e in get_entities(project=project, limit=200)
if e.id == rel.target_entity_id),
None,
)
if affected:
lines.append(
f"- Affects: [{affected.entity_type}] {affected.name}"
)
return "\n".join(lines)
def _requirements_section(project: str) -> str:
reqs = get_entities(entity_type="requirement", project=project)
constraints = get_entities(entity_type="constraint", project=project)
if not reqs and not constraints:
return ""
lines = ["## Requirements & Constraints"]
for r in reqs:
lines.append(f"- **{r.name}**: {r.description}" if r.description else f"- **{r.name}**")
for c in constraints:
lines.append(f"- [constraint] **{c.name}**: {c.description}" if c.description else f"- [constraint] **{c.name}**")
return "\n".join(lines)
def _materials_section(project: str) -> str:
materials = get_entities(entity_type="material", project=project)
if not materials:
return ""
lines = ["## Materials"]
for m in materials:
desc = f"{m.description}" if m.description else ""
lines.append(f"- **{m.name}**{desc}")
return "\n".join(lines)
def _vendors_section(project: str) -> str:
vendors = get_entities(entity_type="vendor", project=project)
if not vendors:
return ""
lines = ["## Vendors"]
for v in vendors:
desc = f"{v.description}" if v.description else ""
lines.append(f"- **{v.name}**{desc}")
return "\n".join(lines)
def _active_memories_section(project: str) -> str:
memories = get_memories(project=project, active_only=True, limit=20)
if not memories:
return ""
lines = ["## Active Memories"]
for m in memories:
conf = f" (conf: {m.confidence:.2f})" if m.confidence < 1.0 else ""
refs = f" | refs: {m.reference_count}" if m.reference_count > 0 else ""
lines.append(f"- [{m.memory_type}]{conf}{refs} {m.content[:200]}")
return "\n".join(lines)
def _footer(project: str) -> str:
from datetime import datetime, timezone
now = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M UTC")
return (
f"---\n\n"
f"*Generated by AtoCore Human Mirror at {now}. "
f"This is a derived view — not canonical truth.*"
)