docs: Major documentation overhaul - restructure folders, update tagline, add Getting Started guide

- Restructure docs/ folder (remove numeric prefixes): - 04_USER_GUIDES -> guides/ - 05_API_REFERENCE -> api/ - 06_PHYSICS -> physics/ - 07_DEVELOPMENT -> development/ - 08_ARCHIVE -> archive/ - 09_DIAGRAMS -> diagrams/ - Replace tagline 'Talk, don't click' with 'LLM-driven optimization framework' in 9 files - Create comprehensive docs/GETTING_STARTED.md: - Prerequisites and quick setup - Project structure overview - First study tutorial (Claude or manual) - Dashboard usage guide - Neural acceleration introduction - Rewrite docs/00_INDEX.md with correct paths and modern structure - Archive obsolete files: - 01_PROTOCOLS.md -> archive/historical/01_PROTOCOLS_legacy.md - 03_GETTING_STARTED.md -> archive/historical/ - ATOMIZER_PODCAST_BRIEFING.md -> archive/marketing/ - Update timestamps to 2026-01-20 across all key files - Update .gitignore to exclude docs/generated/ - Version bump: ATOMIZER_CONTEXT v1.8 -> v2.0
2026-01-20 10:03:45 -05:00
parent 37f73cc2be
commit ea437d360e
103 changed files with 8980 additions and 327 deletions
--- a/docs/reference/DEEP_INVESTIGATION_PROMPT.md
+++ b/docs/reference/DEEP_INVESTIGATION_PROMPT.md
@@ -0,0 +1,197 @@
+# Deep Investigation & Architecture Analysis Prompt
+
+**Purpose**: Reusable instructions for conducting thorough system investigations and producing actionable design documents.
+
+---
+
+## Full Investigation Prompt
+
+Use this when you need a comprehensive analysis of a system, architecture, or problem:
+
+```markdown
+Please conduct a comprehensive investigation and produce a master design document.
+
+### Investigation Phase
+
+1. **Multi-Agent Exploration**: Launch parallel exploration agents to examine:
+   - Frontend components and state management
+   - Backend services, routes, and data flow
+   - Core engine/library code and schemas
+   - Configuration files and existing patterns
+   - Any MCP tools, APIs, or integration points
+
+2. **Source of Truth Analysis**: Identify ALL places where the same concept is represented:
+   - Data schemas/types (JSON, TypeScript, Python)
+   - Configuration formats and their variants
+   - State management across components
+   - Document inconsistencies and naming conflicts
+
+3. **Data Flow Mapping**: Trace how data moves through the system:
+   - Entry points → Processing → Storage → Display
+   - Identify lossy conversions and sync gaps
+   - Note missing endpoints or broken connections
+
+### Documentation Phase
+
+4. **Current State Report**: Document what exists with:
+   - Architecture diagrams (text-based)
+   - Comparison tables showing format inconsistencies
+   - Data flow diagrams showing where information is lost
+   - Clear "What's Wrong / What's OK" assessment
+
+5. **Problem Statement**: Articulate:
+   - Core issues with severity ratings
+   - User pain points with concrete scenarios
+   - Technical debt and architectural gaps
+
+### Design Phase
+
+6. **Proposed Architecture**: Design the solution with:
+   - Single source of truth principle
+   - Complete schema definition (JSON Schema if applicable)
+   - Component responsibilities and interfaces
+   - API/endpoint specifications
+   - Real-time sync strategy if needed
+
+7. **Integration Design**: Show how components connect:
+   - Frontend ↔ Backend ↔ Engine data contracts
+   - Bidirectional sync mechanisms
+   - AI/Assistant integration points
+   - Extensibility patterns (plugins, custom functions)
+
+### Planning Phase
+
+8. **Implementation Roadmap**: Break into phases with:
+   - Clear deliverables per phase
+   - Priority (P0/P1/P2) and effort estimates
+   - Dependencies between tasks
+   - Migration strategy for existing data
+
+9. **Appendices**: Include:
+   - Glossary of terms
+   - File location references
+   - Comparison tables (before/after)
+   - Code examples for key components
+
+### Output Format
+
+Produce:
+1. **Master Document** (Markdown): Complete design document with all sections
+2. **Schema Files**: Actual JSON Schema or type definitions ready to use
+3. **Executive Summary**: Key findings, what's broken, proposed solution, timeline
+
+### Quality Standards
+
+- Read actual source files, don't assume
+- Use tables for comparisons (makes inconsistencies obvious)
+- Include text-based diagrams for architecture and data flow
+- Provide concrete code examples, not just descriptions
+- Make it actionable: someone should be able to implement from this document
+```
+
+---
+
+## Quick Version
+
+For faster investigations or when you already have context:
+
+```markdown
+Deep investigation with master document output:
+
+1. **Explore**: Parallel agents examine frontend, backend, engine, schemas
+2. **Map**: Trace data flow, identify all representations of same concept
+3. **Compare**: Tables showing format inconsistencies and naming conflicts
+4. **Diagram**: Architecture and data flow (text-based)
+5. **Assess**: What's wrong (severity) / What's OK / User pain points
+6. **Design**: Single source of truth, complete schema, API specs
+7. **Plan**: Phased roadmap with priorities and effort estimates
+8. **Deliver**: Master document + actual schema files + executive summary
+
+Standards: Read actual code, use comparison tables, include diagrams, make it actionable.
+```
+
+---
+
+## Trigger Phrases
+
+Use these phrases to invoke deep analysis behavior:
+
+| Phrase | Effect |
+|--------|--------|
+| "Make a very deep and thoughtful research on..." | Triggers comprehensive multi-agent exploration |
+| "Produce a master document that will lead to implementation" | Ensures actionable output with roadmap |
+| "Investigate with multi-agent exploration" | Parallel exploration of different system areas |
+| "Map all representations and identify inconsistencies" | Source of truth analysis with comparison tables |
+| "Design a single source of truth architecture" | Unified schema/format design |
+| "Include comparison tables and data flow diagrams" | Visual documentation requirements |
+| "Make it actionable with implementation roadmap" | Phased planning with priorities |
+
+---
+
+## Example Usage
+
+### For Architecture Overhaul
+```
+Please conduct a deep investigation on how [Component A] and [Component B]
+share data. I need a master document that:
+- Maps all data representations across the system
+- Identifies inconsistencies with comparison tables
+- Proposes a unified architecture
+- Includes implementation roadmap
+
+Make it actionable - someone should be able to implement from this document.
+```
+
+### For Problem Diagnosis
+```
+Deep investigation needed: [describe the problem]
+
+Explore the codebase to understand:
+- Where this data/logic currently lives
+- How it flows through the system
+- What's broken and why
+
+Produce a report with:
+- Root cause analysis
+- Data flow diagrams
+- Proposed fix with implementation steps
+```
+
+### For New Feature Design
+```
+I want to add [feature]. Before implementing, conduct a deep analysis:
+
+1. How does similar functionality work in the codebase?
+2. What components would be affected?
+3. What's the cleanest integration approach?
+
+Produce a design document with architecture, API specs, and phased roadmap.
+```
+
+---
+
+## Output Checklist
+
+A good deep investigation should produce:
+
+- [ ] **Architecture diagram** (text-based, showing component relationships)
+- [ ] **Data flow diagram** (showing how data moves, where it's transformed)
+- [ ] **Comparison tables** (format inconsistencies, naming conflicts)
+- [ ] **Problem severity matrix** (Critical/High/Medium/Low ratings)
+- [ ] **User pain points** (concrete scenarios, not abstract)
+- [ ] **Proposed schema/types** (actual JSON Schema or TypeScript)
+- [ ] **API specifications** (endpoints, request/response formats)
+- [ ] **Implementation roadmap** (phased, with effort estimates)
+- [ ] **Migration strategy** (for existing data/code)
+- [ ] **Code examples** (for key components)
+
+---
+
+## Related Documents
+
+- Example output: `docs/plans/UNIFIED_CONFIGURATION_ARCHITECTURE.md`
+- Example schema: `optimization_engine/schemas/atomizer_spec_v2.json`
+
+---
+
+*This prompt template was developed during the Atomizer Unified Configuration Architecture investigation (January 2026).*
--- a/docs/reference/EXECUTION_PLAN_GENERATOR_PROMPT.md
+++ b/docs/reference/EXECUTION_PLAN_GENERATOR_PROMPT.md
@@ -0,0 +1,322 @@
+# Execution Plan & Ralph Loop Generator
+
+**Purpose**: Instructions for generating comprehensive execution plans with autonomous implementation prompts from any design document.
+
+---
+
+## When to Use
+
+After completing a deep investigation that produced:
+- A master design document (architecture, schemas, component designs)
+- Clear understanding of what needs to be built
+
+Use this to generate:
+- Detailed task list with dependencies
+- File-level implementation instructions
+- Autonomous execution prompts (Ralph Loop)
+- Quick reference guides
+
+---
+
+## Generator Prompt
+
+```markdown
+Based on the design document at [PATH_TO_DESIGN_DOC], create a comprehensive execution plan for autonomous implementation.
+
+### Task Breakdown Requirements
+
+1. **Granularity**: Break work into tasks that take 15-60 minutes each
+2. **Atomic**: Each task should produce a testable/verifiable output
+3. **Sequential IDs**: Use format P{phase}.{number} (e.g., P1.1, P1.2, P2.1)
+4. **Dependencies**: Explicitly list which tasks must complete first
+
+### For Each Task, Specify:
+
+- **Task ID & Name**: P1.1 - Create TypeScript types from JSON Schema
+- **File(s)**: Exact file path(s) to create or modify
+- **Action**: Specific implementation instructions
+- **Reference**: Section of design doc with details/examples
+- **Dependencies**: List of prerequisite task IDs
+- **Acceptance Criteria**: How to verify task is complete
+
+### Structure the Plan As:
+
+1. **Project Overview**: Goal, reference docs, success metrics
+2. **Phase Structure**: Table showing phases, duration, focus areas
+3. **Implementation Order**: Critical path diagram (text-based)
+4. **Phase Sections**: Detailed tasks grouped by phase
+5. **File Summary**: Tables of files to create and modify
+6. **Success Criteria**: Checkboxes for each phase completion
+7. **Risk Mitigation**: Known risks and how to handle them
+
+### Also Generate:
+
+1. **Ralph Loop Prompts**:
+   - Initial prompt to start autonomous execution
+   - Continuation prompt for resuming
+   - Phase-specific prompts
+   - Troubleshooting prompts
+   - Full task list for TodoWrite initialization
+
+2. **Quick Start Guide**:
+   - TL;DR summary
+   - Copy-paste commands to start
+   - Phase summary table
+   - Key concepts glossary
+   - Success metrics checklist
+
+### Output Files:
+
+- `{PROJECT}_EXECUTION_PLAN.md` - Detailed task list
+- `{PROJECT}_RALPH_PROMPT.md` - Autonomous execution prompts
+- `{PROJECT}_QUICKSTART.md` - Quick reference guide
+```
+
+---
+
+## Example Usage
+
+### Input
+```
+Based on the design document at `docs/plans/UNIFIED_CONFIGURATION_ARCHITECTURE.md`,
+create a comprehensive execution plan for autonomous implementation.
+
+The project involves:
+- Backend services (Python/FastAPI)
+- Frontend components (React/TypeScript)
+- MCP tools integration
+- Database migrations
+
+Generate the execution plan, Ralph Loop prompts, and quick start guide.
+```
+
+### Output Structure
+```
+docs/plans/
+├── {PROJECT}_EXECUTION_PLAN.md      # 50+ detailed tasks
+├── {PROJECT}_RALPH_PROMPT.md        # Autonomous prompts
+└── {PROJECT}_QUICKSTART.md          # TL;DR guide
+```
+
+---
+
+## Task Template
+
+Use this template for each task:
+
+```markdown
+### P{X}.{Y} - {Task Name}
+- **File**: `path/to/file.ext`
+- **Action**: {What to implement}
+- **Reference**: {Design doc section} Section X.Y
+- **Dependencies**: P{X}.{Z}, P{X}.{W}
+- **Acceptance**: {How to verify completion}
+```
+
+---
+
+## Phase Template
+
+```markdown
+## PHASE {N}: {Phase Name}
+
+### P{N}.1 - {First Task}
+- **File**: `path/to/file`
+- **Action**: {Description}
+- **Dependencies**: {None or list}
+- **Acceptance**: {Criteria}
+
+### P{N}.2 - {Second Task}
+...
+```
+
+---
+
+## Ralph Prompt Template
+
+```markdown
+# Ralph Loop Prompt: {Project Name}
+
+## Claude CLI Command
+\`\`\`bash
+claude --dangerously-skip-permissions
+\`\`\`
+
+## Initial Prompt (Copy This)
+\`\`\`
+You are implementing {Project Description}.
+
+## Project Context
+
+Read these documents before starting:
+1. `{path/to/design_doc}` - Master design document
+2. `{path/to/execution_plan}` - Detailed task list
+3. `{path/to/schema_or_types}` - Type definitions (if applicable)
+
+## Your Mission
+
+Implement the system following the execution plan. Work through tasks in order (P1.1 → P1.2 → ... → P{N}.{M}).
+
+## Rules
+
+1. **Follow the plan**: Execute tasks in order specified
+2. **Check dependencies**: Don't start until prerequisites complete
+3. **Use TodoWrite**: Track all tasks, mark progress
+4. **Test as you go**: Verify acceptance criteria
+5. **Commit regularly**: After each logical group
+6. **Reference the design**: Use code examples from docs
+7. **Ask if blocked**: Explain blockers clearly
+
+Begin by reading the reference documents and loading tasks into TodoWrite.
+\`\`\`
+
+## Continuation Prompt
+\`\`\`
+Continue implementing {Project Name}.
+
+Reference: `{path/to/execution_plan}`
+
+Check current todo list, find last completed task, continue from there.
+\`\`\`
+
+## Full Task List for TodoWrite
+\`\`\`
+P1.1 - {Task name}
+P1.2 - {Task name}
+...
+\`\`\`
+```
+
+---
+
+## Quick Start Template
+
+```markdown
+# {Project Name} - Quick Start Guide
+
+## TL;DR
+
+**Goal**: {One sentence description}
+
+**Effort**: ~{N} tasks across {M} phases (~{W} weeks)
+
+**Key Files**:
+- Design: `{path}`
+- Tasks: `{path}`
+- Prompts: `{path}`
+
+---
+
+## Start Autonomous Implementation
+
+### 1. Open Claude CLI
+\`\`\`bash
+claude --dangerously-skip-permissions
+\`\`\`
+
+### 2. Paste Initial Prompt
+\`\`\`
+{Condensed version of initial prompt}
+\`\`\`
+
+### 3. Let It Run
+
+---
+
+## Phase Summary
+
+| Phase | Tasks | Focus | Deliverables |
+|-------|-------|-------|--------------|
+| 1 | P1.1-P1.{N} | {Focus} | {Key outputs} |
+| 2 | P2.1-P2.{N} | {Focus} | {Key outputs} |
+...
+
+---
+
+## Success Metrics
+
+- [ ] {Metric 1}
+- [ ] {Metric 2}
+...
+```
+
+---
+
+## Quality Checklist
+
+Before finalizing the execution plan, verify:
+
+- [ ] Every task has a specific file path
+- [ ] Every task has clear acceptance criteria
+- [ ] Dependencies form a valid DAG (no cycles)
+- [ ] Tasks are ordered so dependencies come first
+- [ ] Phase boundaries make logical sense
+- [ ] Commits points are identified (after groups of related tasks)
+- [ ] Risk mitigations are documented
+- [ ] Ralph prompts reference all necessary documents
+- [ ] Quick start is actually quick (fits on one screen)
+- [ ] Task list for TodoWrite is complete and copy-pasteable
+
+---
+
+## Complexity Guidelines
+
+| Project Size | Tasks | Phases | Typical Duration |
+|--------------|-------|--------|------------------|
+| Small | 10-20 | 2 | 1-2 weeks |
+| Medium | 20-50 | 3-4 | 3-6 weeks |
+| Large | 50-100 | 4-6 | 6-12 weeks |
+| Epic | 100+ | 6+ | 12+ weeks |
+
+---
+
+## Common Patterns
+
+### Backend-First Pattern
+```
+Phase 1: Data models, validation, core service
+Phase 2: REST API endpoints
+Phase 3: Frontend integration
+Phase 4: Testing & polish
+```
+
+### Full-Stack Feature Pattern
+```
+Phase 1: Schema/types, backend service, API
+Phase 2: Frontend components, state management
+Phase 3: Integration, real-time sync
+Phase 4: Testing, documentation
+```
+
+### Migration Pattern
+```
+Phase 1: New system alongside old
+Phase 2: Migration tooling
+Phase 3: Gradual cutover
+Phase 4: Deprecate old system
+```
+
+### Integration Pattern
+```
+Phase 1: Define contracts/interfaces
+Phase 2: Implement adapters
+Phase 3: Wire up connections
+Phase 4: Testing, error handling
+```
+
+---
+
+## Tips for Better Plans
+
+1. **Start with the end**: Define success criteria first, then work backward
+2. **Identify the critical path**: What's the minimum to get something working?
+3. **Group related tasks**: Makes commits logical and rollback easier
+4. **Front-load risky tasks**: Discover blockers early
+5. **Include buffer**: Things always take longer than expected
+6. **Make tasks testable**: "It works" is not acceptance criteria
+7. **Reference existing code**: Point to similar patterns in codebase
+8. **Consider parallelism**: Some phases can overlap if teams split work
+
+---
+
+*Use this template to generate execution plans for any project following a design document.*