docs: Consolidate documentation and fix protocol numbering (partial)

Phase 2 of restructuring plan: - Rename SYS_16_STUDY_INSIGHTS -> SYS_17_STUDY_INSIGHTS - Rename SYS_17_CONTEXT_ENGINEERING -> SYS_18_CONTEXT_ENGINEERING - Promote Bootstrap V3.0 (Context Engineering) as default - Archive old Bootstrap V2.0 - Create knowledge_base/playbook.json for ACE framework - Add OP_08 (Generate Report) to routing tables - Add SYS_16-18 to protocol tables - Update docs/protocols/README.md to version 1.1 - Update CLAUDE.md with new protocols - Create docs/plans/RESTRUCTURING_PLAN.md for continuation Remaining: Phase 2.8 (Cheatsheet), Phases 3-6 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-07 08:52:07 -05:00
parent 18c221a218
commit b8a04c62b8
17 changed files with 1568 additions and 623 deletions
--- a/docs/plans/RESTRUCTURING_PLAN.md
+++ b/docs/plans/RESTRUCTURING_PLAN.md
@@ -0,0 +1,549 @@
+# Atomizer Massive Restructuring Plan
+
+**Created:** 2026-01-06
+**Purpose:** Comprehensive TODO list for Ralph mode execution with skip permissions
+**Status:** IN PROGRESS (Phase 2 partially complete)
+
+---
+
+## Progress Summary
+
+**Completed:**
+- [x] Phase 1: Safe Cleanup (FULLY DONE)
+- [x] Phase 2.1-2.7: Protocol renaming, Bootstrap V3.0 promotion, routing updates
+
+**In Progress:**
+- Phase 2.8-2.10: Cheatsheet updates and commit
+
+**Remaining:**
+- Phases 3-6 and final push
+
+---
+
+## RALPH MODE TODO LIST
+
+### PHASE 2 (Remaining - Documentation)
+
+#### 2.8 Add OP_08 to 01_CHEATSHEET.md
+```
+File: .claude/skills/01_CHEATSHEET.md
+Action: Add row to "I want to..." table after OP_07 entry (around line 33)
+
+Add this line:
+| **Generate report** | **OP_08** | `python -m optimization_engine.reporting.report_generator <study>` |
+
+Also add a section around line 280:
+## Report Generation (OP_08)
+
+### Quick Commands
+| Task | Command |
+|------|---------|
+| Generate markdown report | `python -m optimization_engine.reporting.markdown_report <study>` |
+| Generate HTML visualization | `python tools/zernike_html_generator.py <study>` |
+
+**Full details**: `docs/protocols/operations/OP_08_GENERATE_REPORT.md`
+```
+
+#### 2.9 SKIP (Already verified V3.0 Bootstrap has no circular refs)
+
+#### 2.10 Commit Phase 2 Changes
+```bash
+cd c:\Users\antoi\Atomizer
+git add -A
+git commit -m "$(cat <<'EOF'
+docs: Consolidate documentation and fix protocol numbering
+
+- Rename SYS_16_STUDY_INSIGHTS -> SYS_17_STUDY_INSIGHTS
+- Rename SYS_17_CONTEXT_ENGINEERING -> SYS_18_CONTEXT_ENGINEERING
+- Promote Bootstrap V3.0 (Context Engineering) as default
+- Create knowledge_base/playbook.json for ACE framework
+- Add OP_08 (Generate Report) to all routing tables
+- Add SYS_16-18 to all protocol tables
+- Update docs/protocols/README.md version 1.1
+- Update CLAUDE.md with new protocols
+
+🤖 Generated with [Claude Code](https://claude.com/claude-code)
+
+Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
+EOF
+)"
+```
+
+---
+
+### PHASE 3: Code Organization
+
+#### 3.1 Move ensemble_surrogate.py
+```bash
+cd c:\Users\antoi\Atomizer
+git mv optimization_engine/surrogates/ensemble_surrogate.py optimization_engine/processors/surrogates/ensemble_surrogate.py
+```
+
+#### 3.2 Update processors/surrogates/__init__.py
+```
+File: optimization_engine/processors/surrogates/__init__.py
+Action: Add to __getattr__ function and __all__ list:
+
+In __getattr__, add these elif blocks:
+    elif name == 'EnsembleSurrogate':
+        from .ensemble_surrogate import EnsembleSurrogate
+        return EnsembleSurrogate
+    elif name == 'OODDetector':
+        from .ensemble_surrogate import OODDetector
+        return OODDetector
+    elif name == 'create_and_train_ensemble':
+        from .ensemble_surrogate import create_and_train_ensemble
+        return create_and_train_ensemble
+
+In __all__, add:
+    'EnsembleSurrogate',
+    'OODDetector',
+    'create_and_train_ensemble',
+```
+
+#### 3.3 Add deprecation shim to surrogates/__init__.py
+```
+File: optimization_engine/surrogates/__init__.py
+Replace contents with:
+
+"""
+DEPRECATED: This module has been moved to optimization_engine.processors.surrogates
+
+Please update your imports:
+    from optimization_engine.processors.surrogates import EnsembleSurrogate
+
+This module will be removed in a future version.
+"""
+
+import warnings
+
+warnings.warn(
+    "optimization_engine.surrogates is deprecated. "
+    "Use optimization_engine.processors.surrogates instead.",
+    DeprecationWarning,
+    stacklevel=2
+)
+
+# Redirect imports
+from optimization_engine.processors.surrogates import (
+    EnsembleSurrogate,
+    OODDetector,
+    create_and_train_ensemble
+)
+
+__all__ = ['EnsembleSurrogate', 'OODDetector', 'create_and_train_ensemble']
+```
+
+#### 3.4 Check future/ imports
+```bash
+cd c:\Users\antoi\Atomizer
+grep -r "from optimization_engine.future" --include="*.py" | grep -v "future/" | head -20
+```
+Analyze output and decide which modules need to move out of future/
+
+#### 3.5 Move workflow_decomposer.py (if imported by production code)
+If grep shows imports from config/ or core/:
+```bash
+git mv optimization_engine/future/workflow_decomposer.py optimization_engine/config/workflow_decomposer.py
+# Update imports in capability_matcher.py and any other files
+```
+
+#### 3.6 Create tests/ directory structure
+```bash
+cd c:\Users\antoi\Atomizer
+mkdir -p tests/unit/gnn tests/unit/extractors tests/integration tests/fixtures/sample_data
+```
+
+#### 3.7 Move test files from archive/test_scripts/
+```bash
+cd c:\Users\antoi\Atomizer
+git mv archive/test_scripts/test_neural_surrogate.py tests/unit/
+git mv archive/test_scripts/test_nn_surrogate.py tests/unit/
+git mv archive/test_scripts/test_parametric_surrogate.py tests/unit/
+git mv archive/test_scripts/test_adaptive_characterization.py tests/unit/
+git mv archive/test_scripts/test_training_data_export.py tests/unit/
+git mv optimization_engine/gnn/test_*.py tests/unit/gnn/ 2>/dev/null || true
+git mv optimization_engine/extractors/test_phase3_extractors.py tests/unit/extractors/ 2>/dev/null || true
+```
+
+#### 3.8 Create tests/conftest.py
+```
+File: tests/conftest.py
+Content:
+"""
+Pytest configuration and shared fixtures for Atomizer tests.
+"""
+
+import pytest
+import sys
+from pathlib import Path
+
+# Add project root to path
+sys.path.insert(0, str(Path(__file__).parent.parent))
+
+@pytest.fixture
+def sample_study_dir(tmp_path):
+    """Create a temporary study directory structure."""
+    study = tmp_path / "test_study"
+    (study / "1_setup").mkdir(parents=True)
+    (study / "2_iterations").mkdir()
+    (study / "3_results").mkdir()
+    return study
+
+@pytest.fixture
+def sample_config():
+    """Sample optimization config for testing."""
+    return {
+        "study_name": "test_study",
+        "design_variables": [
+            {"name": "param1", "lower": 0, "upper": 10, "type": "continuous"}
+        ],
+        "objectives": [
+            {"name": "minimize_mass", "direction": "minimize"}
+        ]
+    }
+```
+
+#### 3.9 Rename bracket_displacement_maximizing/results to 3_results
+```bash
+cd c:\Users\antoi\Atomizer
+# Check if results/ exists first
+if [ -d "studies/bracket_displacement_maximizing/results" ]; then
+    git mv studies/bracket_displacement_maximizing/results studies/bracket_displacement_maximizing/3_results
+fi
+```
+
+#### 3.10 Rename Drone_Gimbal/2_results to 3_results
+```bash
+cd c:\Users\antoi\Atomizer
+# Check if 2_results/ exists first
+if [ -d "studies/Drone_Gimbal/2_results" ]; then
+    git mv studies/Drone_Gimbal/2_results studies/Drone_Gimbal/3_results
+fi
+```
+
+#### 3.11 Commit Phase 3 Changes
+```bash
+cd c:\Users\antoi\Atomizer
+git add -A
+git commit -m "$(cat <<'EOF'
+refactor: Reorganize code structure and create tests directory
+
+- Consolidate surrogates module to processors/surrogates/
+- Add deprecation shim for old import path
+- Create tests/ directory with pytest structure
+- Move test files from archive/test_scripts/
+- Standardize study folder naming (3_results/)
+
+🤖 Generated with [Claude Code](https://claude.com/claude-code)
+
+Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
+EOF
+)"
+```
+
+---
+
+### PHASE 4: Dependency Management
+
+#### 4.1-4.2 Add neural and gnn optional deps to pyproject.toml
+```
+File: pyproject.toml
+After the [project.optional-dependencies] section, add:
+
+neural = [
+    "torch>=2.0.0",
+    "torch-geometric>=2.3.0",
+    "tensorboard>=2.13.0",
+]
+
+gnn = [
+    "torch>=2.0.0",
+    "torch-geometric>=2.3.0",
+]
+
+all = ["atomizer[neural,gnn,dev,dashboard]"]
+```
+
+#### 4.3 Remove mcp optional deps
+```
+File: pyproject.toml
+Delete this section:
+
+mcp = [
+    "mcp>=0.1.0",
+]
+```
+
+#### 4.4 Remove mcp_server from packages.find
+```
+File: pyproject.toml
+Change:
+    include = ["mcp_server*", "optimization_engine*", "nx_journals*"]
+To:
+    include = ["optimization_engine*", "nx_journals*"]
+```
+
+#### 4.5 Commit Phase 4 Changes
+```bash
+cd c:\Users\antoi\Atomizer
+git add pyproject.toml
+git commit -m "$(cat <<'EOF'
+build: Add optional dependency groups and clean up pyproject.toml
+
+- Add neural optional group (torch, torch-geometric, tensorboard)
+- Add gnn optional group (torch, torch-geometric)
+- Add all optional group for convenience
+- Remove mcp optional group (not implemented)
+- Remove mcp_server from packages.find (not implemented)
+
+🤖 Generated with [Claude Code](https://claude.com/claude-code)
+
+Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
+EOF
+)"
+```
+
+---
+
+### PHASE 5: Study Organization
+
+#### 5.1 Create archive directory
+```bash
+cd c:\Users\antoi\Atomizer
+mkdir -p studies/M1_Mirror/_archive
+```
+
+#### 5.2 Move V1-V8 cost_reduction studies
+```bash
+cd c:\Users\antoi\Atomizer/studies/M1_Mirror
+# Move cost_reduction V2-V8 (V1 doesn't exist as base is just "cost_reduction")
+for v in V2 V3 V4 V5 V6 V7 V8; do
+    if [ -d "m1_mirror_cost_reduction_$v" ]; then
+        mv "m1_mirror_cost_reduction_$v" _archive/
+    fi
+done
+```
+
+#### 5.3 Move V1-V8 flat_back studies
+```bash
+cd c:\Users\antoi\Atomizer/studies/M1_Mirror
+# Move flat_back V1-V8 (note: V2 may not exist)
+for v in V1 V3 V4 V5 V6 V7 V8; do
+    if [ -d "m1_mirror_cost_reduction_flat_back_$v" ]; then
+        mv "m1_mirror_cost_reduction_flat_back_$v" _archive/
+    fi
+done
+```
+
+#### 5.4 Create MANIFEST.md
+```
+File: studies/M1_Mirror/_archive/MANIFEST.md
+Content:
+
+# M1 Mirror Archived Studies
+
+**Archived:** 2026-01-06
+**Reason:** Repository cleanup - keeping only V9+ studies active
+
+## Archived Studies
+
+### Cost Reduction Series
+| Study | Trials | Best WS | Notes |
+|-------|--------|---------|-------|
+| V2 | TBD | TBD | Early exploration |
+| V3 | TBD | TBD | - |
+| V4 | TBD | TBD | - |
+| V5 | TBD | TBD | - |
+| V6 | TBD | TBD | - |
+| V7 | TBD | TBD | - |
+| V8 | TBD | TBD | - |
+
+### Cost Reduction Flat Back Series
+| Study | Trials | Best WS | Notes |
+|-------|--------|---------|-------|
+| V1 | TBD | TBD | Initial flat back design |
+| V3 | TBD | TBD | V2 was skipped |
+| V4 | TBD | TBD | - |
+| V5 | TBD | TBD | - |
+| V6 | TBD | TBD | - |
+| V7 | TBD | TBD | - |
+| V8 | TBD | TBD | - |
+
+## Restoration Instructions
+
+To restore a study:
+1. Move from _archive/ to parent directory
+2. Verify database integrity: `sqlite3 3_results/study.db ".tables"`
+3. Check optimization_config.json exists
+```
+
+#### 5.5 Commit Phase 5 Changes
+```bash
+cd c:\Users\antoi\Atomizer
+git add -A
+git commit -m "$(cat <<'EOF'
+chore: Archive old M1_Mirror studies (V1-V8)
+
+- Create studies/M1_Mirror/_archive/ directory
+- Move cost_reduction V2-V8 to archive
+- Move flat_back V1-V8 to archive
+- Create MANIFEST.md documenting archived studies
+- Keep V9+ studies active for reference
+
+🤖 Generated with [Claude Code](https://claude.com/claude-code)
+
+Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
+EOF
+)"
+```
+
+---
+
+### PHASE 6: Documentation Polish
+
+#### 6.1 Update README.md with LLM section
+```
+File: README.md
+Add this section after the main description:
+
+## For AI Assistants
+
+Atomizer is designed for LLM-first interaction. Key resources:
+
+- **[CLAUDE.md](CLAUDE.md)** - System instructions for Claude Code
+- **[.claude/skills/](/.claude/skills/)** - LLM skill modules
+- **[docs/protocols/](docs/protocols/)** - Protocol Operating System
+
+### Knowledge Base (LAC)
+
+The Learning Atomizer Core (`knowledge_base/lac/`) accumulates optimization knowledge:
+- `session_insights/` - Learnings from past sessions
+- `optimization_memory/` - Optimization outcomes by geometry type
+- `playbook.json` - ACE framework knowledge store
+
+For detailed AI interaction guidance, see CLAUDE.md.
+```
+
+#### 6.2-6.4 Create optimization_memory JSONL files
+```bash
+cd c:\Users\antoi\Atomizer
+mkdir -p knowledge_base/lac/optimization_memory
+```
+
+```
+File: knowledge_base/lac/optimization_memory/bracket.jsonl
+Content (one JSON per line):
+{"geometry_type": "bracket", "study_name": "example", "method": "TPE", "objectives": ["mass"], "trials": 0, "converged": false, "notes": "Schema file - replace with real data"}
+```
+
+```
+File: knowledge_base/lac/optimization_memory/beam.jsonl
+Content:
+{"geometry_type": "beam", "study_name": "example", "method": "TPE", "objectives": ["mass"], "trials": 0, "converged": false, "notes": "Schema file - replace with real data"}
+```
+
+```
+File: knowledge_base/lac/optimization_memory/mirror.jsonl
+Content:
+{"geometry_type": "mirror", "study_name": "m1_mirror_adaptive_V14", "method": "IMSO", "objectives": ["wfe_40_20", "mass_kg"], "trials": 100, "converged": true, "notes": "SAT v3 achieved WS=205.58"}
+```
+
+#### 6.5 Move implementation plans to docs/plans
+```bash
+cd c:\Users\antoi\Atomizer
+git mv .claude/skills/modules/DYNAMIC_RESPONSE_IMPLEMENTATION_PLAN.md docs/plans/
+git mv .claude/skills/modules/OPTIMIZATION_ENGINE_MIGRATION_PLAN.md docs/plans/
+git mv .claude/skills/modules/atomizer_fast_solver_technologies.md docs/plans/
+```
+
+#### 6.6 Final consistency verification
+```bash
+cd c:\Users\antoi\Atomizer
+# Verify protocol files exist
+ls docs/protocols/operations/OP_0*.md
+ls docs/protocols/system/SYS_1*.md
+
+# Verify imports work
+python -c "import optimization_engine; print('OK')"
+
+# Verify no broken references
+grep -r "SYS_16_STUDY" . --include="*.md" | head -5  # Should be empty
+grep -r "SYS_17_CONTEXT" . --include="*.md" | head -5  # Should be empty
+
+# Count todos completed
+echo "Verification complete"
+```
+
+#### 6.7 Commit Phase 6 Changes
+```bash
+cd c:\Users\antoi\Atomizer
+git add -A
+git commit -m "$(cat <<'EOF'
+docs: Final documentation polish and consistency fixes
+
+- Update README.md with LLM assistant section
+- Create optimization_memory JSONL structure
+- Move implementation plans from skills/modules to docs/plans
+- Verify all protocol references are consistent
+
+🤖 Generated with [Claude Code](https://claude.com/claude-code)
+
+Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
+EOF
+)"
+```
+
+---
+
+### FINAL: Push to Both Remotes
+
+```bash
+cd c:\Users\antoi\Atomizer
+git push origin main
+git push github main
+```
+
+---
+
+## Quick Reference
+
+### Files Modified in This Restructuring
+
+**Documentation (Phase 2):**
+- `docs/protocols/README.md` - Updated protocol listings
+- `docs/protocols/system/SYS_17_STUDY_INSIGHTS.md` - Renamed from SYS_16
+- `docs/protocols/system/SYS_18_CONTEXT_ENGINEERING.md` - Renamed from SYS_17
+- `CLAUDE.md` - Updated routing tables
+- `.claude/skills/00_BOOTSTRAP.md` - Replaced with V3.0
+- `.claude/skills/01_CHEATSHEET.md` - Added OP_08
+- `knowledge_base/playbook.json` - Created
+
+**Code (Phase 3):**
+- `optimization_engine/processors/surrogates/__init__.py` - Added exports
+- `optimization_engine/surrogates/__init__.py` - Deprecation shim
+- `tests/conftest.py` - Created
+
+**Dependencies (Phase 4):**
+- `pyproject.toml` - Updated optional groups
+
+**Studies (Phase 5):**
+- `studies/M1_Mirror/_archive/` - Created with V1-V8 studies
+
+**Final Polish (Phase 6):**
+- `README.md` - Added LLM section
+- `knowledge_base/lac/optimization_memory/` - Created structure
+- `docs/plans/` - Moved implementation plans
+
+---
+
+## Success Criteria Checklist
+
+- [ ] All imports work: `python -c "import optimization_engine"`
+- [ ] Dashboard starts: `python launch_dashboard.py`
+- [ ] No SYS_16 duplication (only SELF_AWARE_TURBO)
+- [ ] Bootstrap V3.0 is active version
+- [ ] OP_08 discoverable in all routing tables
+- [ ] Studies use consistent 3_results/ naming
+- [ ] Tests directory exists with conftest.py
+- [ ] All changes pushed to both remotes