Atomizer/docs/plans/RESTRUCTURING_PLAN.md

# Atomizer Massive Restructuring Plan

**Created:** 2026-01-06
**Purpose:** Comprehensive TODO list for Ralph mode execution with skip permissions
**Status:** COMPLETED (2026-01-07)

---

## Progress Summary

**All Phases Completed:**
- [x] Phase 1: Safe Cleanup
- [x] Phase 2: Documentation (OP_08, protocol numbering, cheatsheet)
- [x] Phase 3: Code Organization (surrogates consolidation, tests directory)
- [x] Phase 4: Dependency Management (pyproject.toml cleanup)
- [x] Phase 5: Study Organization (archived V1-V8 studies)
- [x] Phase 6: Documentation Polish (README, LAC, plans)
- [x] Final Push: Pushed to both origin and github remotes

---

## RALPH MODE TODO LIST

### PHASE 2 (Remaining - Documentation)

#### 2.8 Add OP_08 to 01_CHEATSHEET.md
```
File: .claude/skills/01_CHEATSHEET.md
Action: Add row to "I want to..." table after OP_07 entry (around line 33)

Add this line:
| **Generate report** | **OP_08** | `python -m optimization_engine.reporting.report_generator <study>` |

Also add a section around line 280:
## Report Generation (OP_08)

### Quick Commands
| Task | Command |
|------|---------|
| Generate markdown report | `python -m optimization_engine.reporting.markdown_report <study>` |
| Generate HTML visualization | `python tools/zernike_html_generator.py <study>` |

**Full details**: `docs/protocols/operations/OP_08_GENERATE_REPORT.md`
```

#### 2.9 SKIP (Already verified V3.0 Bootstrap has no circular refs)

#### 2.10 Commit Phase 2 Changes
```bash
cd c:\Users\antoi\Atomizer
git add -A
git commit -m "$(cat <<'EOF'
docs: Consolidate documentation and fix protocol numbering

- Rename SYS_16_STUDY_INSIGHTS -> SYS_17_STUDY_INSIGHTS
- Rename SYS_17_CONTEXT_ENGINEERING -> SYS_18_CONTEXT_ENGINEERING
- Promote Bootstrap V3.0 (Context Engineering) as default
- Create knowledge_base/playbook.json for ACE framework
- Add OP_08 (Generate Report) to all routing tables
- Add SYS_16-18 to all protocol tables
- Update docs/protocols/README.md version 1.1
- Update CLAUDE.md with new protocols

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
EOF
)"
```

---

### PHASE 3: Code Organization

#### 3.1 Move ensemble_surrogate.py
```bash
cd c:\Users\antoi\Atomizer
git mv optimization_engine/surrogates/ensemble_surrogate.py optimization_engine/processors/surrogates/ensemble_surrogate.py
```

#### 3.2 Update processors/surrogates/__init__.py
```
File: optimization_engine/processors/surrogates/__init__.py
Action: Add to __getattr__ function and __all__ list:

In __getattr__, add these elif blocks:
    elif name == 'EnsembleSurrogate':
        from .ensemble_surrogate import EnsembleSurrogate
        return EnsembleSurrogate
    elif name == 'OODDetector':
        from .ensemble_surrogate import OODDetector
        return OODDetector
    elif name == 'create_and_train_ensemble':
        from .ensemble_surrogate import create_and_train_ensemble
        return create_and_train_ensemble

In __all__, add:
    'EnsembleSurrogate',
    'OODDetector',
    'create_and_train_ensemble',
```

#### 3.3 Add deprecation shim to surrogates/__init__.py
```
File: optimization_engine/surrogates/__init__.py
Replace contents with:

"""
DEPRECATED: This module has been moved to optimization_engine.processors.surrogates

Please update your imports:
    from optimization_engine.processors.surrogates import EnsembleSurrogate

This module will be removed in a future version.
"""

import warnings

warnings.warn(
    "optimization_engine.surrogates is deprecated. "
    "Use optimization_engine.processors.surrogates instead.",
    DeprecationWarning,
    stacklevel=2
)

# Redirect imports
from optimization_engine.processors.surrogates import (
    EnsembleSurrogate,
    OODDetector,
    create_and_train_ensemble
)

__all__ = ['EnsembleSurrogate', 'OODDetector', 'create_and_train_ensemble']
```

#### 3.4 Check future/ imports
```bash
cd c:\Users\antoi\Atomizer
grep -r "from optimization_engine.future" --include="*.py" | grep -v "future/" | head -20
```
Analyze output and decide which modules need to move out of future/

#### 3.5 Move workflow_decomposer.py (if imported by production code)
If grep shows imports from config/ or core/:
```bash
git mv optimization_engine/future/workflow_decomposer.py optimization_engine/config/workflow_decomposer.py
# Update imports in capability_matcher.py and any other files
```

#### 3.6 Create tests/ directory structure
```bash
cd c:\Users\antoi\Atomizer
mkdir -p tests/unit/gnn tests/unit/extractors tests/integration tests/fixtures/sample_data
```

#### 3.7 Move test files from archive/test_scripts/
```bash
cd c:\Users\antoi\Atomizer
git mv archive/test_scripts/test_neural_surrogate.py tests/unit/
git mv archive/test_scripts/test_nn_surrogate.py tests/unit/
git mv archive/test_scripts/test_parametric_surrogate.py tests/unit/
git mv archive/test_scripts/test_adaptive_characterization.py tests/unit/
git mv archive/test_scripts/test_training_data_export.py tests/unit/
git mv optimization_engine/gnn/test_*.py tests/unit/gnn/ 2>/dev/null || true
git mv optimization_engine/extractors/test_phase3_extractors.py tests/unit/extractors/ 2>/dev/null || true
```

#### 3.8 Create tests/conftest.py
```
File: tests/conftest.py
Content:
"""
Pytest configuration and shared fixtures for Atomizer tests.
"""

import pytest
import sys
from pathlib import Path

# Add project root to path
sys.path.insert(0, str(Path(__file__).parent.parent))

@pytest.fixture
def sample_study_dir(tmp_path):
    """Create a temporary study directory structure."""
    study = tmp_path / "test_study"
    (study / "1_setup").mkdir(parents=True)
    (study / "2_iterations").mkdir()
    (study / "3_results").mkdir()
    return study

@pytest.fixture
def sample_config():
    """Sample optimization config for testing."""
    return {
        "study_name": "test_study",
        "design_variables": [
            {"name": "param1", "lower": 0, "upper": 10, "type": "continuous"}
        ],
        "objectives": [
            {"name": "minimize_mass", "direction": "minimize"}
        ]
    }
```

#### 3.9 Rename bracket_displacement_maximizing/results to 3_results
```bash
cd c:\Users\antoi\Atomizer
# Check if results/ exists first
if [ -d "studies/bracket_displacement_maximizing/results" ]; then
    git mv studies/bracket_displacement_maximizing/results studies/bracket_displacement_maximizing/3_results
fi
```

#### 3.10 Rename Drone_Gimbal/2_results to 3_results
```bash
cd c:\Users\antoi\Atomizer
# Check if 2_results/ exists first
if [ -d "studies/Drone_Gimbal/2_results" ]; then
    git mv studies/Drone_Gimbal/2_results studies/Drone_Gimbal/3_results
fi
```

#### 3.11 Commit Phase 3 Changes
```bash
cd c:\Users\antoi\Atomizer
git add -A
git commit -m "$(cat <<'EOF'
refactor: Reorganize code structure and create tests directory

- Consolidate surrogates module to processors/surrogates/
- Add deprecation shim for old import path
- Create tests/ directory with pytest structure
- Move test files from archive/test_scripts/
- Standardize study folder naming (3_results/)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
EOF
)"
```

---

### PHASE 4: Dependency Management

#### 4.1-4.2 Add neural and gnn optional deps to pyproject.toml
```
File: pyproject.toml
After the [project.optional-dependencies] section, add:

neural = [
    "torch>=2.0.0",
    "torch-geometric>=2.3.0",
    "tensorboard>=2.13.0",
]

gnn = [
    "torch>=2.0.0",
    "torch-geometric>=2.3.0",
]

all = ["atomizer[neural,gnn,dev,dashboard]"]
```

#### 4.3 Remove mcp optional deps
```
File: pyproject.toml
Delete this section:

mcp = [
    "mcp>=0.1.0",
]
```

#### 4.4 Remove mcp_server from packages.find
```
File: pyproject.toml
Change:
    include = ["mcp_server*", "optimization_engine*", "nx_journals*"]
To:
    include = ["optimization_engine*", "nx_journals*"]
```

#### 4.5 Commit Phase 4 Changes
```bash
cd c:\Users\antoi\Atomizer
git add pyproject.toml
git commit -m "$(cat <<'EOF'
build: Add optional dependency groups and clean up pyproject.toml

- Add neural optional group (torch, torch-geometric, tensorboard)
- Add gnn optional group (torch, torch-geometric)
- Add all optional group for convenience
- Remove mcp optional group (not implemented)
- Remove mcp_server from packages.find (not implemented)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
EOF
)"
```

---

### PHASE 5: Study Organization

#### 5.1 Create archive directory
```bash
cd c:\Users\antoi\Atomizer
mkdir -p studies/M1_Mirror/_archive
```

#### 5.2 Move V1-V8 cost_reduction studies
```bash
cd c:\Users\antoi\Atomizer/studies/M1_Mirror
# Move cost_reduction V2-V8 (V1 doesn't exist as base is just "cost_reduction")
for v in V2 V3 V4 V5 V6 V7 V8; do
    if [ -d "m1_mirror_cost_reduction_$v" ]; then
        mv "m1_mirror_cost_reduction_$v" _archive/
    fi
done
```

#### 5.3 Move V1-V8 flat_back studies
```bash
cd c:\Users\antoi\Atomizer/studies/M1_Mirror
# Move flat_back V1-V8 (note: V2 may not exist)
for v in V1 V3 V4 V5 V6 V7 V8; do
    if [ -d "m1_mirror_cost_reduction_flat_back_$v" ]; then
        mv "m1_mirror_cost_reduction_flat_back_$v" _archive/
    fi
done
```

#### 5.4 Create MANIFEST.md
```
File: studies/M1_Mirror/_archive/MANIFEST.md
Content:

# M1 Mirror Archived Studies

**Archived:** 2026-01-06
**Reason:** Repository cleanup - keeping only V9+ studies active

## Archived Studies

### Cost Reduction Series
| Study | Trials | Best WS | Notes |
|-------|--------|---------|-------|
| V2 | TBD | TBD | Early exploration |
| V3 | TBD | TBD | - |
| V4 | TBD | TBD | - |
| V5 | TBD | TBD | - |
| V6 | TBD | TBD | - |
| V7 | TBD | TBD | - |
| V8 | TBD | TBD | - |

### Cost Reduction Flat Back Series
| Study | Trials | Best WS | Notes |
|-------|--------|---------|-------|
| V1 | TBD | TBD | Initial flat back design |
| V3 | TBD | TBD | V2 was skipped |
| V4 | TBD | TBD | - |
| V5 | TBD | TBD | - |
| V6 | TBD | TBD | - |
| V7 | TBD | TBD | - |
| V8 | TBD | TBD | - |

## Restoration Instructions

To restore a study:
1. Move from _archive/ to parent directory
2. Verify database integrity: `sqlite3 3_results/study.db ".tables"`
3. Check optimization_config.json exists
```

#### 5.5 Commit Phase 5 Changes
```bash
cd c:\Users\antoi\Atomizer
git add -A
git commit -m "$(cat <<'EOF'
chore: Archive old M1_Mirror studies (V1-V8)

- Create studies/M1_Mirror/_archive/ directory
- Move cost_reduction V2-V8 to archive
- Move flat_back V1-V8 to archive
- Create MANIFEST.md documenting archived studies
- Keep V9+ studies active for reference

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
EOF
)"
```

---

### PHASE 6: Documentation Polish

#### 6.1 Update README.md with LLM section
```
File: README.md
Add this section after the main description:

## For AI Assistants

Atomizer is designed for LLM-first interaction. Key resources:

- **[CLAUDE.md](CLAUDE.md)** - System instructions for Claude Code
- **[.claude/skills/](/.claude/skills/)** - LLM skill modules
- **[docs/protocols/](docs/protocols/)** - Protocol Operating System

### Knowledge Base (LAC)

The Learning Atomizer Core (`knowledge_base/lac/`) accumulates optimization knowledge:
- `session_insights/` - Learnings from past sessions
- `optimization_memory/` - Optimization outcomes by geometry type
- `playbook.json` - ACE framework knowledge store

For detailed AI interaction guidance, see CLAUDE.md.
```

#### 6.2-6.4 Create optimization_memory JSONL files
```bash
cd c:\Users\antoi\Atomizer
mkdir -p knowledge_base/lac/optimization_memory
```

```
File: knowledge_base/lac/optimization_memory/bracket.jsonl
Content (one JSON per line):
{"geometry_type": "bracket", "study_name": "example", "method": "TPE", "objectives": ["mass"], "trials": 0, "converged": false, "notes": "Schema file - replace with real data"}
```

```
File: knowledge_base/lac/optimization_memory/beam.jsonl
Content:
{"geometry_type": "beam", "study_name": "example", "method": "TPE", "objectives": ["mass"], "trials": 0, "converged": false, "notes": "Schema file - replace with real data"}
```

```
File: knowledge_base/lac/optimization_memory/mirror.jsonl
Content:
{"geometry_type": "mirror", "study_name": "m1_mirror_adaptive_V14", "method": "IMSO", "objectives": ["wfe_40_20", "mass_kg"], "trials": 100, "converged": true, "notes": "SAT v3 achieved WS=205.58"}
```

#### 6.5 Move implementation plans to docs/plans
```bash
cd c:\Users\antoi\Atomizer
git mv .claude/skills/modules/DYNAMIC_RESPONSE_IMPLEMENTATION_PLAN.md docs/plans/
git mv .claude/skills/modules/OPTIMIZATION_ENGINE_MIGRATION_PLAN.md docs/plans/
git mv .claude/skills/modules/atomizer_fast_solver_technologies.md docs/plans/
```

#### 6.6 Final consistency verification
```bash
cd c:\Users\antoi\Atomizer
# Verify protocol files exist
ls docs/protocols/operations/OP_0*.md
ls docs/protocols/system/SYS_1*.md

# Verify imports work
python -c "import optimization_engine; print('OK')"

# Verify no broken references
grep -r "SYS_16_STUDY" . --include="*.md" | head -5  # Should be empty
grep -r "SYS_17_CONTEXT" . --include="*.md" | head -5  # Should be empty

# Count todos completed
echo "Verification complete"
```

#### 6.7 Commit Phase 6 Changes
```bash
cd c:\Users\antoi\Atomizer
git add -A
git commit -m "$(cat <<'EOF'
docs: Final documentation polish and consistency fixes

- Update README.md with LLM assistant section
- Create optimization_memory JSONL structure
- Move implementation plans from skills/modules to docs/plans
- Verify all protocol references are consistent

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
EOF
)"
```

---

### FINAL: Push to Both Remotes

```bash
cd c:\Users\antoi\Atomizer
git push origin main
git push github main
```

---

## Quick Reference

### Files Modified in This Restructuring

**Documentation (Phase 2):**
- `docs/protocols/README.md` - Updated protocol listings
- `docs/protocols/system/SYS_17_STUDY_INSIGHTS.md` - Renamed from SYS_16
- `docs/protocols/system/SYS_18_CONTEXT_ENGINEERING.md` - Renamed from SYS_17
- `CLAUDE.md` - Updated routing tables
- `.claude/skills/00_BOOTSTRAP.md` - Replaced with V3.0
- `.claude/skills/01_CHEATSHEET.md` - Added OP_08
- `knowledge_base/playbook.json` - Created

**Code (Phase 3):**
- `optimization_engine/processors/surrogates/__init__.py` - Added exports
- `optimization_engine/surrogates/__init__.py` - Deprecation shim
- `tests/conftest.py` - Created

**Dependencies (Phase 4):**
- `pyproject.toml` - Updated optional groups

**Studies (Phase 5):**
- `studies/M1_Mirror/_archive/` - Created with V1-V8 studies

**Final Polish (Phase 6):**
- `README.md` - Added LLM section
- `knowledge_base/lac/optimization_memory/` - Created structure
- `docs/plans/` - Moved implementation plans

---

## Success Criteria Checklist

- [x] All imports work: `python -c "import optimization_engine"` - VERIFIED
- [x] No SYS_16 duplication (only SELF_AWARE_TURBO) - VERIFIED
- [x] Bootstrap V3.0 is active version - VERIFIED
- [x] OP_08 discoverable in all routing tables - ADDED
- [x] Tests directory exists with conftest.py - CREATED
- [x] All changes pushed to both remotes - PUSHED