Atomizer/docs/archive/historical/RESTRUCTURING_PLAN.md

Atomizer Massive Restructuring Plan

Created: 2026-01-06 Purpose: Comprehensive TODO list for Ralph mode execution with skip permissions Status: COMPLETED (2026-01-07)

---

Progress Summary

All Phases Completed:

• Phase 1: Safe Cleanup
• Phase 2: Documentation (OP_08, protocol numbering, cheatsheet)
• Phase 3: Code Organization (surrogates consolidation, tests directory)
• Phase 4: Dependency Management (pyproject.toml cleanup)
• Phase 5: Study Organization (archived V1-V8 studies)
• Phase 6: Documentation Polish (README, LAC, plans)
• Final Push: Pushed to both origin and github remotes

---

RALPH MODE TODO LIST

PHASE 2 (Remaining - Documentation)

2.8 Add OP_08 to 01_CHEATSHEET.md

File: .claude/skills/01_CHEATSHEET.md
Action: Add row to "I want to..." table after OP_07 entry (around line 33)

Add this line:
| **Generate report** | **OP_08** | `python -m optimization_engine.reporting.report_generator <study>` |

Also add a section around line 280:
## Report Generation (OP_08)

### Quick Commands
| Task | Command |
|------|---------|
| Generate markdown report | `python -m optimization_engine.reporting.markdown_report <study>` |
| Generate HTML visualization | `python tools/zernike_html_generator.py <study>` |

**Full details**: `docs/protocols/operations/OP_08_GENERATE_REPORT.md`

2.9 SKIP (Already verified V3.0 Bootstrap has no circular refs)

2.10 Commit Phase 2 Changes

cd c:\Users\antoi\Atomizer
git add -A
git commit -m "$(cat <<'EOF'
docs: Consolidate documentation and fix protocol numbering

- Rename SYS_16_STUDY_INSIGHTS -> SYS_17_STUDY_INSIGHTS
- Rename SYS_17_CONTEXT_ENGINEERING -> SYS_18_CONTEXT_ENGINEERING
- Promote Bootstrap V3.0 (Context Engineering) as default
- Create knowledge_base/playbook.json for ACE framework
- Add OP_08 (Generate Report) to all routing tables
- Add SYS_16-18 to all protocol tables
- Update docs/protocols/README.md version 1.1
- Update CLAUDE.md with new protocols

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
EOF
)"

---

PHASE 3: Code Organization

3.1 Move ensemble_surrogate.py

cd c:\Users\antoi\Atomizer
git mv optimization_engine/surrogates/ensemble_surrogate.py optimization_engine/processors/surrogates/ensemble_surrogate.py

3.2 Update processors/surrogates/init.py

File: optimization_engine/processors/surrogates/__init__.py
Action: Add to __getattr__ function and __all__ list:

In __getattr__, add these elif blocks:
    elif name == 'EnsembleSurrogate':
        from .ensemble_surrogate import EnsembleSurrogate
        return EnsembleSurrogate
    elif name == 'OODDetector':
        from .ensemble_surrogate import OODDetector
        return OODDetector
    elif name == 'create_and_train_ensemble':
        from .ensemble_surrogate import create_and_train_ensemble
        return create_and_train_ensemble

In __all__, add:
    'EnsembleSurrogate',
    'OODDetector',
    'create_and_train_ensemble',

3.3 Add deprecation shim to surrogates/init.py

File: optimization_engine/surrogates/__init__.py
Replace contents with:

"""
DEPRECATED: This module has been moved to optimization_engine.processors.surrogates

Please update your imports:
    from optimization_engine.processors.surrogates import EnsembleSurrogate

This module will be removed in a future version.
"""

import warnings

warnings.warn(
    "optimization_engine.surrogates is deprecated. "
    "Use optimization_engine.processors.surrogates instead.",
    DeprecationWarning,
    stacklevel=2
)

# Redirect imports
from optimization_engine.processors.surrogates import (
    EnsembleSurrogate,
    OODDetector,
    create_and_train_ensemble
)

__all__ = ['EnsembleSurrogate', 'OODDetector', 'create_and_train_ensemble']

3.4 Check future/ imports

cd c:\Users\antoi\Atomizer
grep -r "from optimization_engine.future" --include="*.py" | grep -v "future/" | head -20

Analyze output and decide which modules need to move out of future/

3.5 Move workflow_decomposer.py (if imported by production code)

If grep shows imports from config/ or core/:

git mv optimization_engine/future/workflow_decomposer.py optimization_engine/config/workflow_decomposer.py
# Update imports in capability_matcher.py and any other files

3.6 Create tests/ directory structure

cd c:\Users\antoi\Atomizer
mkdir -p tests/unit/gnn tests/unit/extractors tests/integration tests/fixtures/sample_data

3.7 Move test files from archive/test_scripts/

cd c:\Users\antoi\Atomizer
git mv archive/test_scripts/test_neural_surrogate.py tests/unit/
git mv archive/test_scripts/test_nn_surrogate.py tests/unit/
git mv archive/test_scripts/test_parametric_surrogate.py tests/unit/
git mv archive/test_scripts/test_adaptive_characterization.py tests/unit/
git mv archive/test_scripts/test_training_data_export.py tests/unit/
git mv optimization_engine/gnn/test_*.py tests/unit/gnn/ 2>/dev/null || true
git mv optimization_engine/extractors/test_phase3_extractors.py tests/unit/extractors/ 2>/dev/null || true

3.8 Create tests/conftest.py

File: tests/conftest.py
Content:
"""
Pytest configuration and shared fixtures for Atomizer tests.
"""

import pytest
import sys
from pathlib import Path

# Add project root to path
sys.path.insert(0, str(Path(__file__).parent.parent))

@pytest.fixture
def sample_study_dir(tmp_path):
    """Create a temporary study directory structure."""
    study = tmp_path / "test_study"
    (study / "1_setup").mkdir(parents=True)
    (study / "2_iterations").mkdir()
    (study / "3_results").mkdir()
    return study

@pytest.fixture
def sample_config():
    """Sample optimization config for testing."""
    return {
        "study_name": "test_study",
        "design_variables": [
            {"name": "param1", "lower": 0, "upper": 10, "type": "continuous"}
        ],
        "objectives": [
            {"name": "minimize_mass", "direction": "minimize"}
        ]
    }

3.9 Rename bracket_displacement_maximizing/results to 3_results

cd c:\Users\antoi\Atomizer
# Check if results/ exists first
if [ -d "studies/bracket_displacement_maximizing/results" ]; then
    git mv studies/bracket_displacement_maximizing/results studies/bracket_displacement_maximizing/3_results
fi

3.10 Rename Drone_Gimbal/2_results to 3_results

cd c:\Users\antoi\Atomizer
# Check if 2_results/ exists first
if [ -d "studies/Drone_Gimbal/2_results" ]; then
    git mv studies/Drone_Gimbal/2_results studies/Drone_Gimbal/3_results
fi

3.11 Commit Phase 3 Changes

cd c:\Users\antoi\Atomizer
git add -A
git commit -m "$(cat <<'EOF'
refactor: Reorganize code structure and create tests directory

- Consolidate surrogates module to processors/surrogates/
- Add deprecation shim for old import path
- Create tests/ directory with pytest structure
- Move test files from archive/test_scripts/
- Standardize study folder naming (3_results/)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
EOF
)"

---

PHASE 4: Dependency Management

4.1-4.2 Add neural and gnn optional deps to pyproject.toml

File: pyproject.toml
After the [project.optional-dependencies] section, add:

neural = [
    "torch>=2.0.0",
    "torch-geometric>=2.3.0",
    "tensorboard>=2.13.0",
]

gnn = [
    "torch>=2.0.0",
    "torch-geometric>=2.3.0",
]

all = ["atomizer[neural,gnn,dev,dashboard]"]

4.3 Remove mcp optional deps

File: pyproject.toml
Delete this section:

mcp = [
    "mcp>=0.1.0",
]

4.4 Remove mcp_server from packages.find

File: pyproject.toml
Change:
    include = ["mcp_server*", "optimization_engine*", "nx_journals*"]
To:
    include = ["optimization_engine*", "nx_journals*"]

4.5 Commit Phase 4 Changes

cd c:\Users\antoi\Atomizer
git add pyproject.toml
git commit -m "$(cat <<'EOF'
build: Add optional dependency groups and clean up pyproject.toml

- Add neural optional group (torch, torch-geometric, tensorboard)
- Add gnn optional group (torch, torch-geometric)
- Add all optional group for convenience
- Remove mcp optional group (not implemented)
- Remove mcp_server from packages.find (not implemented)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
EOF
)"

---

PHASE 5: Study Organization

5.1 Create archive directory

cd c:\Users\antoi\Atomizer
mkdir -p studies/M1_Mirror/_archive

5.2 Move V1-V8 cost_reduction studies

cd c:\Users\antoi\Atomizer/studies/M1_Mirror
# Move cost_reduction V2-V8 (V1 doesn't exist as base is just "cost_reduction")
for v in V2 V3 V4 V5 V6 V7 V8; do
    if [ -d "m1_mirror_cost_reduction_$v" ]; then
        mv "m1_mirror_cost_reduction_$v" _archive/
    fi
done

5.3 Move V1-V8 flat_back studies

cd c:\Users\antoi\Atomizer/studies/M1_Mirror
# Move flat_back V1-V8 (note: V2 may not exist)
for v in V1 V3 V4 V5 V6 V7 V8; do
    if [ -d "m1_mirror_cost_reduction_flat_back_$v" ]; then
        mv "m1_mirror_cost_reduction_flat_back_$v" _archive/
    fi
done

5.4 Create MANIFEST.md

File: studies/M1_Mirror/_archive/MANIFEST.md
Content:

# M1 Mirror Archived Studies

**Archived:** 2026-01-06
**Reason:** Repository cleanup - keeping only V9+ studies active

## Archived Studies

### Cost Reduction Series
| Study | Trials | Best WS | Notes |
|-------|--------|---------|-------|
| V2 | TBD | TBD | Early exploration |
| V3 | TBD | TBD | - |
| V4 | TBD | TBD | - |
| V5 | TBD | TBD | - |
| V6 | TBD | TBD | - |
| V7 | TBD | TBD | - |
| V8 | TBD | TBD | - |

### Cost Reduction Flat Back Series
| Study | Trials | Best WS | Notes |
|-------|--------|---------|-------|
| V1 | TBD | TBD | Initial flat back design |
| V3 | TBD | TBD | V2 was skipped |
| V4 | TBD | TBD | - |
| V5 | TBD | TBD | - |
| V6 | TBD | TBD | - |
| V7 | TBD | TBD | - |
| V8 | TBD | TBD | - |

## Restoration Instructions

To restore a study:
1. Move from _archive/ to parent directory
2. Verify database integrity: `sqlite3 3_results/study.db ".tables"`
3. Check optimization_config.json exists

5.5 Commit Phase 5 Changes

cd c:\Users\antoi\Atomizer
git add -A
git commit -m "$(cat <<'EOF'
chore: Archive old M1_Mirror studies (V1-V8)

- Create studies/M1_Mirror/_archive/ directory
- Move cost_reduction V2-V8 to archive
- Move flat_back V1-V8 to archive
- Create MANIFEST.md documenting archived studies
- Keep V9+ studies active for reference

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
EOF
)"

---

PHASE 6: Documentation Polish

6.1 Update README.md with LLM section

File: README.md
Add this section after the main description:

## For AI Assistants

Atomizer is designed for LLM-first interaction. Key resources:

- **[CLAUDE.md](CLAUDE.md)** - System instructions for Claude Code
- **[.claude/skills/](/.claude/skills/)** - LLM skill modules
- **[docs/protocols/](docs/protocols/)** - Protocol Operating System

### Knowledge Base (LAC)

The Learning Atomizer Core (`knowledge_base/lac/`) accumulates optimization knowledge:
- `session_insights/` - Learnings from past sessions
- `optimization_memory/` - Optimization outcomes by geometry type
- `playbook.json` - ACE framework knowledge store

For detailed AI interaction guidance, see CLAUDE.md.

6.2-6.4 Create optimization_memory JSONL files

cd c:\Users\antoi\Atomizer
mkdir -p knowledge_base/lac/optimization_memory

File: knowledge_base/lac/optimization_memory/bracket.jsonl
Content (one JSON per line):
{"geometry_type": "bracket", "study_name": "example", "method": "TPE", "objectives": ["mass"], "trials": 0, "converged": false, "notes": "Schema file - replace with real data"}

File: knowledge_base/lac/optimization_memory/beam.jsonl
Content:
{"geometry_type": "beam", "study_name": "example", "method": "TPE", "objectives": ["mass"], "trials": 0, "converged": false, "notes": "Schema file - replace with real data"}

File: knowledge_base/lac/optimization_memory/mirror.jsonl
Content:
{"geometry_type": "mirror", "study_name": "m1_mirror_adaptive_V14", "method": "IMSO", "objectives": ["wfe_40_20", "mass_kg"], "trials": 100, "converged": true, "notes": "SAT v3 achieved WS=205.58"}

6.5 Move implementation plans to docs/plans

cd c:\Users\antoi\Atomizer
git mv .claude/skills/modules/DYNAMIC_RESPONSE_IMPLEMENTATION_PLAN.md docs/plans/
git mv .claude/skills/modules/OPTIMIZATION_ENGINE_MIGRATION_PLAN.md docs/plans/
git mv .claude/skills/modules/atomizer_fast_solver_technologies.md docs/plans/

6.6 Final consistency verification

cd c:\Users\antoi\Atomizer
# Verify protocol files exist
ls docs/protocols/operations/OP_0*.md
ls docs/protocols/system/SYS_1*.md

# Verify imports work
python -c "import optimization_engine; print('OK')"

# Verify no broken references
grep -r "SYS_16_STUDY" . --include="*.md" | head -5  # Should be empty
grep -r "SYS_17_CONTEXT" . --include="*.md" | head -5  # Should be empty

# Count todos completed
echo "Verification complete"

6.7 Commit Phase 6 Changes

cd c:\Users\antoi\Atomizer
git add -A
git commit -m "$(cat <<'EOF'
docs: Final documentation polish and consistency fixes

- Update README.md with LLM assistant section
- Create optimization_memory JSONL structure
- Move implementation plans from skills/modules to docs/plans
- Verify all protocol references are consistent

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
EOF
)"

---

FINAL: Push to Both Remotes

cd c:\Users\antoi\Atomizer
git push origin main
git push github main

---

Quick Reference

Files Modified in This Restructuring

Documentation (Phase 2):

• docs/protocols/README.md - Updated protocol listings
• docs/protocols/system/SYS_17_STUDY_INSIGHTS.md - Renamed from SYS_16
• docs/protocols/system/SYS_18_CONTEXT_ENGINEERING.md - Renamed from SYS_17
• CLAUDE.md - Updated routing tables
• .claude/skills/00_BOOTSTRAP.md - Replaced with V3.0
• .claude/skills/01_CHEATSHEET.md - Added OP_08
• knowledge_base/playbook.json - Created

Code (Phase 3):

• optimization_engine/processors/surrogates/__init__.py - Added exports
• optimization_engine/surrogates/__init__.py - Deprecation shim
• tests/conftest.py - Created

Dependencies (Phase 4):

• pyproject.toml - Updated optional groups

Studies (Phase 5):

• studies/M1_Mirror/_archive/ - Created with V1-V8 studies

Final Polish (Phase 6):

• README.md - Added LLM section
• knowledge_base/lac/optimization_memory/ - Created structure
• docs/plans/ - Moved implementation plans

---

Success Criteria Checklist

• All imports work: python -c "import optimization_engine" - VERIFIED
• No SYS_16 duplication (only SELF_AWARE_TURBO) - VERIFIED
• Bootstrap V3.0 is active version - VERIFIED
• OP_08 discoverable in all routing tables - ADDED
• Tests directory exists with conftest.py - CREATED
• All changes pushed to both remotes - PUSHED