Atomizer/.claude/skills/modules/OPTIMIZATION_ENGINE_MIGRATION_PLAN.md

# Optimization Engine Reorganization - Migration Plan

## Comprehensive Guide for Safe Codebase Restructuring

**Document Version**: 1.0
**Created**: 2025-12-23
**Status**: PLANNING - Do Not Execute Without Review
**Risk Level**: HIGH - Affects 557+ locations across 276 files

---

## Executive Summary

This document provides a complete migration plan for reorganizing `optimization_engine/` from 50+ loose files to a clean modular structure. This is a **high-impact refactoring** that requires careful execution.

### Impact Summary

| Category | Files Affected | Lines to Change |
|----------|----------------|-----------------|
| Python imports (internal) | 90+ files | ~145 changes |
| Python imports (studies) | 30+ folders | ~153 changes |
| Python imports (tests) | 30+ files | ~79 changes |
| Python imports (dashboard) | 3 files | ~11 changes |
| Documentation (protocols, skills) | 119 files | ~200 changes |
| JSON configs | 4 files | ~50 changes |
| **TOTAL** | **276 files** | **~640 changes** |

---

## Part 1: Current State Analysis

### 1.1 Top-Level Files Requiring Migration

These 50+ files at `optimization_engine/` root need to move:

```
SURROGATES (6 files) → processors/surrogates/
├── neural_surrogate.py
├── generic_surrogate.py
├── adaptive_surrogate.py
├── simple_mlp_surrogate.py
├── active_learning_surrogate.py
└── surrogate_tuner.py

OPTIMIZATION CORE (7 files) → core/
├── runner.py
├── runner_with_neural.py
├── base_runner.py
├── intelligent_optimizer.py
├── method_selector.py
├── strategy_selector.py
└── strategy_portfolio.py

NX INTEGRATION (6 files) → nx/
├── nx_solver.py
├── nx_updater.py
├── nx_session_manager.py
├── solve_simulation.py
├── solve_simulation_simple.py
└── model_cleanup.py

STUDY MANAGEMENT (5 files) → study/
├── study_creator.py
├── study_wizard.py
├── study_state.py
├── study_reset.py
└── study_continuation.py

REPORTING (5 files) → reporting/
├── generate_report.py
├── generate_report_markdown.py
├── comprehensive_results_analyzer.py
├── visualizer.py
└── landscape_analyzer.py

CONFIG (4 files) → config/
├── config_manager.py
├── optimization_config_builder.py
├── optimization_setup_wizard.py
└── capability_matcher.py

AGENTS/RESEARCH (5 files) → agents/ or future/
├── research_agent.py
├── pynastran_research_agent.py
├── targeted_research_planner.py
├── workflow_decomposer.py
└── step_classifier.py

MISC (remaining ~15 files) - evaluate individually
├── logger.py → utils/
├── op2_extractor.py → extractors/
├── extractor_library.py → extractors/
├── export_expressions.py → nx/
├── import_expressions.py → nx/
├── mesh_converter.py → nx/
├── simulation_validator.py → validators/
├── auto_doc.py → utils/
├── auto_trainer.py → processors/surrogates/
├── realtime_tracking.py → utils/
├── benchmarking_substudy.py → study/
├── codebase_analyzer.py → utils/
├── training_data_exporter.py → processors/surrogates/
├── pruning_logger.py → utils/
├── adaptive_characterization.py → processors/
└── generate_history_from_trials.py → study/
```

### 1.2 Well-Organized Directories (Keep As-Is)

These are already properly organized:

```
extractors/          ✓ 20+ extractors, clean __init__.py
insights/            ✓ 8 insight types, registry pattern
hooks/               ✓ nx_cad/, nx_cae/ subdirs
gnn/                 ✓ Neural surrogate for Zernike
templates/           ✓ Config templates
schemas/             ✓ JSON schemas
validators/          ✓ Validation code
plugins/             ✓ Hook manager system
utils/               ✓ Utility functions
custom_functions/    ✓ NX material generator
model_discovery/     ✓ Model introspection
future/              ✓ Experimental code
```

---

## Part 2: Target Structure

### 2.1 Proposed Final Structure

```
optimization_engine/
│
├── __init__.py                    # Updated with backwards-compat aliases
│
├── core/                          # NEW - Optimization engine core
│   ├── __init__.py
│   ├── runner.py
│   ├── base_runner.py
│   ├── runner_with_neural.py
│   ├── intelligent_optimizer.py
│   ├── method_selector.py
│   ├── strategy_selector.py
│   └── strategy_portfolio.py
│
├── processors/                    # NEW - Data processing & algorithms
│   ├── __init__.py
│   ├── surrogates/
│   │   ├── __init__.py
│   │   ├── neural_surrogate.py
│   │   ├── generic_surrogate.py
│   │   ├── adaptive_surrogate.py
│   │   ├── simple_mlp_surrogate.py
│   │   ├── active_learning_surrogate.py
│   │   ├── surrogate_tuner.py
│   │   ├── auto_trainer.py
│   │   └── training_data_exporter.py
│   │
│   └── dynamic_response/          # NEW - From master plan
│       ├── __init__.py
│       ├── modal_database.py
│       ├── transfer_functions.py
│       ├── random_vibration.py
│       ├── psd_profiles.py
│       └── utils/
│
├── nx/                            # NEW - NX/Nastran integration
│   ├── __init__.py
│   ├── solver.py                  # Was nx_solver.py
│   ├── updater.py                 # Was nx_updater.py
│   ├── session_manager.py         # Was nx_session_manager.py
│   ├── solve_simulation.py
│   ├── solve_simulation_simple.py
│   ├── model_cleanup.py
│   ├── export_expressions.py
│   ├── import_expressions.py
│   └── mesh_converter.py
│
├── study/                         # NEW - Study management
│   ├── __init__.py
│   ├── creator.py                 # Was study_creator.py
│   ├── wizard.py                  # Was study_wizard.py
│   ├── state.py                   # Was study_state.py
│   ├── reset.py                   # Was study_reset.py
│   ├── continuation.py            # Was study_continuation.py
│   ├── benchmarking.py            # Was benchmarking_substudy.py
│   └── history_generator.py       # Was generate_history_from_trials.py
│
├── reporting/                     # NEW - Reports and analysis
│   ├── __init__.py
│   ├── report_generator.py
│   ├── markdown_report.py
│   ├── results_analyzer.py
│   ├── visualizer.py
│   └── landscape_analyzer.py
│
├── config/                        # NEW - Configuration
│   ├── __init__.py
│   ├── manager.py
│   ├── builder.py
│   ├── setup_wizard.py
│   └── capability_matcher.py
│
├── extractors/                    # EXISTING - Add op2_extractor, extractor_library
├── insights/                      # EXISTING
├── hooks/                         # EXISTING
├── gnn/                           # EXISTING
├── templates/                     # EXISTING
├── schemas/                       # EXISTING
├── validators/                    # EXISTING - Add simulation_validator
├── plugins/                       # EXISTING
├── utils/                         # EXISTING - Add logger, auto_doc, etc.
├── custom_functions/              # EXISTING
├── model_discovery/               # EXISTING
└── future/                        # EXISTING - Move agents here
    ├── research_agent.py
    ├── pynastran_research_agent.py
    ├── targeted_research_planner.py
    ├── workflow_decomposer.py
    └── step_classifier.py
```

---

## Part 3: Import Mapping Tables

### 3.1 Old → New Import Mapping

This is the critical reference for all updates:

```python
# CORE OPTIMIZATION
"from optimization_engine.runner"                    → "from optimization_engine.core.runner"
"from optimization_engine.base_runner"               → "from optimization_engine.core.base_runner"
"from optimization_engine.runner_with_neural"        → "from optimization_engine.core.runner_with_neural"
"from optimization_engine.intelligent_optimizer"     → "from optimization_engine.core.intelligent_optimizer"
"from optimization_engine.method_selector"           → "from optimization_engine.core.method_selector"
"from optimization_engine.strategy_selector"         → "from optimization_engine.core.strategy_selector"
"from optimization_engine.strategy_portfolio"        → "from optimization_engine.core.strategy_portfolio"

# SURROGATES
"from optimization_engine.neural_surrogate"          → "from optimization_engine.processors.surrogates.neural_surrogate"
"from optimization_engine.generic_surrogate"         → "from optimization_engine.processors.surrogates.generic_surrogate"
"from optimization_engine.adaptive_surrogate"        → "from optimization_engine.processors.surrogates.adaptive_surrogate"
"from optimization_engine.simple_mlp_surrogate"      → "from optimization_engine.processors.surrogates.simple_mlp_surrogate"
"from optimization_engine.active_learning_surrogate" → "from optimization_engine.processors.surrogates.active_learning_surrogate"
"from optimization_engine.surrogate_tuner"           → "from optimization_engine.processors.surrogates.surrogate_tuner"

# NX INTEGRATION
"from optimization_engine.nx_solver"                 → "from optimization_engine.nx.solver"
"from optimization_engine.nx_updater"                → "from optimization_engine.nx.updater"
"from optimization_engine.nx_session_manager"        → "from optimization_engine.nx.session_manager"
"from optimization_engine.solve_simulation"          → "from optimization_engine.nx.solve_simulation"
"from optimization_engine.model_cleanup"             → "from optimization_engine.nx.model_cleanup"
"from optimization_engine.export_expressions"        → "from optimization_engine.nx.export_expressions"
"from optimization_engine.import_expressions"        → "from optimization_engine.nx.import_expressions"
"from optimization_engine.mesh_converter"            → "from optimization_engine.nx.mesh_converter"

# STUDY MANAGEMENT
"from optimization_engine.study_creator"             → "from optimization_engine.study.creator"
"from optimization_engine.study_wizard"              → "from optimization_engine.study.wizard"
"from optimization_engine.study_state"               → "from optimization_engine.study.state"
"from optimization_engine.study_reset"               → "from optimization_engine.study.reset"
"from optimization_engine.study_continuation"        → "from optimization_engine.study.continuation"

# REPORTING
"from optimization_engine.generate_report"           → "from optimization_engine.reporting.report_generator"
"from optimization_engine.generate_report_markdown"  → "from optimization_engine.reporting.markdown_report"
"from optimization_engine.comprehensive_results"     → "from optimization_engine.reporting.results_analyzer"
"from optimization_engine.visualizer"                → "from optimization_engine.reporting.visualizer"
"from optimization_engine.landscape_analyzer"        → "from optimization_engine.reporting.landscape_analyzer"

# CONFIG
"from optimization_engine.config_manager"            → "from optimization_engine.config.manager"
"from optimization_engine.optimization_config_builder" → "from optimization_engine.config.builder"
"from optimization_engine.optimization_setup_wizard" → "from optimization_engine.config.setup_wizard"
"from optimization_engine.capability_matcher"        → "from optimization_engine.config.capability_matcher"

# UTILITIES (moving to utils/)
"from optimization_engine.logger"                    → "from optimization_engine.utils.logger"
"from optimization_engine.auto_doc"                  → "from optimization_engine.utils.auto_doc"
"from optimization_engine.realtime_tracking"         → "from optimization_engine.utils.realtime_tracking"
"from optimization_engine.codebase_analyzer"         → "from optimization_engine.utils.codebase_analyzer"
"from optimization_engine.pruning_logger"            → "from optimization_engine.utils.pruning_logger"

# RESEARCH/AGENTS (moving to future/)
"from optimization_engine.research_agent"            → "from optimization_engine.future.research_agent"
"from optimization_engine.workflow_decomposer"       → "from optimization_engine.future.workflow_decomposer"
"from optimization_engine.step_classifier"           → "from optimization_engine.future.step_classifier"
```

### 3.2 Backwards Compatibility Aliases

Add to `optimization_engine/__init__.py` for transition period:

```python
# BACKWARDS COMPATIBILITY ALIASES
# These allow old imports to work during migration period
# Remove after all code is updated

# Core
from optimization_engine.core.runner import *
from optimization_engine.core.base_runner import *
from optimization_engine.core.intelligent_optimizer import *

# NX
from optimization_engine.nx import solver as nx_solver
from optimization_engine.nx import updater as nx_updater
from optimization_engine.nx import session_manager as nx_session_manager
from optimization_engine.nx import solve_simulation

# Study
from optimization_engine.study import creator as study_creator
from optimization_engine.study import wizard as study_wizard
from optimization_engine.study import state as study_state

# Surrogates
from optimization_engine.processors.surrogates import neural_surrogate
from optimization_engine.processors.surrogates import generic_surrogate

# Config
from optimization_engine.config import manager as config_manager

# Utils
from optimization_engine.utils import logger

# Deprecation warnings (optional)
import warnings
def __getattr__(name):
    deprecated = {
        'nx_solver': 'optimization_engine.nx.solver',
        'study_creator': 'optimization_engine.study.creator',
        # ... more mappings
    }
    if name in deprecated:
        warnings.warn(
            f"Importing {name} from optimization_engine is deprecated. "
            f"Use {deprecated[name]} instead.",
            DeprecationWarning,
            stacklevel=2
        )
        # Return the module anyway for compatibility
    raise AttributeError(f"module 'optimization_engine' has no attribute '{name}'")
```

---

## Part 4: Files Requiring Updates

### 4.1 Internal optimization_engine Files (90+ files)

**Highest-impact internal files:**

| File | Imports to Update | Priority |
|------|-------------------|----------|
| `runner.py` | nx_solver, config_manager, extractors | Critical |
| `base_runner.py` | config_manager, validators | Critical |
| `intelligent_optimizer.py` | runner, neural_surrogate, method_selector | Critical |
| `runner_with_neural.py` | runner, neural_surrogate | High |
| `gnn/gnn_optimizer.py` | polar_graph, zernike_gnn, nx_solver | High |
| `hooks/nx_cad/*.py` | nx_session_manager | High |
| `plugins/hook_manager.py` | validators, config | Medium |

### 4.2 Study Scripts (30+ folders, 153 imports)

**Pattern in every study's `run_optimization.py`:**

```python
# BEFORE
from optimization_engine.nx_solver import run_nx_simulation
from optimization_engine.extractors import ZernikeExtractor
from optimization_engine.gnn.gnn_optimizer import ZernikeGNNOptimizer

# AFTER
from optimization_engine.nx.solver import run_nx_simulation
from optimization_engine.extractors import ZernikeExtractor  # unchanged
from optimization_engine.gnn.gnn_optimizer import ZernikeGNNOptimizer  # unchanged
```

**Studies requiring updates:**

```
studies/
├── M1_Mirror/
│   ├── m1_mirror_adaptive_V*/run_optimization.py (12 files)
│   └── [other variants]
├── Simple_Bracket/
│   ├── bracket_*/run_optimization.py (8 files)
├── UAV_Arm/
│   ├── */run_optimization.py (4 files)
├── Drone_Gimbal/
├── Simple_Beam/
└── [others]
```

### 4.3 Test Files (30+ files, 79 imports)

Located in `tests/`:

```
tests/
├── test_extractors.py
├── test_zernike_*.py (5+ files)
├── test_neural_surrogate.py
├── test_gnn_*.py
├── test_nx_solver.py
├── test_study_*.py
└── [others]
```

### 4.4 Dashboard Backend (3 files, 11 imports)

```
atomizer-dashboard/backend/api/
├── main.py                # sys.path setup
├── routes/optimization.py # study management imports
└── routes/insights.py     # insight imports
```

### 4.5 Documentation Files (119 files)

**Protocols requiring updates:**

| Protocol | References | Changes Needed |
|----------|------------|----------------|
| SYS_10_IMSO.md | intelligent_optimizer | core.intelligent_optimizer |
| SYS_12_EXTRACTOR_LIBRARY.md | extractors/* | None (unchanged) |
| SYS_14_NEURAL_ACCELERATION.md | neural_surrogate, gnn/* | processors.surrogates.* |
| SYS_15_METHOD_SELECTOR.md | method_selector | core.method_selector |
| OP_01_CREATE_STUDY.md | study_creator, study_wizard | study.creator, study.wizard |
| OP_02_RUN_OPTIMIZATION.md | runner, nx_solver | core.runner, nx.solver |

**Skill files requiring updates:**

| Skill File | Changes |
|------------|---------|
| 01_CHEATSHEET.md | Path references |
| core/study-creation-core.md | Import examples |
| modules/extractors-catalog.md | Directory paths |
| modules/neural-acceleration.md | Surrogate imports |

### 4.6 JSON Configuration Files (4 files)

**feature_registry.json** (878 lines, 50+ path references):

```json
// BEFORE
"file_path": "optimization_engine/neural_surrogate.py"

// AFTER
"file_path": "optimization_engine/processors/surrogates/neural_surrogate.py"
```

---

## Part 5: Migration Execution Plan

### Phase 0: Pre-Migration (30 min)

1. **Create full backup**
   ```bash
   git stash  # Save any uncommitted changes
   git checkout -b refactor/optimization-engine-reorganization
   cp -r optimization_engine optimization_engine_backup
   ```

2. **Run baseline tests**
   ```bash
   python -m pytest tests/ -v --tb=short > baseline_tests.log 2>&1
   ```

3. **Document current working state**
   - Run one study end-to-end
   - Verify dashboard loads
   - Note any existing failures

### Phase 1: Create Directory Structure (15 min)

```bash
# Create new directories
mkdir -p optimization_engine/core
mkdir -p optimization_engine/processors/surrogates
mkdir -p optimization_engine/processors/dynamic_response
mkdir -p optimization_engine/nx
mkdir -p optimization_engine/study
mkdir -p optimization_engine/reporting
mkdir -p optimization_engine/config

# Create __init__.py files
touch optimization_engine/core/__init__.py
touch optimization_engine/processors/__init__.py
touch optimization_engine/processors/surrogates/__init__.py
touch optimization_engine/processors/dynamic_response/__init__.py
touch optimization_engine/nx/__init__.py
touch optimization_engine/study/__init__.py
touch optimization_engine/reporting/__init__.py
touch optimization_engine/config/__init__.py
```

### Phase 2: Move Files (30 min)

**Execute in this order to minimize circular import issues:**

```bash
# 1. UTILITIES FIRST (no dependencies)
mv optimization_engine/logger.py optimization_engine/utils/
mv optimization_engine/auto_doc.py optimization_engine/utils/
mv optimization_engine/realtime_tracking.py optimization_engine/utils/
mv optimization_engine/codebase_analyzer.py optimization_engine/utils/
mv optimization_engine/pruning_logger.py optimization_engine/utils/

# 2. CONFIG (low dependencies)
mv optimization_engine/config_manager.py optimization_engine/config/manager.py
mv optimization_engine/optimization_config_builder.py optimization_engine/config/builder.py
mv optimization_engine/optimization_setup_wizard.py optimization_engine/config/setup_wizard.py
mv optimization_engine/capability_matcher.py optimization_engine/config/capability_matcher.py

# 3. NX INTEGRATION
mv optimization_engine/nx_solver.py optimization_engine/nx/solver.py
mv optimization_engine/nx_updater.py optimization_engine/nx/updater.py
mv optimization_engine/nx_session_manager.py optimization_engine/nx/session_manager.py
mv optimization_engine/solve_simulation.py optimization_engine/nx/
mv optimization_engine/solve_simulation_simple.py optimization_engine/nx/
mv optimization_engine/model_cleanup.py optimization_engine/nx/
mv optimization_engine/export_expressions.py optimization_engine/nx/
mv optimization_engine/import_expressions.py optimization_engine/nx/
mv optimization_engine/mesh_converter.py optimization_engine/nx/

# 4. SURROGATES
mv optimization_engine/neural_surrogate.py optimization_engine/processors/surrogates/
mv optimization_engine/generic_surrogate.py optimization_engine/processors/surrogates/
mv optimization_engine/adaptive_surrogate.py optimization_engine/processors/surrogates/
mv optimization_engine/simple_mlp_surrogate.py optimization_engine/processors/surrogates/
mv optimization_engine/active_learning_surrogate.py optimization_engine/processors/surrogates/
mv optimization_engine/surrogate_tuner.py optimization_engine/processors/surrogates/
mv optimization_engine/auto_trainer.py optimization_engine/processors/surrogates/
mv optimization_engine/training_data_exporter.py optimization_engine/processors/surrogates/

# 5. STUDY MANAGEMENT
mv optimization_engine/study_creator.py optimization_engine/study/creator.py
mv optimization_engine/study_wizard.py optimization_engine/study/wizard.py
mv optimization_engine/study_state.py optimization_engine/study/state.py
mv optimization_engine/study_reset.py optimization_engine/study/reset.py
mv optimization_engine/study_continuation.py optimization_engine/study/continuation.py
mv optimization_engine/benchmarking_substudy.py optimization_engine/study/benchmarking.py
mv optimization_engine/generate_history_from_trials.py optimization_engine/study/history_generator.py

# 6. REPORTING
mv optimization_engine/generate_report.py optimization_engine/reporting/report_generator.py
mv optimization_engine/generate_report_markdown.py optimization_engine/reporting/markdown_report.py
mv optimization_engine/comprehensive_results_analyzer.py optimization_engine/reporting/results_analyzer.py
mv optimization_engine/visualizer.py optimization_engine/reporting/
mv optimization_engine/landscape_analyzer.py optimization_engine/reporting/

# 7. CORE (depends on many things, do last)
mv optimization_engine/runner.py optimization_engine/core/
mv optimization_engine/base_runner.py optimization_engine/core/
mv optimization_engine/runner_with_neural.py optimization_engine/core/
mv optimization_engine/intelligent_optimizer.py optimization_engine/core/
mv optimization_engine/method_selector.py optimization_engine/core/
mv optimization_engine/strategy_selector.py optimization_engine/core/
mv optimization_engine/strategy_portfolio.py optimization_engine/core/

# 8. RESEARCH/FUTURE
mv optimization_engine/research_agent.py optimization_engine/future/
mv optimization_engine/pynastran_research_agent.py optimization_engine/future/
mv optimization_engine/targeted_research_planner.py optimization_engine/future/
mv optimization_engine/workflow_decomposer.py optimization_engine/future/
mv optimization_engine/step_classifier.py optimization_engine/future/

# 9. REMAINING MISC
mv optimization_engine/op2_extractor.py optimization_engine/extractors/
mv optimization_engine/extractor_library.py optimization_engine/extractors/
mv optimization_engine/simulation_validator.py optimization_engine/validators/
mv optimization_engine/adaptive_characterization.py optimization_engine/processors/
```

### Phase 3: Update Internal Imports (1-2 hours)

**Use sed/grep for bulk updates:**

```bash
# Example sed commands (run from project root)

# NX imports
find . -name "*.py" -exec sed -i 's/from optimization_engine\.nx_solver/from optimization_engine.nx.solver/g' {} +
find . -name "*.py" -exec sed -i 's/from optimization_engine\.nx_updater/from optimization_engine.nx.updater/g' {} +
find . -name "*.py" -exec sed -i 's/from optimization_engine\.nx_session_manager/from optimization_engine.nx.session_manager/g' {} +
find . -name "*.py" -exec sed -i 's/from optimization_engine\.solve_simulation/from optimization_engine.nx.solve_simulation/g' {} +

# Study imports
find . -name "*.py" -exec sed -i 's/from optimization_engine\.study_creator/from optimization_engine.study.creator/g' {} +
find . -name "*.py" -exec sed -i 's/from optimization_engine\.study_wizard/from optimization_engine.study.wizard/g' {} +
find . -name "*.py" -exec sed -i 's/from optimization_engine\.study_state/from optimization_engine.study.state/g' {} +

# Config imports
find . -name "*.py" -exec sed -i 's/from optimization_engine\.config_manager/from optimization_engine.config.manager/g' {} +

# Core imports
find . -name "*.py" -exec sed -i 's/from optimization_engine\.runner import/from optimization_engine.core.runner import/g' {} +
find . -name "*.py" -exec sed -i 's/from optimization_engine\.base_runner/from optimization_engine.core.base_runner/g' {} +
find . -name "*.py" -exec sed -i 's/from optimization_engine\.intelligent_optimizer/from optimization_engine.core.intelligent_optimizer/g' {} +

# Surrogate imports
find . -name "*.py" -exec sed -i 's/from optimization_engine\.neural_surrogate/from optimization_engine.processors.surrogates.neural_surrogate/g' {} +
find . -name "*.py" -exec sed -i 's/from optimization_engine\.generic_surrogate/from optimization_engine.processors.surrogates.generic_surrogate/g' {} +

# Logger
find . -name "*.py" -exec sed -i 's/from optimization_engine\.logger/from optimization_engine.utils.logger/g' {} +
```

**Handle edge cases manually:**
- `import optimization_engine.nx_solver` (without `from`)
- Dynamic imports using `importlib`
- String references in configs

### Phase 4: Create __init__.py Files (30 min)

**Example: `optimization_engine/core/__init__.py`**

```python
"""
Optimization Engine Core

Main optimization runners and algorithm selection.
"""

from .runner import OptimizationRunner
from .base_runner import BaseRunner
from .intelligent_optimizer import IntelligentOptimizer, IMSO
from .method_selector import MethodSelector
from .strategy_selector import StrategySelector
from .strategy_portfolio import StrategyPortfolio

__all__ = [
    'OptimizationRunner',
    'BaseRunner',
    'IntelligentOptimizer',
    'IMSO',
    'MethodSelector',
    'StrategySelector',
    'StrategyPortfolio',
]
```

**Create similar for each new directory.**

### Phase 5: Add Backwards Compatibility (30 min)

Update `optimization_engine/__init__.py`:

```python
"""
Optimization Engine for Atomizer

Reorganized structure (v2.0):
- core/         - Optimization runners
- processors/   - Data processing (surrogates, dynamic response)
- nx/           - NX/Nastran integration
- study/        - Study management
- reporting/    - Reports and analysis
- config/       - Configuration
- extractors/   - Physics extraction
- insights/     - Visualizations
- gnn/          - Graph neural networks
- hooks/        - NX hooks
"""

# Re-export commonly used items at top level for convenience
from optimization_engine.core.runner import OptimizationRunner
from optimization_engine.core.intelligent_optimizer import IMSO
from optimization_engine.nx.solver import run_nx_simulation, NXSolver
from optimization_engine.study.creator import create_study
from optimization_engine.config.manager import ConfigManager

# Backwards compatibility aliases (deprecated)
# These will be removed in a future version
import warnings as _warnings

def _deprecated_import(old_name, new_location):
    _warnings.warn(
        f"Importing '{old_name}' directly from optimization_engine is deprecated. "
        f"Use '{new_location}' instead.",
        DeprecationWarning,
        stacklevel=3
    )

# Lazy loading for backwards compatibility
def __getattr__(name):
    # Map old names to new locations
    _compat_map = {
        'nx_solver': ('optimization_engine.nx.solver', 'nx_solver'),
        'nx_updater': ('optimization_engine.nx.updater', 'nx_updater'),
        'study_creator': ('optimization_engine.study.creator', 'study_creator'),
        'config_manager': ('optimization_engine.config.manager', 'config_manager'),
        'runner': ('optimization_engine.core.runner', 'runner'),
        'neural_surrogate': ('optimization_engine.processors.surrogates.neural_surrogate', 'neural_surrogate'),
    }

    if name in _compat_map:
        module_path, attr = _compat_map[name]
        _deprecated_import(name, module_path)
        import importlib
        module = importlib.import_module(module_path)
        return module

    raise AttributeError(f"module 'optimization_engine' has no attribute '{name}'")
```

### Phase 6: Update Documentation (1 hour)

**Use sed for bulk updates in markdown:**

```bash
# Update protocol files
find docs/protocols -name "*.md" -exec sed -i 's/optimization_engine\/nx_solver/optimization_engine\/nx\/solver/g' {} +
find docs/protocols -name "*.md" -exec sed -i 's/optimization_engine\/study_creator/optimization_engine\/study\/creator/g' {} +
# ... etc for all mappings

# Update .claude skills
find .claude -name "*.md" -exec sed -i 's/optimization_engine\.nx_solver/optimization_engine.nx.solver/g' {} +
# ... etc
```

**Manual review needed for:**
- Code examples in markdown
- Directory structure diagrams
- Command-line examples

### Phase 7: Update JSON Configs (30 min)

**feature_registry.json** - Use a Python script:

```python
import json

with open('optimization_engine/feature_registry.json', 'r') as f:
    registry = json.load(f)

# Define path mappings
path_map = {
    'optimization_engine/neural_surrogate.py': 'optimization_engine/processors/surrogates/neural_surrogate.py',
    'optimization_engine/nx_solver.py': 'optimization_engine/nx/solver.py',
    # ... all other mappings
}

def update_paths(obj):
    if isinstance(obj, dict):
        for key, value in obj.items():
            if key == 'file_path' and value in path_map:
                obj[key] = path_map[value]
            else:
                update_paths(value)
    elif isinstance(obj, list):
        for item in obj:
            update_paths(item)

update_paths(registry)

with open('optimization_engine/feature_registry.json', 'w') as f:
    json.dump(registry, f, indent=2)
```

### Phase 8: Testing & Validation (1-2 hours)

```bash
# 1. Run Python import tests
python -c "from optimization_engine.core.runner import OptimizationRunner"
python -c "from optimization_engine.nx.solver import run_nx_simulation"
python -c "from optimization_engine.study.creator import create_study"
python -c "from optimization_engine.processors.surrogates.neural_surrogate import NeuralSurrogate"

# 2. Run backwards compatibility tests
python -c "from optimization_engine import nx_solver"  # Should work with deprecation warning

# 3. Run test suite
python -m pytest tests/ -v --tb=short

# 4. Test a study end-to-end
cd studies/Simple_Bracket/bracket_displacement_maximizing
python run_optimization.py --trials 2 --dry-run

# 5. Test dashboard
cd atomizer-dashboard
python backend/api/main.py  # Should start without import errors
```

### Phase 9: Cleanup (15 min)

```bash
# Remove backup after successful testing
rm -rf optimization_engine_backup

# Remove any .pyc files that might cache old imports
find . -name "*.pyc" -delete
find . -name "__pycache__" -type d -exec rm -rf {} +

# Commit
git add -A
git commit -m "refactor: Reorganize optimization_engine into modular structure

- Move 50+ top-level files into logical subdirectories
- Create core/, processors/, nx/, study/, reporting/, config/
- Add backwards compatibility aliases with deprecation warnings
- Update all imports across codebase (640+ changes)
- Update documentation and protocols

BREAKING: Direct imports from optimization_engine.* are deprecated.
Use new paths like optimization_engine.core.runner instead.

🤖 Generated with [Claude Code](https://claude.com/claude-code)"
```

---

## Part 6: Rollback Plan

If migration fails:

```bash
# Option 1: Git reset (if not committed)
git checkout -- .

# Option 2: Restore backup
rm -rf optimization_engine
mv optimization_engine_backup optimization_engine

# Option 3: Git revert (if committed)
git revert HEAD
```

---

## Part 7: Post-Migration Tasks

### 7.1 Update LAC Knowledge Base

Record the migration in LAC:

```python
from knowledge_base.lac import get_lac
lac = get_lac()
lac.record_insight(
    category="architecture",
    context="optimization_engine reorganization",
    insight="Migrated 50+ files into modular structure. New paths: "
            "core/, processors/, nx/, study/, reporting/, config/. "
            "Backwards compat aliases provided with deprecation warnings.",
    confidence=1.0,
    tags=["refactoring", "architecture", "breaking-change"]
)
```

### 7.2 Update CLAUDE.md

Add section about new structure.

### 7.3 Schedule Deprecation Removal

After 2-4 weeks of stable operation:
- Remove backwards compatibility aliases
- Update remaining old imports
- Clean up __init__.py files

---

## Part 8: Time Estimate Summary

| Phase | Task | Time |
|-------|------|------|
| 0 | Pre-migration (backup, baseline) | 30 min |
| 1 | Create directory structure | 15 min |
| 2 | Move files | 30 min |
| 3 | Update internal imports | 1-2 hours |
| 4 | Create __init__.py files | 30 min |
| 5 | Add backwards compatibility | 30 min |
| 6 | Update documentation | 1 hour |
| 7 | Update JSON configs | 30 min |
| 8 | Testing & validation | 1-2 hours |
| 9 | Cleanup & commit | 15 min |
| **TOTAL** | | **6-8 hours** |

---

## Part 9: Decision Points

### Option A: Full Migration Now
- **Pros**: Clean structure, enables dynamic_response cleanly
- **Cons**: 6-8 hours work, risk of breakage
- **When**: When you have a dedicated half-day

### Option B: Minimal Migration (processors/ only)
- **Pros**: Low risk, enables dynamic_response
- **Cons**: Leaves technical debt, inconsistent structure
- **When**: If you need dynamic_response urgently

### Option C: Defer Migration
- **Pros**: Zero risk now
- **Cons**: Dynamic_response goes in awkward location
- **When**: If stability is critical

---

## Appendix A: Automated Migration Script

A Python script to automate most of the migration:

```python
#!/usr/bin/env python
"""
optimization_engine Migration Script

Run with: python migrate_optimization_engine.py --dry-run
Then:     python migrate_optimization_engine.py --execute
"""

import os
import re
import shutil
from pathlib import Path

# ... full script would be ~200 lines
# Handles: directory creation, file moves, import updates, __init__ generation
```

Would you like me to create this full automation script?

---

*This plan ensures a safe, reversible migration with clear validation steps.*