Atomizer/docs/plans/OPTIMIZATION_ENGINE_MIGRATION_PLAN.md

# Optimization Engine Reorganization - DEFINITIVE Migration Plan

## Complete Guide for Safe Codebase Restructuring

**Document Version**: 3.0 (COMPLETED)
**Created**: 2025-12-23
**Updated**: 2025-12-29
**Status**: ✅ COMPLETED - Merged to main
**Risk Level**: HIGH - Affects 500+ files across entire codebase
**Next Phase**: ATOMIZER_CONTEXT_ENGINEERING_PLAN.md (execute AFTER this migration)

---

## ✅ Migration Completed: 2025-12-29

### Final Statistics
| Metric | Value |
|--------|-------|
| Files modified | 120 |
| Import changes | ~200 |
| Commits | 2 (pre-migration checkpoint + migration) |
| Branch | `refactor/optimization-engine-v2` merged to `main` |
| Version bump | `optimization_engine` v1.0 → v2.0.0 |

### Key Changes Made
1. Created new directory structure: `core/`, `processors/surrogates/`, `nx/`, `study/`, `reporting/`, `config/`
2. Moved 60+ Python files to new locations
3. Updated all imports across 125 files
4. Added lazy-loading `__init__.py` files to avoid circular imports
5. Implemented backwards compatibility layer with deprecation warnings
6. Updated `feature_registry.json` paths

### Post-Migration Notes
- All existing `run_optimization.py` scripts continue to work
- Old imports emit deprecation warnings but still function
- Dashboard integration preserved
- Study databases unchanged

---

## Execution Order (IMPORTANT)

This migration plan should be executed **BEFORE** the Context Engineering plan.

**Why Migration First?**
1. Creates clean, organized module structure
2. Context Engineering adds NEW modules (`context/`) that won't be moved
3. Avoids double-updating imports (migrate first, then integrate new code)
4. Documentation written once with final paths

**After Migration Completes:**
- Execute `ATOMIZER_CONTEXT_ENGINEERING_PLAN.md`
- Add `optimization_engine/context/` module (playbook, reflector, session_state)
- Integrate with `optimization_engine/core/runner.py` (already at final path)

---

## Executive Summary

This is the **DEFINITIVE** migration plan covering every single file that needs updating. The scope is significantly larger than initially estimated.

### True Impact Summary

| Category | Files | Estimated Changes |
|----------|-------|-------------------|
| Python files (optimization_engine internal) | 90+ | ~300 import changes |
| Python files (studies) | 80+ | ~200 import changes |
| Python files (tests) | 45+ | ~150 import changes |
| Python files (tools, examples, root) | 10+ | ~50 import changes |
| Python files (dashboard backend) | 3 | ~15 import changes |
| Python files (archive - optional) | 50+ | ~100 import changes |
| Documentation (markdown) | 100+ | ~500 path references |
| JSON configuration files | 5 | ~50 path entries |
| Build config (pyproject.toml) | 1 | 2 lines |
| **TOTAL** | **500+ files** | **~1,400 changes** |

### Estimated Time: 8-12 hours (with automation scripts)

---

## Part 1: Complete File Inventory

### 1.1 Files to MOVE (Source Files)

These files move from `optimization_engine/` root to new subdirectories:

```
MOVE TO: optimization_engine/core/
├── runner.py
├── base_runner.py
├── runner_with_neural.py
├── intelligent_optimizer.py
├── method_selector.py
├── strategy_selector.py
├── strategy_portfolio.py
└── gradient_optimizer.py

MOVE TO: optimization_engine/processors/surrogates/
├── neural_surrogate.py
├── generic_surrogate.py
├── adaptive_surrogate.py
├── simple_mlp_surrogate.py
├── active_learning_surrogate.py
├── surrogate_tuner.py
├── auto_trainer.py
└── training_data_exporter.py

MOVE TO: optimization_engine/nx/
├── nx_solver.py → solver.py
├── nx_updater.py → updater.py
├── nx_session_manager.py → session_manager.py
├── solve_simulation.py
├── solve_simulation_simple.py
├── model_cleanup.py
├── export_expressions.py
├── import_expressions.py
└── mesh_converter.py

MOVE TO: optimization_engine/study/
├── study_creator.py → creator.py
├── study_wizard.py → wizard.py
├── study_state.py → state.py
├── study_reset.py → reset.py
├── study_continuation.py → continuation.py
├── benchmarking_substudy.py → benchmarking.py
└── generate_history_from_trials.py → history_generator.py

MOVE TO: optimization_engine/reporting/
├── generate_report.py → report_generator.py
├── generate_report_markdown.py → markdown_report.py
├── comprehensive_results_analyzer.py → results_analyzer.py
├── visualizer.py
└── landscape_analyzer.py

MOVE TO: optimization_engine/config/
├── config_manager.py → manager.py
├── optimization_config_builder.py → builder.py
├── optimization_setup_wizard.py → setup_wizard.py
├── capability_matcher.py
└── template_loader.py

MOVE TO: optimization_engine/utils/ (existing, add to it)
├── logger.py
├── auto_doc.py
├── realtime_tracking.py
├── codebase_analyzer.py
└── pruning_logger.py

MOVE TO: optimization_engine/future/ (existing, add to it)
├── research_agent.py
├── pynastran_research_agent.py
├── targeted_research_planner.py
├── workflow_decomposer.py
├── step_classifier.py
├── llm_optimization_runner.py
├── llm_workflow_analyzer.py
├── extractor_orchestrator.py
├── inline_code_generator.py
└── hook_generator.py

MOVE TO: optimization_engine/extractors/ (existing, add to it)
├── op2_extractor.py
└── extractor_library.py

MOVE TO: optimization_engine/validators/ (existing, add to it)
└── simulation_validator.py

MOVE TO: optimization_engine/processors/ (new)
└── adaptive_characterization.py
```

### 1.2 Files to UPDATE (Import Changes Required)

#### A. OPTIMIZATION_ENGINE INTERNAL (90+ files)

**Core/Root Files:**
```
optimization_engine/run_optimization.py
optimization_engine/__init__.py
```

**Extractors (20 files):**
```
optimization_engine/extractors/__init__.py (31 imports)
optimization_engine/extractors/extract_displacement.py
optimization_engine/extractors/extract_frequency.py
optimization_engine/extractors/extract_von_mises_stress.py
optimization_engine/extractors/bdf_mass_extractor.py
optimization_engine/extractors/extract_mass_from_bdf.py
optimization_engine/extractors/extract_mass_from_expression.py
optimization_engine/extractors/extract_part_mass_material.py
optimization_engine/extractors/extract_zernike.py
optimization_engine/extractors/extract_zernike_opd.py
optimization_engine/extractors/extract_zernike_figure.py
optimization_engine/extractors/extract_zernike_surface.py
optimization_engine/extractors/zernike_helpers.py
optimization_engine/extractors/extract_principal_stress.py
optimization_engine/extractors/extract_strain_energy.py
optimization_engine/extractors/extract_spc_forces.py
optimization_engine/extractors/extract_temperature.py
optimization_engine/extractors/extract_modal_mass.py
optimization_engine/extractors/introspect_part.py
optimization_engine/extractors/field_data_extractor.py
optimization_engine/extractors/stiffness_calculator.py
optimization_engine/extractors/test_phase3_extractors.py
```

**GNN Module (12 files):**
```
optimization_engine/gnn/__init__.py
optimization_engine/gnn/zernike_gnn.py
optimization_engine/gnn/polar_graph.py
optimization_engine/gnn/train_zernike_gnn.py
optimization_engine/gnn/gnn_optimizer.py
optimization_engine/gnn/extract_displacement_field.py
optimization_engine/gnn/differentiable_zernike.py
optimization_engine/gnn/backfill_field_data.py
optimization_engine/gnn/test_polar_graph.py
optimization_engine/gnn/test_field_extraction.py
optimization_engine/gnn/test_new_extraction.py
```

**Hooks Module (12 files):**
```
optimization_engine/hooks/__init__.py
optimization_engine/hooks/examples.py
optimization_engine/hooks/test_introspection.py
optimization_engine/hooks/test_hooks.py
optimization_engine/hooks/nx_cad/__init__.py
optimization_engine/hooks/nx_cad/model_introspection.py
optimization_engine/hooks/nx_cad/part_manager.py
optimization_engine/hooks/nx_cad/expression_manager.py
optimization_engine/hooks/nx_cad/geometry_query.py
optimization_engine/hooks/nx_cad/feature_manager.py
optimization_engine/hooks/nx_cae/__init__.py
optimization_engine/hooks/nx_cae/solver_manager.py
```

**Plugins Module (5 files):**
```
optimization_engine/plugins/__init__.py
optimization_engine/plugins/hook_manager.py
optimization_engine/plugins/post_calculation/safety_factor_constraint.py
optimization_engine/plugins/pre_solve/*.py
optimization_engine/plugins/post_solve/*.py
```

**Insights Module (12 files):**
```
optimization_engine/insights/__init__.py
optimization_engine/insights/base.py
optimization_engine/insights/zernike_wfe.py
optimization_engine/insights/zernike_opd_comparison.py
optimization_engine/insights/zernike_dashboard.py
optimization_engine/insights/msf_zernike.py
optimization_engine/insights/stress_field.py
optimization_engine/insights/modal_analysis.py
optimization_engine/insights/thermal_field.py
optimization_engine/insights/design_space.py
```

**Utils Module (6 files):**
```
optimization_engine/utils/__init__.py
optimization_engine/utils/trial_manager.py
optimization_engine/utils/dashboard_db.py
optimization_engine/utils/nx_session_manager.py
optimization_engine/utils/nx_file_discovery.py
optimization_engine/utils/study_cleanup.py
```

**Validators Module (5 files):**
```
optimization_engine/validators/__init__.py
optimization_engine/validators/study_validator.py
optimization_engine/validators/results_validator.py
optimization_engine/validators/model_validator.py
optimization_engine/validators/config_validator.py
```

**Templates (3 files):**
```
optimization_engine/templates/__init__.py
optimization_engine/templates/run_optimization_template.py
optimization_engine/templates/run_nn_optimization_template.py
```

**Model Discovery (2 files):**
```
optimization_engine/model_discovery/__init__.py
optimization_engine/model_discovery/discover_parts.py
```

#### B. STUDIES DIRECTORY (80+ files)

**M1_Mirror Studies (35+ files):**
```
studies/M1_Mirror/m1_mirror_cost_reduction/run_optimization.py
studies/M1_Mirror/m1_mirror_zernike_optimization/run_optimization.py
studies/M1_Mirror/m1_mirror_cost_reduction_flat_back_V1/run_optimization.py
studies/M1_Mirror/m1_mirror_cost_reduction_flat_back_V3/run_optimization.py
studies/M1_Mirror/m1_mirror_cost_reduction_flat_back_V4/run_turbo_optimization.py
studies/M1_Mirror/m1_mirror_cost_reduction_flat_back_V5/run_turbo_optimization.py
studies/M1_Mirror/m1_mirror_cost_reduction_flat_back_V6/run_optimization.py
studies/M1_Mirror/m1_mirror_turbo_V1/run_optimization.py
studies/M1_Mirror/m1_mirror_turbo_V2/run_optimization.py
studies/M1_Mirror/m1_mirror_cost_reduction_V2/run_optimization.py
studies/M1_Mirror/m1_mirror_cost_reduction_V3/run_optimization.py
studies/M1_Mirror/m1_mirror_cost_reduction_V4/run_optimization.py
studies/M1_Mirror/m1_mirror_cost_reduction_V5/run_optimization.py
studies/M1_Mirror/m1_mirror_cost_reduction_V5/_analyze_baseline.py
studies/M1_Mirror/m1_mirror_cost_reduction_V6/run_optimization.py
studies/M1_Mirror/m1_mirror_cost_reduction_V6/_analyze_baseline.py
studies/M1_Mirror/m1_mirror_cost_reduction_V7/run_optimization.py
studies/M1_Mirror/m1_mirror_cost_reduction_V8/run_optimization.py
studies/M1_Mirror/m1_mirror_cost_reduction_V9/run_optimization.py
studies/M1_Mirror/m1_mirror_cost_reduction_V10/run_optimization.py
studies/M1_Mirror/m1_mirror_cost_reduction_V11/run_optimization.py
studies/M1_Mirror/m1_mirror_cost_reduction_V12/run_optimization.py
studies/M1_Mirror/m1_mirror_adaptive_V11/run_optimization.py
studies/M1_Mirror/m1_mirror_adaptive_V12/run_optimization.py
studies/M1_Mirror/m1_mirror_adaptive_V12/run_gnn_turbo.py
studies/M1_Mirror/m1_mirror_adaptive_V12/validate_gnn_best.py
studies/M1_Mirror/m1_mirror_adaptive_V12/compute_full_calibration.py
studies/M1_Mirror/m1_mirror_adaptive_V13/run_optimization.py
studies/M1_Mirror/m1_mirror_adaptive_V14/run_optimization.py
studies/M1_Mirror/m1_mirror_adaptive_V14/run_lbfgs_polish.py
studies/M1_Mirror/m1_mirror_adaptive_V15/run_optimization.py
studies/M1_Mirror/m1_mirror_surrogate_turbo/scripts/01_reextract_studies.py
studies/M1_Mirror/m1_mirror_surrogate_turbo/scripts/02_aggregate_gnn_data.py
studies/M1_Mirror/m1_mirror_surrogate_turbo/scripts/03_train_gnn.py
```

**Simple_Bracket Studies (12+ files):**
```
studies/Simple_Bracket/bracket_stiffness_optimization/run_optimization.py
studies/Simple_Bracket/bracket_stiffness_optimization/bracket_stiffness_extractor.py
studies/Simple_Bracket/bracket_stiffness_optimization_V2/run_optimization.py
studies/Simple_Bracket/bracket_stiffness_optimization_V2/bracket_stiffness_extractor.py
studies/Simple_Bracket/bracket_stiffness_optimization_V3/run_optimization.py
studies/Simple_Bracket/bracket_stiffness_optimization_V3/bracket_stiffness_extractor.py
studies/Simple_Bracket/bracket_stiffness_optimization_atomizerfield/run_optimization.py
studies/Simple_Bracket/bracket_pareto_3obj/run_optimization.py
studies/Simple_Bracket/bracket_pareto_3obj/run_nn_optimization.py
```

**Other Studies (15+ files):**
```
studies/Simple_Beam/simple_beam_optimization/run_optimization.py
studies/UAV_Arm/uav_arm_optimization/run_optimization.py
studies/UAV_Arm/uav_arm_optimization_V2/run_optimization.py
studies/UAV_Arm/uav_arm_atomizerfield_test/run_optimization.py
studies/Drone_Gimbal/drone_gimbal_arm_optimization/run_optimization.py
studies/bracket_displacement_maximizing/run_optimization.py
studies/bracket_displacement_maximizing/run_optimization_old.py
studies/bracket_displacement_maximizing/run_substudy.py
studies/bracket_displacement_maximizing/test_fix.py
studies/_Other/training_data_export_test/run_optimization.py
```

#### C. TESTS DIRECTORY (45+ files)

```
tests/audit_v10_wfe.py
tests/audit_v10_method_diff.py
tests/audit_v10_fix.py
tests/analyze_v11.py
tests/debug_insights.py
tests/debug_figure_coords.py
tests/compare_v8_zernike_methods.py
tests/test_zernike_opd_with_prescription.py
tests/test_zernike_opd_comparison.py
tests/test_zernike_methods_comparison.py
tests/test_zernike_insight.py
tests/test_zernike_import.py
tests/test_training_data_exporter.py
tests/test_timestamp_verification.py
tests/test_task_1_2_integration.py
tests/test_step_classifier.py
tests/test_research_agent.py
tests/test_phase_3_2_llm_mode.py
tests/test_phase_3_1_integration.py
tests/test_phase_2_5_intelligent_gap_detection.py
tests/test_plugin_system.py
tests/test_optimization_setup_wizard.py
tests/test_modal_deformation_request.py
tests/test_llm_runner_init.py
tests/test_llm_complex_request.py
tests/test_lifecycle_hook_integration.py
tests/test_knowledge_base_search.py
tests/test_journal_optimization.py
tests/test_interactive_session.py
tests/test_hooks_with_bracket.py
tests/test_complex_multiobj_request.py
tests/test_complete_research_workflow.py
tests/test_code_generation.py
tests/test_cbar_genetic_algorithm.py
tests/test_cbush_optimization.py
tests/test_bracket_llm_runner.py
tests/test_bracket_full_optimization.py
tests/test_beam_workflow.py
tests/test_api_verification.py
tests/interactive_optimization_setup.py
tests/demo_research_agent.py
tests/run_beam_benchmarking.py
tests/setup_beam_optimization.py
tests/run_5trial_test.py
tests/run_benchmarking_simple.py
tests/run_beam_benchmarking_clean.py
```

#### D. ROOT & TOOLS (10+ files)

```
atomizer.py
run_cleanup.py
run_training_fea.py
tools/zernike_html_generator.py
tools/extract_mirror_optical_specs.py
examples/interactive_research_session.py
```

#### E. DASHBOARD BACKEND (3 files)

```
atomizer-dashboard/backend/api/main.py
atomizer-dashboard/backend/api/routes/optimization.py
atomizer-dashboard/backend/api/routes/insights.py
```

#### F. ARCHIVE (50+ files - OPTIONAL)

```
archive/deprecated/result_extractors/extractors.py
archive/deprecated/result_extractors/generated/extract_expression.py
archive/scripts/create_intelligent_study.py
archive/scripts/create_circular_plate_study_v2.py
archive/scripts/create_circular_plate_study.py
archive/scripts/run_nn_optimization.py
archive/scripts/run_calibration_loop.py
archive/scripts/run_validated_nn_optimization.py
archive/scripts/validate_surrogate_real_data.py
archive/scripts/visualize_nn_optimization.py
archive/test_scripts/*.py (10+ files)
```

### 1.3 Documentation Files to UPDATE (100+ files)

#### A. ROOT DOCUMENTATION

```
CLAUDE.md (lines 141, 152, 166, 200, 221, 235, 253, 283, 592-603)
README.md (lines 98, 193, 213, 272)
CHANGELOG.md (lines 31, 40, 82)
DEVELOPMENT.md (lines 40, 64, 77, 90, 215-277, 398, 532, 535)
PROTOCOL.md (lines 481, 704-705, 760, 812, 924-939, 1118, 1262, 1380, 1406, 1501-1596, 1851)
```

#### B. .CLAUDE DIRECTORY

```
.claude/ATOMIZER_CONTEXT.md (lines 37, 40, 71, 235, 238, 248, 273, 282, 288, 421-472)
.claude/CHEATSHEET.md (lines 7-10, 33-35, 144, 156, 353-472)
.claude/skills/00_BOOTSTRAP.md (line 236)
.claude/skills/01_CHEATSHEET.md (lines 7-10, 33-35, 144, 156, 353-472)
.claude/skills/analyze-workflow.md (line 37)
.claude/skills/analyze-model.md (lines 52, 99, 183)
.claude/skills/atomizer.md (lines 15, 27, 62, 193, 286, 317, 323)
.claude/skills/DEV_DOCUMENTATION.md (lines 115-117, 141)
.claude/skills/core/study-creation-core.md (lines 7-9, 43, 157, 160, 259, 279, 454-459, 643, 706)
.claude/skills/modules/extractors-catalog.md (lines 24-31, 42, 63, 86, 110-271)
.claude/skills/modules/insights-catalog.md (lines 53-237)
.claude/skills/modules/neural-acceleration.md (lines 48, 318)
.claude/skills/modules/nx-docs-lookup.md (line 190)
.claude/skills/modules/zernike-optimization.md (lines 35, 73, 111)
.claude/skills/modules/atomizer_fast_solver_technologies.md (lines 739, 744)
.claude/skills/modules/learning-atomizer-core.md
.claude/skills/archive/create-study-wizard.md (lines 13, 74, 177, 360, 402)
```

#### C. DOCS DIRECTORY - PROTOCOLS

```
docs/00_INDEX.md (line 284)
docs/01_PROTOCOLS.md (lines 277, 352-355, 528, 535-600, 1355, 1434, 1460)
docs/02_ARCHITECTURE.md (lines 68-69, 104, 123, 193, 254, 279, 299, 376-495)
docs/03_GETTING_STARTED.md (line 286)
docs/protocols/operations/OP_01_CREATE_STUDY.md (lines 162, 301-303)
docs/protocols/operations/OP_02_RUN_OPTIMIZATION.md
docs/protocols/operations/OP_03_MONITOR_PROGRESS.md
docs/protocols/operations/OP_04_ANALYZE_RESULTS.md
docs/protocols/operations/OP_05_EXPORT_TRAINING_DATA.md
docs/protocols/operations/OP_06_TROUBLESHOOT.md
docs/protocols/system/SYS_10_IMSO.md (lines 224, 302-305)
docs/protocols/system/SYS_11_MULTI_OBJECTIVE.md (lines 322-330)
docs/protocols/system/SYS_12_EXTRACTOR_LIBRARY.md (lines 71-851 - MANY references)
docs/protocols/system/SYS_13_DASHBOARD_TRACKING.md (lines 60, 85, 315)
docs/protocols/system/SYS_14_NEURAL_ACCELERATION.md (lines 231-1080)
docs/protocols/system/SYS_15_METHOD_SELECTOR.md (lines 42-422)
docs/protocols/system/SYS_17_STUDY_INSIGHTS.md (lines 62-498)
docs/protocols/extensions/EXT_01_CREATE_EXTRACTOR.md (lines 35-287)
docs/protocols/extensions/EXT_02_CREATE_HOOK.md (lines 45-357)
docs/protocols/extensions/EXT_03_CREATE_INSIGHT.md
docs/protocols/extensions/EXT_04_CREATE_SKILL.md (lines 263, 274)
docs/protocols/extensions/templates/hook_template.py (lines 128, 164)
```

#### D. DOCS SUBDIRECTORIES

```
docs/04_USER_GUIDES/*.md (estimated 20+ files)
docs/05_API_REFERENCE/*.md (estimated 10+ files)
docs/06_PHYSICS/*.md (estimated 15+ files)
docs/07_DEVELOPMENT/*.md (estimated 20+ files)
docs/08_ARCHIVE/*.md (estimated 30+ files)
```

### 1.4 Configuration Files to UPDATE

```
optimization_engine/feature_registry.json (50+ path entries on lines 21-683)
optimization_engine/schemas/README.md (lines 19, 26)
studies/Simple_Bracket/bracket_stiffness_optimization_atomizerfield/1_setup/optimization_config.json
studies/Simple_Bracket/bracket_pareto_3obj/1_setup/optimization_config.json
pyproject.toml (lines 53, 76)
```

---

## Part 2: Import Mapping Reference

### 2.1 Complete Old → New Import Mapping

```python
# =============================================================================
# CORE MODULE
# =============================================================================
"from optimization_engine.runner"                     → "from optimization_engine.core.runner"
"from optimization_engine.base_runner"                → "from optimization_engine.core.base_runner"
"from optimization_engine.runner_with_neural"         → "from optimization_engine.core.runner_with_neural"
"from optimization_engine.intelligent_optimizer"      → "from optimization_engine.core.intelligent_optimizer"
"from optimization_engine.method_selector"            → "from optimization_engine.core.method_selector"
"from optimization_engine.strategy_selector"          → "from optimization_engine.core.strategy_selector"
"from optimization_engine.strategy_portfolio"         → "from optimization_engine.core.strategy_portfolio"
"from optimization_engine.gradient_optimizer"         → "from optimization_engine.core.gradient_optimizer"

"import optimization_engine.runner"                   → "import optimization_engine.core.runner"
"import optimization_engine.intelligent_optimizer"    → "import optimization_engine.core.intelligent_optimizer"

# =============================================================================
# SURROGATES MODULE
# =============================================================================
"from optimization_engine.neural_surrogate"           → "from optimization_engine.processors.surrogates.neural_surrogate"
"from optimization_engine.generic_surrogate"          → "from optimization_engine.processors.surrogates.generic_surrogate"
"from optimization_engine.adaptive_surrogate"         → "from optimization_engine.processors.surrogates.adaptive_surrogate"
"from optimization_engine.simple_mlp_surrogate"       → "from optimization_engine.processors.surrogates.simple_mlp_surrogate"
"from optimization_engine.active_learning_surrogate"  → "from optimization_engine.processors.surrogates.active_learning_surrogate"
"from optimization_engine.surrogate_tuner"            → "from optimization_engine.processors.surrogates.surrogate_tuner"
"from optimization_engine.auto_trainer"               → "from optimization_engine.processors.surrogates.auto_trainer"
"from optimization_engine.training_data_exporter"     → "from optimization_engine.processors.surrogates.training_data_exporter"

# =============================================================================
# NX MODULE
# =============================================================================
"from optimization_engine.nx_solver"                  → "from optimization_engine.nx.solver"
"from optimization_engine.nx_updater"                 → "from optimization_engine.nx.updater"
"from optimization_engine.nx_session_manager"         → "from optimization_engine.nx.session_manager"
"from optimization_engine.solve_simulation"           → "from optimization_engine.nx.solve_simulation"
"from optimization_engine.solve_simulation_simple"    → "from optimization_engine.nx.solve_simulation_simple"
"from optimization_engine.model_cleanup"              → "from optimization_engine.nx.model_cleanup"
"from optimization_engine.export_expressions"         → "from optimization_engine.nx.export_expressions"
"from optimization_engine.import_expressions"         → "from optimization_engine.nx.import_expressions"
"from optimization_engine.mesh_converter"             → "from optimization_engine.nx.mesh_converter"

"import optimization_engine.nx_solver"                → "import optimization_engine.nx.solver"

# =============================================================================
# STUDY MODULE
# =============================================================================
"from optimization_engine.study_creator"              → "from optimization_engine.study.creator"
"from optimization_engine.study_wizard"               → "from optimization_engine.study.wizard"
"from optimization_engine.study_state"                → "from optimization_engine.study.state"
"from optimization_engine.study_reset"                → "from optimization_engine.study.reset"
"from optimization_engine.study_continuation"         → "from optimization_engine.study.continuation"
"from optimization_engine.benchmarking_substudy"      → "from optimization_engine.study.benchmarking"
"from optimization_engine.generate_history_from_trials" → "from optimization_engine.study.history_generator"

# =============================================================================
# REPORTING MODULE
# =============================================================================
"from optimization_engine.generate_report"            → "from optimization_engine.reporting.report_generator"
"from optimization_engine.generate_report_markdown"   → "from optimization_engine.reporting.markdown_report"
"from optimization_engine.comprehensive_results_analyzer" → "from optimization_engine.reporting.results_analyzer"
"from optimization_engine.visualizer"                 → "from optimization_engine.reporting.visualizer"
"from optimization_engine.landscape_analyzer"         → "from optimization_engine.reporting.landscape_analyzer"

# =============================================================================
# CONFIG MODULE
# =============================================================================
"from optimization_engine.config_manager"             → "from optimization_engine.config.manager"
"from optimization_engine.optimization_config_builder" → "from optimization_engine.config.builder"
"from optimization_engine.optimization_setup_wizard"  → "from optimization_engine.config.setup_wizard"
"from optimization_engine.capability_matcher"         → "from optimization_engine.config.capability_matcher"
"from optimization_engine.template_loader"            → "from optimization_engine.config.template_loader"

# =============================================================================
# UTILS MODULE (moving to existing utils/)
# =============================================================================
"from optimization_engine.logger"                     → "from optimization_engine.utils.logger"
"from optimization_engine.auto_doc"                   → "from optimization_engine.utils.auto_doc"
"from optimization_engine.realtime_tracking"          → "from optimization_engine.utils.realtime_tracking"
"from optimization_engine.codebase_analyzer"          → "from optimization_engine.utils.codebase_analyzer"
"from optimization_engine.pruning_logger"             → "from optimization_engine.utils.pruning_logger"

# =============================================================================
# FUTURE MODULE (research/experimental)
# =============================================================================
"from optimization_engine.research_agent"             → "from optimization_engine.future.research_agent"
"from optimization_engine.pynastran_research_agent"   → "from optimization_engine.future.pynastran_research_agent"
"from optimization_engine.targeted_research_planner"  → "from optimization_engine.future.targeted_research_planner"
"from optimization_engine.workflow_decomposer"        → "from optimization_engine.future.workflow_decomposer"
"from optimization_engine.step_classifier"            → "from optimization_engine.future.step_classifier"
"from optimization_engine.llm_optimization_runner"    → "from optimization_engine.future.llm_optimization_runner"
"from optimization_engine.llm_workflow_analyzer"      → "from optimization_engine.future.llm_workflow_analyzer"
"from optimization_engine.extractor_orchestrator"     → "from optimization_engine.future.extractor_orchestrator"
"from optimization_engine.inline_code_generator"      → "from optimization_engine.future.inline_code_generator"
"from optimization_engine.hook_generator"             → "from optimization_engine.future.hook_generator"

# =============================================================================
# EXTRACTORS MODULE (moving to existing extractors/)
# =============================================================================
"from optimization_engine.op2_extractor"              → "from optimization_engine.extractors.op2_extractor"
"from optimization_engine.extractor_library"          → "from optimization_engine.extractors.extractor_library"

# =============================================================================
# VALIDATORS MODULE (moving to existing validators/)
# =============================================================================
"from optimization_engine.simulation_validator"       → "from optimization_engine.validators.simulation_validator"

# =============================================================================
# PROCESSORS MODULE (new)
# =============================================================================
"from optimization_engine.adaptive_characterization"  → "from optimization_engine.processors.adaptive_characterization"

# =============================================================================
# UNCHANGED (these modules stay in place)
# =============================================================================
# extractors/ - stays as optimization_engine.extractors.*
# gnn/ - stays as optimization_engine.gnn.*
# hooks/ - stays as optimization_engine.hooks.*
# insights/ - stays as optimization_engine.insights.*
# plugins/ - stays as optimization_engine.plugins.*
# validators/ - stays as optimization_engine.validators.*
# utils/ - stays as optimization_engine.utils.*
# templates/ - stays as optimization_engine.templates.*
# schemas/ - stays as optimization_engine.schemas.*
# model_discovery/ - stays as optimization_engine.model_discovery.*
```

### 2.2 Documentation Path Mapping

```
# In markdown files, update these path references:

optimization_engine/runner.py           → optimization_engine/core/runner.py
optimization_engine/nx_solver.py        → optimization_engine/nx/solver.py
optimization_engine/nx_updater.py       → optimization_engine/nx/updater.py
optimization_engine/study_creator.py    → optimization_engine/study/creator.py
optimization_engine/study_wizard.py     → optimization_engine/study/wizard.py
optimization_engine/neural_surrogate.py → optimization_engine/processors/surrogates/neural_surrogate.py
optimization_engine/config_manager.py   → optimization_engine/config/manager.py
optimization_engine/logger.py           → optimization_engine/utils/logger.py
# ... etc for all moved files
```

---

## Part 3: Execution Plan

### Phase 0: Preparation (30 minutes)

```bash
# 1. Create backup branch
git checkout -b backup/pre-reorganization
git push origin backup/pre-reorganization
git checkout main
git checkout -b refactor/optimization-engine-v2

# 2. Create full backup
cp -r optimization_engine optimization_engine_BACKUP_$(date +%Y%m%d)

# 3. Run baseline tests and save output
python -m pytest tests/ -v --tb=short 2>&1 | tee baseline_tests_$(date +%Y%m%d).log

# 4. Count current test status
echo "Baseline test count:"
grep -E "passed|failed|error" baseline_tests_*.log | tail -5

# 5. Verify NX is not running (to prevent file locks)
tasklist | grep -i ugraf || echo "NX not running - safe to proceed"
```

### Phase 1: Create Directory Structure (15 minutes)

```bash
# Create all new directories
mkdir -p optimization_engine/core
mkdir -p optimization_engine/processors/surrogates
mkdir -p optimization_engine/processors/dynamic_response
mkdir -p optimization_engine/nx
mkdir -p optimization_engine/study
mkdir -p optimization_engine/reporting
mkdir -p optimization_engine/config

# Create __init__.py files for new directories
touch optimization_engine/core/__init__.py
touch optimization_engine/processors/__init__.py
touch optimization_engine/processors/surrogates/__init__.py
touch optimization_engine/processors/dynamic_response/__init__.py
touch optimization_engine/nx/__init__.py
touch optimization_engine/study/__init__.py
touch optimization_engine/reporting/__init__.py
touch optimization_engine/config/__init__.py
```

### Phase 2: Move Files (45 minutes)

**Execute in this specific order to minimize issues:**

```bash
# =============================================================================
# STEP 2.1: UTILITIES FIRST (no dependencies on other moved files)
# =============================================================================
mv optimization_engine/logger.py optimization_engine/utils/
mv optimization_engine/auto_doc.py optimization_engine/utils/
mv optimization_engine/realtime_tracking.py optimization_engine/utils/
mv optimization_engine/codebase_analyzer.py optimization_engine/utils/
mv optimization_engine/pruning_logger.py optimization_engine/utils/

# =============================================================================
# STEP 2.2: CONFIG (depends only on logger)
# =============================================================================
mv optimization_engine/config_manager.py optimization_engine/config/manager.py
mv optimization_engine/optimization_config_builder.py optimization_engine/config/builder.py
mv optimization_engine/optimization_setup_wizard.py optimization_engine/config/setup_wizard.py
mv optimization_engine/capability_matcher.py optimization_engine/config/capability_matcher.py
mv optimization_engine/template_loader.py optimization_engine/config/template_loader.py

# =============================================================================
# STEP 2.3: NX INTEGRATION (core infrastructure)
# =============================================================================
mv optimization_engine/nx_solver.py optimization_engine/nx/solver.py
mv optimization_engine/nx_updater.py optimization_engine/nx/updater.py
mv optimization_engine/nx_session_manager.py optimization_engine/nx/session_manager.py
mv optimization_engine/solve_simulation.py optimization_engine/nx/
mv optimization_engine/solve_simulation_simple.py optimization_engine/nx/
mv optimization_engine/model_cleanup.py optimization_engine/nx/
mv optimization_engine/export_expressions.py optimization_engine/nx/
mv optimization_engine/import_expressions.py optimization_engine/nx/
mv optimization_engine/mesh_converter.py optimization_engine/nx/

# =============================================================================
# STEP 2.4: SURROGATES (ML components)
# =============================================================================
mv optimization_engine/neural_surrogate.py optimization_engine/processors/surrogates/
mv optimization_engine/generic_surrogate.py optimization_engine/processors/surrogates/
mv optimization_engine/adaptive_surrogate.py optimization_engine/processors/surrogates/
mv optimization_engine/simple_mlp_surrogate.py optimization_engine/processors/surrogates/
mv optimization_engine/active_learning_surrogate.py optimization_engine/processors/surrogates/
mv optimization_engine/surrogate_tuner.py optimization_engine/processors/surrogates/
mv optimization_engine/auto_trainer.py optimization_engine/processors/surrogates/
mv optimization_engine/training_data_exporter.py optimization_engine/processors/surrogates/

# =============================================================================
# STEP 2.5: STUDY MANAGEMENT
# =============================================================================
mv optimization_engine/study_creator.py optimization_engine/study/creator.py
mv optimization_engine/study_wizard.py optimization_engine/study/wizard.py
mv optimization_engine/study_state.py optimization_engine/study/state.py
mv optimization_engine/study_reset.py optimization_engine/study/reset.py
mv optimization_engine/study_continuation.py optimization_engine/study/continuation.py
mv optimization_engine/benchmarking_substudy.py optimization_engine/study/benchmarking.py
mv optimization_engine/generate_history_from_trials.py optimization_engine/study/history_generator.py

# =============================================================================
# STEP 2.6: REPORTING
# =============================================================================
mv optimization_engine/generate_report.py optimization_engine/reporting/report_generator.py
mv optimization_engine/generate_report_markdown.py optimization_engine/reporting/markdown_report.py
mv optimization_engine/comprehensive_results_analyzer.py optimization_engine/reporting/results_analyzer.py
mv optimization_engine/visualizer.py optimization_engine/reporting/
mv optimization_engine/landscape_analyzer.py optimization_engine/reporting/

# =============================================================================
# STEP 2.7: CORE (depends on many things, do after others)
# =============================================================================
mv optimization_engine/runner.py optimization_engine/core/
mv optimization_engine/base_runner.py optimization_engine/core/
mv optimization_engine/runner_with_neural.py optimization_engine/core/
mv optimization_engine/intelligent_optimizer.py optimization_engine/core/
mv optimization_engine/method_selector.py optimization_engine/core/
mv optimization_engine/strategy_selector.py optimization_engine/core/
mv optimization_engine/strategy_portfolio.py optimization_engine/core/
mv optimization_engine/gradient_optimizer.py optimization_engine/core/

# =============================================================================
# STEP 2.8: FUTURE/RESEARCH (experimental)
# =============================================================================
mv optimization_engine/research_agent.py optimization_engine/future/
mv optimization_engine/pynastran_research_agent.py optimization_engine/future/
mv optimization_engine/targeted_research_planner.py optimization_engine/future/
mv optimization_engine/workflow_decomposer.py optimization_engine/future/
mv optimization_engine/step_classifier.py optimization_engine/future/
# Check if these exist before moving:
test -f optimization_engine/llm_optimization_runner.py && mv optimization_engine/llm_optimization_runner.py optimization_engine/future/
test -f optimization_engine/llm_workflow_analyzer.py && mv optimization_engine/llm_workflow_analyzer.py optimization_engine/future/
test -f optimization_engine/extractor_orchestrator.py && mv optimization_engine/extractor_orchestrator.py optimization_engine/future/
test -f optimization_engine/inline_code_generator.py && mv optimization_engine/inline_code_generator.py optimization_engine/future/
test -f optimization_engine/hook_generator.py && mv optimization_engine/hook_generator.py optimization_engine/future/

# =============================================================================
# STEP 2.9: MOVE TO EXISTING DIRECTORIES
# =============================================================================
mv optimization_engine/op2_extractor.py optimization_engine/extractors/
mv optimization_engine/extractor_library.py optimization_engine/extractors/
mv optimization_engine/simulation_validator.py optimization_engine/validators/
mv optimization_engine/adaptive_characterization.py optimization_engine/processors/
```

### Phase 3: Update Imports with Automation Script (2-3 hours)

Create and run a Python migration script:

```python
#!/usr/bin/env python3
"""
optimization_engine Migration Script
=====================================
Automatically updates all imports across the codebase.

Usage:
    python migrate_imports.py --dry-run    # Preview changes
    python migrate_imports.py --execute    # Apply changes
"""

import os
import re
import sys
from pathlib import Path
from typing import Dict, List, Tuple

# Import mappings (old -> new)
IMPORT_MAPPINGS = {
    # Core
    r'from optimization_engine\.runner\b': 'from optimization_engine.core.runner',
    r'from optimization_engine\.base_runner\b': 'from optimization_engine.core.base_runner',
    r'from optimization_engine\.runner_with_neural\b': 'from optimization_engine.core.runner_with_neural',
    r'from optimization_engine\.intelligent_optimizer\b': 'from optimization_engine.core.intelligent_optimizer',
    r'from optimization_engine\.method_selector\b': 'from optimization_engine.core.method_selector',
    r'from optimization_engine\.strategy_selector\b': 'from optimization_engine.core.strategy_selector',
    r'from optimization_engine\.strategy_portfolio\b': 'from optimization_engine.core.strategy_portfolio',
    r'from optimization_engine\.gradient_optimizer\b': 'from optimization_engine.core.gradient_optimizer',
    r'import optimization_engine\.runner\b': 'import optimization_engine.core.runner',
    r'import optimization_engine\.intelligent_optimizer\b': 'import optimization_engine.core.intelligent_optimizer',

    # Surrogates
    r'from optimization_engine\.neural_surrogate\b': 'from optimization_engine.processors.surrogates.neural_surrogate',
    r'from optimization_engine\.generic_surrogate\b': 'from optimization_engine.processors.surrogates.generic_surrogate',
    r'from optimization_engine\.adaptive_surrogate\b': 'from optimization_engine.processors.surrogates.adaptive_surrogate',
    r'from optimization_engine\.simple_mlp_surrogate\b': 'from optimization_engine.processors.surrogates.simple_mlp_surrogate',
    r'from optimization_engine\.active_learning_surrogate\b': 'from optimization_engine.processors.surrogates.active_learning_surrogate',
    r'from optimization_engine\.surrogate_tuner\b': 'from optimization_engine.processors.surrogates.surrogate_tuner',
    r'from optimization_engine\.auto_trainer\b': 'from optimization_engine.processors.surrogates.auto_trainer',
    r'from optimization_engine\.training_data_exporter\b': 'from optimization_engine.processors.surrogates.training_data_exporter',

    # NX
    r'from optimization_engine\.nx_solver\b': 'from optimization_engine.nx.solver',
    r'from optimization_engine\.nx_updater\b': 'from optimization_engine.nx.updater',
    r'from optimization_engine\.nx_session_manager\b': 'from optimization_engine.nx.session_manager',
    r'from optimization_engine\.solve_simulation\b': 'from optimization_engine.nx.solve_simulation',
    r'from optimization_engine\.solve_simulation_simple\b': 'from optimization_engine.nx.solve_simulation_simple',
    r'from optimization_engine\.model_cleanup\b': 'from optimization_engine.nx.model_cleanup',
    r'from optimization_engine\.export_expressions\b': 'from optimization_engine.nx.export_expressions',
    r'from optimization_engine\.import_expressions\b': 'from optimization_engine.nx.import_expressions',
    r'from optimization_engine\.mesh_converter\b': 'from optimization_engine.nx.mesh_converter',
    r'import optimization_engine\.nx_solver\b': 'import optimization_engine.nx.solver',

    # Study
    r'from optimization_engine\.study_creator\b': 'from optimization_engine.study.creator',
    r'from optimization_engine\.study_wizard\b': 'from optimization_engine.study.wizard',
    r'from optimization_engine\.study_state\b': 'from optimization_engine.study.state',
    r'from optimization_engine\.study_reset\b': 'from optimization_engine.study.reset',
    r'from optimization_engine\.study_continuation\b': 'from optimization_engine.study.continuation',
    r'from optimization_engine\.benchmarking_substudy\b': 'from optimization_engine.study.benchmarking',
    r'from optimization_engine\.generate_history_from_trials\b': 'from optimization_engine.study.history_generator',

    # Reporting
    r'from optimization_engine\.generate_report\b': 'from optimization_engine.reporting.report_generator',
    r'from optimization_engine\.generate_report_markdown\b': 'from optimization_engine.reporting.markdown_report',
    r'from optimization_engine\.comprehensive_results_analyzer\b': 'from optimization_engine.reporting.results_analyzer',
    r'from optimization_engine\.visualizer\b': 'from optimization_engine.reporting.visualizer',
    r'from optimization_engine\.landscape_analyzer\b': 'from optimization_engine.reporting.landscape_analyzer',

    # Config
    r'from optimization_engine\.config_manager\b': 'from optimization_engine.config.manager',
    r'from optimization_engine\.optimization_config_builder\b': 'from optimization_engine.config.builder',
    r'from optimization_engine\.optimization_setup_wizard\b': 'from optimization_engine.config.setup_wizard',
    r'from optimization_engine\.capability_matcher\b': 'from optimization_engine.config.capability_matcher',
    r'from optimization_engine\.template_loader\b': 'from optimization_engine.config.template_loader',

    # Utils
    r'from optimization_engine\.logger\b': 'from optimization_engine.utils.logger',
    r'from optimization_engine\.auto_doc\b': 'from optimization_engine.utils.auto_doc',
    r'from optimization_engine\.realtime_tracking\b': 'from optimization_engine.utils.realtime_tracking',
    r'from optimization_engine\.codebase_analyzer\b': 'from optimization_engine.utils.codebase_analyzer',
    r'from optimization_engine\.pruning_logger\b': 'from optimization_engine.utils.pruning_logger',

    # Future
    r'from optimization_engine\.research_agent\b': 'from optimization_engine.future.research_agent',
    r'from optimization_engine\.pynastran_research_agent\b': 'from optimization_engine.future.pynastran_research_agent',
    r'from optimization_engine\.targeted_research_planner\b': 'from optimization_engine.future.targeted_research_planner',
    r'from optimization_engine\.workflow_decomposer\b': 'from optimization_engine.future.workflow_decomposer',
    r'from optimization_engine\.step_classifier\b': 'from optimization_engine.future.step_classifier',
    r'from optimization_engine\.llm_optimization_runner\b': 'from optimization_engine.future.llm_optimization_runner',
    r'from optimization_engine\.llm_workflow_analyzer\b': 'from optimization_engine.future.llm_workflow_analyzer',

    # Extractors/Validators additions
    r'from optimization_engine\.op2_extractor\b': 'from optimization_engine.extractors.op2_extractor',
    r'from optimization_engine\.extractor_library\b': 'from optimization_engine.extractors.extractor_library',
    r'from optimization_engine\.simulation_validator\b': 'from optimization_engine.validators.simulation_validator',

    # Processors
    r'from optimization_engine\.adaptive_characterization\b': 'from optimization_engine.processors.adaptive_characterization',
}

# Path mappings for documentation
PATH_MAPPINGS = {
    'optimization_engine/runner.py': 'optimization_engine/core/runner.py',
    'optimization_engine/base_runner.py': 'optimization_engine/core/base_runner.py',
    'optimization_engine/intelligent_optimizer.py': 'optimization_engine/core/intelligent_optimizer.py',
    'optimization_engine/nx_solver.py': 'optimization_engine/nx/solver.py',
    'optimization_engine/nx_updater.py': 'optimization_engine/nx/updater.py',
    'optimization_engine/study_creator.py': 'optimization_engine/study/creator.py',
    'optimization_engine/study_wizard.py': 'optimization_engine/study/wizard.py',
    'optimization_engine/neural_surrogate.py': 'optimization_engine/processors/surrogates/neural_surrogate.py',
    'optimization_engine/config_manager.py': 'optimization_engine/config/manager.py',
    'optimization_engine/logger.py': 'optimization_engine/utils/logger.py',
    # Add all other path mappings...
}

def find_files(root: Path, extensions: List[str]) -> List[Path]:
    """Find all files with given extensions."""
    files = []
    for ext in extensions:
        files.extend(root.rglob(f'*{ext}'))
    return files

def update_file(filepath: Path, mappings: Dict[str, str], dry_run: bool = True) -> Tuple[int, List[str]]:
    """Update imports in a single file."""
    try:
        content = filepath.read_text(encoding='utf-8', errors='ignore')
    except Exception as e:
        print(f"  ERROR reading {filepath}: {e}")
        return 0, []

    changes = []
    new_content = content

    for pattern, replacement in mappings.items():
        matches = re.findall(pattern, content)
        if matches:
            new_content = re.sub(pattern, replacement, new_content)
            changes.append(f"  {pattern} → {replacement} ({len(matches)} occurrences)")

    if changes and not dry_run:
        filepath.write_text(new_content, encoding='utf-8')

    return len(changes), changes

def main():
    dry_run = '--dry-run' in sys.argv or '--execute' not in sys.argv

    if dry_run:
        print("=" * 60)
        print("DRY RUN MODE - No files will be modified")
        print("=" * 60)
    else:
        print("=" * 60)
        print("EXECUTE MODE - Files will be modified!")
        print("=" * 60)
        confirm = input("Are you sure? (yes/no): ")
        if confirm.lower() != 'yes':
            print("Aborted.")
            return

    root = Path('.')

    # Find all Python files
    py_files = find_files(root, ['.py'])
    py_files = [f for f in py_files if 'optimization_engine_BACKUP' not in str(f)]
    py_files = [f for f in py_files if '.venv' not in str(f)]
    py_files = [f for f in py_files if 'node_modules' not in str(f)]

    print(f"\nFound {len(py_files)} Python files to check")

    total_changes = 0
    files_changed = 0

    for filepath in sorted(py_files):
        count, changes = update_file(filepath, IMPORT_MAPPINGS, dry_run)
        if count > 0:
            files_changed += 1
            total_changes += count
            print(f"\n{filepath} ({count} changes):")
            for change in changes:
                print(change)

    # Find all Markdown files
    md_files = find_files(root, ['.md'])
    md_files = [f for f in md_files if 'optimization_engine_BACKUP' not in str(f)]
    md_files = [f for f in md_files if 'node_modules' not in str(f)]

    print(f"\n\nFound {len(md_files)} Markdown files to check")

    for filepath in sorted(md_files):
        count, changes = update_file(filepath, PATH_MAPPINGS, dry_run)
        if count > 0:
            files_changed += 1
            total_changes += count
            print(f"\n{filepath} ({count} changes):")
            for change in changes[:5]:  # Limit output
                print(change)
            if len(changes) > 5:
                print(f"  ... and {len(changes) - 5} more")

    print("\n" + "=" * 60)
    print(f"SUMMARY: {total_changes} changes in {files_changed} files")
    print("=" * 60)

    if dry_run:
        print("\nTo apply changes, run: python migrate_imports.py --execute")

if __name__ == '__main__':
    main()
```

### Phase 4: Create __init__.py Files (1 hour)

Create proper `__init__.py` for each new module. See separate section below.

### Phase 5: Update feature_registry.json (30 minutes)

Use a dedicated script to update all path references in the JSON file.

### Phase 6: Update Documentation (2 hours)

Run sed/grep commands for bulk markdown updates, then manual review.

### Phase 7: Testing & Validation (2 hours)

```bash
# 1. Test Python imports
python -c "from optimization_engine.core.runner import OptimizationRunner; print('core OK')"
python -c "from optimization_engine.nx.solver import NXSolver; print('nx OK')"
python -c "from optimization_engine.study.creator import StudyCreator; print('study OK')"
python -c "from optimization_engine.processors.surrogates.neural_surrogate import NeuralSurrogate; print('surrogates OK')"
python -c "from optimization_engine.config.manager import ConfigManager; print('config OK')"
python -c "from optimization_engine.utils.logger import get_logger; print('utils OK')"

# 2. Test backwards compatibility
python -c "from optimization_engine import nx_solver; print('compat OK')"

# 3. Run full test suite
python -m pytest tests/ -v --tb=short 2>&1 | tee post_migration_tests.log

# 4. Compare test results
diff baseline_tests_*.log post_migration_tests.log

# 5. Test a study
cd studies/bracket_displacement_maximizing
python run_optimization.py --trials 1 --dry-run

# 6. Test dashboard
cd atomizer-dashboard/backend
python -c "from api.routes.optimization import *; print('dashboard OK')"
```

### Phase 8: Commit & Cleanup (30 minutes)

```bash
# Remove backup after successful testing
rm -rf optimization_engine_BACKUP_*

# Clear Python cache
find . -name "*.pyc" -delete
find . -name "__pycache__" -type d -exec rm -rf {} + 2>/dev/null || true

# Final commit
git add -A
git commit -m "refactor: Major reorganization of optimization_engine

BREAKING CHANGE: Module structure reorganized for better maintainability.

New structure:
- core/           - Optimization runners (runner, intelligent_optimizer, etc.)
- processors/     - Data processing (surrogates/, dynamic_response/)
- nx/             - NX/Nastran integration (solver, updater, etc.)
- study/          - Study management (creator, wizard, state, etc.)
- reporting/      - Reports and analysis
- config/         - Configuration management
- utils/          - Utilities (logger, etc.)
- future/         - Research/experimental code

Migration:
- 500+ files updated
- ~1,400 import changes
- Backwards compatibility aliases provided with deprecation warnings
- All tests passing

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>"

# Push
git push origin refactor/optimization-engine-v2
```

---

## Part 4: __init__.py Files Content

### 4.1 optimization_engine/core/__init__.py

```python
"""
Optimization Engine Core
========================

Main optimization runners and algorithm selection.

Modules:
- runner: Main OptimizationRunner class
- base_runner: BaseRunner abstract class
- intelligent_optimizer: IMSO adaptive optimizer
- method_selector: Algorithm selection logic
- strategy_selector: Strategy portfolio management
"""

from .runner import OptimizationRunner
from .base_runner import BaseRunner
from .runner_with_neural import NeuralOptimizationRunner
from .intelligent_optimizer import IntelligentOptimizer, IMSO
from .method_selector import MethodSelector, select_method
from .strategy_selector import StrategySelector
from .strategy_portfolio import StrategyPortfolio
from .gradient_optimizer import GradientOptimizer, LBFGSPolisher

__all__ = [
    'OptimizationRunner',
    'BaseRunner',
    'NeuralOptimizationRunner',
    'IntelligentOptimizer',
    'IMSO',
    'MethodSelector',
    'select_method',
    'StrategySelector',
    'StrategyPortfolio',
    'GradientOptimizer',
    'LBFGSPolisher',
]
```

### 4.2 optimization_engine/processors/__init__.py

```python
"""
Optimization Processors
=======================

Data processing algorithms and ML models.

Submodules:
- surrogates/: Neural network surrogate models
- dynamic_response/: Dynamic response processing (random vib, sine sweep)
"""

from . import surrogates

__all__ = [
    'surrogates',
]
```

### 4.3 optimization_engine/processors/surrogates/__init__.py

```python
"""
Surrogate Models
================

Neural network and ML surrogate models for FEA acceleration.

Classes:
- NeuralSurrogate: Main neural network surrogate
- GenericSurrogate: Flexible surrogate interface
- AdaptiveSurrogate: Self-improving surrogate
- SimpleMLP: Simple multi-layer perceptron
- ActiveLearningSurrogate: Active learning surrogate
- SurrogateTuner: Hyperparameter tuning
"""

from .neural_surrogate import NeuralSurrogate, create_surrogate
from .generic_surrogate import GenericSurrogate
from .adaptive_surrogate import AdaptiveSurrogate
from .simple_mlp_surrogate import SimpleMLP, SimpleMLPSurrogate
from .active_learning_surrogate import ActiveLearningSurrogate
from .surrogate_tuner import SurrogateTuner, tune_surrogate
from .auto_trainer import AutoTrainer
from .training_data_exporter import TrainingDataExporter, export_training_data

__all__ = [
    'NeuralSurrogate',
    'create_surrogate',
    'GenericSurrogate',
    'AdaptiveSurrogate',
    'SimpleMLP',
    'SimpleMLPSurrogate',
    'ActiveLearningSurrogate',
    'SurrogateTuner',
    'tune_surrogate',
    'AutoTrainer',
    'TrainingDataExporter',
    'export_training_data',
]
```

### 4.4 optimization_engine/nx/__init__.py

```python
"""
NX Integration
==============

Siemens NX and Nastran integration modules.

Modules:
- solver: NXSolver for running simulations
- updater: NXParameterUpdater for design updates
- session_manager: NX session lifecycle management
- solve_simulation: Low-level simulation execution
"""

from .solver import NXSolver, run_nx_simulation
from .updater import NXParameterUpdater, update_parameters
from .session_manager import NXSessionManager, get_session
from .solve_simulation import solve_simulation, SolveSimulationError
from .model_cleanup import cleanup_model, ModelCleanup

__all__ = [
    'NXSolver',
    'run_nx_simulation',
    'NXParameterUpdater',
    'update_parameters',
    'NXSessionManager',
    'get_session',
    'solve_simulation',
    'SolveSimulationError',
    'cleanup_model',
    'ModelCleanup',
]
```

### 4.5 optimization_engine/study/__init__.py

```python
"""
Study Management
================

Study creation, state management, and lifecycle.

Modules:
- creator: Study creation from templates
- wizard: Interactive study setup wizard
- state: Study state tracking
- reset: Study reset functionality
- continuation: Resume interrupted studies
"""

from .creator import StudyCreator, create_study
from .wizard import StudyWizard, run_wizard
from .state import StudyState, get_study_state
from .reset import reset_study, StudyReset
from .continuation import continue_study, StudyContinuation
from .benchmarking import BenchmarkingSubstudy
from .history_generator import generate_history

__all__ = [
    'StudyCreator',
    'create_study',
    'StudyWizard',
    'run_wizard',
    'StudyState',
    'get_study_state',
    'reset_study',
    'StudyReset',
    'continue_study',
    'StudyContinuation',
    'BenchmarkingSubstudy',
    'generate_history',
]
```

### 4.6 optimization_engine/reporting/__init__.py

```python
"""
Reporting & Analysis
====================

Report generation and results analysis.

Modules:
- report_generator: HTML/PDF report generation
- markdown_report: Markdown report format
- results_analyzer: Comprehensive results analysis
- visualizer: Plotting and visualization
- landscape_analyzer: Design space analysis
"""

from .report_generator import generate_report, ReportGenerator
from .markdown_report import generate_markdown_report, MarkdownReporter
from .results_analyzer import ResultsAnalyzer, analyze_results
from .visualizer import Visualizer, plot_results
from .landscape_analyzer import LandscapeAnalyzer, analyze_landscape

__all__ = [
    'generate_report',
    'ReportGenerator',
    'generate_markdown_report',
    'MarkdownReporter',
    'ResultsAnalyzer',
    'analyze_results',
    'Visualizer',
    'plot_results',
    'LandscapeAnalyzer',
    'analyze_landscape',
]
```

### 4.7 optimization_engine/config/__init__.py

```python
"""
Configuration Management
========================

Configuration loading, validation, and building.

Modules:
- manager: ConfigManager for loading/saving configs
- builder: OptimizationConfigBuilder for creating configs
- setup_wizard: Interactive configuration setup
- capability_matcher: Match capabilities to requirements
"""

from .manager import ConfigManager, load_config, save_config
from .builder import OptimizationConfigBuilder, build_config
from .setup_wizard import SetupWizard, run_setup
from .capability_matcher import CapabilityMatcher, match_capabilities
from .template_loader import TemplateLoader, load_template

__all__ = [
    'ConfigManager',
    'load_config',
    'save_config',
    'OptimizationConfigBuilder',
    'build_config',
    'SetupWizard',
    'run_setup',
    'CapabilityMatcher',
    'match_capabilities',
    'TemplateLoader',
    'load_template',
]
```

### 4.8 optimization_engine/__init__.py (Updated with Backwards Compat)

```python
"""
Atomizer Optimization Engine
============================

Structural optimization framework for Siemens NX.

New Module Structure (v2.0):
- core/           - Optimization runners
- processors/     - Data processing (surrogates, dynamic_response)
- nx/             - NX/Nastran integration
- study/          - Study management
- reporting/      - Reports and analysis
- config/         - Configuration
- extractors/     - Physics extraction (unchanged)
- insights/       - Visualizations (unchanged)
- gnn/            - Graph neural networks (unchanged)
- hooks/          - NX hooks (unchanged)
- utils/          - Utilities
- validators/     - Validation (unchanged)

Quick Start:
    from optimization_engine.core import OptimizationRunner
    from optimization_engine.nx import NXSolver
    from optimization_engine.extractors import extract_displacement
"""

__version__ = '2.0.0'

# Re-export commonly used items at top level
from optimization_engine.core.runner import OptimizationRunner
from optimization_engine.core.intelligent_optimizer import IMSO, IntelligentOptimizer
from optimization_engine.nx.solver import NXSolver, run_nx_simulation
from optimization_engine.study.creator import create_study
from optimization_engine.config.manager import ConfigManager

# Submodule access
from optimization_engine import (
    core,
    processors,
    nx,
    study,
    reporting,
    config,
    extractors,
    insights,
    gnn,
    hooks,
    utils,
    validators,
)

__all__ = [
    # Version
    '__version__',
    # Top-level exports
    'OptimizationRunner',
    'IMSO',
    'IntelligentOptimizer',
    'NXSolver',
    'run_nx_simulation',
    'create_study',
    'ConfigManager',
    # Submodules
    'core',
    'processors',
    'nx',
    'study',
    'reporting',
    'config',
    'extractors',
    'insights',
    'gnn',
    'hooks',
    'utils',
    'validators',
]

# =============================================================================
# BACKWARDS COMPATIBILITY LAYER
# =============================================================================
# These aliases allow old imports to work with deprecation warnings.
# Will be removed in v3.0.

import warnings as _warnings
import importlib as _importlib

_DEPRECATED_MAPPINGS = {
    # Core
    'runner': 'optimization_engine.core.runner',
    'base_runner': 'optimization_engine.core.base_runner',
    'intelligent_optimizer': 'optimization_engine.core.intelligent_optimizer',
    'method_selector': 'optimization_engine.core.method_selector',
    'strategy_selector': 'optimization_engine.core.strategy_selector',
    'strategy_portfolio': 'optimization_engine.core.strategy_portfolio',
    'gradient_optimizer': 'optimization_engine.core.gradient_optimizer',

    # Surrogates
    'neural_surrogate': 'optimization_engine.processors.surrogates.neural_surrogate',
    'generic_surrogate': 'optimization_engine.processors.surrogates.generic_surrogate',
    'adaptive_surrogate': 'optimization_engine.processors.surrogates.adaptive_surrogate',
    'simple_mlp_surrogate': 'optimization_engine.processors.surrogates.simple_mlp_surrogate',
    'active_learning_surrogate': 'optimization_engine.processors.surrogates.active_learning_surrogate',
    'surrogate_tuner': 'optimization_engine.processors.surrogates.surrogate_tuner',

    # NX
    'nx_solver': 'optimization_engine.nx.solver',
    'nx_updater': 'optimization_engine.nx.updater',
    'nx_session_manager': 'optimization_engine.nx.session_manager',
    'solve_simulation': 'optimization_engine.nx.solve_simulation',
    'model_cleanup': 'optimization_engine.nx.model_cleanup',

    # Study
    'study_creator': 'optimization_engine.study.creator',
    'study_wizard': 'optimization_engine.study.wizard',
    'study_state': 'optimization_engine.study.state',
    'study_reset': 'optimization_engine.study.reset',
    'study_continuation': 'optimization_engine.study.continuation',

    # Reporting
    'generate_report': 'optimization_engine.reporting.report_generator',
    'generate_report_markdown': 'optimization_engine.reporting.markdown_report',
    'visualizer': 'optimization_engine.reporting.visualizer',

    # Config
    'config_manager': 'optimization_engine.config.manager',
    'optimization_config_builder': 'optimization_engine.config.builder',
    'optimization_setup_wizard': 'optimization_engine.config.setup_wizard',
    'capability_matcher': 'optimization_engine.config.capability_matcher',

    # Utils
    'logger': 'optimization_engine.utils.logger',
}


def __getattr__(name):
    """Provide backwards compatibility with deprecation warnings."""
    if name in _DEPRECATED_MAPPINGS:
        new_module = _DEPRECATED_MAPPINGS[name]
        _warnings.warn(
            f"Importing '{name}' from optimization_engine is deprecated. "
            f"Use '{new_module}' instead. "
            f"This will be removed in v3.0.",
            DeprecationWarning,
            stacklevel=2
        )
        return _importlib.import_module(new_module)

    raise AttributeError(f"module 'optimization_engine' has no attribute '{name}'")
```

---

## Part 5: Post-Migration Checklist

### Immediate (same day)

- [ ] All imports updated in optimization_engine/
- [ ] All imports updated in studies/
- [ ] All imports updated in tests/
- [ ] All imports updated in tools/
- [ ] All imports updated in dashboard backend
- [ ] feature_registry.json updated
- [ ] All __init__.py files created
- [ ] Basic import tests pass
- [ ] Test suite passes (or known failures documented)

### Short-term (within 1 week)

- [ ] Update CLAUDE.md with new structure
- [ ] Update all protocol files (SYS_10-16, OP_01-06, EXT_01-04)
- [ ] Update all skill files in .claude/skills/
- [ ] Update ATOMIZER_CONTEXT.md
- [ ] Update README.md
- [ ] Record migration in LAC knowledge base
- [ ] Run 2-3 studies end-to-end to verify

### Medium-term (within 1 month)

- [ ] Remove backwards compatibility after team confirms no issues
- [ ] Update any external documentation/wikis
- [ ] Update pyproject.toml metadata if needed
- [ ] Archive the migration plan document

---

## Part 6: Rollback Procedure

If critical issues are discovered:

```bash
# Option 1: Git revert (preferred if committed)
git revert HEAD

# Option 2: Reset to backup branch
git checkout backup/pre-reorganization
git checkout -b main-restored
git branch -D main
git branch -m main-restored main

# Option 3: Restore from backup directory
rm -rf optimization_engine
mv optimization_engine_BACKUP_* optimization_engine
```

---

## Part 7: Time Summary

| Phase | Task | Time |
|-------|------|------|
| 0 | Preparation (backup, baseline) | 30 min |
| 1 | Create directory structure | 15 min |
| 2 | Move files | 45 min |
| 3 | Update imports (with script) | 2-3 hours |
| 4 | Create __init__.py files | 1 hour |
| 5 | Update feature_registry.json | 30 min |
| 6 | Update documentation | 2 hours |
| 7 | Testing & validation | 2 hours |
| 8 | Commit & cleanup | 30 min |
| **TOTAL** | | **10-12 hours** |

---

## Appendix: Quick Reference Card

### New Import Patterns

```python
# OLD                                    # NEW
from optimization_engine.runner          from optimization_engine.core.runner
from optimization_engine.nx_solver       from optimization_engine.nx.solver
from optimization_engine.study_creator   from optimization_engine.study.creator
from optimization_engine.neural_surrogate from optimization_engine.processors.surrogates.neural_surrogate
from optimization_engine.config_manager  from optimization_engine.config.manager
from optimization_engine.logger          from optimization_engine.utils.logger
```

### Directory Quick Reference

```
optimization_engine/
├── core/           → runner, intelligent_optimizer, method_selector
├── processors/
│   └── surrogates/ → neural_surrogate, generic_surrogate, etc.
├── nx/             → solver (was nx_solver), updater, session_manager
├── study/          → creator (was study_creator), wizard, state
├── reporting/      → report_generator, visualizer
├── config/         → manager (was config_manager), builder
├── utils/          → logger, auto_doc
├── future/         → research_agent, workflow_decomposer
├── extractors/     → (unchanged)
├── insights/       → (unchanged)
├── gnn/            → (unchanged)
├── hooks/          → (unchanged)
└── validators/     → (unchanged)
```

---

## Part 8: Handoff to Context Engineering Plan

After this migration is complete, proceed with `ATOMIZER_CONTEXT_ENGINEERING_PLAN.md`.

### What Context Engineering Will Add

```
optimization_engine/
├── context/                    ← NEW (added by Context Engineering)
│   ├── __init__.py
│   ├── playbook.py            # AtomizerPlaybook - structured knowledge storage
│   ├── reflector.py           # Analyzes optimization outcomes
│   ├── session_state.py       # Context isolation for LLM sessions
│   ├── compaction.py          # Context compaction for long sessions
│   ├── feedback_loop.py       # Learning from execution outcomes
│   └── cache_monitor.py       # KV-cache efficiency tracking
├── core/                       ← Already created by Migration
│   └── runner.py              # Will be modified to integrate context
└── ...
```

### Context Engineering Integration Points

After migration, Context Engineering will modify:

1. **`optimization_engine/core/runner.py`** - Add playbook and reflector integration
2. **`.claude/skills/00_BOOTSTRAP.md`** - Update to v2 with playbook loading
3. **`.claude/skills/02_CONTEXT_LOADER.md`** - Add playbook integration
4. **`atomizer-dashboard/backend/api/routes/`** - Add context API routes

### Pre-Migration Checklist for Context Engineering Compatibility

Before starting Context Engineering, verify:

- [ ] Migration completed successfully
- [ ] `optimization_engine/core/runner.py` exists at new location
- [ ] All imports in codebase updated to new paths
- [ ] Test suite passes
- [ ] No deprecation warnings in core workflows

### Context Engineering Will Use These New Paths

```python
# Context Engineering code will import from new locations:
from optimization_engine.core.runner import OptimizationRunner
from optimization_engine.core.intelligent_optimizer import IMSO
from optimization_engine.utils.logger import get_logger
from optimization_engine.nx.solver import NXSolver

# And add new context module:
from optimization_engine.context.playbook import AtomizerPlaybook
from optimization_engine.context.reflector import AtomizerReflector
```

---

*This plan is DEFINITIVE and covers ALL 500+ files that need updating.*
*After completion, proceed to ATOMIZER_CONTEXT_ENGINEERING_PLAN.md*