Atomizer/.claude/skills/modules/OPTIMIZATION_ENGINE_MIGRATION_PLAN.md

Optimization Engine Reorganization - Migration Plan

Comprehensive Guide for Safe Codebase Restructuring

Document Version: 1.0 Created: 2025-12-23 Status: PLANNING - Do Not Execute Without Review Risk Level: HIGH - Affects 557+ locations across 276 files

---

Executive Summary

This document provides a complete migration plan for reorganizing optimization_engine/ from 50+ loose files to a clean modular structure. This is a high-impact refactoring that requires careful execution.

Impact Summary

Category	Files Affected	Lines to Change
Python imports (internal)	90+ files	~145 changes
Python imports (studies)	30+ folders	~153 changes
Python imports (tests)	30+ files	~79 changes
Python imports (dashboard)	3 files	~11 changes
Documentation (protocols, skills)	119 files	~200 changes
JSON configs	4 files	~50 changes
TOTAL	276 files	~640 changes

---

Part 1: Current State Analysis

1.1 Top-Level Files Requiring Migration

These 50+ files at optimization_engine/ root need to move:

SURROGATES (6 files) → processors/surrogates/
├── neural_surrogate.py
├── generic_surrogate.py
├── adaptive_surrogate.py
├── simple_mlp_surrogate.py
├── active_learning_surrogate.py
└── surrogate_tuner.py

OPTIMIZATION CORE (7 files) → core/
├── runner.py
├── runner_with_neural.py
├── base_runner.py
├── intelligent_optimizer.py
├── method_selector.py
├── strategy_selector.py
└── strategy_portfolio.py

NX INTEGRATION (6 files) → nx/
├── nx_solver.py
├── nx_updater.py
├── nx_session_manager.py
├── solve_simulation.py
├── solve_simulation_simple.py
└── model_cleanup.py

STUDY MANAGEMENT (5 files) → study/
├── study_creator.py
├── study_wizard.py
├── study_state.py
├── study_reset.py
└── study_continuation.py

REPORTING (5 files) → reporting/
├── generate_report.py
├── generate_report_markdown.py
├── comprehensive_results_analyzer.py
├── visualizer.py
└── landscape_analyzer.py

CONFIG (4 files) → config/
├── config_manager.py
├── optimization_config_builder.py
├── optimization_setup_wizard.py
└── capability_matcher.py

AGENTS/RESEARCH (5 files) → agents/ or future/
├── research_agent.py
├── pynastran_research_agent.py
├── targeted_research_planner.py
├── workflow_decomposer.py
└── step_classifier.py

MISC (remaining ~15 files) - evaluate individually
├── logger.py → utils/
├── op2_extractor.py → extractors/
├── extractor_library.py → extractors/
├── export_expressions.py → nx/
├── import_expressions.py → nx/
├── mesh_converter.py → nx/
├── simulation_validator.py → validators/
├── auto_doc.py → utils/
├── auto_trainer.py → processors/surrogates/
├── realtime_tracking.py → utils/
├── benchmarking_substudy.py → study/
├── codebase_analyzer.py → utils/
├── training_data_exporter.py → processors/surrogates/
├── pruning_logger.py → utils/
├── adaptive_characterization.py → processors/
└── generate_history_from_trials.py → study/

1.2 Well-Organized Directories (Keep As-Is)

These are already properly organized:

extractors/          ✓ 20+ extractors, clean __init__.py
insights/            ✓ 8 insight types, registry pattern
hooks/               ✓ nx_cad/, nx_cae/ subdirs
gnn/                 ✓ Neural surrogate for Zernike
templates/           ✓ Config templates
schemas/             ✓ JSON schemas
validators/          ✓ Validation code
plugins/             ✓ Hook manager system
utils/               ✓ Utility functions
custom_functions/    ✓ NX material generator
model_discovery/     ✓ Model introspection
future/              ✓ Experimental code

---

Part 2: Target Structure

2.1 Proposed Final Structure

optimization_engine/
│
├── __init__.py                    # Updated with backwards-compat aliases
│
├── core/                          # NEW - Optimization engine core
│   ├── __init__.py
│   ├── runner.py
│   ├── base_runner.py
│   ├── runner_with_neural.py
│   ├── intelligent_optimizer.py
│   ├── method_selector.py
│   ├── strategy_selector.py
│   └── strategy_portfolio.py
│
├── processors/                    # NEW - Data processing & algorithms
│   ├── __init__.py
│   ├── surrogates/
│   │   ├── __init__.py
│   │   ├── neural_surrogate.py
│   │   ├── generic_surrogate.py
│   │   ├── adaptive_surrogate.py
│   │   ├── simple_mlp_surrogate.py
│   │   ├── active_learning_surrogate.py
│   │   ├── surrogate_tuner.py
│   │   ├── auto_trainer.py
│   │   └── training_data_exporter.py
│   │
│   └── dynamic_response/          # NEW - From master plan
│       ├── __init__.py
│       ├── modal_database.py
│       ├── transfer_functions.py
│       ├── random_vibration.py
│       ├── psd_profiles.py
│       └── utils/
│
├── nx/                            # NEW - NX/Nastran integration
│   ├── __init__.py
│   ├── solver.py                  # Was nx_solver.py
│   ├── updater.py                 # Was nx_updater.py
│   ├── session_manager.py         # Was nx_session_manager.py
│   ├── solve_simulation.py
│   ├── solve_simulation_simple.py
│   ├── model_cleanup.py
│   ├── export_expressions.py
│   ├── import_expressions.py
│   └── mesh_converter.py
│
├── study/                         # NEW - Study management
│   ├── __init__.py
│   ├── creator.py                 # Was study_creator.py
│   ├── wizard.py                  # Was study_wizard.py
│   ├── state.py                   # Was study_state.py
│   ├── reset.py                   # Was study_reset.py
│   ├── continuation.py            # Was study_continuation.py
│   ├── benchmarking.py            # Was benchmarking_substudy.py
│   └── history_generator.py       # Was generate_history_from_trials.py
│
├── reporting/                     # NEW - Reports and analysis
│   ├── __init__.py
│   ├── report_generator.py
│   ├── markdown_report.py
│   ├── results_analyzer.py
│   ├── visualizer.py
│   └── landscape_analyzer.py
│
├── config/                        # NEW - Configuration
│   ├── __init__.py
│   ├── manager.py
│   ├── builder.py
│   ├── setup_wizard.py
│   └── capability_matcher.py
│
├── extractors/                    # EXISTING - Add op2_extractor, extractor_library
├── insights/                      # EXISTING
├── hooks/                         # EXISTING
├── gnn/                           # EXISTING
├── templates/                     # EXISTING
├── schemas/                       # EXISTING
├── validators/                    # EXISTING - Add simulation_validator
├── plugins/                       # EXISTING
├── utils/                         # EXISTING - Add logger, auto_doc, etc.
├── custom_functions/              # EXISTING
├── model_discovery/               # EXISTING
└── future/                        # EXISTING - Move agents here
    ├── research_agent.py
    ├── pynastran_research_agent.py
    ├── targeted_research_planner.py
    ├── workflow_decomposer.py
    └── step_classifier.py

---

Part 3: Import Mapping Tables

3.1 Old → New Import Mapping

This is the critical reference for all updates:

# CORE OPTIMIZATION
"from optimization_engine.runner"                    → "from optimization_engine.core.runner"
"from optimization_engine.base_runner"               → "from optimization_engine.core.base_runner"
"from optimization_engine.runner_with_neural"        → "from optimization_engine.core.runner_with_neural"
"from optimization_engine.intelligent_optimizer"     → "from optimization_engine.core.intelligent_optimizer"
"from optimization_engine.method_selector"           → "from optimization_engine.core.method_selector"
"from optimization_engine.strategy_selector"         → "from optimization_engine.core.strategy_selector"
"from optimization_engine.strategy_portfolio"        → "from optimization_engine.core.strategy_portfolio"

# SURROGATES
"from optimization_engine.neural_surrogate"          → "from optimization_engine.processors.surrogates.neural_surrogate"
"from optimization_engine.generic_surrogate"         → "from optimization_engine.processors.surrogates.generic_surrogate"
"from optimization_engine.adaptive_surrogate"        → "from optimization_engine.processors.surrogates.adaptive_surrogate"
"from optimization_engine.simple_mlp_surrogate"      → "from optimization_engine.processors.surrogates.simple_mlp_surrogate"
"from optimization_engine.active_learning_surrogate" → "from optimization_engine.processors.surrogates.active_learning_surrogate"
"from optimization_engine.surrogate_tuner"           → "from optimization_engine.processors.surrogates.surrogate_tuner"

# NX INTEGRATION
"from optimization_engine.nx_solver"                 → "from optimization_engine.nx.solver"
"from optimization_engine.nx_updater"                → "from optimization_engine.nx.updater"
"from optimization_engine.nx_session_manager"        → "from optimization_engine.nx.session_manager"
"from optimization_engine.solve_simulation"          → "from optimization_engine.nx.solve_simulation"
"from optimization_engine.model_cleanup"             → "from optimization_engine.nx.model_cleanup"
"from optimization_engine.export_expressions"        → "from optimization_engine.nx.export_expressions"
"from optimization_engine.import_expressions"        → "from optimization_engine.nx.import_expressions"
"from optimization_engine.mesh_converter"            → "from optimization_engine.nx.mesh_converter"

# STUDY MANAGEMENT
"from optimization_engine.study_creator"             → "from optimization_engine.study.creator"
"from optimization_engine.study_wizard"              → "from optimization_engine.study.wizard"
"from optimization_engine.study_state"               → "from optimization_engine.study.state"
"from optimization_engine.study_reset"               → "from optimization_engine.study.reset"
"from optimization_engine.study_continuation"        → "from optimization_engine.study.continuation"

# REPORTING
"from optimization_engine.generate_report"           → "from optimization_engine.reporting.report_generator"
"from optimization_engine.generate_report_markdown"  → "from optimization_engine.reporting.markdown_report"
"from optimization_engine.comprehensive_results"     → "from optimization_engine.reporting.results_analyzer"
"from optimization_engine.visualizer"                → "from optimization_engine.reporting.visualizer"
"from optimization_engine.landscape_analyzer"        → "from optimization_engine.reporting.landscape_analyzer"

# CONFIG
"from optimization_engine.config_manager"            → "from optimization_engine.config.manager"
"from optimization_engine.optimization_config_builder" → "from optimization_engine.config.builder"
"from optimization_engine.optimization_setup_wizard" → "from optimization_engine.config.setup_wizard"
"from optimization_engine.capability_matcher"        → "from optimization_engine.config.capability_matcher"

# UTILITIES (moving to utils/)
"from optimization_engine.logger"                    → "from optimization_engine.utils.logger"
"from optimization_engine.auto_doc"                  → "from optimization_engine.utils.auto_doc"
"from optimization_engine.realtime_tracking"         → "from optimization_engine.utils.realtime_tracking"
"from optimization_engine.codebase_analyzer"         → "from optimization_engine.utils.codebase_analyzer"
"from optimization_engine.pruning_logger"            → "from optimization_engine.utils.pruning_logger"

# RESEARCH/AGENTS (moving to future/)
"from optimization_engine.research_agent"            → "from optimization_engine.future.research_agent"
"from optimization_engine.workflow_decomposer"       → "from optimization_engine.future.workflow_decomposer"
"from optimization_engine.step_classifier"           → "from optimization_engine.future.step_classifier"

3.2 Backwards Compatibility Aliases

Add to optimization_engine/__init__.py for transition period:

# BACKWARDS COMPATIBILITY ALIASES
# These allow old imports to work during migration period
# Remove after all code is updated

# Core
from optimization_engine.core.runner import *
from optimization_engine.core.base_runner import *
from optimization_engine.core.intelligent_optimizer import *

# NX
from optimization_engine.nx import solver as nx_solver
from optimization_engine.nx import updater as nx_updater
from optimization_engine.nx import session_manager as nx_session_manager
from optimization_engine.nx import solve_simulation

# Study
from optimization_engine.study import creator as study_creator
from optimization_engine.study import wizard as study_wizard
from optimization_engine.study import state as study_state

# Surrogates
from optimization_engine.processors.surrogates import neural_surrogate
from optimization_engine.processors.surrogates import generic_surrogate

# Config
from optimization_engine.config import manager as config_manager

# Utils
from optimization_engine.utils import logger

# Deprecation warnings (optional)
import warnings
def __getattr__(name):
    deprecated = {
        'nx_solver': 'optimization_engine.nx.solver',
        'study_creator': 'optimization_engine.study.creator',
        # ... more mappings
    }
    if name in deprecated:
        warnings.warn(
            f"Importing {name} from optimization_engine is deprecated. "
            f"Use {deprecated[name]} instead.",
            DeprecationWarning,
            stacklevel=2
        )
        # Return the module anyway for compatibility
    raise AttributeError(f"module 'optimization_engine' has no attribute '{name}'")

---

Part 4: Files Requiring Updates

4.1 Internal optimization_engine Files (90+ files)

Highest-impact internal files:

File	Imports to Update	Priority
runner.py	nx_solver, config_manager, extractors	Critical
base_runner.py	config_manager, validators	Critical
intelligent_optimizer.py	runner, neural_surrogate, method_selector	Critical
runner_with_neural.py	runner, neural_surrogate	High
gnn/gnn_optimizer.py	polar_graph, zernike_gnn, nx_solver	High
hooks/nx_cad/*.py	nx_session_manager	High
plugins/hook_manager.py	validators, config	Medium

4.2 Study Scripts (30+ folders, 153 imports)

Pattern in every study's run_optimization.py:

# BEFORE
from optimization_engine.nx_solver import run_nx_simulation
from optimization_engine.extractors import ZernikeExtractor
from optimization_engine.gnn.gnn_optimizer import ZernikeGNNOptimizer

# AFTER
from optimization_engine.nx.solver import run_nx_simulation
from optimization_engine.extractors import ZernikeExtractor  # unchanged
from optimization_engine.gnn.gnn_optimizer import ZernikeGNNOptimizer  # unchanged

Studies requiring updates:

studies/
├── M1_Mirror/
│   ├── m1_mirror_adaptive_V*/run_optimization.py (12 files)
│   └── [other variants]
├── Simple_Bracket/
│   ├── bracket_*/run_optimization.py (8 files)
├── UAV_Arm/
│   ├── */run_optimization.py (4 files)
├── Drone_Gimbal/
├── Simple_Beam/
└── [others]

4.3 Test Files (30+ files, 79 imports)

Located in tests/:

tests/
├── test_extractors.py
├── test_zernike_*.py (5+ files)
├── test_neural_surrogate.py
├── test_gnn_*.py
├── test_nx_solver.py
├── test_study_*.py
└── [others]

4.4 Dashboard Backend (3 files, 11 imports)

atomizer-dashboard/backend/api/
├── main.py                # sys.path setup
├── routes/optimization.py # study management imports
└── routes/insights.py     # insight imports

4.5 Documentation Files (119 files)

Protocols requiring updates:

Protocol	References	Changes Needed
SYS_10_IMSO.md	intelligent_optimizer	core.intelligent_optimizer
SYS_12_EXTRACTOR_LIBRARY.md	extractors/*	None (unchanged)
SYS_14_NEURAL_ACCELERATION.md	neural_surrogate, gnn/*	processors.surrogates.*
SYS_15_METHOD_SELECTOR.md	method_selector	core.method_selector
OP_01_CREATE_STUDY.md	study_creator, study_wizard	study.creator, study.wizard
OP_02_RUN_OPTIMIZATION.md	runner, nx_solver	core.runner, nx.solver

Skill files requiring updates:

Skill File	Changes
01_CHEATSHEET.md	Path references
core/study-creation-core.md	Import examples
modules/extractors-catalog.md	Directory paths
modules/neural-acceleration.md	Surrogate imports

4.6 JSON Configuration Files (4 files)

feature_registry.json (878 lines, 50+ path references):

// BEFORE
"file_path": "optimization_engine/neural_surrogate.py"

// AFTER
"file_path": "optimization_engine/processors/surrogates/neural_surrogate.py"

---

Part 5: Migration Execution Plan

Phase 0: Pre-Migration (30 min)

1. Create full backup

   git stash  # Save any uncommitted changes
   git checkout -b refactor/optimization-engine-reorganization
   cp -r optimization_engine optimization_engine_backup
   

2. Run baseline tests

   python -m pytest tests/ -v --tb=short > baseline_tests.log 2>&1
   

3. Document current working state

   • Run one study end-to-end
   • Verify dashboard loads
   • Note any existing failures

Phase 1: Create Directory Structure (15 min)

# Create new directories
mkdir -p optimization_engine/core
mkdir -p optimization_engine/processors/surrogates
mkdir -p optimization_engine/processors/dynamic_response
mkdir -p optimization_engine/nx
mkdir -p optimization_engine/study
mkdir -p optimization_engine/reporting
mkdir -p optimization_engine/config

# Create __init__.py files
touch optimization_engine/core/__init__.py
touch optimization_engine/processors/__init__.py
touch optimization_engine/processors/surrogates/__init__.py
touch optimization_engine/processors/dynamic_response/__init__.py
touch optimization_engine/nx/__init__.py
touch optimization_engine/study/__init__.py
touch optimization_engine/reporting/__init__.py
touch optimization_engine/config/__init__.py

Phase 2: Move Files (30 min)

Execute in this order to minimize circular import issues:

# 1. UTILITIES FIRST (no dependencies)
mv optimization_engine/logger.py optimization_engine/utils/
mv optimization_engine/auto_doc.py optimization_engine/utils/
mv optimization_engine/realtime_tracking.py optimization_engine/utils/
mv optimization_engine/codebase_analyzer.py optimization_engine/utils/
mv optimization_engine/pruning_logger.py optimization_engine/utils/

# 2. CONFIG (low dependencies)
mv optimization_engine/config_manager.py optimization_engine/config/manager.py
mv optimization_engine/optimization_config_builder.py optimization_engine/config/builder.py
mv optimization_engine/optimization_setup_wizard.py optimization_engine/config/setup_wizard.py
mv optimization_engine/capability_matcher.py optimization_engine/config/capability_matcher.py

# 3. NX INTEGRATION
mv optimization_engine/nx_solver.py optimization_engine/nx/solver.py
mv optimization_engine/nx_updater.py optimization_engine/nx/updater.py
mv optimization_engine/nx_session_manager.py optimization_engine/nx/session_manager.py
mv optimization_engine/solve_simulation.py optimization_engine/nx/
mv optimization_engine/solve_simulation_simple.py optimization_engine/nx/
mv optimization_engine/model_cleanup.py optimization_engine/nx/
mv optimization_engine/export_expressions.py optimization_engine/nx/
mv optimization_engine/import_expressions.py optimization_engine/nx/
mv optimization_engine/mesh_converter.py optimization_engine/nx/

# 4. SURROGATES
mv optimization_engine/neural_surrogate.py optimization_engine/processors/surrogates/
mv optimization_engine/generic_surrogate.py optimization_engine/processors/surrogates/
mv optimization_engine/adaptive_surrogate.py optimization_engine/processors/surrogates/
mv optimization_engine/simple_mlp_surrogate.py optimization_engine/processors/surrogates/
mv optimization_engine/active_learning_surrogate.py optimization_engine/processors/surrogates/
mv optimization_engine/surrogate_tuner.py optimization_engine/processors/surrogates/
mv optimization_engine/auto_trainer.py optimization_engine/processors/surrogates/
mv optimization_engine/training_data_exporter.py optimization_engine/processors/surrogates/

# 5. STUDY MANAGEMENT
mv optimization_engine/study_creator.py optimization_engine/study/creator.py
mv optimization_engine/study_wizard.py optimization_engine/study/wizard.py
mv optimization_engine/study_state.py optimization_engine/study/state.py
mv optimization_engine/study_reset.py optimization_engine/study/reset.py
mv optimization_engine/study_continuation.py optimization_engine/study/continuation.py
mv optimization_engine/benchmarking_substudy.py optimization_engine/study/benchmarking.py
mv optimization_engine/generate_history_from_trials.py optimization_engine/study/history_generator.py

# 6. REPORTING
mv optimization_engine/generate_report.py optimization_engine/reporting/report_generator.py
mv optimization_engine/generate_report_markdown.py optimization_engine/reporting/markdown_report.py
mv optimization_engine/comprehensive_results_analyzer.py optimization_engine/reporting/results_analyzer.py
mv optimization_engine/visualizer.py optimization_engine/reporting/
mv optimization_engine/landscape_analyzer.py optimization_engine/reporting/

# 7. CORE (depends on many things, do last)
mv optimization_engine/runner.py optimization_engine/core/
mv optimization_engine/base_runner.py optimization_engine/core/
mv optimization_engine/runner_with_neural.py optimization_engine/core/
mv optimization_engine/intelligent_optimizer.py optimization_engine/core/
mv optimization_engine/method_selector.py optimization_engine/core/
mv optimization_engine/strategy_selector.py optimization_engine/core/
mv optimization_engine/strategy_portfolio.py optimization_engine/core/

# 8. RESEARCH/FUTURE
mv optimization_engine/research_agent.py optimization_engine/future/
mv optimization_engine/pynastran_research_agent.py optimization_engine/future/
mv optimization_engine/targeted_research_planner.py optimization_engine/future/
mv optimization_engine/workflow_decomposer.py optimization_engine/future/
mv optimization_engine/step_classifier.py optimization_engine/future/

# 9. REMAINING MISC
mv optimization_engine/op2_extractor.py optimization_engine/extractors/
mv optimization_engine/extractor_library.py optimization_engine/extractors/
mv optimization_engine/simulation_validator.py optimization_engine/validators/
mv optimization_engine/adaptive_characterization.py optimization_engine/processors/

Phase 3: Update Internal Imports (1-2 hours)

Use sed/grep for bulk updates:

# Example sed commands (run from project root)

# NX imports
find . -name "*.py" -exec sed -i 's/from optimization_engine\.nx_solver/from optimization_engine.nx.solver/g' {} +
find . -name "*.py" -exec sed -i 's/from optimization_engine\.nx_updater/from optimization_engine.nx.updater/g' {} +
find . -name "*.py" -exec sed -i 's/from optimization_engine\.nx_session_manager/from optimization_engine.nx.session_manager/g' {} +
find . -name "*.py" -exec sed -i 's/from optimization_engine\.solve_simulation/from optimization_engine.nx.solve_simulation/g' {} +

# Study imports
find . -name "*.py" -exec sed -i 's/from optimization_engine\.study_creator/from optimization_engine.study.creator/g' {} +
find . -name "*.py" -exec sed -i 's/from optimization_engine\.study_wizard/from optimization_engine.study.wizard/g' {} +
find . -name "*.py" -exec sed -i 's/from optimization_engine\.study_state/from optimization_engine.study.state/g' {} +

# Config imports
find . -name "*.py" -exec sed -i 's/from optimization_engine\.config_manager/from optimization_engine.config.manager/g' {} +

# Core imports
find . -name "*.py" -exec sed -i 's/from optimization_engine\.runner import/from optimization_engine.core.runner import/g' {} +
find . -name "*.py" -exec sed -i 's/from optimization_engine\.base_runner/from optimization_engine.core.base_runner/g' {} +
find . -name "*.py" -exec sed -i 's/from optimization_engine\.intelligent_optimizer/from optimization_engine.core.intelligent_optimizer/g' {} +

# Surrogate imports
find . -name "*.py" -exec sed -i 's/from optimization_engine\.neural_surrogate/from optimization_engine.processors.surrogates.neural_surrogate/g' {} +
find . -name "*.py" -exec sed -i 's/from optimization_engine\.generic_surrogate/from optimization_engine.processors.surrogates.generic_surrogate/g' {} +

# Logger
find . -name "*.py" -exec sed -i 's/from optimization_engine\.logger/from optimization_engine.utils.logger/g' {} +

Handle edge cases manually:

• import optimization_engine.nx_solver (without from)
• Dynamic imports using importlib
• String references in configs

Phase 4: Create init.py Files (30 min)

Example: optimization_engine/core/__init__.py

"""
Optimization Engine Core

Main optimization runners and algorithm selection.
"""

from .runner import OptimizationRunner
from .base_runner import BaseRunner
from .intelligent_optimizer import IntelligentOptimizer, IMSO
from .method_selector import MethodSelector
from .strategy_selector import StrategySelector
from .strategy_portfolio import StrategyPortfolio

__all__ = [
    'OptimizationRunner',
    'BaseRunner',
    'IntelligentOptimizer',
    'IMSO',
    'MethodSelector',
    'StrategySelector',
    'StrategyPortfolio',
]

Create similar for each new directory.

Phase 5: Add Backwards Compatibility (30 min)

Update optimization_engine/__init__.py:

"""
Optimization Engine for Atomizer

Reorganized structure (v2.0):
- core/         - Optimization runners
- processors/   - Data processing (surrogates, dynamic response)
- nx/           - NX/Nastran integration
- study/        - Study management
- reporting/    - Reports and analysis
- config/       - Configuration
- extractors/   - Physics extraction
- insights/     - Visualizations
- gnn/          - Graph neural networks
- hooks/        - NX hooks
"""

# Re-export commonly used items at top level for convenience
from optimization_engine.core.runner import OptimizationRunner
from optimization_engine.core.intelligent_optimizer import IMSO
from optimization_engine.nx.solver import run_nx_simulation, NXSolver
from optimization_engine.study.creator import create_study
from optimization_engine.config.manager import ConfigManager

# Backwards compatibility aliases (deprecated)
# These will be removed in a future version
import warnings as _warnings

def _deprecated_import(old_name, new_location):
    _warnings.warn(
        f"Importing '{old_name}' directly from optimization_engine is deprecated. "
        f"Use '{new_location}' instead.",
        DeprecationWarning,
        stacklevel=3
    )

# Lazy loading for backwards compatibility
def __getattr__(name):
    # Map old names to new locations
    _compat_map = {
        'nx_solver': ('optimization_engine.nx.solver', 'nx_solver'),
        'nx_updater': ('optimization_engine.nx.updater', 'nx_updater'),
        'study_creator': ('optimization_engine.study.creator', 'study_creator'),
        'config_manager': ('optimization_engine.config.manager', 'config_manager'),
        'runner': ('optimization_engine.core.runner', 'runner'),
        'neural_surrogate': ('optimization_engine.processors.surrogates.neural_surrogate', 'neural_surrogate'),
    }

    if name in _compat_map:
        module_path, attr = _compat_map[name]
        _deprecated_import(name, module_path)
        import importlib
        module = importlib.import_module(module_path)
        return module

    raise AttributeError(f"module 'optimization_engine' has no attribute '{name}'")

Phase 6: Update Documentation (1 hour)

Use sed for bulk updates in markdown:

# Update protocol files
find docs/protocols -name "*.md" -exec sed -i 's/optimization_engine\/nx_solver/optimization_engine\/nx\/solver/g' {} +
find docs/protocols -name "*.md" -exec sed -i 's/optimization_engine\/study_creator/optimization_engine\/study\/creator/g' {} +
# ... etc for all mappings

# Update .claude skills
find .claude -name "*.md" -exec sed -i 's/optimization_engine\.nx_solver/optimization_engine.nx.solver/g' {} +
# ... etc

Manual review needed for:

• Code examples in markdown
• Directory structure diagrams
• Command-line examples

Phase 7: Update JSON Configs (30 min)

feature_registry.json - Use a Python script:

import json

with open('optimization_engine/feature_registry.json', 'r') as f:
    registry = json.load(f)

# Define path mappings
path_map = {
    'optimization_engine/neural_surrogate.py': 'optimization_engine/processors/surrogates/neural_surrogate.py',
    'optimization_engine/nx_solver.py': 'optimization_engine/nx/solver.py',
    # ... all other mappings
}

def update_paths(obj):
    if isinstance(obj, dict):
        for key, value in obj.items():
            if key == 'file_path' and value in path_map:
                obj[key] = path_map[value]
            else:
                update_paths(value)
    elif isinstance(obj, list):
        for item in obj:
            update_paths(item)

update_paths(registry)

with open('optimization_engine/feature_registry.json', 'w') as f:
    json.dump(registry, f, indent=2)

Phase 8: Testing & Validation (1-2 hours)

# 1. Run Python import tests
python -c "from optimization_engine.core.runner import OptimizationRunner"
python -c "from optimization_engine.nx.solver import run_nx_simulation"
python -c "from optimization_engine.study.creator import create_study"
python -c "from optimization_engine.processors.surrogates.neural_surrogate import NeuralSurrogate"

# 2. Run backwards compatibility tests
python -c "from optimization_engine import nx_solver"  # Should work with deprecation warning

# 3. Run test suite
python -m pytest tests/ -v --tb=short

# 4. Test a study end-to-end
cd studies/Simple_Bracket/bracket_displacement_maximizing
python run_optimization.py --trials 2 --dry-run

# 5. Test dashboard
cd atomizer-dashboard
python backend/api/main.py  # Should start without import errors

Phase 9: Cleanup (15 min)

# Remove backup after successful testing
rm -rf optimization_engine_backup

# Remove any .pyc files that might cache old imports
find . -name "*.pyc" -delete
find . -name "__pycache__" -type d -exec rm -rf {} +

# Commit
git add -A
git commit -m "refactor: Reorganize optimization_engine into modular structure

- Move 50+ top-level files into logical subdirectories
- Create core/, processors/, nx/, study/, reporting/, config/
- Add backwards compatibility aliases with deprecation warnings
- Update all imports across codebase (640+ changes)
- Update documentation and protocols

BREAKING: Direct imports from optimization_engine.* are deprecated.
Use new paths like optimization_engine.core.runner instead.

🤖 Generated with [Claude Code](https://claude.com/claude-code)"

---

Part 6: Rollback Plan

If migration fails:

# Option 1: Git reset (if not committed)
git checkout -- .

# Option 2: Restore backup
rm -rf optimization_engine
mv optimization_engine_backup optimization_engine

# Option 3: Git revert (if committed)
git revert HEAD

---

Part 7: Post-Migration Tasks

7.1 Update LAC Knowledge Base

Record the migration in LAC:

from knowledge_base.lac import get_lac
lac = get_lac()
lac.record_insight(
    category="architecture",
    context="optimization_engine reorganization",
    insight="Migrated 50+ files into modular structure. New paths: "
            "core/, processors/, nx/, study/, reporting/, config/. "
            "Backwards compat aliases provided with deprecation warnings.",
    confidence=1.0,
    tags=["refactoring", "architecture", "breaking-change"]
)

7.2 Update CLAUDE.md

Add section about new structure.

7.3 Schedule Deprecation Removal

After 2-4 weeks of stable operation:

• Remove backwards compatibility aliases
• Update remaining old imports
• Clean up init.py files

---

Part 8: Time Estimate Summary

Phase	Task	Time
0	Pre-migration (backup, baseline)	30 min
1	Create directory structure	15 min
2	Move files	30 min
3	Update internal imports	1-2 hours
4	Create init.py files	30 min
5	Add backwards compatibility	30 min
6	Update documentation	1 hour
7	Update JSON configs	30 min
8	Testing & validation	1-2 hours
9	Cleanup & commit	15 min
TOTAL		6-8 hours

---

Part 9: Decision Points

Option A: Full Migration Now

• Pros: Clean structure, enables dynamic_response cleanly
• Cons: 6-8 hours work, risk of breakage
• When: When you have a dedicated half-day

Option B: Minimal Migration (processors/ only)

• Pros: Low risk, enables dynamic_response
• Cons: Leaves technical debt, inconsistent structure
• When: If you need dynamic_response urgently

Option C: Defer Migration

• Pros: Zero risk now
• Cons: Dynamic_response goes in awkward location
• When: If stability is critical

---

Appendix A: Automated Migration Script

A Python script to automate most of the migration:

#!/usr/bin/env python
"""
optimization_engine Migration Script

Run with: python migrate_optimization_engine.py --dry-run
Then:     python migrate_optimization_engine.py --execute
"""

import os
import re
import shutil
from pathlib import Path

# ... full script would be ~200 lines
# Handles: directory creation, file moves, import updates, __init__ generation

Would you like me to create this full automation script?

---

This plan ensures a safe, reversible migration with clear validation steps.