Files

Anto01 820c34c39a docs: Update documentation for v2.0 module reorganization

- Update feature_registry.json paths to new module locations (v0.3.0)
- Update cheatsheet with new import paths (v2.3)
- Mark migration plan as completed (v3.0)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

2025-12-29 13:01:36 -05:00

69 KiB

Raw Blame History

Optimization Engine Reorganization - DEFINITIVE Migration Plan

Complete Guide for Safe Codebase Restructuring

Document Version: 3.0 (COMPLETED) Created: 2025-12-23 Updated: 2025-12-29 Status: ✅ COMPLETED - Merged to main Risk Level: HIGH - Affects 500+ files across entire codebase Next Phase: ATOMIZER_CONTEXT_ENGINEERING_PLAN.md (execute AFTER this migration)

✅ Migration Completed: 2025-12-29

Final Statistics

Metric	Value
Files modified	120
Import changes	~200
Commits	2 (pre-migration checkpoint + migration)
Branch	`refactor/optimization-engine-v2` merged to `main`
Version bump	`optimization_engine` v1.0 → v2.0.0

Key Changes Made

Created new directory structure: core/, processors/surrogates/, nx/, study/, reporting/, config/
Moved 60+ Python files to new locations
Updated all imports across 125 files
Added lazy-loading __init__.py files to avoid circular imports
Implemented backwards compatibility layer with deprecation warnings
Updated feature_registry.json paths

Post-Migration Notes

All existing run_optimization.py scripts continue to work
Old imports emit deprecation warnings but still function
Dashboard integration preserved
Study databases unchanged

Execution Order (IMPORTANT)

This migration plan should be executed BEFORE the Context Engineering plan.

Why Migration First?

Creates clean, organized module structure
Context Engineering adds NEW modules (context/) that won't be moved
Avoids double-updating imports (migrate first, then integrate new code)
Documentation written once with final paths

After Migration Completes:

Execute ATOMIZER_CONTEXT_ENGINEERING_PLAN.md
Add optimization_engine/context/ module (playbook, reflector, session_state)
Integrate with optimization_engine/core/runner.py (already at final path)

Executive Summary

This is the DEFINITIVE migration plan covering every single file that needs updating. The scope is significantly larger than initially estimated.

True Impact Summary

Category	Files	Estimated Changes
Python files (optimization_engine internal)	90+	~300 import changes
Python files (studies)	80+	~200 import changes
Python files (tests)	45+	~150 import changes
Python files (tools, examples, root)	10+	~50 import changes
Python files (dashboard backend)	3	~15 import changes
Python files (archive - optional)	50+	~100 import changes
Documentation (markdown)	100+	~500 path references
JSON configuration files	5	~50 path entries
Build config (pyproject.toml)	1	2 lines
TOTAL	500+ files	~1,400 changes

Estimated Time: 8-12 hours (with automation scripts)

Part 1: Complete File Inventory

1.1 Files to MOVE (Source Files)

These files move from optimization_engine/ root to new subdirectories:

MOVE TO: optimization_engine/core/
├── runner.py
├── base_runner.py
├── runner_with_neural.py
├── intelligent_optimizer.py
├── method_selector.py
├── strategy_selector.py
├── strategy_portfolio.py
└── gradient_optimizer.py

MOVE TO: optimization_engine/processors/surrogates/
├── neural_surrogate.py
├── generic_surrogate.py
├── adaptive_surrogate.py
├── simple_mlp_surrogate.py
├── active_learning_surrogate.py
├── surrogate_tuner.py
├── auto_trainer.py
└── training_data_exporter.py

MOVE TO: optimization_engine/nx/
├── nx_solver.py → solver.py
├── nx_updater.py → updater.py
├── nx_session_manager.py → session_manager.py
├── solve_simulation.py
├── solve_simulation_simple.py
├── model_cleanup.py
├── export_expressions.py
├── import_expressions.py
└── mesh_converter.py

MOVE TO: optimization_engine/study/
├── study_creator.py → creator.py
├── study_wizard.py → wizard.py
├── study_state.py → state.py
├── study_reset.py → reset.py
├── study_continuation.py → continuation.py
├── benchmarking_substudy.py → benchmarking.py
└── generate_history_from_trials.py → history_generator.py

MOVE TO: optimization_engine/reporting/
├── generate_report.py → report_generator.py
├── generate_report_markdown.py → markdown_report.py
├── comprehensive_results_analyzer.py → results_analyzer.py
├── visualizer.py
└── landscape_analyzer.py

MOVE TO: optimization_engine/config/
├── config_manager.py → manager.py
├── optimization_config_builder.py → builder.py
├── optimization_setup_wizard.py → setup_wizard.py
├── capability_matcher.py
└── template_loader.py

MOVE TO: optimization_engine/utils/ (existing, add to it)
├── logger.py
├── auto_doc.py
├── realtime_tracking.py
├── codebase_analyzer.py
└── pruning_logger.py

MOVE TO: optimization_engine/future/ (existing, add to it)
├── research_agent.py
├── pynastran_research_agent.py
├── targeted_research_planner.py
├── workflow_decomposer.py
├── step_classifier.py
├── llm_optimization_runner.py
├── llm_workflow_analyzer.py
├── extractor_orchestrator.py
├── inline_code_generator.py
└── hook_generator.py

MOVE TO: optimization_engine/extractors/ (existing, add to it)
├── op2_extractor.py
└── extractor_library.py

MOVE TO: optimization_engine/validators/ (existing, add to it)
└── simulation_validator.py

MOVE TO: optimization_engine/processors/ (new)
└── adaptive_characterization.py

1.2 Files to UPDATE (Import Changes Required)

A. OPTIMIZATION_ENGINE INTERNAL (90+ files)

Core/Root Files:

optimization_engine/run_optimization.py
optimization_engine/__init__.py

Extractors (20 files):

optimization_engine/extractors/__init__.py (31 imports)
optimization_engine/extractors/extract_displacement.py
optimization_engine/extractors/extract_frequency.py
optimization_engine/extractors/extract_von_mises_stress.py
optimization_engine/extractors/bdf_mass_extractor.py
optimization_engine/extractors/extract_mass_from_bdf.py
optimization_engine/extractors/extract_mass_from_expression.py
optimization_engine/extractors/extract_part_mass_material.py
optimization_engine/extractors/extract_zernike.py
optimization_engine/extractors/extract_zernike_opd.py
optimization_engine/extractors/extract_zernike_figure.py
optimization_engine/extractors/extract_zernike_surface.py
optimization_engine/extractors/zernike_helpers.py
optimization_engine/extractors/extract_principal_stress.py
optimization_engine/extractors/extract_strain_energy.py
optimization_engine/extractors/extract_spc_forces.py
optimization_engine/extractors/extract_temperature.py
optimization_engine/extractors/extract_modal_mass.py
optimization_engine/extractors/introspect_part.py
optimization_engine/extractors/field_data_extractor.py
optimization_engine/extractors/stiffness_calculator.py
optimization_engine/extractors/test_phase3_extractors.py

GNN Module (12 files):

optimization_engine/gnn/__init__.py
optimization_engine/gnn/zernike_gnn.py
optimization_engine/gnn/polar_graph.py
optimization_engine/gnn/train_zernike_gnn.py
optimization_engine/gnn/gnn_optimizer.py
optimization_engine/gnn/extract_displacement_field.py
optimization_engine/gnn/differentiable_zernike.py
optimization_engine/gnn/backfill_field_data.py
optimization_engine/gnn/test_polar_graph.py
optimization_engine/gnn/test_field_extraction.py
optimization_engine/gnn/test_new_extraction.py

Hooks Module (12 files):

optimization_engine/hooks/__init__.py
optimization_engine/hooks/examples.py
optimization_engine/hooks/test_introspection.py
optimization_engine/hooks/test_hooks.py
optimization_engine/hooks/nx_cad/__init__.py
optimization_engine/hooks/nx_cad/model_introspection.py
optimization_engine/hooks/nx_cad/part_manager.py
optimization_engine/hooks/nx_cad/expression_manager.py
optimization_engine/hooks/nx_cad/geometry_query.py
optimization_engine/hooks/nx_cad/feature_manager.py
optimization_engine/hooks/nx_cae/__init__.py
optimization_engine/hooks/nx_cae/solver_manager.py

Plugins Module (5 files):

optimization_engine/plugins/__init__.py
optimization_engine/plugins/hook_manager.py
optimization_engine/plugins/post_calculation/safety_factor_constraint.py
optimization_engine/plugins/pre_solve/*.py
optimization_engine/plugins/post_solve/*.py

Insights Module (12 files):

optimization_engine/insights/__init__.py
optimization_engine/insights/base.py
optimization_engine/insights/zernike_wfe.py
optimization_engine/insights/zernike_opd_comparison.py
optimization_engine/insights/zernike_dashboard.py
optimization_engine/insights/msf_zernike.py
optimization_engine/insights/stress_field.py
optimization_engine/insights/modal_analysis.py
optimization_engine/insights/thermal_field.py
optimization_engine/insights/design_space.py

Utils Module (6 files):

optimization_engine/utils/__init__.py
optimization_engine/utils/trial_manager.py
optimization_engine/utils/dashboard_db.py
optimization_engine/utils/nx_session_manager.py
optimization_engine/utils/nx_file_discovery.py
optimization_engine/utils/study_cleanup.py

Validators Module (5 files):

optimization_engine/validators/__init__.py
optimization_engine/validators/study_validator.py
optimization_engine/validators/results_validator.py
optimization_engine/validators/model_validator.py
optimization_engine/validators/config_validator.py

Templates (3 files):

optimization_engine/templates/__init__.py
optimization_engine/templates/run_optimization_template.py
optimization_engine/templates/run_nn_optimization_template.py

Model Discovery (2 files):

optimization_engine/model_discovery/__init__.py
optimization_engine/model_discovery/discover_parts.py

B. STUDIES DIRECTORY (80+ files)

M1_Mirror Studies (35+ files):

studies/M1_Mirror/m1_mirror_cost_reduction/run_optimization.py
studies/M1_Mirror/m1_mirror_zernike_optimization/run_optimization.py
studies/M1_Mirror/m1_mirror_cost_reduction_flat_back_V1/run_optimization.py
studies/M1_Mirror/m1_mirror_cost_reduction_flat_back_V3/run_optimization.py
studies/M1_Mirror/m1_mirror_cost_reduction_flat_back_V4/run_turbo_optimization.py
studies/M1_Mirror/m1_mirror_cost_reduction_flat_back_V5/run_turbo_optimization.py
studies/M1_Mirror/m1_mirror_cost_reduction_flat_back_V6/run_optimization.py
studies/M1_Mirror/m1_mirror_turbo_V1/run_optimization.py
studies/M1_Mirror/m1_mirror_turbo_V2/run_optimization.py
studies/M1_Mirror/m1_mirror_cost_reduction_V2/run_optimization.py
studies/M1_Mirror/m1_mirror_cost_reduction_V3/run_optimization.py
studies/M1_Mirror/m1_mirror_cost_reduction_V4/run_optimization.py
studies/M1_Mirror/m1_mirror_cost_reduction_V5/run_optimization.py
studies/M1_Mirror/m1_mirror_cost_reduction_V5/_analyze_baseline.py
studies/M1_Mirror/m1_mirror_cost_reduction_V6/run_optimization.py
studies/M1_Mirror/m1_mirror_cost_reduction_V6/_analyze_baseline.py
studies/M1_Mirror/m1_mirror_cost_reduction_V7/run_optimization.py
studies/M1_Mirror/m1_mirror_cost_reduction_V8/run_optimization.py
studies/M1_Mirror/m1_mirror_cost_reduction_V9/run_optimization.py
studies/M1_Mirror/m1_mirror_cost_reduction_V10/run_optimization.py
studies/M1_Mirror/m1_mirror_cost_reduction_V11/run_optimization.py
studies/M1_Mirror/m1_mirror_cost_reduction_V12/run_optimization.py
studies/M1_Mirror/m1_mirror_adaptive_V11/run_optimization.py
studies/M1_Mirror/m1_mirror_adaptive_V12/run_optimization.py
studies/M1_Mirror/m1_mirror_adaptive_V12/run_gnn_turbo.py
studies/M1_Mirror/m1_mirror_adaptive_V12/validate_gnn_best.py
studies/M1_Mirror/m1_mirror_adaptive_V12/compute_full_calibration.py
studies/M1_Mirror/m1_mirror_adaptive_V13/run_optimization.py
studies/M1_Mirror/m1_mirror_adaptive_V14/run_optimization.py
studies/M1_Mirror/m1_mirror_adaptive_V14/run_lbfgs_polish.py
studies/M1_Mirror/m1_mirror_adaptive_V15/run_optimization.py
studies/M1_Mirror/m1_mirror_surrogate_turbo/scripts/01_reextract_studies.py
studies/M1_Mirror/m1_mirror_surrogate_turbo/scripts/02_aggregate_gnn_data.py
studies/M1_Mirror/m1_mirror_surrogate_turbo/scripts/03_train_gnn.py

Simple_Bracket Studies (12+ files):

studies/Simple_Bracket/bracket_stiffness_optimization/run_optimization.py
studies/Simple_Bracket/bracket_stiffness_optimization/bracket_stiffness_extractor.py
studies/Simple_Bracket/bracket_stiffness_optimization_V2/run_optimization.py
studies/Simple_Bracket/bracket_stiffness_optimization_V2/bracket_stiffness_extractor.py
studies/Simple_Bracket/bracket_stiffness_optimization_V3/run_optimization.py
studies/Simple_Bracket/bracket_stiffness_optimization_V3/bracket_stiffness_extractor.py
studies/Simple_Bracket/bracket_stiffness_optimization_atomizerfield/run_optimization.py
studies/Simple_Bracket/bracket_pareto_3obj/run_optimization.py
studies/Simple_Bracket/bracket_pareto_3obj/run_nn_optimization.py

Other Studies (15+ files):

studies/Simple_Beam/simple_beam_optimization/run_optimization.py
studies/UAV_Arm/uav_arm_optimization/run_optimization.py
studies/UAV_Arm/uav_arm_optimization_V2/run_optimization.py
studies/UAV_Arm/uav_arm_atomizerfield_test/run_optimization.py
studies/Drone_Gimbal/drone_gimbal_arm_optimization/run_optimization.py
studies/bracket_displacement_maximizing/run_optimization.py
studies/bracket_displacement_maximizing/run_optimization_old.py
studies/bracket_displacement_maximizing/run_substudy.py
studies/bracket_displacement_maximizing/test_fix.py
studies/_Other/training_data_export_test/run_optimization.py

C. TESTS DIRECTORY (45+ files)

tests/audit_v10_wfe.py
tests/audit_v10_method_diff.py
tests/audit_v10_fix.py
tests/analyze_v11.py
tests/debug_insights.py
tests/debug_figure_coords.py
tests/compare_v8_zernike_methods.py
tests/test_zernike_opd_with_prescription.py
tests/test_zernike_opd_comparison.py
tests/test_zernike_methods_comparison.py
tests/test_zernike_insight.py
tests/test_zernike_import.py
tests/test_training_data_exporter.py
tests/test_timestamp_verification.py
tests/test_task_1_2_integration.py
tests/test_step_classifier.py
tests/test_research_agent.py
tests/test_phase_3_2_llm_mode.py
tests/test_phase_3_1_integration.py
tests/test_phase_2_5_intelligent_gap_detection.py
tests/test_plugin_system.py
tests/test_optimization_setup_wizard.py
tests/test_modal_deformation_request.py
tests/test_llm_runner_init.py
tests/test_llm_complex_request.py
tests/test_lifecycle_hook_integration.py
tests/test_knowledge_base_search.py
tests/test_journal_optimization.py
tests/test_interactive_session.py
tests/test_hooks_with_bracket.py
tests/test_complex_multiobj_request.py
tests/test_complete_research_workflow.py
tests/test_code_generation.py
tests/test_cbar_genetic_algorithm.py
tests/test_cbush_optimization.py
tests/test_bracket_llm_runner.py
tests/test_bracket_full_optimization.py
tests/test_beam_workflow.py
tests/test_api_verification.py
tests/interactive_optimization_setup.py
tests/demo_research_agent.py
tests/run_beam_benchmarking.py
tests/setup_beam_optimization.py
tests/run_5trial_test.py
tests/run_benchmarking_simple.py
tests/run_beam_benchmarking_clean.py

D. ROOT & TOOLS (10+ files)

atomizer.py
run_cleanup.py
run_training_fea.py
tools/zernike_html_generator.py
tools/extract_mirror_optical_specs.py
examples/interactive_research_session.py

E. DASHBOARD BACKEND (3 files)

atomizer-dashboard/backend/api/main.py
atomizer-dashboard/backend/api/routes/optimization.py
atomizer-dashboard/backend/api/routes/insights.py

F. ARCHIVE (50+ files - OPTIONAL)

archive/deprecated/result_extractors/extractors.py
archive/deprecated/result_extractors/generated/extract_expression.py
archive/scripts/create_intelligent_study.py
archive/scripts/create_circular_plate_study_v2.py
archive/scripts/create_circular_plate_study.py
archive/scripts/run_nn_optimization.py
archive/scripts/run_calibration_loop.py
archive/scripts/run_validated_nn_optimization.py
archive/scripts/validate_surrogate_real_data.py
archive/scripts/visualize_nn_optimization.py
archive/test_scripts/*.py (10+ files)

1.3 Documentation Files to UPDATE (100+ files)

A. ROOT DOCUMENTATION

CLAUDE.md (lines 141, 152, 166, 200, 221, 235, 253, 283, 592-603)
README.md (lines 98, 193, 213, 272)
CHANGELOG.md (lines 31, 40, 82)
DEVELOPMENT.md (lines 40, 64, 77, 90, 215-277, 398, 532, 535)
PROTOCOL.md (lines 481, 704-705, 760, 812, 924-939, 1118, 1262, 1380, 1406, 1501-1596, 1851)

B. .CLAUDE DIRECTORY

.claude/ATOMIZER_CONTEXT.md (lines 37, 40, 71, 235, 238, 248, 273, 282, 288, 421-472)
.claude/CHEATSHEET.md (lines 7-10, 33-35, 144, 156, 353-472)
.claude/skills/00_BOOTSTRAP.md (line 236)
.claude/skills/01_CHEATSHEET.md (lines 7-10, 33-35, 144, 156, 353-472)
.claude/skills/analyze-workflow.md (line 37)
.claude/skills/analyze-model.md (lines 52, 99, 183)
.claude/skills/atomizer.md (lines 15, 27, 62, 193, 286, 317, 323)
.claude/skills/DEV_DOCUMENTATION.md (lines 115-117, 141)
.claude/skills/core/study-creation-core.md (lines 7-9, 43, 157, 160, 259, 279, 454-459, 643, 706)
.claude/skills/modules/extractors-catalog.md (lines 24-31, 42, 63, 86, 110-271)
.claude/skills/modules/insights-catalog.md (lines 53-237)
.claude/skills/modules/neural-acceleration.md (lines 48, 318)
.claude/skills/modules/nx-docs-lookup.md (line 190)
.claude/skills/modules/zernike-optimization.md (lines 35, 73, 111)
.claude/skills/modules/atomizer_fast_solver_technologies.md (lines 739, 744)
.claude/skills/modules/learning-atomizer-core.md
.claude/skills/archive/create-study-wizard.md (lines 13, 74, 177, 360, 402)

C. DOCS DIRECTORY - PROTOCOLS

docs/00_INDEX.md (line 284)
docs/01_PROTOCOLS.md (lines 277, 352-355, 528, 535-600, 1355, 1434, 1460)
docs/02_ARCHITECTURE.md (lines 68-69, 104, 123, 193, 254, 279, 299, 376-495)
docs/03_GETTING_STARTED.md (line 286)
docs/protocols/operations/OP_01_CREATE_STUDY.md (lines 162, 301-303)
docs/protocols/operations/OP_02_RUN_OPTIMIZATION.md
docs/protocols/operations/OP_03_MONITOR_PROGRESS.md
docs/protocols/operations/OP_04_ANALYZE_RESULTS.md
docs/protocols/operations/OP_05_EXPORT_TRAINING_DATA.md
docs/protocols/operations/OP_06_TROUBLESHOOT.md
docs/protocols/system/SYS_10_IMSO.md (lines 224, 302-305)
docs/protocols/system/SYS_11_MULTI_OBJECTIVE.md (lines 322-330)
docs/protocols/system/SYS_12_EXTRACTOR_LIBRARY.md (lines 71-851 - MANY references)
docs/protocols/system/SYS_13_DASHBOARD_TRACKING.md (lines 60, 85, 315)
docs/protocols/system/SYS_14_NEURAL_ACCELERATION.md (lines 231-1080)
docs/protocols/system/SYS_15_METHOD_SELECTOR.md (lines 42-422)
docs/protocols/system/SYS_16_STUDY_INSIGHTS.md (lines 62-498)
docs/protocols/extensions/EXT_01_CREATE_EXTRACTOR.md (lines 35-287)
docs/protocols/extensions/EXT_02_CREATE_HOOK.md (lines 45-357)
docs/protocols/extensions/EXT_03_CREATE_INSIGHT.md
docs/protocols/extensions/EXT_04_CREATE_SKILL.md (lines 263, 274)
docs/protocols/extensions/templates/hook_template.py (lines 128, 164)

D. DOCS SUBDIRECTORIES

docs/04_USER_GUIDES/*.md (estimated 20+ files)
docs/05_API_REFERENCE/*.md (estimated 10+ files)
docs/06_PHYSICS/*.md (estimated 15+ files)
docs/07_DEVELOPMENT/*.md (estimated 20+ files)
docs/08_ARCHIVE/*.md (estimated 30+ files)

1.4 Configuration Files to UPDATE

optimization_engine/feature_registry.json (50+ path entries on lines 21-683)
optimization_engine/schemas/README.md (lines 19, 26)
studies/Simple_Bracket/bracket_stiffness_optimization_atomizerfield/1_setup/optimization_config.json
studies/Simple_Bracket/bracket_pareto_3obj/1_setup/optimization_config.json
pyproject.toml (lines 53, 76)

Part 2: Import Mapping Reference

2.1 Complete Old → New Import Mapping

# =============================================================================
# CORE MODULE
# =============================================================================
"from optimization_engine.runner"                     → "from optimization_engine.core.runner"
"from optimization_engine.base_runner"                → "from optimization_engine.core.base_runner"
"from optimization_engine.runner_with_neural"         → "from optimization_engine.core.runner_with_neural"
"from optimization_engine.intelligent_optimizer"      → "from optimization_engine.core.intelligent_optimizer"
"from optimization_engine.method_selector"            → "from optimization_engine.core.method_selector"
"from optimization_engine.strategy_selector"          → "from optimization_engine.core.strategy_selector"
"from optimization_engine.strategy_portfolio"         → "from optimization_engine.core.strategy_portfolio"
"from optimization_engine.gradient_optimizer"         → "from optimization_engine.core.gradient_optimizer"

"import optimization_engine.runner"                   → "import optimization_engine.core.runner"
"import optimization_engine.intelligent_optimizer"    → "import optimization_engine.core.intelligent_optimizer"

# =============================================================================
# SURROGATES MODULE
# =============================================================================
"from optimization_engine.neural_surrogate"           → "from optimization_engine.processors.surrogates.neural_surrogate"
"from optimization_engine.generic_surrogate"          → "from optimization_engine.processors.surrogates.generic_surrogate"
"from optimization_engine.adaptive_surrogate"         → "from optimization_engine.processors.surrogates.adaptive_surrogate"
"from optimization_engine.simple_mlp_surrogate"       → "from optimization_engine.processors.surrogates.simple_mlp_surrogate"
"from optimization_engine.active_learning_surrogate"  → "from optimization_engine.processors.surrogates.active_learning_surrogate"
"from optimization_engine.surrogate_tuner"            → "from optimization_engine.processors.surrogates.surrogate_tuner"
"from optimization_engine.auto_trainer"               → "from optimization_engine.processors.surrogates.auto_trainer"
"from optimization_engine.training_data_exporter"     → "from optimization_engine.processors.surrogates.training_data_exporter"

# =============================================================================
# NX MODULE
# =============================================================================
"from optimization_engine.nx_solver"                  → "from optimization_engine.nx.solver"
"from optimization_engine.nx_updater"                 → "from optimization_engine.nx.updater"
"from optimization_engine.nx_session_manager"         → "from optimization_engine.nx.session_manager"
"from optimization_engine.solve_simulation"           → "from optimization_engine.nx.solve_simulation"
"from optimization_engine.solve_simulation_simple"    → "from optimization_engine.nx.solve_simulation_simple"
"from optimization_engine.model_cleanup"              → "from optimization_engine.nx.model_cleanup"
"from optimization_engine.export_expressions"         → "from optimization_engine.nx.export_expressions"
"from optimization_engine.import_expressions"         → "from optimization_engine.nx.import_expressions"
"from optimization_engine.mesh_converter"             → "from optimization_engine.nx.mesh_converter"

"import optimization_engine.nx_solver"                → "import optimization_engine.nx.solver"

# =============================================================================
# STUDY MODULE
# =============================================================================
"from optimization_engine.study_creator"              → "from optimization_engine.study.creator"
"from optimization_engine.study_wizard"               → "from optimization_engine.study.wizard"
"from optimization_engine.study_state"                → "from optimization_engine.study.state"
"from optimization_engine.study_reset"                → "from optimization_engine.study.reset"
"from optimization_engine.study_continuation"         → "from optimization_engine.study.continuation"
"from optimization_engine.benchmarking_substudy"      → "from optimization_engine.study.benchmarking"
"from optimization_engine.generate_history_from_trials" → "from optimization_engine.study.history_generator"

# =============================================================================
# REPORTING MODULE
# =============================================================================
"from optimization_engine.generate_report"            → "from optimization_engine.reporting.report_generator"
"from optimization_engine.generate_report_markdown"   → "from optimization_engine.reporting.markdown_report"
"from optimization_engine.comprehensive_results_analyzer" → "from optimization_engine.reporting.results_analyzer"
"from optimization_engine.visualizer"                 → "from optimization_engine.reporting.visualizer"
"from optimization_engine.landscape_analyzer"         → "from optimization_engine.reporting.landscape_analyzer"

# =============================================================================
# CONFIG MODULE
# =============================================================================
"from optimization_engine.config_manager"             → "from optimization_engine.config.manager"
"from optimization_engine.optimization_config_builder" → "from optimization_engine.config.builder"
"from optimization_engine.optimization_setup_wizard"  → "from optimization_engine.config.setup_wizard"
"from optimization_engine.capability_matcher"         → "from optimization_engine.config.capability_matcher"
"from optimization_engine.template_loader"            → "from optimization_engine.config.template_loader"

# =============================================================================
# UTILS MODULE (moving to existing utils/)
# =============================================================================
"from optimization_engine.logger"                     → "from optimization_engine.utils.logger"
"from optimization_engine.auto_doc"                   → "from optimization_engine.utils.auto_doc"
"from optimization_engine.realtime_tracking"          → "from optimization_engine.utils.realtime_tracking"
"from optimization_engine.codebase_analyzer"          → "from optimization_engine.utils.codebase_analyzer"
"from optimization_engine.pruning_logger"             → "from optimization_engine.utils.pruning_logger"

# =============================================================================
# FUTURE MODULE (research/experimental)
# =============================================================================
"from optimization_engine.research_agent"             → "from optimization_engine.future.research_agent"
"from optimization_engine.pynastran_research_agent"   → "from optimization_engine.future.pynastran_research_agent"
"from optimization_engine.targeted_research_planner"  → "from optimization_engine.future.targeted_research_planner"
"from optimization_engine.workflow_decomposer"        → "from optimization_engine.future.workflow_decomposer"
"from optimization_engine.step_classifier"            → "from optimization_engine.future.step_classifier"
"from optimization_engine.llm_optimization_runner"    → "from optimization_engine.future.llm_optimization_runner"
"from optimization_engine.llm_workflow_analyzer"      → "from optimization_engine.future.llm_workflow_analyzer"
"from optimization_engine.extractor_orchestrator"     → "from optimization_engine.future.extractor_orchestrator"
"from optimization_engine.inline_code_generator"      → "from optimization_engine.future.inline_code_generator"
"from optimization_engine.hook_generator"             → "from optimization_engine.future.hook_generator"

# =============================================================================
# EXTRACTORS MODULE (moving to existing extractors/)
# =============================================================================
"from optimization_engine.op2_extractor"              → "from optimization_engine.extractors.op2_extractor"
"from optimization_engine.extractor_library"          → "from optimization_engine.extractors.extractor_library"

# =============================================================================
# VALIDATORS MODULE (moving to existing validators/)
# =============================================================================
"from optimization_engine.simulation_validator"       → "from optimization_engine.validators.simulation_validator"

# =============================================================================
# PROCESSORS MODULE (new)
# =============================================================================
"from optimization_engine.adaptive_characterization"  → "from optimization_engine.processors.adaptive_characterization"

# =============================================================================
# UNCHANGED (these modules stay in place)
# =============================================================================
# extractors/ - stays as optimization_engine.extractors.*
# gnn/ - stays as optimization_engine.gnn.*
# hooks/ - stays as optimization_engine.hooks.*
# insights/ - stays as optimization_engine.insights.*
# plugins/ - stays as optimization_engine.plugins.*
# validators/ - stays as optimization_engine.validators.*
# utils/ - stays as optimization_engine.utils.*
# templates/ - stays as optimization_engine.templates.*
# schemas/ - stays as optimization_engine.schemas.*
# model_discovery/ - stays as optimization_engine.model_discovery.*

2.2 Documentation Path Mapping

# In markdown files, update these path references:

optimization_engine/runner.py           → optimization_engine/core/runner.py
optimization_engine/nx_solver.py        → optimization_engine/nx/solver.py
optimization_engine/nx_updater.py       → optimization_engine/nx/updater.py
optimization_engine/study_creator.py    → optimization_engine/study/creator.py
optimization_engine/study_wizard.py     → optimization_engine/study/wizard.py
optimization_engine/neural_surrogate.py → optimization_engine/processors/surrogates/neural_surrogate.py
optimization_engine/config_manager.py   → optimization_engine/config/manager.py
optimization_engine/logger.py           → optimization_engine/utils/logger.py
# ... etc for all moved files

Part 3: Execution Plan

Phase 0: Preparation (30 minutes)

# 1. Create backup branch
git checkout -b backup/pre-reorganization
git push origin backup/pre-reorganization
git checkout main
git checkout -b refactor/optimization-engine-v2

# 2. Create full backup
cp -r optimization_engine optimization_engine_BACKUP_$(date +%Y%m%d)

# 3. Run baseline tests and save output
python -m pytest tests/ -v --tb=short 2>&1 | tee baseline_tests_$(date +%Y%m%d).log

# 4. Count current test status
echo "Baseline test count:"
grep -E "passed|failed|error" baseline_tests_*.log | tail -5

# 5. Verify NX is not running (to prevent file locks)
tasklist | grep -i ugraf || echo "NX not running - safe to proceed"

Phase 1: Create Directory Structure (15 minutes)

# Create all new directories
mkdir -p optimization_engine/core
mkdir -p optimization_engine/processors/surrogates
mkdir -p optimization_engine/processors/dynamic_response
mkdir -p optimization_engine/nx
mkdir -p optimization_engine/study
mkdir -p optimization_engine/reporting
mkdir -p optimization_engine/config

# Create __init__.py files for new directories
touch optimization_engine/core/__init__.py
touch optimization_engine/processors/__init__.py
touch optimization_engine/processors/surrogates/__init__.py
touch optimization_engine/processors/dynamic_response/__init__.py
touch optimization_engine/nx/__init__.py
touch optimization_engine/study/__init__.py
touch optimization_engine/reporting/__init__.py
touch optimization_engine/config/__init__.py

Phase 2: Move Files (45 minutes)

Execute in this specific order to minimize issues:

# =============================================================================
# STEP 2.1: UTILITIES FIRST (no dependencies on other moved files)
# =============================================================================
mv optimization_engine/logger.py optimization_engine/utils/
mv optimization_engine/auto_doc.py optimization_engine/utils/
mv optimization_engine/realtime_tracking.py optimization_engine/utils/
mv optimization_engine/codebase_analyzer.py optimization_engine/utils/
mv optimization_engine/pruning_logger.py optimization_engine/utils/

# =============================================================================
# STEP 2.2: CONFIG (depends only on logger)
# =============================================================================
mv optimization_engine/config_manager.py optimization_engine/config/manager.py
mv optimization_engine/optimization_config_builder.py optimization_engine/config/builder.py
mv optimization_engine/optimization_setup_wizard.py optimization_engine/config/setup_wizard.py
mv optimization_engine/capability_matcher.py optimization_engine/config/capability_matcher.py
mv optimization_engine/template_loader.py optimization_engine/config/template_loader.py

# =============================================================================
# STEP 2.3: NX INTEGRATION (core infrastructure)
# =============================================================================
mv optimization_engine/nx_solver.py optimization_engine/nx/solver.py
mv optimization_engine/nx_updater.py optimization_engine/nx/updater.py
mv optimization_engine/nx_session_manager.py optimization_engine/nx/session_manager.py
mv optimization_engine/solve_simulation.py optimization_engine/nx/
mv optimization_engine/solve_simulation_simple.py optimization_engine/nx/
mv optimization_engine/model_cleanup.py optimization_engine/nx/
mv optimization_engine/export_expressions.py optimization_engine/nx/
mv optimization_engine/import_expressions.py optimization_engine/nx/
mv optimization_engine/mesh_converter.py optimization_engine/nx/

# =============================================================================
# STEP 2.4: SURROGATES (ML components)
# =============================================================================
mv optimization_engine/neural_surrogate.py optimization_engine/processors/surrogates/
mv optimization_engine/generic_surrogate.py optimization_engine/processors/surrogates/
mv optimization_engine/adaptive_surrogate.py optimization_engine/processors/surrogates/
mv optimization_engine/simple_mlp_surrogate.py optimization_engine/processors/surrogates/
mv optimization_engine/active_learning_surrogate.py optimization_engine/processors/surrogates/
mv optimization_engine/surrogate_tuner.py optimization_engine/processors/surrogates/
mv optimization_engine/auto_trainer.py optimization_engine/processors/surrogates/
mv optimization_engine/training_data_exporter.py optimization_engine/processors/surrogates/

# =============================================================================
# STEP 2.5: STUDY MANAGEMENT
# =============================================================================
mv optimization_engine/study_creator.py optimization_engine/study/creator.py
mv optimization_engine/study_wizard.py optimization_engine/study/wizard.py
mv optimization_engine/study_state.py optimization_engine/study/state.py
mv optimization_engine/study_reset.py optimization_engine/study/reset.py
mv optimization_engine/study_continuation.py optimization_engine/study/continuation.py
mv optimization_engine/benchmarking_substudy.py optimization_engine/study/benchmarking.py
mv optimization_engine/generate_history_from_trials.py optimization_engine/study/history_generator.py

# =============================================================================
# STEP 2.6: REPORTING
# =============================================================================
mv optimization_engine/generate_report.py optimization_engine/reporting/report_generator.py
mv optimization_engine/generate_report_markdown.py optimization_engine/reporting/markdown_report.py
mv optimization_engine/comprehensive_results_analyzer.py optimization_engine/reporting/results_analyzer.py
mv optimization_engine/visualizer.py optimization_engine/reporting/
mv optimization_engine/landscape_analyzer.py optimization_engine/reporting/

# =============================================================================
# STEP 2.7: CORE (depends on many things, do after others)
# =============================================================================
mv optimization_engine/runner.py optimization_engine/core/
mv optimization_engine/base_runner.py optimization_engine/core/
mv optimization_engine/runner_with_neural.py optimization_engine/core/
mv optimization_engine/intelligent_optimizer.py optimization_engine/core/
mv optimization_engine/method_selector.py optimization_engine/core/
mv optimization_engine/strategy_selector.py optimization_engine/core/
mv optimization_engine/strategy_portfolio.py optimization_engine/core/
mv optimization_engine/gradient_optimizer.py optimization_engine/core/

# =============================================================================
# STEP 2.8: FUTURE/RESEARCH (experimental)
# =============================================================================
mv optimization_engine/research_agent.py optimization_engine/future/
mv optimization_engine/pynastran_research_agent.py optimization_engine/future/
mv optimization_engine/targeted_research_planner.py optimization_engine/future/
mv optimization_engine/workflow_decomposer.py optimization_engine/future/
mv optimization_engine/step_classifier.py optimization_engine/future/
# Check if these exist before moving:
test -f optimization_engine/llm_optimization_runner.py && mv optimization_engine/llm_optimization_runner.py optimization_engine/future/
test -f optimization_engine/llm_workflow_analyzer.py && mv optimization_engine/llm_workflow_analyzer.py optimization_engine/future/
test -f optimization_engine/extractor_orchestrator.py && mv optimization_engine/extractor_orchestrator.py optimization_engine/future/
test -f optimization_engine/inline_code_generator.py && mv optimization_engine/inline_code_generator.py optimization_engine/future/
test -f optimization_engine/hook_generator.py && mv optimization_engine/hook_generator.py optimization_engine/future/

# =============================================================================
# STEP 2.9: MOVE TO EXISTING DIRECTORIES
# =============================================================================
mv optimization_engine/op2_extractor.py optimization_engine/extractors/
mv optimization_engine/extractor_library.py optimization_engine/extractors/
mv optimization_engine/simulation_validator.py optimization_engine/validators/
mv optimization_engine/adaptive_characterization.py optimization_engine/processors/

Phase 3: Update Imports with Automation Script (2-3 hours)

Create and run a Python migration script:

#!/usr/bin/env python3
"""
optimization_engine Migration Script
=====================================
Automatically updates all imports across the codebase.

Usage:
    python migrate_imports.py --dry-run    # Preview changes
    python migrate_imports.py --execute    # Apply changes
"""

import os
import re
import sys
from pathlib import Path
from typing import Dict, List, Tuple

# Import mappings (old -> new)
IMPORT_MAPPINGS = {
    # Core
    r'from optimization_engine\.runner\b': 'from optimization_engine.core.runner',
    r'from optimization_engine\.base_runner\b': 'from optimization_engine.core.base_runner',
    r'from optimization_engine\.runner_with_neural\b': 'from optimization_engine.core.runner_with_neural',
    r'from optimization_engine\.intelligent_optimizer\b': 'from optimization_engine.core.intelligent_optimizer',
    r'from optimization_engine\.method_selector\b': 'from optimization_engine.core.method_selector',
    r'from optimization_engine\.strategy_selector\b': 'from optimization_engine.core.strategy_selector',
    r'from optimization_engine\.strategy_portfolio\b': 'from optimization_engine.core.strategy_portfolio',
    r'from optimization_engine\.gradient_optimizer\b': 'from optimization_engine.core.gradient_optimizer',
    r'import optimization_engine\.runner\b': 'import optimization_engine.core.runner',
    r'import optimization_engine\.intelligent_optimizer\b': 'import optimization_engine.core.intelligent_optimizer',

    # Surrogates
    r'from optimization_engine\.neural_surrogate\b': 'from optimization_engine.processors.surrogates.neural_surrogate',
    r'from optimization_engine\.generic_surrogate\b': 'from optimization_engine.processors.surrogates.generic_surrogate',
    r'from optimization_engine\.adaptive_surrogate\b': 'from optimization_engine.processors.surrogates.adaptive_surrogate',
    r'from optimization_engine\.simple_mlp_surrogate\b': 'from optimization_engine.processors.surrogates.simple_mlp_surrogate',
    r'from optimization_engine\.active_learning_surrogate\b': 'from optimization_engine.processors.surrogates.active_learning_surrogate',
    r'from optimization_engine\.surrogate_tuner\b': 'from optimization_engine.processors.surrogates.surrogate_tuner',
    r'from optimization_engine\.auto_trainer\b': 'from optimization_engine.processors.surrogates.auto_trainer',
    r'from optimization_engine\.training_data_exporter\b': 'from optimization_engine.processors.surrogates.training_data_exporter',

    # NX
    r'from optimization_engine\.nx_solver\b': 'from optimization_engine.nx.solver',
    r'from optimization_engine\.nx_updater\b': 'from optimization_engine.nx.updater',
    r'from optimization_engine\.nx_session_manager\b': 'from optimization_engine.nx.session_manager',
    r'from optimization_engine\.solve_simulation\b': 'from optimization_engine.nx.solve_simulation',
    r'from optimization_engine\.solve_simulation_simple\b': 'from optimization_engine.nx.solve_simulation_simple',
    r'from optimization_engine\.model_cleanup\b': 'from optimization_engine.nx.model_cleanup',
    r'from optimization_engine\.export_expressions\b': 'from optimization_engine.nx.export_expressions',
    r'from optimization_engine\.import_expressions\b': 'from optimization_engine.nx.import_expressions',
    r'from optimization_engine\.mesh_converter\b': 'from optimization_engine.nx.mesh_converter',
    r'import optimization_engine\.nx_solver\b': 'import optimization_engine.nx.solver',

    # Study
    r'from optimization_engine\.study_creator\b': 'from optimization_engine.study.creator',
    r'from optimization_engine\.study_wizard\b': 'from optimization_engine.study.wizard',
    r'from optimization_engine\.study_state\b': 'from optimization_engine.study.state',
    r'from optimization_engine\.study_reset\b': 'from optimization_engine.study.reset',
    r'from optimization_engine\.study_continuation\b': 'from optimization_engine.study.continuation',
    r'from optimization_engine\.benchmarking_substudy\b': 'from optimization_engine.study.benchmarking',
    r'from optimization_engine\.generate_history_from_trials\b': 'from optimization_engine.study.history_generator',

    # Reporting
    r'from optimization_engine\.generate_report\b': 'from optimization_engine.reporting.report_generator',
    r'from optimization_engine\.generate_report_markdown\b': 'from optimization_engine.reporting.markdown_report',
    r'from optimization_engine\.comprehensive_results_analyzer\b': 'from optimization_engine.reporting.results_analyzer',
    r'from optimization_engine\.visualizer\b': 'from optimization_engine.reporting.visualizer',
    r'from optimization_engine\.landscape_analyzer\b': 'from optimization_engine.reporting.landscape_analyzer',

    # Config
    r'from optimization_engine\.config_manager\b': 'from optimization_engine.config.manager',
    r'from optimization_engine\.optimization_config_builder\b': 'from optimization_engine.config.builder',
    r'from optimization_engine\.optimization_setup_wizard\b': 'from optimization_engine.config.setup_wizard',
    r'from optimization_engine\.capability_matcher\b': 'from optimization_engine.config.capability_matcher',
    r'from optimization_engine\.template_loader\b': 'from optimization_engine.config.template_loader',

    # Utils
    r'from optimization_engine\.logger\b': 'from optimization_engine.utils.logger',
    r'from optimization_engine\.auto_doc\b': 'from optimization_engine.utils.auto_doc',
    r'from optimization_engine\.realtime_tracking\b': 'from optimization_engine.utils.realtime_tracking',
    r'from optimization_engine\.codebase_analyzer\b': 'from optimization_engine.utils.codebase_analyzer',
    r'from optimization_engine\.pruning_logger\b': 'from optimization_engine.utils.pruning_logger',

    # Future
    r'from optimization_engine\.research_agent\b': 'from optimization_engine.future.research_agent',
    r'from optimization_engine\.pynastran_research_agent\b': 'from optimization_engine.future.pynastran_research_agent',
    r'from optimization_engine\.targeted_research_planner\b': 'from optimization_engine.future.targeted_research_planner',
    r'from optimization_engine\.workflow_decomposer\b': 'from optimization_engine.future.workflow_decomposer',
    r'from optimization_engine\.step_classifier\b': 'from optimization_engine.future.step_classifier',
    r'from optimization_engine\.llm_optimization_runner\b': 'from optimization_engine.future.llm_optimization_runner',
    r'from optimization_engine\.llm_workflow_analyzer\b': 'from optimization_engine.future.llm_workflow_analyzer',

    # Extractors/Validators additions
    r'from optimization_engine\.op2_extractor\b': 'from optimization_engine.extractors.op2_extractor',
    r'from optimization_engine\.extractor_library\b': 'from optimization_engine.extractors.extractor_library',
    r'from optimization_engine\.simulation_validator\b': 'from optimization_engine.validators.simulation_validator',

    # Processors
    r'from optimization_engine\.adaptive_characterization\b': 'from optimization_engine.processors.adaptive_characterization',
}

# Path mappings for documentation
PATH_MAPPINGS = {
    'optimization_engine/runner.py': 'optimization_engine/core/runner.py',
    'optimization_engine/base_runner.py': 'optimization_engine/core/base_runner.py',
    'optimization_engine/intelligent_optimizer.py': 'optimization_engine/core/intelligent_optimizer.py',
    'optimization_engine/nx_solver.py': 'optimization_engine/nx/solver.py',
    'optimization_engine/nx_updater.py': 'optimization_engine/nx/updater.py',
    'optimization_engine/study_creator.py': 'optimization_engine/study/creator.py',
    'optimization_engine/study_wizard.py': 'optimization_engine/study/wizard.py',
    'optimization_engine/neural_surrogate.py': 'optimization_engine/processors/surrogates/neural_surrogate.py',
    'optimization_engine/config_manager.py': 'optimization_engine/config/manager.py',
    'optimization_engine/logger.py': 'optimization_engine/utils/logger.py',
    # Add all other path mappings...
}

def find_files(root: Path, extensions: List[str]) -> List[Path]:
    """Find all files with given extensions."""
    files = []
    for ext in extensions:
        files.extend(root.rglob(f'*{ext}'))
    return files

def update_file(filepath: Path, mappings: Dict[str, str], dry_run: bool = True) -> Tuple[int, List[str]]:
    """Update imports in a single file."""
    try:
        content = filepath.read_text(encoding='utf-8', errors='ignore')
    except Exception as e:
        print(f"  ERROR reading {filepath}: {e}")
        return 0, []

    changes = []
    new_content = content

    for pattern, replacement in mappings.items():
        matches = re.findall(pattern, content)
        if matches:
            new_content = re.sub(pattern, replacement, new_content)
            changes.append(f"  {pattern} → {replacement} ({len(matches)} occurrences)")

    if changes and not dry_run:
        filepath.write_text(new_content, encoding='utf-8')

    return len(changes), changes

def main():
    dry_run = '--dry-run' in sys.argv or '--execute' not in sys.argv

    if dry_run:
        print("=" * 60)
        print("DRY RUN MODE - No files will be modified")
        print("=" * 60)
    else:
        print("=" * 60)
        print("EXECUTE MODE - Files will be modified!")
        print("=" * 60)
        confirm = input("Are you sure? (yes/no): ")
        if confirm.lower() != 'yes':
            print("Aborted.")
            return

    root = Path('.')

    # Find all Python files
    py_files = find_files(root, ['.py'])
    py_files = [f for f in py_files if 'optimization_engine_BACKUP' not in str(f)]
    py_files = [f for f in py_files if '.venv' not in str(f)]
    py_files = [f for f in py_files if 'node_modules' not in str(f)]

    print(f"\nFound {len(py_files)} Python files to check")

    total_changes = 0
    files_changed = 0

    for filepath in sorted(py_files):
        count, changes = update_file(filepath, IMPORT_MAPPINGS, dry_run)
        if count > 0:
            files_changed += 1
            total_changes += count
            print(f"\n{filepath} ({count} changes):")
            for change in changes:
                print(change)

    # Find all Markdown files
    md_files = find_files(root, ['.md'])
    md_files = [f for f in md_files if 'optimization_engine_BACKUP' not in str(f)]
    md_files = [f for f in md_files if 'node_modules' not in str(f)]

    print(f"\n\nFound {len(md_files)} Markdown files to check")

    for filepath in sorted(md_files):
        count, changes = update_file(filepath, PATH_MAPPINGS, dry_run)
        if count > 0:
            files_changed += 1
            total_changes += count
            print(f"\n{filepath} ({count} changes):")
            for change in changes[:5]:  # Limit output
                print(change)
            if len(changes) > 5:
                print(f"  ... and {len(changes) - 5} more")

    print("\n" + "=" * 60)
    print(f"SUMMARY: {total_changes} changes in {files_changed} files")
    print("=" * 60)

    if dry_run:
        print("\nTo apply changes, run: python migrate_imports.py --execute")

if __name__ == '__main__':
    main()

Phase 4: Create init.py Files (1 hour)

Create proper __init__.py for each new module. See separate section below.

Phase 5: Update feature_registry.json (30 minutes)

Use a dedicated script to update all path references in the JSON file.

Phase 6: Update Documentation (2 hours)

Run sed/grep commands for bulk markdown updates, then manual review.

Phase 7: Testing & Validation (2 hours)

# 1. Test Python imports
python -c "from optimization_engine.core.runner import OptimizationRunner; print('core OK')"
python -c "from optimization_engine.nx.solver import NXSolver; print('nx OK')"
python -c "from optimization_engine.study.creator import StudyCreator; print('study OK')"
python -c "from optimization_engine.processors.surrogates.neural_surrogate import NeuralSurrogate; print('surrogates OK')"
python -c "from optimization_engine.config.manager import ConfigManager; print('config OK')"
python -c "from optimization_engine.utils.logger import get_logger; print('utils OK')"

# 2. Test backwards compatibility
python -c "from optimization_engine import nx_solver; print('compat OK')"

# 3. Run full test suite
python -m pytest tests/ -v --tb=short 2>&1 | tee post_migration_tests.log

# 4. Compare test results
diff baseline_tests_*.log post_migration_tests.log

# 5. Test a study
cd studies/bracket_displacement_maximizing
python run_optimization.py --trials 1 --dry-run

# 6. Test dashboard
cd atomizer-dashboard/backend
python -c "from api.routes.optimization import *; print('dashboard OK')"

Phase 8: Commit & Cleanup (30 minutes)

# Remove backup after successful testing
rm -rf optimization_engine_BACKUP_*

# Clear Python cache
find . -name "*.pyc" -delete
find . -name "__pycache__" -type d -exec rm -rf {} + 2>/dev/null || true

# Final commit
git add -A
git commit -m "refactor: Major reorganization of optimization_engine

BREAKING CHANGE: Module structure reorganized for better maintainability.

New structure:
- core/           - Optimization runners (runner, intelligent_optimizer, etc.)
- processors/     - Data processing (surrogates/, dynamic_response/)
- nx/             - NX/Nastran integration (solver, updater, etc.)
- study/          - Study management (creator, wizard, state, etc.)
- reporting/      - Reports and analysis
- config/         - Configuration management
- utils/          - Utilities (logger, etc.)
- future/         - Research/experimental code

Migration:
- 500+ files updated
- ~1,400 import changes
- Backwards compatibility aliases provided with deprecation warnings
- All tests passing

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>"

# Push
git push origin refactor/optimization-engine-v2

Part 4: init.py Files Content

4.1 optimization_engine/core/init.py

"""
Optimization Engine Core
========================

Main optimization runners and algorithm selection.

Modules:
- runner: Main OptimizationRunner class
- base_runner: BaseRunner abstract class
- intelligent_optimizer: IMSO adaptive optimizer
- method_selector: Algorithm selection logic
- strategy_selector: Strategy portfolio management
"""

from .runner import OptimizationRunner
from .base_runner import BaseRunner
from .runner_with_neural import NeuralOptimizationRunner
from .intelligent_optimizer import IntelligentOptimizer, IMSO
from .method_selector import MethodSelector, select_method
from .strategy_selector import StrategySelector
from .strategy_portfolio import StrategyPortfolio
from .gradient_optimizer import GradientOptimizer, LBFGSPolisher

__all__ = [
    'OptimizationRunner',
    'BaseRunner',
    'NeuralOptimizationRunner',
    'IntelligentOptimizer',
    'IMSO',
    'MethodSelector',
    'select_method',
    'StrategySelector',
    'StrategyPortfolio',
    'GradientOptimizer',
    'LBFGSPolisher',
]

4.2 optimization_engine/processors/init.py

"""
Optimization Processors
=======================

Data processing algorithms and ML models.

Submodules:
- surrogates/: Neural network surrogate models
- dynamic_response/: Dynamic response processing (random vib, sine sweep)
"""

from . import surrogates

__all__ = [
    'surrogates',
]

4.3 optimization_engine/processors/surrogates/init.py

"""
Surrogate Models
================

Neural network and ML surrogate models for FEA acceleration.

Classes:
- NeuralSurrogate: Main neural network surrogate
- GenericSurrogate: Flexible surrogate interface
- AdaptiveSurrogate: Self-improving surrogate
- SimpleMLP: Simple multi-layer perceptron
- ActiveLearningSurrogate: Active learning surrogate
- SurrogateTuner: Hyperparameter tuning
"""

from .neural_surrogate import NeuralSurrogate, create_surrogate
from .generic_surrogate import GenericSurrogate
from .adaptive_surrogate import AdaptiveSurrogate
from .simple_mlp_surrogate import SimpleMLP, SimpleMLPSurrogate
from .active_learning_surrogate import ActiveLearningSurrogate
from .surrogate_tuner import SurrogateTuner, tune_surrogate
from .auto_trainer import AutoTrainer
from .training_data_exporter import TrainingDataExporter, export_training_data

__all__ = [
    'NeuralSurrogate',
    'create_surrogate',
    'GenericSurrogate',
    'AdaptiveSurrogate',
    'SimpleMLP',
    'SimpleMLPSurrogate',
    'ActiveLearningSurrogate',
    'SurrogateTuner',
    'tune_surrogate',
    'AutoTrainer',
    'TrainingDataExporter',
    'export_training_data',
]

4.4 optimization_engine/nx/init.py

"""
NX Integration
==============

Siemens NX and Nastran integration modules.

Modules:
- solver: NXSolver for running simulations
- updater: NXParameterUpdater for design updates
- session_manager: NX session lifecycle management
- solve_simulation: Low-level simulation execution
"""

from .solver import NXSolver, run_nx_simulation
from .updater import NXParameterUpdater, update_parameters
from .session_manager import NXSessionManager, get_session
from .solve_simulation import solve_simulation, SolveSimulationError
from .model_cleanup import cleanup_model, ModelCleanup

__all__ = [
    'NXSolver',
    'run_nx_simulation',
    'NXParameterUpdater',
    'update_parameters',
    'NXSessionManager',
    'get_session',
    'solve_simulation',
    'SolveSimulationError',
    'cleanup_model',
    'ModelCleanup',
]

4.5 optimization_engine/study/init.py

"""
Study Management
================

Study creation, state management, and lifecycle.

Modules:
- creator: Study creation from templates
- wizard: Interactive study setup wizard
- state: Study state tracking
- reset: Study reset functionality
- continuation: Resume interrupted studies
"""

from .creator import StudyCreator, create_study
from .wizard import StudyWizard, run_wizard
from .state import StudyState, get_study_state
from .reset import reset_study, StudyReset
from .continuation import continue_study, StudyContinuation
from .benchmarking import BenchmarkingSubstudy
from .history_generator import generate_history

__all__ = [
    'StudyCreator',
    'create_study',
    'StudyWizard',
    'run_wizard',
    'StudyState',
    'get_study_state',
    'reset_study',
    'StudyReset',
    'continue_study',
    'StudyContinuation',
    'BenchmarkingSubstudy',
    'generate_history',
]

4.6 optimization_engine/reporting/init.py

"""
Reporting & Analysis
====================

Report generation and results analysis.

Modules:
- report_generator: HTML/PDF report generation
- markdown_report: Markdown report format
- results_analyzer: Comprehensive results analysis
- visualizer: Plotting and visualization
- landscape_analyzer: Design space analysis
"""

from .report_generator import generate_report, ReportGenerator
from .markdown_report import generate_markdown_report, MarkdownReporter
from .results_analyzer import ResultsAnalyzer, analyze_results
from .visualizer import Visualizer, plot_results
from .landscape_analyzer import LandscapeAnalyzer, analyze_landscape

__all__ = [
    'generate_report',
    'ReportGenerator',
    'generate_markdown_report',
    'MarkdownReporter',
    'ResultsAnalyzer',
    'analyze_results',
    'Visualizer',
    'plot_results',
    'LandscapeAnalyzer',
    'analyze_landscape',
]

4.7 optimization_engine/config/init.py

"""
Configuration Management
========================

Configuration loading, validation, and building.

Modules:
- manager: ConfigManager for loading/saving configs
- builder: OptimizationConfigBuilder for creating configs
- setup_wizard: Interactive configuration setup
- capability_matcher: Match capabilities to requirements
"""

from .manager import ConfigManager, load_config, save_config
from .builder import OptimizationConfigBuilder, build_config
from .setup_wizard import SetupWizard, run_setup
from .capability_matcher import CapabilityMatcher, match_capabilities
from .template_loader import TemplateLoader, load_template

__all__ = [
    'ConfigManager',
    'load_config',
    'save_config',
    'OptimizationConfigBuilder',
    'build_config',
    'SetupWizard',
    'run_setup',
    'CapabilityMatcher',
    'match_capabilities',
    'TemplateLoader',
    'load_template',
]

4.8 optimization_engine/init.py (Updated with Backwards Compat)

"""
Atomizer Optimization Engine
============================

Structural optimization framework for Siemens NX.

New Module Structure (v2.0):
- core/           - Optimization runners
- processors/     - Data processing (surrogates, dynamic_response)
- nx/             - NX/Nastran integration
- study/          - Study management
- reporting/      - Reports and analysis
- config/         - Configuration
- extractors/     - Physics extraction (unchanged)
- insights/       - Visualizations (unchanged)
- gnn/            - Graph neural networks (unchanged)
- hooks/          - NX hooks (unchanged)
- utils/          - Utilities
- validators/     - Validation (unchanged)

Quick Start:
    from optimization_engine.core import OptimizationRunner
    from optimization_engine.nx import NXSolver
    from optimization_engine.extractors import extract_displacement
"""

__version__ = '2.0.0'

# Re-export commonly used items at top level
from optimization_engine.core.runner import OptimizationRunner
from optimization_engine.core.intelligent_optimizer import IMSO, IntelligentOptimizer
from optimization_engine.nx.solver import NXSolver, run_nx_simulation
from optimization_engine.study.creator import create_study
from optimization_engine.config.manager import ConfigManager

# Submodule access
from optimization_engine import (
    core,
    processors,
    nx,
    study,
    reporting,
    config,
    extractors,
    insights,
    gnn,
    hooks,
    utils,
    validators,
)

__all__ = [
    # Version
    '__version__',
    # Top-level exports
    'OptimizationRunner',
    'IMSO',
    'IntelligentOptimizer',
    'NXSolver',
    'run_nx_simulation',
    'create_study',
    'ConfigManager',
    # Submodules
    'core',
    'processors',
    'nx',
    'study',
    'reporting',
    'config',
    'extractors',
    'insights',
    'gnn',
    'hooks',
    'utils',
    'validators',
]

# =============================================================================
# BACKWARDS COMPATIBILITY LAYER
# =============================================================================
# These aliases allow old imports to work with deprecation warnings.
# Will be removed in v3.0.

import warnings as _warnings
import importlib as _importlib

_DEPRECATED_MAPPINGS = {
    # Core
    'runner': 'optimization_engine.core.runner',
    'base_runner': 'optimization_engine.core.base_runner',
    'intelligent_optimizer': 'optimization_engine.core.intelligent_optimizer',
    'method_selector': 'optimization_engine.core.method_selector',
    'strategy_selector': 'optimization_engine.core.strategy_selector',
    'strategy_portfolio': 'optimization_engine.core.strategy_portfolio',
    'gradient_optimizer': 'optimization_engine.core.gradient_optimizer',

    # Surrogates
    'neural_surrogate': 'optimization_engine.processors.surrogates.neural_surrogate',
    'generic_surrogate': 'optimization_engine.processors.surrogates.generic_surrogate',
    'adaptive_surrogate': 'optimization_engine.processors.surrogates.adaptive_surrogate',
    'simple_mlp_surrogate': 'optimization_engine.processors.surrogates.simple_mlp_surrogate',
    'active_learning_surrogate': 'optimization_engine.processors.surrogates.active_learning_surrogate',
    'surrogate_tuner': 'optimization_engine.processors.surrogates.surrogate_tuner',

    # NX
    'nx_solver': 'optimization_engine.nx.solver',
    'nx_updater': 'optimization_engine.nx.updater',
    'nx_session_manager': 'optimization_engine.nx.session_manager',
    'solve_simulation': 'optimization_engine.nx.solve_simulation',
    'model_cleanup': 'optimization_engine.nx.model_cleanup',

    # Study
    'study_creator': 'optimization_engine.study.creator',
    'study_wizard': 'optimization_engine.study.wizard',
    'study_state': 'optimization_engine.study.state',
    'study_reset': 'optimization_engine.study.reset',
    'study_continuation': 'optimization_engine.study.continuation',

    # Reporting
    'generate_report': 'optimization_engine.reporting.report_generator',
    'generate_report_markdown': 'optimization_engine.reporting.markdown_report',
    'visualizer': 'optimization_engine.reporting.visualizer',

    # Config
    'config_manager': 'optimization_engine.config.manager',
    'optimization_config_builder': 'optimization_engine.config.builder',
    'optimization_setup_wizard': 'optimization_engine.config.setup_wizard',
    'capability_matcher': 'optimization_engine.config.capability_matcher',

    # Utils
    'logger': 'optimization_engine.utils.logger',
}


def __getattr__(name):
    """Provide backwards compatibility with deprecation warnings."""
    if name in _DEPRECATED_MAPPINGS:
        new_module = _DEPRECATED_MAPPINGS[name]
        _warnings.warn(
            f"Importing '{name}' from optimization_engine is deprecated. "
            f"Use '{new_module}' instead. "
            f"This will be removed in v3.0.",
            DeprecationWarning,
            stacklevel=2
        )
        return _importlib.import_module(new_module)

    raise AttributeError(f"module 'optimization_engine' has no attribute '{name}'")

Part 5: Post-Migration Checklist

Immediate (same day)

All imports updated in optimization_engine/
All imports updated in studies/
All imports updated in tests/
All imports updated in tools/
All imports updated in dashboard backend
feature_registry.json updated
All init.py files created
Basic import tests pass
Test suite passes (or known failures documented)

Short-term (within 1 week)

Update CLAUDE.md with new structure
Update all protocol files (SYS_10-16, OP_01-06, EXT_01-04)
Update all skill files in .claude/skills/
Update ATOMIZER_CONTEXT.md
Update README.md
Record migration in LAC knowledge base
Run 2-3 studies end-to-end to verify

Medium-term (within 1 month)

Remove backwards compatibility after team confirms no issues
Update any external documentation/wikis
Update pyproject.toml metadata if needed
Archive the migration plan document

Part 6: Rollback Procedure

If critical issues are discovered:

# Option 1: Git revert (preferred if committed)
git revert HEAD

# Option 2: Reset to backup branch
git checkout backup/pre-reorganization
git checkout -b main-restored
git branch -D main
git branch -m main-restored main

# Option 3: Restore from backup directory
rm -rf optimization_engine
mv optimization_engine_BACKUP_* optimization_engine

Part 7: Time Summary

Phase	Task	Time
0	Preparation (backup, baseline)	30 min
1	Create directory structure	15 min
2	Move files	45 min
3	Update imports (with script)	2-3 hours
4	Create init.py files	1 hour
5	Update feature_registry.json	30 min
6	Update documentation	2 hours
7	Testing & validation	2 hours
8	Commit & cleanup	30 min
TOTAL		10-12 hours

Appendix: Quick Reference Card

New Import Patterns

# OLD                                    # NEW
from optimization_engine.runner          from optimization_engine.core.runner
from optimization_engine.nx_solver       from optimization_engine.nx.solver
from optimization_engine.study_creator   from optimization_engine.study.creator
from optimization_engine.neural_surrogate from optimization_engine.processors.surrogates.neural_surrogate
from optimization_engine.config_manager  from optimization_engine.config.manager
from optimization_engine.logger          from optimization_engine.utils.logger

Directory Quick Reference

optimization_engine/
├── core/           → runner, intelligent_optimizer, method_selector
├── processors/
│   └── surrogates/ → neural_surrogate, generic_surrogate, etc.
├── nx/             → solver (was nx_solver), updater, session_manager
├── study/          → creator (was study_creator), wizard, state
├── reporting/      → report_generator, visualizer
├── config/         → manager (was config_manager), builder
├── utils/          → logger, auto_doc
├── future/         → research_agent, workflow_decomposer
├── extractors/     → (unchanged)
├── insights/       → (unchanged)
├── gnn/            → (unchanged)
├── hooks/          → (unchanged)
└── validators/     → (unchanged)

Part 8: Handoff to Context Engineering Plan

After this migration is complete, proceed with ATOMIZER_CONTEXT_ENGINEERING_PLAN.md.

What Context Engineering Will Add

optimization_engine/
├── context/                    ← NEW (added by Context Engineering)
│   ├── __init__.py
│   ├── playbook.py            # AtomizerPlaybook - structured knowledge storage
│   ├── reflector.py           # Analyzes optimization outcomes
│   ├── session_state.py       # Context isolation for LLM sessions
│   ├── compaction.py          # Context compaction for long sessions
│   ├── feedback_loop.py       # Learning from execution outcomes
│   └── cache_monitor.py       # KV-cache efficiency tracking
├── core/                       ← Already created by Migration
│   └── runner.py              # Will be modified to integrate context
└── ...

Context Engineering Integration Points

After migration, Context Engineering will modify:

optimization_engine/core/runner.py - Add playbook and reflector integration
.claude/skills/00_BOOTSTRAP.md - Update to v2 with playbook loading
.claude/skills/02_CONTEXT_LOADER.md - Add playbook integration
atomizer-dashboard/backend/api/routes/ - Add context API routes

Pre-Migration Checklist for Context Engineering Compatibility

Before starting Context Engineering, verify:

Migration completed successfully
optimization_engine/core/runner.py exists at new location
All imports in codebase updated to new paths
Test suite passes
No deprecation warnings in core workflows

Context Engineering Will Use These New Paths

# Context Engineering code will import from new locations:
from optimization_engine.core.runner import OptimizationRunner
from optimization_engine.core.intelligent_optimizer import IMSO
from optimization_engine.utils.logger import get_logger
from optimization_engine.nx.solver import NXSolver

# And add new context module:
from optimization_engine.context.playbook import AtomizerPlaybook
from optimization_engine.context.reflector import AtomizerReflector

This plan is DEFINITIVE and covers ALL 500+ files that need updating. After completion, proceed to ATOMIZER_CONTEXT_ENGINEERING_PLAN.md

69 KiB Raw Blame History