# Optimization Engine Reorganization - DEFINITIVE Migration Plan ## Complete Guide for Safe Codebase Restructuring **Document Version**: 3.0 (COMPLETED) **Created**: 2025-12-23 **Updated**: 2025-12-29 **Status**: ✅ COMPLETED - Merged to main **Risk Level**: HIGH - Affects 500+ files across entire codebase **Next Phase**: ATOMIZER_CONTEXT_ENGINEERING_PLAN.md (execute AFTER this migration) --- ## ✅ Migration Completed: 2025-12-29 ### Final Statistics | Metric | Value | |--------|-------| | Files modified | 120 | | Import changes | ~200 | | Commits | 2 (pre-migration checkpoint + migration) | | Branch | `refactor/optimization-engine-v2` merged to `main` | | Version bump | `optimization_engine` v1.0 → v2.0.0 | ### Key Changes Made 1. Created new directory structure: `core/`, `processors/surrogates/`, `nx/`, `study/`, `reporting/`, `config/` 2. Moved 60+ Python files to new locations 3. Updated all imports across 125 files 4. Added lazy-loading `__init__.py` files to avoid circular imports 5. Implemented backwards compatibility layer with deprecation warnings 6. Updated `feature_registry.json` paths ### Post-Migration Notes - All existing `run_optimization.py` scripts continue to work - Old imports emit deprecation warnings but still function - Dashboard integration preserved - Study databases unchanged --- ## Execution Order (IMPORTANT) This migration plan should be executed **BEFORE** the Context Engineering plan. **Why Migration First?** 1. Creates clean, organized module structure 2. Context Engineering adds NEW modules (`context/`) that won't be moved 3. Avoids double-updating imports (migrate first, then integrate new code) 4. Documentation written once with final paths **After Migration Completes:** - Execute `ATOMIZER_CONTEXT_ENGINEERING_PLAN.md` - Add `optimization_engine/context/` module (playbook, reflector, session_state) - Integrate with `optimization_engine/core/runner.py` (already at final path) --- ## Executive Summary This is the **DEFINITIVE** migration plan covering every single file that needs updating. The scope is significantly larger than initially estimated. ### True Impact Summary | Category | Files | Estimated Changes | |----------|-------|-------------------| | Python files (optimization_engine internal) | 90+ | ~300 import changes | | Python files (studies) | 80+ | ~200 import changes | | Python files (tests) | 45+ | ~150 import changes | | Python files (tools, examples, root) | 10+ | ~50 import changes | | Python files (dashboard backend) | 3 | ~15 import changes | | Python files (archive - optional) | 50+ | ~100 import changes | | Documentation (markdown) | 100+ | ~500 path references | | JSON configuration files | 5 | ~50 path entries | | Build config (pyproject.toml) | 1 | 2 lines | | **TOTAL** | **500+ files** | **~1,400 changes** | ### Estimated Time: 8-12 hours (with automation scripts) --- ## Part 1: Complete File Inventory ### 1.1 Files to MOVE (Source Files) These files move from `optimization_engine/` root to new subdirectories: ``` MOVE TO: optimization_engine/core/ ├── runner.py ├── base_runner.py ├── runner_with_neural.py ├── intelligent_optimizer.py ├── method_selector.py ├── strategy_selector.py ├── strategy_portfolio.py └── gradient_optimizer.py MOVE TO: optimization_engine/processors/surrogates/ ├── neural_surrogate.py ├── generic_surrogate.py ├── adaptive_surrogate.py ├── simple_mlp_surrogate.py ├── active_learning_surrogate.py ├── surrogate_tuner.py ├── auto_trainer.py └── training_data_exporter.py MOVE TO: optimization_engine/nx/ ├── nx_solver.py → solver.py ├── nx_updater.py → updater.py ├── nx_session_manager.py → session_manager.py ├── solve_simulation.py ├── solve_simulation_simple.py ├── model_cleanup.py ├── export_expressions.py ├── import_expressions.py └── mesh_converter.py MOVE TO: optimization_engine/study/ ├── study_creator.py → creator.py ├── study_wizard.py → wizard.py ├── study_state.py → state.py ├── study_reset.py → reset.py ├── study_continuation.py → continuation.py ├── benchmarking_substudy.py → benchmarking.py └── generate_history_from_trials.py → history_generator.py MOVE TO: optimization_engine/reporting/ ├── generate_report.py → report_generator.py ├── generate_report_markdown.py → markdown_report.py ├── comprehensive_results_analyzer.py → results_analyzer.py ├── visualizer.py └── landscape_analyzer.py MOVE TO: optimization_engine/config/ ├── config_manager.py → manager.py ├── optimization_config_builder.py → builder.py ├── optimization_setup_wizard.py → setup_wizard.py ├── capability_matcher.py └── template_loader.py MOVE TO: optimization_engine/utils/ (existing, add to it) ├── logger.py ├── auto_doc.py ├── realtime_tracking.py ├── codebase_analyzer.py └── pruning_logger.py MOVE TO: optimization_engine/future/ (existing, add to it) ├── research_agent.py ├── pynastran_research_agent.py ├── targeted_research_planner.py ├── workflow_decomposer.py ├── step_classifier.py ├── llm_optimization_runner.py ├── llm_workflow_analyzer.py ├── extractor_orchestrator.py ├── inline_code_generator.py └── hook_generator.py MOVE TO: optimization_engine/extractors/ (existing, add to it) ├── op2_extractor.py └── extractor_library.py MOVE TO: optimization_engine/validators/ (existing, add to it) └── simulation_validator.py MOVE TO: optimization_engine/processors/ (new) └── adaptive_characterization.py ``` ### 1.2 Files to UPDATE (Import Changes Required) #### A. OPTIMIZATION_ENGINE INTERNAL (90+ files) **Core/Root Files:** ``` optimization_engine/run_optimization.py optimization_engine/__init__.py ``` **Extractors (20 files):** ``` optimization_engine/extractors/__init__.py (31 imports) optimization_engine/extractors/extract_displacement.py optimization_engine/extractors/extract_frequency.py optimization_engine/extractors/extract_von_mises_stress.py optimization_engine/extractors/bdf_mass_extractor.py optimization_engine/extractors/extract_mass_from_bdf.py optimization_engine/extractors/extract_mass_from_expression.py optimization_engine/extractors/extract_part_mass_material.py optimization_engine/extractors/extract_zernike.py optimization_engine/extractors/extract_zernike_opd.py optimization_engine/extractors/extract_zernike_figure.py optimization_engine/extractors/extract_zernike_surface.py optimization_engine/extractors/zernike_helpers.py optimization_engine/extractors/extract_principal_stress.py optimization_engine/extractors/extract_strain_energy.py optimization_engine/extractors/extract_spc_forces.py optimization_engine/extractors/extract_temperature.py optimization_engine/extractors/extract_modal_mass.py optimization_engine/extractors/introspect_part.py optimization_engine/extractors/field_data_extractor.py optimization_engine/extractors/stiffness_calculator.py optimization_engine/extractors/test_phase3_extractors.py ``` **GNN Module (12 files):** ``` optimization_engine/gnn/__init__.py optimization_engine/gnn/zernike_gnn.py optimization_engine/gnn/polar_graph.py optimization_engine/gnn/train_zernike_gnn.py optimization_engine/gnn/gnn_optimizer.py optimization_engine/gnn/extract_displacement_field.py optimization_engine/gnn/differentiable_zernike.py optimization_engine/gnn/backfill_field_data.py optimization_engine/gnn/test_polar_graph.py optimization_engine/gnn/test_field_extraction.py optimization_engine/gnn/test_new_extraction.py ``` **Hooks Module (12 files):** ``` optimization_engine/hooks/__init__.py optimization_engine/hooks/examples.py optimization_engine/hooks/test_introspection.py optimization_engine/hooks/test_hooks.py optimization_engine/hooks/nx_cad/__init__.py optimization_engine/hooks/nx_cad/model_introspection.py optimization_engine/hooks/nx_cad/part_manager.py optimization_engine/hooks/nx_cad/expression_manager.py optimization_engine/hooks/nx_cad/geometry_query.py optimization_engine/hooks/nx_cad/feature_manager.py optimization_engine/hooks/nx_cae/__init__.py optimization_engine/hooks/nx_cae/solver_manager.py ``` **Plugins Module (5 files):** ``` optimization_engine/plugins/__init__.py optimization_engine/plugins/hook_manager.py optimization_engine/plugins/post_calculation/safety_factor_constraint.py optimization_engine/plugins/pre_solve/*.py optimization_engine/plugins/post_solve/*.py ``` **Insights Module (12 files):** ``` optimization_engine/insights/__init__.py optimization_engine/insights/base.py optimization_engine/insights/zernike_wfe.py optimization_engine/insights/zernike_opd_comparison.py optimization_engine/insights/zernike_dashboard.py optimization_engine/insights/msf_zernike.py optimization_engine/insights/stress_field.py optimization_engine/insights/modal_analysis.py optimization_engine/insights/thermal_field.py optimization_engine/insights/design_space.py ``` **Utils Module (6 files):** ``` optimization_engine/utils/__init__.py optimization_engine/utils/trial_manager.py optimization_engine/utils/dashboard_db.py optimization_engine/utils/nx_session_manager.py optimization_engine/utils/nx_file_discovery.py optimization_engine/utils/study_cleanup.py ``` **Validators Module (5 files):** ``` optimization_engine/validators/__init__.py optimization_engine/validators/study_validator.py optimization_engine/validators/results_validator.py optimization_engine/validators/model_validator.py optimization_engine/validators/config_validator.py ``` **Templates (3 files):** ``` optimization_engine/templates/__init__.py optimization_engine/templates/run_optimization_template.py optimization_engine/templates/run_nn_optimization_template.py ``` **Model Discovery (2 files):** ``` optimization_engine/model_discovery/__init__.py optimization_engine/model_discovery/discover_parts.py ``` #### B. STUDIES DIRECTORY (80+ files) **M1_Mirror Studies (35+ files):** ``` studies/M1_Mirror/m1_mirror_cost_reduction/run_optimization.py studies/M1_Mirror/m1_mirror_zernike_optimization/run_optimization.py studies/M1_Mirror/m1_mirror_cost_reduction_flat_back_V1/run_optimization.py studies/M1_Mirror/m1_mirror_cost_reduction_flat_back_V3/run_optimization.py studies/M1_Mirror/m1_mirror_cost_reduction_flat_back_V4/run_turbo_optimization.py studies/M1_Mirror/m1_mirror_cost_reduction_flat_back_V5/run_turbo_optimization.py studies/M1_Mirror/m1_mirror_cost_reduction_flat_back_V6/run_optimization.py studies/M1_Mirror/m1_mirror_turbo_V1/run_optimization.py studies/M1_Mirror/m1_mirror_turbo_V2/run_optimization.py studies/M1_Mirror/m1_mirror_cost_reduction_V2/run_optimization.py studies/M1_Mirror/m1_mirror_cost_reduction_V3/run_optimization.py studies/M1_Mirror/m1_mirror_cost_reduction_V4/run_optimization.py studies/M1_Mirror/m1_mirror_cost_reduction_V5/run_optimization.py studies/M1_Mirror/m1_mirror_cost_reduction_V5/_analyze_baseline.py studies/M1_Mirror/m1_mirror_cost_reduction_V6/run_optimization.py studies/M1_Mirror/m1_mirror_cost_reduction_V6/_analyze_baseline.py studies/M1_Mirror/m1_mirror_cost_reduction_V7/run_optimization.py studies/M1_Mirror/m1_mirror_cost_reduction_V8/run_optimization.py studies/M1_Mirror/m1_mirror_cost_reduction_V9/run_optimization.py studies/M1_Mirror/m1_mirror_cost_reduction_V10/run_optimization.py studies/M1_Mirror/m1_mirror_cost_reduction_V11/run_optimization.py studies/M1_Mirror/m1_mirror_cost_reduction_V12/run_optimization.py studies/M1_Mirror/m1_mirror_adaptive_V11/run_optimization.py studies/M1_Mirror/m1_mirror_adaptive_V12/run_optimization.py studies/M1_Mirror/m1_mirror_adaptive_V12/run_gnn_turbo.py studies/M1_Mirror/m1_mirror_adaptive_V12/validate_gnn_best.py studies/M1_Mirror/m1_mirror_adaptive_V12/compute_full_calibration.py studies/M1_Mirror/m1_mirror_adaptive_V13/run_optimization.py studies/M1_Mirror/m1_mirror_adaptive_V14/run_optimization.py studies/M1_Mirror/m1_mirror_adaptive_V14/run_lbfgs_polish.py studies/M1_Mirror/m1_mirror_adaptive_V15/run_optimization.py studies/M1_Mirror/m1_mirror_surrogate_turbo/scripts/01_reextract_studies.py studies/M1_Mirror/m1_mirror_surrogate_turbo/scripts/02_aggregate_gnn_data.py studies/M1_Mirror/m1_mirror_surrogate_turbo/scripts/03_train_gnn.py ``` **Simple_Bracket Studies (12+ files):** ``` studies/Simple_Bracket/bracket_stiffness_optimization/run_optimization.py studies/Simple_Bracket/bracket_stiffness_optimization/bracket_stiffness_extractor.py studies/Simple_Bracket/bracket_stiffness_optimization_V2/run_optimization.py studies/Simple_Bracket/bracket_stiffness_optimization_V2/bracket_stiffness_extractor.py studies/Simple_Bracket/bracket_stiffness_optimization_V3/run_optimization.py studies/Simple_Bracket/bracket_stiffness_optimization_V3/bracket_stiffness_extractor.py studies/Simple_Bracket/bracket_stiffness_optimization_atomizerfield/run_optimization.py studies/Simple_Bracket/bracket_pareto_3obj/run_optimization.py studies/Simple_Bracket/bracket_pareto_3obj/run_nn_optimization.py ``` **Other Studies (15+ files):** ``` studies/Simple_Beam/simple_beam_optimization/run_optimization.py studies/UAV_Arm/uav_arm_optimization/run_optimization.py studies/UAV_Arm/uav_arm_optimization_V2/run_optimization.py studies/UAV_Arm/uav_arm_atomizerfield_test/run_optimization.py studies/Drone_Gimbal/drone_gimbal_arm_optimization/run_optimization.py studies/bracket_displacement_maximizing/run_optimization.py studies/bracket_displacement_maximizing/run_optimization_old.py studies/bracket_displacement_maximizing/run_substudy.py studies/bracket_displacement_maximizing/test_fix.py studies/_Other/training_data_export_test/run_optimization.py ``` #### C. TESTS DIRECTORY (45+ files) ``` tests/audit_v10_wfe.py tests/audit_v10_method_diff.py tests/audit_v10_fix.py tests/analyze_v11.py tests/debug_insights.py tests/debug_figure_coords.py tests/compare_v8_zernike_methods.py tests/test_zernike_opd_with_prescription.py tests/test_zernike_opd_comparison.py tests/test_zernike_methods_comparison.py tests/test_zernike_insight.py tests/test_zernike_import.py tests/test_training_data_exporter.py tests/test_timestamp_verification.py tests/test_task_1_2_integration.py tests/test_step_classifier.py tests/test_research_agent.py tests/test_phase_3_2_llm_mode.py tests/test_phase_3_1_integration.py tests/test_phase_2_5_intelligent_gap_detection.py tests/test_plugin_system.py tests/test_optimization_setup_wizard.py tests/test_modal_deformation_request.py tests/test_llm_runner_init.py tests/test_llm_complex_request.py tests/test_lifecycle_hook_integration.py tests/test_knowledge_base_search.py tests/test_journal_optimization.py tests/test_interactive_session.py tests/test_hooks_with_bracket.py tests/test_complex_multiobj_request.py tests/test_complete_research_workflow.py tests/test_code_generation.py tests/test_cbar_genetic_algorithm.py tests/test_cbush_optimization.py tests/test_bracket_llm_runner.py tests/test_bracket_full_optimization.py tests/test_beam_workflow.py tests/test_api_verification.py tests/interactive_optimization_setup.py tests/demo_research_agent.py tests/run_beam_benchmarking.py tests/setup_beam_optimization.py tests/run_5trial_test.py tests/run_benchmarking_simple.py tests/run_beam_benchmarking_clean.py ``` #### D. ROOT & TOOLS (10+ files) ``` atomizer.py run_cleanup.py run_training_fea.py tools/zernike_html_generator.py tools/extract_mirror_optical_specs.py examples/interactive_research_session.py ``` #### E. DASHBOARD BACKEND (3 files) ``` atomizer-dashboard/backend/api/main.py atomizer-dashboard/backend/api/routes/optimization.py atomizer-dashboard/backend/api/routes/insights.py ``` #### F. ARCHIVE (50+ files - OPTIONAL) ``` archive/deprecated/result_extractors/extractors.py archive/deprecated/result_extractors/generated/extract_expression.py archive/scripts/create_intelligent_study.py archive/scripts/create_circular_plate_study_v2.py archive/scripts/create_circular_plate_study.py archive/scripts/run_nn_optimization.py archive/scripts/run_calibration_loop.py archive/scripts/run_validated_nn_optimization.py archive/scripts/validate_surrogate_real_data.py archive/scripts/visualize_nn_optimization.py archive/test_scripts/*.py (10+ files) ``` ### 1.3 Documentation Files to UPDATE (100+ files) #### A. ROOT DOCUMENTATION ``` CLAUDE.md (lines 141, 152, 166, 200, 221, 235, 253, 283, 592-603) README.md (lines 98, 193, 213, 272) CHANGELOG.md (lines 31, 40, 82) DEVELOPMENT.md (lines 40, 64, 77, 90, 215-277, 398, 532, 535) PROTOCOL.md (lines 481, 704-705, 760, 812, 924-939, 1118, 1262, 1380, 1406, 1501-1596, 1851) ``` #### B. .CLAUDE DIRECTORY ``` .claude/ATOMIZER_CONTEXT.md (lines 37, 40, 71, 235, 238, 248, 273, 282, 288, 421-472) .claude/CHEATSHEET.md (lines 7-10, 33-35, 144, 156, 353-472) .claude/skills/00_BOOTSTRAP.md (line 236) .claude/skills/01_CHEATSHEET.md (lines 7-10, 33-35, 144, 156, 353-472) .claude/skills/analyze-workflow.md (line 37) .claude/skills/analyze-model.md (lines 52, 99, 183) .claude/skills/atomizer.md (lines 15, 27, 62, 193, 286, 317, 323) .claude/skills/DEV_DOCUMENTATION.md (lines 115-117, 141) .claude/skills/core/study-creation-core.md (lines 7-9, 43, 157, 160, 259, 279, 454-459, 643, 706) .claude/skills/modules/extractors-catalog.md (lines 24-31, 42, 63, 86, 110-271) .claude/skills/modules/insights-catalog.md (lines 53-237) .claude/skills/modules/neural-acceleration.md (lines 48, 318) .claude/skills/modules/nx-docs-lookup.md (line 190) .claude/skills/modules/zernike-optimization.md (lines 35, 73, 111) .claude/skills/modules/atomizer_fast_solver_technologies.md (lines 739, 744) .claude/skills/modules/learning-atomizer-core.md .claude/skills/archive/create-study-wizard.md (lines 13, 74, 177, 360, 402) ``` #### C. DOCS DIRECTORY - PROTOCOLS ``` docs/00_INDEX.md (line 284) docs/01_PROTOCOLS.md (lines 277, 352-355, 528, 535-600, 1355, 1434, 1460) docs/02_ARCHITECTURE.md (lines 68-69, 104, 123, 193, 254, 279, 299, 376-495) docs/03_GETTING_STARTED.md (line 286) docs/protocols/operations/OP_01_CREATE_STUDY.md (lines 162, 301-303) docs/protocols/operations/OP_02_RUN_OPTIMIZATION.md docs/protocols/operations/OP_03_MONITOR_PROGRESS.md docs/protocols/operations/OP_04_ANALYZE_RESULTS.md docs/protocols/operations/OP_05_EXPORT_TRAINING_DATA.md docs/protocols/operations/OP_06_TROUBLESHOOT.md docs/protocols/system/SYS_10_IMSO.md (lines 224, 302-305) docs/protocols/system/SYS_11_MULTI_OBJECTIVE.md (lines 322-330) docs/protocols/system/SYS_12_EXTRACTOR_LIBRARY.md (lines 71-851 - MANY references) docs/protocols/system/SYS_13_DASHBOARD_TRACKING.md (lines 60, 85, 315) docs/protocols/system/SYS_14_NEURAL_ACCELERATION.md (lines 231-1080) docs/protocols/system/SYS_15_METHOD_SELECTOR.md (lines 42-422) docs/protocols/system/SYS_17_STUDY_INSIGHTS.md (lines 62-498) docs/protocols/extensions/EXT_01_CREATE_EXTRACTOR.md (lines 35-287) docs/protocols/extensions/EXT_02_CREATE_HOOK.md (lines 45-357) docs/protocols/extensions/EXT_03_CREATE_INSIGHT.md docs/protocols/extensions/EXT_04_CREATE_SKILL.md (lines 263, 274) docs/protocols/extensions/templates/hook_template.py (lines 128, 164) ``` #### D. DOCS SUBDIRECTORIES ``` docs/04_USER_GUIDES/*.md (estimated 20+ files) docs/05_API_REFERENCE/*.md (estimated 10+ files) docs/06_PHYSICS/*.md (estimated 15+ files) docs/07_DEVELOPMENT/*.md (estimated 20+ files) docs/08_ARCHIVE/*.md (estimated 30+ files) ``` ### 1.4 Configuration Files to UPDATE ``` optimization_engine/feature_registry.json (50+ path entries on lines 21-683) optimization_engine/schemas/README.md (lines 19, 26) studies/Simple_Bracket/bracket_stiffness_optimization_atomizerfield/1_setup/optimization_config.json studies/Simple_Bracket/bracket_pareto_3obj/1_setup/optimization_config.json pyproject.toml (lines 53, 76) ``` --- ## Part 2: Import Mapping Reference ### 2.1 Complete Old → New Import Mapping ```python # ============================================================================= # CORE MODULE # ============================================================================= "from optimization_engine.runner" → "from optimization_engine.core.runner" "from optimization_engine.base_runner" → "from optimization_engine.core.base_runner" "from optimization_engine.runner_with_neural" → "from optimization_engine.core.runner_with_neural" "from optimization_engine.intelligent_optimizer" → "from optimization_engine.core.intelligent_optimizer" "from optimization_engine.method_selector" → "from optimization_engine.core.method_selector" "from optimization_engine.strategy_selector" → "from optimization_engine.core.strategy_selector" "from optimization_engine.strategy_portfolio" → "from optimization_engine.core.strategy_portfolio" "from optimization_engine.gradient_optimizer" → "from optimization_engine.core.gradient_optimizer" "import optimization_engine.runner" → "import optimization_engine.core.runner" "import optimization_engine.intelligent_optimizer" → "import optimization_engine.core.intelligent_optimizer" # ============================================================================= # SURROGATES MODULE # ============================================================================= "from optimization_engine.neural_surrogate" → "from optimization_engine.processors.surrogates.neural_surrogate" "from optimization_engine.generic_surrogate" → "from optimization_engine.processors.surrogates.generic_surrogate" "from optimization_engine.adaptive_surrogate" → "from optimization_engine.processors.surrogates.adaptive_surrogate" "from optimization_engine.simple_mlp_surrogate" → "from optimization_engine.processors.surrogates.simple_mlp_surrogate" "from optimization_engine.active_learning_surrogate" → "from optimization_engine.processors.surrogates.active_learning_surrogate" "from optimization_engine.surrogate_tuner" → "from optimization_engine.processors.surrogates.surrogate_tuner" "from optimization_engine.auto_trainer" → "from optimization_engine.processors.surrogates.auto_trainer" "from optimization_engine.training_data_exporter" → "from optimization_engine.processors.surrogates.training_data_exporter" # ============================================================================= # NX MODULE # ============================================================================= "from optimization_engine.nx_solver" → "from optimization_engine.nx.solver" "from optimization_engine.nx_updater" → "from optimization_engine.nx.updater" "from optimization_engine.nx_session_manager" → "from optimization_engine.nx.session_manager" "from optimization_engine.solve_simulation" → "from optimization_engine.nx.solve_simulation" "from optimization_engine.solve_simulation_simple" → "from optimization_engine.nx.solve_simulation_simple" "from optimization_engine.model_cleanup" → "from optimization_engine.nx.model_cleanup" "from optimization_engine.export_expressions" → "from optimization_engine.nx.export_expressions" "from optimization_engine.import_expressions" → "from optimization_engine.nx.import_expressions" "from optimization_engine.mesh_converter" → "from optimization_engine.nx.mesh_converter" "import optimization_engine.nx_solver" → "import optimization_engine.nx.solver" # ============================================================================= # STUDY MODULE # ============================================================================= "from optimization_engine.study_creator" → "from optimization_engine.study.creator" "from optimization_engine.study_wizard" → "from optimization_engine.study.wizard" "from optimization_engine.study_state" → "from optimization_engine.study.state" "from optimization_engine.study_reset" → "from optimization_engine.study.reset" "from optimization_engine.study_continuation" → "from optimization_engine.study.continuation" "from optimization_engine.benchmarking_substudy" → "from optimization_engine.study.benchmarking" "from optimization_engine.generate_history_from_trials" → "from optimization_engine.study.history_generator" # ============================================================================= # REPORTING MODULE # ============================================================================= "from optimization_engine.generate_report" → "from optimization_engine.reporting.report_generator" "from optimization_engine.generate_report_markdown" → "from optimization_engine.reporting.markdown_report" "from optimization_engine.comprehensive_results_analyzer" → "from optimization_engine.reporting.results_analyzer" "from optimization_engine.visualizer" → "from optimization_engine.reporting.visualizer" "from optimization_engine.landscape_analyzer" → "from optimization_engine.reporting.landscape_analyzer" # ============================================================================= # CONFIG MODULE # ============================================================================= "from optimization_engine.config_manager" → "from optimization_engine.config.manager" "from optimization_engine.optimization_config_builder" → "from optimization_engine.config.builder" "from optimization_engine.optimization_setup_wizard" → "from optimization_engine.config.setup_wizard" "from optimization_engine.capability_matcher" → "from optimization_engine.config.capability_matcher" "from optimization_engine.template_loader" → "from optimization_engine.config.template_loader" # ============================================================================= # UTILS MODULE (moving to existing utils/) # ============================================================================= "from optimization_engine.logger" → "from optimization_engine.utils.logger" "from optimization_engine.auto_doc" → "from optimization_engine.utils.auto_doc" "from optimization_engine.realtime_tracking" → "from optimization_engine.utils.realtime_tracking" "from optimization_engine.codebase_analyzer" → "from optimization_engine.utils.codebase_analyzer" "from optimization_engine.pruning_logger" → "from optimization_engine.utils.pruning_logger" # ============================================================================= # FUTURE MODULE (research/experimental) # ============================================================================= "from optimization_engine.research_agent" → "from optimization_engine.future.research_agent" "from optimization_engine.pynastran_research_agent" → "from optimization_engine.future.pynastran_research_agent" "from optimization_engine.targeted_research_planner" → "from optimization_engine.future.targeted_research_planner" "from optimization_engine.workflow_decomposer" → "from optimization_engine.future.workflow_decomposer" "from optimization_engine.step_classifier" → "from optimization_engine.future.step_classifier" "from optimization_engine.llm_optimization_runner" → "from optimization_engine.future.llm_optimization_runner" "from optimization_engine.llm_workflow_analyzer" → "from optimization_engine.future.llm_workflow_analyzer" "from optimization_engine.extractor_orchestrator" → "from optimization_engine.future.extractor_orchestrator" "from optimization_engine.inline_code_generator" → "from optimization_engine.future.inline_code_generator" "from optimization_engine.hook_generator" → "from optimization_engine.future.hook_generator" # ============================================================================= # EXTRACTORS MODULE (moving to existing extractors/) # ============================================================================= "from optimization_engine.op2_extractor" → "from optimization_engine.extractors.op2_extractor" "from optimization_engine.extractor_library" → "from optimization_engine.extractors.extractor_library" # ============================================================================= # VALIDATORS MODULE (moving to existing validators/) # ============================================================================= "from optimization_engine.simulation_validator" → "from optimization_engine.validators.simulation_validator" # ============================================================================= # PROCESSORS MODULE (new) # ============================================================================= "from optimization_engine.adaptive_characterization" → "from optimization_engine.processors.adaptive_characterization" # ============================================================================= # UNCHANGED (these modules stay in place) # ============================================================================= # extractors/ - stays as optimization_engine.extractors.* # gnn/ - stays as optimization_engine.gnn.* # hooks/ - stays as optimization_engine.hooks.* # insights/ - stays as optimization_engine.insights.* # plugins/ - stays as optimization_engine.plugins.* # validators/ - stays as optimization_engine.validators.* # utils/ - stays as optimization_engine.utils.* # templates/ - stays as optimization_engine.templates.* # schemas/ - stays as optimization_engine.schemas.* # model_discovery/ - stays as optimization_engine.model_discovery.* ``` ### 2.2 Documentation Path Mapping ``` # In markdown files, update these path references: optimization_engine/runner.py → optimization_engine/core/runner.py optimization_engine/nx_solver.py → optimization_engine/nx/solver.py optimization_engine/nx_updater.py → optimization_engine/nx/updater.py optimization_engine/study_creator.py → optimization_engine/study/creator.py optimization_engine/study_wizard.py → optimization_engine/study/wizard.py optimization_engine/neural_surrogate.py → optimization_engine/processors/surrogates/neural_surrogate.py optimization_engine/config_manager.py → optimization_engine/config/manager.py optimization_engine/logger.py → optimization_engine/utils/logger.py # ... etc for all moved files ``` --- ## Part 3: Execution Plan ### Phase 0: Preparation (30 minutes) ```bash # 1. Create backup branch git checkout -b backup/pre-reorganization git push origin backup/pre-reorganization git checkout main git checkout -b refactor/optimization-engine-v2 # 2. Create full backup cp -r optimization_engine optimization_engine_BACKUP_$(date +%Y%m%d) # 3. Run baseline tests and save output python -m pytest tests/ -v --tb=short 2>&1 | tee baseline_tests_$(date +%Y%m%d).log # 4. Count current test status echo "Baseline test count:" grep -E "passed|failed|error" baseline_tests_*.log | tail -5 # 5. Verify NX is not running (to prevent file locks) tasklist | grep -i ugraf || echo "NX not running - safe to proceed" ``` ### Phase 1: Create Directory Structure (15 minutes) ```bash # Create all new directories mkdir -p optimization_engine/core mkdir -p optimization_engine/processors/surrogates mkdir -p optimization_engine/processors/dynamic_response mkdir -p optimization_engine/nx mkdir -p optimization_engine/study mkdir -p optimization_engine/reporting mkdir -p optimization_engine/config # Create __init__.py files for new directories touch optimization_engine/core/__init__.py touch optimization_engine/processors/__init__.py touch optimization_engine/processors/surrogates/__init__.py touch optimization_engine/processors/dynamic_response/__init__.py touch optimization_engine/nx/__init__.py touch optimization_engine/study/__init__.py touch optimization_engine/reporting/__init__.py touch optimization_engine/config/__init__.py ``` ### Phase 2: Move Files (45 minutes) **Execute in this specific order to minimize issues:** ```bash # ============================================================================= # STEP 2.1: UTILITIES FIRST (no dependencies on other moved files) # ============================================================================= mv optimization_engine/logger.py optimization_engine/utils/ mv optimization_engine/auto_doc.py optimization_engine/utils/ mv optimization_engine/realtime_tracking.py optimization_engine/utils/ mv optimization_engine/codebase_analyzer.py optimization_engine/utils/ mv optimization_engine/pruning_logger.py optimization_engine/utils/ # ============================================================================= # STEP 2.2: CONFIG (depends only on logger) # ============================================================================= mv optimization_engine/config_manager.py optimization_engine/config/manager.py mv optimization_engine/optimization_config_builder.py optimization_engine/config/builder.py mv optimization_engine/optimization_setup_wizard.py optimization_engine/config/setup_wizard.py mv optimization_engine/capability_matcher.py optimization_engine/config/capability_matcher.py mv optimization_engine/template_loader.py optimization_engine/config/template_loader.py # ============================================================================= # STEP 2.3: NX INTEGRATION (core infrastructure) # ============================================================================= mv optimization_engine/nx_solver.py optimization_engine/nx/solver.py mv optimization_engine/nx_updater.py optimization_engine/nx/updater.py mv optimization_engine/nx_session_manager.py optimization_engine/nx/session_manager.py mv optimization_engine/solve_simulation.py optimization_engine/nx/ mv optimization_engine/solve_simulation_simple.py optimization_engine/nx/ mv optimization_engine/model_cleanup.py optimization_engine/nx/ mv optimization_engine/export_expressions.py optimization_engine/nx/ mv optimization_engine/import_expressions.py optimization_engine/nx/ mv optimization_engine/mesh_converter.py optimization_engine/nx/ # ============================================================================= # STEP 2.4: SURROGATES (ML components) # ============================================================================= mv optimization_engine/neural_surrogate.py optimization_engine/processors/surrogates/ mv optimization_engine/generic_surrogate.py optimization_engine/processors/surrogates/ mv optimization_engine/adaptive_surrogate.py optimization_engine/processors/surrogates/ mv optimization_engine/simple_mlp_surrogate.py optimization_engine/processors/surrogates/ mv optimization_engine/active_learning_surrogate.py optimization_engine/processors/surrogates/ mv optimization_engine/surrogate_tuner.py optimization_engine/processors/surrogates/ mv optimization_engine/auto_trainer.py optimization_engine/processors/surrogates/ mv optimization_engine/training_data_exporter.py optimization_engine/processors/surrogates/ # ============================================================================= # STEP 2.5: STUDY MANAGEMENT # ============================================================================= mv optimization_engine/study_creator.py optimization_engine/study/creator.py mv optimization_engine/study_wizard.py optimization_engine/study/wizard.py mv optimization_engine/study_state.py optimization_engine/study/state.py mv optimization_engine/study_reset.py optimization_engine/study/reset.py mv optimization_engine/study_continuation.py optimization_engine/study/continuation.py mv optimization_engine/benchmarking_substudy.py optimization_engine/study/benchmarking.py mv optimization_engine/generate_history_from_trials.py optimization_engine/study/history_generator.py # ============================================================================= # STEP 2.6: REPORTING # ============================================================================= mv optimization_engine/generate_report.py optimization_engine/reporting/report_generator.py mv optimization_engine/generate_report_markdown.py optimization_engine/reporting/markdown_report.py mv optimization_engine/comprehensive_results_analyzer.py optimization_engine/reporting/results_analyzer.py mv optimization_engine/visualizer.py optimization_engine/reporting/ mv optimization_engine/landscape_analyzer.py optimization_engine/reporting/ # ============================================================================= # STEP 2.7: CORE (depends on many things, do after others) # ============================================================================= mv optimization_engine/runner.py optimization_engine/core/ mv optimization_engine/base_runner.py optimization_engine/core/ mv optimization_engine/runner_with_neural.py optimization_engine/core/ mv optimization_engine/intelligent_optimizer.py optimization_engine/core/ mv optimization_engine/method_selector.py optimization_engine/core/ mv optimization_engine/strategy_selector.py optimization_engine/core/ mv optimization_engine/strategy_portfolio.py optimization_engine/core/ mv optimization_engine/gradient_optimizer.py optimization_engine/core/ # ============================================================================= # STEP 2.8: FUTURE/RESEARCH (experimental) # ============================================================================= mv optimization_engine/research_agent.py optimization_engine/future/ mv optimization_engine/pynastran_research_agent.py optimization_engine/future/ mv optimization_engine/targeted_research_planner.py optimization_engine/future/ mv optimization_engine/workflow_decomposer.py optimization_engine/future/ mv optimization_engine/step_classifier.py optimization_engine/future/ # Check if these exist before moving: test -f optimization_engine/llm_optimization_runner.py && mv optimization_engine/llm_optimization_runner.py optimization_engine/future/ test -f optimization_engine/llm_workflow_analyzer.py && mv optimization_engine/llm_workflow_analyzer.py optimization_engine/future/ test -f optimization_engine/extractor_orchestrator.py && mv optimization_engine/extractor_orchestrator.py optimization_engine/future/ test -f optimization_engine/inline_code_generator.py && mv optimization_engine/inline_code_generator.py optimization_engine/future/ test -f optimization_engine/hook_generator.py && mv optimization_engine/hook_generator.py optimization_engine/future/ # ============================================================================= # STEP 2.9: MOVE TO EXISTING DIRECTORIES # ============================================================================= mv optimization_engine/op2_extractor.py optimization_engine/extractors/ mv optimization_engine/extractor_library.py optimization_engine/extractors/ mv optimization_engine/simulation_validator.py optimization_engine/validators/ mv optimization_engine/adaptive_characterization.py optimization_engine/processors/ ``` ### Phase 3: Update Imports with Automation Script (2-3 hours) Create and run a Python migration script: ```python #!/usr/bin/env python3 """ optimization_engine Migration Script ===================================== Automatically updates all imports across the codebase. Usage: python migrate_imports.py --dry-run # Preview changes python migrate_imports.py --execute # Apply changes """ import os import re import sys from pathlib import Path from typing import Dict, List, Tuple # Import mappings (old -> new) IMPORT_MAPPINGS = { # Core r'from optimization_engine\.runner\b': 'from optimization_engine.core.runner', r'from optimization_engine\.base_runner\b': 'from optimization_engine.core.base_runner', r'from optimization_engine\.runner_with_neural\b': 'from optimization_engine.core.runner_with_neural', r'from optimization_engine\.intelligent_optimizer\b': 'from optimization_engine.core.intelligent_optimizer', r'from optimization_engine\.method_selector\b': 'from optimization_engine.core.method_selector', r'from optimization_engine\.strategy_selector\b': 'from optimization_engine.core.strategy_selector', r'from optimization_engine\.strategy_portfolio\b': 'from optimization_engine.core.strategy_portfolio', r'from optimization_engine\.gradient_optimizer\b': 'from optimization_engine.core.gradient_optimizer', r'import optimization_engine\.runner\b': 'import optimization_engine.core.runner', r'import optimization_engine\.intelligent_optimizer\b': 'import optimization_engine.core.intelligent_optimizer', # Surrogates r'from optimization_engine\.neural_surrogate\b': 'from optimization_engine.processors.surrogates.neural_surrogate', r'from optimization_engine\.generic_surrogate\b': 'from optimization_engine.processors.surrogates.generic_surrogate', r'from optimization_engine\.adaptive_surrogate\b': 'from optimization_engine.processors.surrogates.adaptive_surrogate', r'from optimization_engine\.simple_mlp_surrogate\b': 'from optimization_engine.processors.surrogates.simple_mlp_surrogate', r'from optimization_engine\.active_learning_surrogate\b': 'from optimization_engine.processors.surrogates.active_learning_surrogate', r'from optimization_engine\.surrogate_tuner\b': 'from optimization_engine.processors.surrogates.surrogate_tuner', r'from optimization_engine\.auto_trainer\b': 'from optimization_engine.processors.surrogates.auto_trainer', r'from optimization_engine\.training_data_exporter\b': 'from optimization_engine.processors.surrogates.training_data_exporter', # NX r'from optimization_engine\.nx_solver\b': 'from optimization_engine.nx.solver', r'from optimization_engine\.nx_updater\b': 'from optimization_engine.nx.updater', r'from optimization_engine\.nx_session_manager\b': 'from optimization_engine.nx.session_manager', r'from optimization_engine\.solve_simulation\b': 'from optimization_engine.nx.solve_simulation', r'from optimization_engine\.solve_simulation_simple\b': 'from optimization_engine.nx.solve_simulation_simple', r'from optimization_engine\.model_cleanup\b': 'from optimization_engine.nx.model_cleanup', r'from optimization_engine\.export_expressions\b': 'from optimization_engine.nx.export_expressions', r'from optimization_engine\.import_expressions\b': 'from optimization_engine.nx.import_expressions', r'from optimization_engine\.mesh_converter\b': 'from optimization_engine.nx.mesh_converter', r'import optimization_engine\.nx_solver\b': 'import optimization_engine.nx.solver', # Study r'from optimization_engine\.study_creator\b': 'from optimization_engine.study.creator', r'from optimization_engine\.study_wizard\b': 'from optimization_engine.study.wizard', r'from optimization_engine\.study_state\b': 'from optimization_engine.study.state', r'from optimization_engine\.study_reset\b': 'from optimization_engine.study.reset', r'from optimization_engine\.study_continuation\b': 'from optimization_engine.study.continuation', r'from optimization_engine\.benchmarking_substudy\b': 'from optimization_engine.study.benchmarking', r'from optimization_engine\.generate_history_from_trials\b': 'from optimization_engine.study.history_generator', # Reporting r'from optimization_engine\.generate_report\b': 'from optimization_engine.reporting.report_generator', r'from optimization_engine\.generate_report_markdown\b': 'from optimization_engine.reporting.markdown_report', r'from optimization_engine\.comprehensive_results_analyzer\b': 'from optimization_engine.reporting.results_analyzer', r'from optimization_engine\.visualizer\b': 'from optimization_engine.reporting.visualizer', r'from optimization_engine\.landscape_analyzer\b': 'from optimization_engine.reporting.landscape_analyzer', # Config r'from optimization_engine\.config_manager\b': 'from optimization_engine.config.manager', r'from optimization_engine\.optimization_config_builder\b': 'from optimization_engine.config.builder', r'from optimization_engine\.optimization_setup_wizard\b': 'from optimization_engine.config.setup_wizard', r'from optimization_engine\.capability_matcher\b': 'from optimization_engine.config.capability_matcher', r'from optimization_engine\.template_loader\b': 'from optimization_engine.config.template_loader', # Utils r'from optimization_engine\.logger\b': 'from optimization_engine.utils.logger', r'from optimization_engine\.auto_doc\b': 'from optimization_engine.utils.auto_doc', r'from optimization_engine\.realtime_tracking\b': 'from optimization_engine.utils.realtime_tracking', r'from optimization_engine\.codebase_analyzer\b': 'from optimization_engine.utils.codebase_analyzer', r'from optimization_engine\.pruning_logger\b': 'from optimization_engine.utils.pruning_logger', # Future r'from optimization_engine\.research_agent\b': 'from optimization_engine.future.research_agent', r'from optimization_engine\.pynastran_research_agent\b': 'from optimization_engine.future.pynastran_research_agent', r'from optimization_engine\.targeted_research_planner\b': 'from optimization_engine.future.targeted_research_planner', r'from optimization_engine\.workflow_decomposer\b': 'from optimization_engine.future.workflow_decomposer', r'from optimization_engine\.step_classifier\b': 'from optimization_engine.future.step_classifier', r'from optimization_engine\.llm_optimization_runner\b': 'from optimization_engine.future.llm_optimization_runner', r'from optimization_engine\.llm_workflow_analyzer\b': 'from optimization_engine.future.llm_workflow_analyzer', # Extractors/Validators additions r'from optimization_engine\.op2_extractor\b': 'from optimization_engine.extractors.op2_extractor', r'from optimization_engine\.extractor_library\b': 'from optimization_engine.extractors.extractor_library', r'from optimization_engine\.simulation_validator\b': 'from optimization_engine.validators.simulation_validator', # Processors r'from optimization_engine\.adaptive_characterization\b': 'from optimization_engine.processors.adaptive_characterization', } # Path mappings for documentation PATH_MAPPINGS = { 'optimization_engine/runner.py': 'optimization_engine/core/runner.py', 'optimization_engine/base_runner.py': 'optimization_engine/core/base_runner.py', 'optimization_engine/intelligent_optimizer.py': 'optimization_engine/core/intelligent_optimizer.py', 'optimization_engine/nx_solver.py': 'optimization_engine/nx/solver.py', 'optimization_engine/nx_updater.py': 'optimization_engine/nx/updater.py', 'optimization_engine/study_creator.py': 'optimization_engine/study/creator.py', 'optimization_engine/study_wizard.py': 'optimization_engine/study/wizard.py', 'optimization_engine/neural_surrogate.py': 'optimization_engine/processors/surrogates/neural_surrogate.py', 'optimization_engine/config_manager.py': 'optimization_engine/config/manager.py', 'optimization_engine/logger.py': 'optimization_engine/utils/logger.py', # Add all other path mappings... } def find_files(root: Path, extensions: List[str]) -> List[Path]: """Find all files with given extensions.""" files = [] for ext in extensions: files.extend(root.rglob(f'*{ext}')) return files def update_file(filepath: Path, mappings: Dict[str, str], dry_run: bool = True) -> Tuple[int, List[str]]: """Update imports in a single file.""" try: content = filepath.read_text(encoding='utf-8', errors='ignore') except Exception as e: print(f" ERROR reading {filepath}: {e}") return 0, [] changes = [] new_content = content for pattern, replacement in mappings.items(): matches = re.findall(pattern, content) if matches: new_content = re.sub(pattern, replacement, new_content) changes.append(f" {pattern} → {replacement} ({len(matches)} occurrences)") if changes and not dry_run: filepath.write_text(new_content, encoding='utf-8') return len(changes), changes def main(): dry_run = '--dry-run' in sys.argv or '--execute' not in sys.argv if dry_run: print("=" * 60) print("DRY RUN MODE - No files will be modified") print("=" * 60) else: print("=" * 60) print("EXECUTE MODE - Files will be modified!") print("=" * 60) confirm = input("Are you sure? (yes/no): ") if confirm.lower() != 'yes': print("Aborted.") return root = Path('.') # Find all Python files py_files = find_files(root, ['.py']) py_files = [f for f in py_files if 'optimization_engine_BACKUP' not in str(f)] py_files = [f for f in py_files if '.venv' not in str(f)] py_files = [f for f in py_files if 'node_modules' not in str(f)] print(f"\nFound {len(py_files)} Python files to check") total_changes = 0 files_changed = 0 for filepath in sorted(py_files): count, changes = update_file(filepath, IMPORT_MAPPINGS, dry_run) if count > 0: files_changed += 1 total_changes += count print(f"\n{filepath} ({count} changes):") for change in changes: print(change) # Find all Markdown files md_files = find_files(root, ['.md']) md_files = [f for f in md_files if 'optimization_engine_BACKUP' not in str(f)] md_files = [f for f in md_files if 'node_modules' not in str(f)] print(f"\n\nFound {len(md_files)} Markdown files to check") for filepath in sorted(md_files): count, changes = update_file(filepath, PATH_MAPPINGS, dry_run) if count > 0: files_changed += 1 total_changes += count print(f"\n{filepath} ({count} changes):") for change in changes[:5]: # Limit output print(change) if len(changes) > 5: print(f" ... and {len(changes) - 5} more") print("\n" + "=" * 60) print(f"SUMMARY: {total_changes} changes in {files_changed} files") print("=" * 60) if dry_run: print("\nTo apply changes, run: python migrate_imports.py --execute") if __name__ == '__main__': main() ``` ### Phase 4: Create __init__.py Files (1 hour) Create proper `__init__.py` for each new module. See separate section below. ### Phase 5: Update feature_registry.json (30 minutes) Use a dedicated script to update all path references in the JSON file. ### Phase 6: Update Documentation (2 hours) Run sed/grep commands for bulk markdown updates, then manual review. ### Phase 7: Testing & Validation (2 hours) ```bash # 1. Test Python imports python -c "from optimization_engine.core.runner import OptimizationRunner; print('core OK')" python -c "from optimization_engine.nx.solver import NXSolver; print('nx OK')" python -c "from optimization_engine.study.creator import StudyCreator; print('study OK')" python -c "from optimization_engine.processors.surrogates.neural_surrogate import NeuralSurrogate; print('surrogates OK')" python -c "from optimization_engine.config.manager import ConfigManager; print('config OK')" python -c "from optimization_engine.utils.logger import get_logger; print('utils OK')" # 2. Test backwards compatibility python -c "from optimization_engine import nx_solver; print('compat OK')" # 3. Run full test suite python -m pytest tests/ -v --tb=short 2>&1 | tee post_migration_tests.log # 4. Compare test results diff baseline_tests_*.log post_migration_tests.log # 5. Test a study cd studies/bracket_displacement_maximizing python run_optimization.py --trials 1 --dry-run # 6. Test dashboard cd atomizer-dashboard/backend python -c "from api.routes.optimization import *; print('dashboard OK')" ``` ### Phase 8: Commit & Cleanup (30 minutes) ```bash # Remove backup after successful testing rm -rf optimization_engine_BACKUP_* # Clear Python cache find . -name "*.pyc" -delete find . -name "__pycache__" -type d -exec rm -rf {} + 2>/dev/null || true # Final commit git add -A git commit -m "refactor: Major reorganization of optimization_engine BREAKING CHANGE: Module structure reorganized for better maintainability. New structure: - core/ - Optimization runners (runner, intelligent_optimizer, etc.) - processors/ - Data processing (surrogates/, dynamic_response/) - nx/ - NX/Nastran integration (solver, updater, etc.) - study/ - Study management (creator, wizard, state, etc.) - reporting/ - Reports and analysis - config/ - Configuration management - utils/ - Utilities (logger, etc.) - future/ - Research/experimental code Migration: - 500+ files updated - ~1,400 import changes - Backwards compatibility aliases provided with deprecation warnings - All tests passing 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude " # Push git push origin refactor/optimization-engine-v2 ``` --- ## Part 4: __init__.py Files Content ### 4.1 optimization_engine/core/__init__.py ```python """ Optimization Engine Core ======================== Main optimization runners and algorithm selection. Modules: - runner: Main OptimizationRunner class - base_runner: BaseRunner abstract class - intelligent_optimizer: IMSO adaptive optimizer - method_selector: Algorithm selection logic - strategy_selector: Strategy portfolio management """ from .runner import OptimizationRunner from .base_runner import BaseRunner from .runner_with_neural import NeuralOptimizationRunner from .intelligent_optimizer import IntelligentOptimizer, IMSO from .method_selector import MethodSelector, select_method from .strategy_selector import StrategySelector from .strategy_portfolio import StrategyPortfolio from .gradient_optimizer import GradientOptimizer, LBFGSPolisher __all__ = [ 'OptimizationRunner', 'BaseRunner', 'NeuralOptimizationRunner', 'IntelligentOptimizer', 'IMSO', 'MethodSelector', 'select_method', 'StrategySelector', 'StrategyPortfolio', 'GradientOptimizer', 'LBFGSPolisher', ] ``` ### 4.2 optimization_engine/processors/__init__.py ```python """ Optimization Processors ======================= Data processing algorithms and ML models. Submodules: - surrogates/: Neural network surrogate models - dynamic_response/: Dynamic response processing (random vib, sine sweep) """ from . import surrogates __all__ = [ 'surrogates', ] ``` ### 4.3 optimization_engine/processors/surrogates/__init__.py ```python """ Surrogate Models ================ Neural network and ML surrogate models for FEA acceleration. Classes: - NeuralSurrogate: Main neural network surrogate - GenericSurrogate: Flexible surrogate interface - AdaptiveSurrogate: Self-improving surrogate - SimpleMLP: Simple multi-layer perceptron - ActiveLearningSurrogate: Active learning surrogate - SurrogateTuner: Hyperparameter tuning """ from .neural_surrogate import NeuralSurrogate, create_surrogate from .generic_surrogate import GenericSurrogate from .adaptive_surrogate import AdaptiveSurrogate from .simple_mlp_surrogate import SimpleMLP, SimpleMLPSurrogate from .active_learning_surrogate import ActiveLearningSurrogate from .surrogate_tuner import SurrogateTuner, tune_surrogate from .auto_trainer import AutoTrainer from .training_data_exporter import TrainingDataExporter, export_training_data __all__ = [ 'NeuralSurrogate', 'create_surrogate', 'GenericSurrogate', 'AdaptiveSurrogate', 'SimpleMLP', 'SimpleMLPSurrogate', 'ActiveLearningSurrogate', 'SurrogateTuner', 'tune_surrogate', 'AutoTrainer', 'TrainingDataExporter', 'export_training_data', ] ``` ### 4.4 optimization_engine/nx/__init__.py ```python """ NX Integration ============== Siemens NX and Nastran integration modules. Modules: - solver: NXSolver for running simulations - updater: NXParameterUpdater for design updates - session_manager: NX session lifecycle management - solve_simulation: Low-level simulation execution """ from .solver import NXSolver, run_nx_simulation from .updater import NXParameterUpdater, update_parameters from .session_manager import NXSessionManager, get_session from .solve_simulation import solve_simulation, SolveSimulationError from .model_cleanup import cleanup_model, ModelCleanup __all__ = [ 'NXSolver', 'run_nx_simulation', 'NXParameterUpdater', 'update_parameters', 'NXSessionManager', 'get_session', 'solve_simulation', 'SolveSimulationError', 'cleanup_model', 'ModelCleanup', ] ``` ### 4.5 optimization_engine/study/__init__.py ```python """ Study Management ================ Study creation, state management, and lifecycle. Modules: - creator: Study creation from templates - wizard: Interactive study setup wizard - state: Study state tracking - reset: Study reset functionality - continuation: Resume interrupted studies """ from .creator import StudyCreator, create_study from .wizard import StudyWizard, run_wizard from .state import StudyState, get_study_state from .reset import reset_study, StudyReset from .continuation import continue_study, StudyContinuation from .benchmarking import BenchmarkingSubstudy from .history_generator import generate_history __all__ = [ 'StudyCreator', 'create_study', 'StudyWizard', 'run_wizard', 'StudyState', 'get_study_state', 'reset_study', 'StudyReset', 'continue_study', 'StudyContinuation', 'BenchmarkingSubstudy', 'generate_history', ] ``` ### 4.6 optimization_engine/reporting/__init__.py ```python """ Reporting & Analysis ==================== Report generation and results analysis. Modules: - report_generator: HTML/PDF report generation - markdown_report: Markdown report format - results_analyzer: Comprehensive results analysis - visualizer: Plotting and visualization - landscape_analyzer: Design space analysis """ from .report_generator import generate_report, ReportGenerator from .markdown_report import generate_markdown_report, MarkdownReporter from .results_analyzer import ResultsAnalyzer, analyze_results from .visualizer import Visualizer, plot_results from .landscape_analyzer import LandscapeAnalyzer, analyze_landscape __all__ = [ 'generate_report', 'ReportGenerator', 'generate_markdown_report', 'MarkdownReporter', 'ResultsAnalyzer', 'analyze_results', 'Visualizer', 'plot_results', 'LandscapeAnalyzer', 'analyze_landscape', ] ``` ### 4.7 optimization_engine/config/__init__.py ```python """ Configuration Management ======================== Configuration loading, validation, and building. Modules: - manager: ConfigManager for loading/saving configs - builder: OptimizationConfigBuilder for creating configs - setup_wizard: Interactive configuration setup - capability_matcher: Match capabilities to requirements """ from .manager import ConfigManager, load_config, save_config from .builder import OptimizationConfigBuilder, build_config from .setup_wizard import SetupWizard, run_setup from .capability_matcher import CapabilityMatcher, match_capabilities from .template_loader import TemplateLoader, load_template __all__ = [ 'ConfigManager', 'load_config', 'save_config', 'OptimizationConfigBuilder', 'build_config', 'SetupWizard', 'run_setup', 'CapabilityMatcher', 'match_capabilities', 'TemplateLoader', 'load_template', ] ``` ### 4.8 optimization_engine/__init__.py (Updated with Backwards Compat) ```python """ Atomizer Optimization Engine ============================ Structural optimization framework for Siemens NX. New Module Structure (v2.0): - core/ - Optimization runners - processors/ - Data processing (surrogates, dynamic_response) - nx/ - NX/Nastran integration - study/ - Study management - reporting/ - Reports and analysis - config/ - Configuration - extractors/ - Physics extraction (unchanged) - insights/ - Visualizations (unchanged) - gnn/ - Graph neural networks (unchanged) - hooks/ - NX hooks (unchanged) - utils/ - Utilities - validators/ - Validation (unchanged) Quick Start: from optimization_engine.core import OptimizationRunner from optimization_engine.nx import NXSolver from optimization_engine.extractors import extract_displacement """ __version__ = '2.0.0' # Re-export commonly used items at top level from optimization_engine.core.runner import OptimizationRunner from optimization_engine.core.intelligent_optimizer import IMSO, IntelligentOptimizer from optimization_engine.nx.solver import NXSolver, run_nx_simulation from optimization_engine.study.creator import create_study from optimization_engine.config.manager import ConfigManager # Submodule access from optimization_engine import ( core, processors, nx, study, reporting, config, extractors, insights, gnn, hooks, utils, validators, ) __all__ = [ # Version '__version__', # Top-level exports 'OptimizationRunner', 'IMSO', 'IntelligentOptimizer', 'NXSolver', 'run_nx_simulation', 'create_study', 'ConfigManager', # Submodules 'core', 'processors', 'nx', 'study', 'reporting', 'config', 'extractors', 'insights', 'gnn', 'hooks', 'utils', 'validators', ] # ============================================================================= # BACKWARDS COMPATIBILITY LAYER # ============================================================================= # These aliases allow old imports to work with deprecation warnings. # Will be removed in v3.0. import warnings as _warnings import importlib as _importlib _DEPRECATED_MAPPINGS = { # Core 'runner': 'optimization_engine.core.runner', 'base_runner': 'optimization_engine.core.base_runner', 'intelligent_optimizer': 'optimization_engine.core.intelligent_optimizer', 'method_selector': 'optimization_engine.core.method_selector', 'strategy_selector': 'optimization_engine.core.strategy_selector', 'strategy_portfolio': 'optimization_engine.core.strategy_portfolio', 'gradient_optimizer': 'optimization_engine.core.gradient_optimizer', # Surrogates 'neural_surrogate': 'optimization_engine.processors.surrogates.neural_surrogate', 'generic_surrogate': 'optimization_engine.processors.surrogates.generic_surrogate', 'adaptive_surrogate': 'optimization_engine.processors.surrogates.adaptive_surrogate', 'simple_mlp_surrogate': 'optimization_engine.processors.surrogates.simple_mlp_surrogate', 'active_learning_surrogate': 'optimization_engine.processors.surrogates.active_learning_surrogate', 'surrogate_tuner': 'optimization_engine.processors.surrogates.surrogate_tuner', # NX 'nx_solver': 'optimization_engine.nx.solver', 'nx_updater': 'optimization_engine.nx.updater', 'nx_session_manager': 'optimization_engine.nx.session_manager', 'solve_simulation': 'optimization_engine.nx.solve_simulation', 'model_cleanup': 'optimization_engine.nx.model_cleanup', # Study 'study_creator': 'optimization_engine.study.creator', 'study_wizard': 'optimization_engine.study.wizard', 'study_state': 'optimization_engine.study.state', 'study_reset': 'optimization_engine.study.reset', 'study_continuation': 'optimization_engine.study.continuation', # Reporting 'generate_report': 'optimization_engine.reporting.report_generator', 'generate_report_markdown': 'optimization_engine.reporting.markdown_report', 'visualizer': 'optimization_engine.reporting.visualizer', # Config 'config_manager': 'optimization_engine.config.manager', 'optimization_config_builder': 'optimization_engine.config.builder', 'optimization_setup_wizard': 'optimization_engine.config.setup_wizard', 'capability_matcher': 'optimization_engine.config.capability_matcher', # Utils 'logger': 'optimization_engine.utils.logger', } def __getattr__(name): """Provide backwards compatibility with deprecation warnings.""" if name in _DEPRECATED_MAPPINGS: new_module = _DEPRECATED_MAPPINGS[name] _warnings.warn( f"Importing '{name}' from optimization_engine is deprecated. " f"Use '{new_module}' instead. " f"This will be removed in v3.0.", DeprecationWarning, stacklevel=2 ) return _importlib.import_module(new_module) raise AttributeError(f"module 'optimization_engine' has no attribute '{name}'") ``` --- ## Part 5: Post-Migration Checklist ### Immediate (same day) - [ ] All imports updated in optimization_engine/ - [ ] All imports updated in studies/ - [ ] All imports updated in tests/ - [ ] All imports updated in tools/ - [ ] All imports updated in dashboard backend - [ ] feature_registry.json updated - [ ] All __init__.py files created - [ ] Basic import tests pass - [ ] Test suite passes (or known failures documented) ### Short-term (within 1 week) - [ ] Update CLAUDE.md with new structure - [ ] Update all protocol files (SYS_10-16, OP_01-06, EXT_01-04) - [ ] Update all skill files in .claude/skills/ - [ ] Update ATOMIZER_CONTEXT.md - [ ] Update README.md - [ ] Record migration in LAC knowledge base - [ ] Run 2-3 studies end-to-end to verify ### Medium-term (within 1 month) - [ ] Remove backwards compatibility after team confirms no issues - [ ] Update any external documentation/wikis - [ ] Update pyproject.toml metadata if needed - [ ] Archive the migration plan document --- ## Part 6: Rollback Procedure If critical issues are discovered: ```bash # Option 1: Git revert (preferred if committed) git revert HEAD # Option 2: Reset to backup branch git checkout backup/pre-reorganization git checkout -b main-restored git branch -D main git branch -m main-restored main # Option 3: Restore from backup directory rm -rf optimization_engine mv optimization_engine_BACKUP_* optimization_engine ``` --- ## Part 7: Time Summary | Phase | Task | Time | |-------|------|------| | 0 | Preparation (backup, baseline) | 30 min | | 1 | Create directory structure | 15 min | | 2 | Move files | 45 min | | 3 | Update imports (with script) | 2-3 hours | | 4 | Create __init__.py files | 1 hour | | 5 | Update feature_registry.json | 30 min | | 6 | Update documentation | 2 hours | | 7 | Testing & validation | 2 hours | | 8 | Commit & cleanup | 30 min | | **TOTAL** | | **10-12 hours** | --- ## Appendix: Quick Reference Card ### New Import Patterns ```python # OLD # NEW from optimization_engine.runner from optimization_engine.core.runner from optimization_engine.nx_solver from optimization_engine.nx.solver from optimization_engine.study_creator from optimization_engine.study.creator from optimization_engine.neural_surrogate from optimization_engine.processors.surrogates.neural_surrogate from optimization_engine.config_manager from optimization_engine.config.manager from optimization_engine.logger from optimization_engine.utils.logger ``` ### Directory Quick Reference ``` optimization_engine/ ├── core/ → runner, intelligent_optimizer, method_selector ├── processors/ │ └── surrogates/ → neural_surrogate, generic_surrogate, etc. ├── nx/ → solver (was nx_solver), updater, session_manager ├── study/ → creator (was study_creator), wizard, state ├── reporting/ → report_generator, visualizer ├── config/ → manager (was config_manager), builder ├── utils/ → logger, auto_doc ├── future/ → research_agent, workflow_decomposer ├── extractors/ → (unchanged) ├── insights/ → (unchanged) ├── gnn/ → (unchanged) ├── hooks/ → (unchanged) └── validators/ → (unchanged) ``` --- ## Part 8: Handoff to Context Engineering Plan After this migration is complete, proceed with `ATOMIZER_CONTEXT_ENGINEERING_PLAN.md`. ### What Context Engineering Will Add ``` optimization_engine/ ├── context/ ← NEW (added by Context Engineering) │ ├── __init__.py │ ├── playbook.py # AtomizerPlaybook - structured knowledge storage │ ├── reflector.py # Analyzes optimization outcomes │ ├── session_state.py # Context isolation for LLM sessions │ ├── compaction.py # Context compaction for long sessions │ ├── feedback_loop.py # Learning from execution outcomes │ └── cache_monitor.py # KV-cache efficiency tracking ├── core/ ← Already created by Migration │ └── runner.py # Will be modified to integrate context └── ... ``` ### Context Engineering Integration Points After migration, Context Engineering will modify: 1. **`optimization_engine/core/runner.py`** - Add playbook and reflector integration 2. **`.claude/skills/00_BOOTSTRAP.md`** - Update to v2 with playbook loading 3. **`.claude/skills/02_CONTEXT_LOADER.md`** - Add playbook integration 4. **`atomizer-dashboard/backend/api/routes/`** - Add context API routes ### Pre-Migration Checklist for Context Engineering Compatibility Before starting Context Engineering, verify: - [ ] Migration completed successfully - [ ] `optimization_engine/core/runner.py` exists at new location - [ ] All imports in codebase updated to new paths - [ ] Test suite passes - [ ] No deprecation warnings in core workflows ### Context Engineering Will Use These New Paths ```python # Context Engineering code will import from new locations: from optimization_engine.core.runner import OptimizationRunner from optimization_engine.core.intelligent_optimizer import IMSO from optimization_engine.utils.logger import get_logger from optimization_engine.nx.solver import NXSolver # And add new context module: from optimization_engine.context.playbook import AtomizerPlaybook from optimization_engine.context.reflector import AtomizerReflector ``` --- *This plan is DEFINITIVE and covers ALL 500+ files that need updating.* *After completion, proceed to ATOMIZER_CONTEXT_ENGINEERING_PLAN.md*