feat: Major update - Physics docs, Zernike OPD, insights, NX journals, tools
Documentation: - Add docs/06_PHYSICS/ with Zernike fundamentals and OPD method docs - Add docs/guides/CMA-ES_EXPLAINED.md optimization guide - Update CLAUDE.md and ATOMIZER_CONTEXT.md with current architecture - Update OP_01_CREATE_STUDY protocol Planning: - Add DYNAMIC_RESPONSE plans for random vibration/PSD support - Add OPTIMIZATION_ENGINE_MIGRATION_PLAN for code reorganization Insights System: - Update design_space, modal_analysis, stress_field, thermal_field insights - Improve error handling and data validation NX Journals: - Add analyze_wfe_zernike.py for Zernike WFE analysis - Add capture_study_images.py for automated screenshots - Add extract_expressions.py and introspect_part.py utilities - Add user_generated_journals/journal_top_view_image_taking.py Tests & Tools: - Add comprehensive Zernike OPD test suite - Add audit_v10 tests for WFE validation - Add tools for Pareto graphs and mirror data extraction - Add migrate_studies_to_topics.py utility Knowledge Base: - Initialize LAC (Learning Atomizer Core) with failure/success patterns Dashboard: - Update Setup.tsx and launch_dashboard.py - Add restart-dev.bat helper script 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
1084
.claude/skills/modules/DYNAMIC_RESPONSE_IMPLEMENTATION_PLAN.md
Normal file
1084
.claude/skills/modules/DYNAMIC_RESPONSE_IMPLEMENTATION_PLAN.md
Normal file
File diff suppressed because it is too large
Load Diff
1649
.claude/skills/modules/DYNAMIC_RESPONSE_MASTER_PLAN.md
Normal file
1649
.claude/skills/modules/DYNAMIC_RESPONSE_MASTER_PLAN.md
Normal file
File diff suppressed because it is too large
Load Diff
932
.claude/skills/modules/OPTIMIZATION_ENGINE_MIGRATION_PLAN.md
Normal file
932
.claude/skills/modules/OPTIMIZATION_ENGINE_MIGRATION_PLAN.md
Normal file
@@ -0,0 +1,932 @@
|
||||
# Optimization Engine Reorganization - Migration Plan
|
||||
|
||||
## Comprehensive Guide for Safe Codebase Restructuring
|
||||
|
||||
**Document Version**: 1.0
|
||||
**Created**: 2025-12-23
|
||||
**Status**: PLANNING - Do Not Execute Without Review
|
||||
**Risk Level**: HIGH - Affects 557+ locations across 276 files
|
||||
|
||||
---
|
||||
|
||||
## Executive Summary
|
||||
|
||||
This document provides a complete migration plan for reorganizing `optimization_engine/` from 50+ loose files to a clean modular structure. This is a **high-impact refactoring** that requires careful execution.
|
||||
|
||||
### Impact Summary
|
||||
|
||||
| Category | Files Affected | Lines to Change |
|
||||
|----------|----------------|-----------------|
|
||||
| Python imports (internal) | 90+ files | ~145 changes |
|
||||
| Python imports (studies) | 30+ folders | ~153 changes |
|
||||
| Python imports (tests) | 30+ files | ~79 changes |
|
||||
| Python imports (dashboard) | 3 files | ~11 changes |
|
||||
| Documentation (protocols, skills) | 119 files | ~200 changes |
|
||||
| JSON configs | 4 files | ~50 changes |
|
||||
| **TOTAL** | **276 files** | **~640 changes** |
|
||||
|
||||
---
|
||||
|
||||
## Part 1: Current State Analysis
|
||||
|
||||
### 1.1 Top-Level Files Requiring Migration
|
||||
|
||||
These 50+ files at `optimization_engine/` root need to move:
|
||||
|
||||
```
|
||||
SURROGATES (6 files) → processors/surrogates/
|
||||
├── neural_surrogate.py
|
||||
├── generic_surrogate.py
|
||||
├── adaptive_surrogate.py
|
||||
├── simple_mlp_surrogate.py
|
||||
├── active_learning_surrogate.py
|
||||
└── surrogate_tuner.py
|
||||
|
||||
OPTIMIZATION CORE (7 files) → core/
|
||||
├── runner.py
|
||||
├── runner_with_neural.py
|
||||
├── base_runner.py
|
||||
├── intelligent_optimizer.py
|
||||
├── method_selector.py
|
||||
├── strategy_selector.py
|
||||
└── strategy_portfolio.py
|
||||
|
||||
NX INTEGRATION (6 files) → nx/
|
||||
├── nx_solver.py
|
||||
├── nx_updater.py
|
||||
├── nx_session_manager.py
|
||||
├── solve_simulation.py
|
||||
├── solve_simulation_simple.py
|
||||
└── model_cleanup.py
|
||||
|
||||
STUDY MANAGEMENT (5 files) → study/
|
||||
├── study_creator.py
|
||||
├── study_wizard.py
|
||||
├── study_state.py
|
||||
├── study_reset.py
|
||||
└── study_continuation.py
|
||||
|
||||
REPORTING (5 files) → reporting/
|
||||
├── generate_report.py
|
||||
├── generate_report_markdown.py
|
||||
├── comprehensive_results_analyzer.py
|
||||
├── visualizer.py
|
||||
└── landscape_analyzer.py
|
||||
|
||||
CONFIG (4 files) → config/
|
||||
├── config_manager.py
|
||||
├── optimization_config_builder.py
|
||||
├── optimization_setup_wizard.py
|
||||
└── capability_matcher.py
|
||||
|
||||
AGENTS/RESEARCH (5 files) → agents/ or future/
|
||||
├── research_agent.py
|
||||
├── pynastran_research_agent.py
|
||||
├── targeted_research_planner.py
|
||||
├── workflow_decomposer.py
|
||||
└── step_classifier.py
|
||||
|
||||
MISC (remaining ~15 files) - evaluate individually
|
||||
├── logger.py → utils/
|
||||
├── op2_extractor.py → extractors/
|
||||
├── extractor_library.py → extractors/
|
||||
├── export_expressions.py → nx/
|
||||
├── import_expressions.py → nx/
|
||||
├── mesh_converter.py → nx/
|
||||
├── simulation_validator.py → validators/
|
||||
├── auto_doc.py → utils/
|
||||
├── auto_trainer.py → processors/surrogates/
|
||||
├── realtime_tracking.py → utils/
|
||||
├── benchmarking_substudy.py → study/
|
||||
├── codebase_analyzer.py → utils/
|
||||
├── training_data_exporter.py → processors/surrogates/
|
||||
├── pruning_logger.py → utils/
|
||||
├── adaptive_characterization.py → processors/
|
||||
└── generate_history_from_trials.py → study/
|
||||
```
|
||||
|
||||
### 1.2 Well-Organized Directories (Keep As-Is)
|
||||
|
||||
These are already properly organized:
|
||||
|
||||
```
|
||||
extractors/ ✓ 20+ extractors, clean __init__.py
|
||||
insights/ ✓ 8 insight types, registry pattern
|
||||
hooks/ ✓ nx_cad/, nx_cae/ subdirs
|
||||
gnn/ ✓ Neural surrogate for Zernike
|
||||
templates/ ✓ Config templates
|
||||
schemas/ ✓ JSON schemas
|
||||
validators/ ✓ Validation code
|
||||
plugins/ ✓ Hook manager system
|
||||
utils/ ✓ Utility functions
|
||||
custom_functions/ ✓ NX material generator
|
||||
model_discovery/ ✓ Model introspection
|
||||
future/ ✓ Experimental code
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Part 2: Target Structure
|
||||
|
||||
### 2.1 Proposed Final Structure
|
||||
|
||||
```
|
||||
optimization_engine/
|
||||
│
|
||||
├── __init__.py # Updated with backwards-compat aliases
|
||||
│
|
||||
├── core/ # NEW - Optimization engine core
|
||||
│ ├── __init__.py
|
||||
│ ├── runner.py
|
||||
│ ├── base_runner.py
|
||||
│ ├── runner_with_neural.py
|
||||
│ ├── intelligent_optimizer.py
|
||||
│ ├── method_selector.py
|
||||
│ ├── strategy_selector.py
|
||||
│ └── strategy_portfolio.py
|
||||
│
|
||||
├── processors/ # NEW - Data processing & algorithms
|
||||
│ ├── __init__.py
|
||||
│ ├── surrogates/
|
||||
│ │ ├── __init__.py
|
||||
│ │ ├── neural_surrogate.py
|
||||
│ │ ├── generic_surrogate.py
|
||||
│ │ ├── adaptive_surrogate.py
|
||||
│ │ ├── simple_mlp_surrogate.py
|
||||
│ │ ├── active_learning_surrogate.py
|
||||
│ │ ├── surrogate_tuner.py
|
||||
│ │ ├── auto_trainer.py
|
||||
│ │ └── training_data_exporter.py
|
||||
│ │
|
||||
│ └── dynamic_response/ # NEW - From master plan
|
||||
│ ├── __init__.py
|
||||
│ ├── modal_database.py
|
||||
│ ├── transfer_functions.py
|
||||
│ ├── random_vibration.py
|
||||
│ ├── psd_profiles.py
|
||||
│ └── utils/
|
||||
│
|
||||
├── nx/ # NEW - NX/Nastran integration
|
||||
│ ├── __init__.py
|
||||
│ ├── solver.py # Was nx_solver.py
|
||||
│ ├── updater.py # Was nx_updater.py
|
||||
│ ├── session_manager.py # Was nx_session_manager.py
|
||||
│ ├── solve_simulation.py
|
||||
│ ├── solve_simulation_simple.py
|
||||
│ ├── model_cleanup.py
|
||||
│ ├── export_expressions.py
|
||||
│ ├── import_expressions.py
|
||||
│ └── mesh_converter.py
|
||||
│
|
||||
├── study/ # NEW - Study management
|
||||
│ ├── __init__.py
|
||||
│ ├── creator.py # Was study_creator.py
|
||||
│ ├── wizard.py # Was study_wizard.py
|
||||
│ ├── state.py # Was study_state.py
|
||||
│ ├── reset.py # Was study_reset.py
|
||||
│ ├── continuation.py # Was study_continuation.py
|
||||
│ ├── benchmarking.py # Was benchmarking_substudy.py
|
||||
│ └── history_generator.py # Was generate_history_from_trials.py
|
||||
│
|
||||
├── reporting/ # NEW - Reports and analysis
|
||||
│ ├── __init__.py
|
||||
│ ├── report_generator.py
|
||||
│ ├── markdown_report.py
|
||||
│ ├── results_analyzer.py
|
||||
│ ├── visualizer.py
|
||||
│ └── landscape_analyzer.py
|
||||
│
|
||||
├── config/ # NEW - Configuration
|
||||
│ ├── __init__.py
|
||||
│ ├── manager.py
|
||||
│ ├── builder.py
|
||||
│ ├── setup_wizard.py
|
||||
│ └── capability_matcher.py
|
||||
│
|
||||
├── extractors/ # EXISTING - Add op2_extractor, extractor_library
|
||||
├── insights/ # EXISTING
|
||||
├── hooks/ # EXISTING
|
||||
├── gnn/ # EXISTING
|
||||
├── templates/ # EXISTING
|
||||
├── schemas/ # EXISTING
|
||||
├── validators/ # EXISTING - Add simulation_validator
|
||||
├── plugins/ # EXISTING
|
||||
├── utils/ # EXISTING - Add logger, auto_doc, etc.
|
||||
├── custom_functions/ # EXISTING
|
||||
├── model_discovery/ # EXISTING
|
||||
└── future/ # EXISTING - Move agents here
|
||||
├── research_agent.py
|
||||
├── pynastran_research_agent.py
|
||||
├── targeted_research_planner.py
|
||||
├── workflow_decomposer.py
|
||||
└── step_classifier.py
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Part 3: Import Mapping Tables
|
||||
|
||||
### 3.1 Old → New Import Mapping
|
||||
|
||||
This is the critical reference for all updates:
|
||||
|
||||
```python
|
||||
# CORE OPTIMIZATION
|
||||
"from optimization_engine.runner" → "from optimization_engine.core.runner"
|
||||
"from optimization_engine.base_runner" → "from optimization_engine.core.base_runner"
|
||||
"from optimization_engine.runner_with_neural" → "from optimization_engine.core.runner_with_neural"
|
||||
"from optimization_engine.intelligent_optimizer" → "from optimization_engine.core.intelligent_optimizer"
|
||||
"from optimization_engine.method_selector" → "from optimization_engine.core.method_selector"
|
||||
"from optimization_engine.strategy_selector" → "from optimization_engine.core.strategy_selector"
|
||||
"from optimization_engine.strategy_portfolio" → "from optimization_engine.core.strategy_portfolio"
|
||||
|
||||
# SURROGATES
|
||||
"from optimization_engine.neural_surrogate" → "from optimization_engine.processors.surrogates.neural_surrogate"
|
||||
"from optimization_engine.generic_surrogate" → "from optimization_engine.processors.surrogates.generic_surrogate"
|
||||
"from optimization_engine.adaptive_surrogate" → "from optimization_engine.processors.surrogates.adaptive_surrogate"
|
||||
"from optimization_engine.simple_mlp_surrogate" → "from optimization_engine.processors.surrogates.simple_mlp_surrogate"
|
||||
"from optimization_engine.active_learning_surrogate" → "from optimization_engine.processors.surrogates.active_learning_surrogate"
|
||||
"from optimization_engine.surrogate_tuner" → "from optimization_engine.processors.surrogates.surrogate_tuner"
|
||||
|
||||
# NX INTEGRATION
|
||||
"from optimization_engine.nx_solver" → "from optimization_engine.nx.solver"
|
||||
"from optimization_engine.nx_updater" → "from optimization_engine.nx.updater"
|
||||
"from optimization_engine.nx_session_manager" → "from optimization_engine.nx.session_manager"
|
||||
"from optimization_engine.solve_simulation" → "from optimization_engine.nx.solve_simulation"
|
||||
"from optimization_engine.model_cleanup" → "from optimization_engine.nx.model_cleanup"
|
||||
"from optimization_engine.export_expressions" → "from optimization_engine.nx.export_expressions"
|
||||
"from optimization_engine.import_expressions" → "from optimization_engine.nx.import_expressions"
|
||||
"from optimization_engine.mesh_converter" → "from optimization_engine.nx.mesh_converter"
|
||||
|
||||
# STUDY MANAGEMENT
|
||||
"from optimization_engine.study_creator" → "from optimization_engine.study.creator"
|
||||
"from optimization_engine.study_wizard" → "from optimization_engine.study.wizard"
|
||||
"from optimization_engine.study_state" → "from optimization_engine.study.state"
|
||||
"from optimization_engine.study_reset" → "from optimization_engine.study.reset"
|
||||
"from optimization_engine.study_continuation" → "from optimization_engine.study.continuation"
|
||||
|
||||
# REPORTING
|
||||
"from optimization_engine.generate_report" → "from optimization_engine.reporting.report_generator"
|
||||
"from optimization_engine.generate_report_markdown" → "from optimization_engine.reporting.markdown_report"
|
||||
"from optimization_engine.comprehensive_results" → "from optimization_engine.reporting.results_analyzer"
|
||||
"from optimization_engine.visualizer" → "from optimization_engine.reporting.visualizer"
|
||||
"from optimization_engine.landscape_analyzer" → "from optimization_engine.reporting.landscape_analyzer"
|
||||
|
||||
# CONFIG
|
||||
"from optimization_engine.config_manager" → "from optimization_engine.config.manager"
|
||||
"from optimization_engine.optimization_config_builder" → "from optimization_engine.config.builder"
|
||||
"from optimization_engine.optimization_setup_wizard" → "from optimization_engine.config.setup_wizard"
|
||||
"from optimization_engine.capability_matcher" → "from optimization_engine.config.capability_matcher"
|
||||
|
||||
# UTILITIES (moving to utils/)
|
||||
"from optimization_engine.logger" → "from optimization_engine.utils.logger"
|
||||
"from optimization_engine.auto_doc" → "from optimization_engine.utils.auto_doc"
|
||||
"from optimization_engine.realtime_tracking" → "from optimization_engine.utils.realtime_tracking"
|
||||
"from optimization_engine.codebase_analyzer" → "from optimization_engine.utils.codebase_analyzer"
|
||||
"from optimization_engine.pruning_logger" → "from optimization_engine.utils.pruning_logger"
|
||||
|
||||
# RESEARCH/AGENTS (moving to future/)
|
||||
"from optimization_engine.research_agent" → "from optimization_engine.future.research_agent"
|
||||
"from optimization_engine.workflow_decomposer" → "from optimization_engine.future.workflow_decomposer"
|
||||
"from optimization_engine.step_classifier" → "from optimization_engine.future.step_classifier"
|
||||
```
|
||||
|
||||
### 3.2 Backwards Compatibility Aliases
|
||||
|
||||
Add to `optimization_engine/__init__.py` for transition period:
|
||||
|
||||
```python
|
||||
# BACKWARDS COMPATIBILITY ALIASES
|
||||
# These allow old imports to work during migration period
|
||||
# Remove after all code is updated
|
||||
|
||||
# Core
|
||||
from optimization_engine.core.runner import *
|
||||
from optimization_engine.core.base_runner import *
|
||||
from optimization_engine.core.intelligent_optimizer import *
|
||||
|
||||
# NX
|
||||
from optimization_engine.nx import solver as nx_solver
|
||||
from optimization_engine.nx import updater as nx_updater
|
||||
from optimization_engine.nx import session_manager as nx_session_manager
|
||||
from optimization_engine.nx import solve_simulation
|
||||
|
||||
# Study
|
||||
from optimization_engine.study import creator as study_creator
|
||||
from optimization_engine.study import wizard as study_wizard
|
||||
from optimization_engine.study import state as study_state
|
||||
|
||||
# Surrogates
|
||||
from optimization_engine.processors.surrogates import neural_surrogate
|
||||
from optimization_engine.processors.surrogates import generic_surrogate
|
||||
|
||||
# Config
|
||||
from optimization_engine.config import manager as config_manager
|
||||
|
||||
# Utils
|
||||
from optimization_engine.utils import logger
|
||||
|
||||
# Deprecation warnings (optional)
|
||||
import warnings
|
||||
def __getattr__(name):
|
||||
deprecated = {
|
||||
'nx_solver': 'optimization_engine.nx.solver',
|
||||
'study_creator': 'optimization_engine.study.creator',
|
||||
# ... more mappings
|
||||
}
|
||||
if name in deprecated:
|
||||
warnings.warn(
|
||||
f"Importing {name} from optimization_engine is deprecated. "
|
||||
f"Use {deprecated[name]} instead.",
|
||||
DeprecationWarning,
|
||||
stacklevel=2
|
||||
)
|
||||
# Return the module anyway for compatibility
|
||||
raise AttributeError(f"module 'optimization_engine' has no attribute '{name}'")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Part 4: Files Requiring Updates
|
||||
|
||||
### 4.1 Internal optimization_engine Files (90+ files)
|
||||
|
||||
**Highest-impact internal files:**
|
||||
|
||||
| File | Imports to Update | Priority |
|
||||
|------|-------------------|----------|
|
||||
| `runner.py` | nx_solver, config_manager, extractors | Critical |
|
||||
| `base_runner.py` | config_manager, validators | Critical |
|
||||
| `intelligent_optimizer.py` | runner, neural_surrogate, method_selector | Critical |
|
||||
| `runner_with_neural.py` | runner, neural_surrogate | High |
|
||||
| `gnn/gnn_optimizer.py` | polar_graph, zernike_gnn, nx_solver | High |
|
||||
| `hooks/nx_cad/*.py` | nx_session_manager | High |
|
||||
| `plugins/hook_manager.py` | validators, config | Medium |
|
||||
|
||||
### 4.2 Study Scripts (30+ folders, 153 imports)
|
||||
|
||||
**Pattern in every study's `run_optimization.py`:**
|
||||
|
||||
```python
|
||||
# BEFORE
|
||||
from optimization_engine.nx_solver import run_nx_simulation
|
||||
from optimization_engine.extractors import ZernikeExtractor
|
||||
from optimization_engine.gnn.gnn_optimizer import ZernikeGNNOptimizer
|
||||
|
||||
# AFTER
|
||||
from optimization_engine.nx.solver import run_nx_simulation
|
||||
from optimization_engine.extractors import ZernikeExtractor # unchanged
|
||||
from optimization_engine.gnn.gnn_optimizer import ZernikeGNNOptimizer # unchanged
|
||||
```
|
||||
|
||||
**Studies requiring updates:**
|
||||
|
||||
```
|
||||
studies/
|
||||
├── M1_Mirror/
|
||||
│ ├── m1_mirror_adaptive_V*/run_optimization.py (12 files)
|
||||
│ └── [other variants]
|
||||
├── Simple_Bracket/
|
||||
│ ├── bracket_*/run_optimization.py (8 files)
|
||||
├── UAV_Arm/
|
||||
│ ├── */run_optimization.py (4 files)
|
||||
├── Drone_Gimbal/
|
||||
├── Simple_Beam/
|
||||
└── [others]
|
||||
```
|
||||
|
||||
### 4.3 Test Files (30+ files, 79 imports)
|
||||
|
||||
Located in `tests/`:
|
||||
|
||||
```
|
||||
tests/
|
||||
├── test_extractors.py
|
||||
├── test_zernike_*.py (5+ files)
|
||||
├── test_neural_surrogate.py
|
||||
├── test_gnn_*.py
|
||||
├── test_nx_solver.py
|
||||
├── test_study_*.py
|
||||
└── [others]
|
||||
```
|
||||
|
||||
### 4.4 Dashboard Backend (3 files, 11 imports)
|
||||
|
||||
```
|
||||
atomizer-dashboard/backend/api/
|
||||
├── main.py # sys.path setup
|
||||
├── routes/optimization.py # study management imports
|
||||
└── routes/insights.py # insight imports
|
||||
```
|
||||
|
||||
### 4.5 Documentation Files (119 files)
|
||||
|
||||
**Protocols requiring updates:**
|
||||
|
||||
| Protocol | References | Changes Needed |
|
||||
|----------|------------|----------------|
|
||||
| SYS_10_IMSO.md | intelligent_optimizer | core.intelligent_optimizer |
|
||||
| SYS_12_EXTRACTOR_LIBRARY.md | extractors/* | None (unchanged) |
|
||||
| SYS_14_NEURAL_ACCELERATION.md | neural_surrogate, gnn/* | processors.surrogates.* |
|
||||
| SYS_15_METHOD_SELECTOR.md | method_selector | core.method_selector |
|
||||
| OP_01_CREATE_STUDY.md | study_creator, study_wizard | study.creator, study.wizard |
|
||||
| OP_02_RUN_OPTIMIZATION.md | runner, nx_solver | core.runner, nx.solver |
|
||||
|
||||
**Skill files requiring updates:**
|
||||
|
||||
| Skill File | Changes |
|
||||
|------------|---------|
|
||||
| 01_CHEATSHEET.md | Path references |
|
||||
| core/study-creation-core.md | Import examples |
|
||||
| modules/extractors-catalog.md | Directory paths |
|
||||
| modules/neural-acceleration.md | Surrogate imports |
|
||||
|
||||
### 4.6 JSON Configuration Files (4 files)
|
||||
|
||||
**feature_registry.json** (878 lines, 50+ path references):
|
||||
|
||||
```json
|
||||
// BEFORE
|
||||
"file_path": "optimization_engine/neural_surrogate.py"
|
||||
|
||||
// AFTER
|
||||
"file_path": "optimization_engine/processors/surrogates/neural_surrogate.py"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Part 5: Migration Execution Plan
|
||||
|
||||
### Phase 0: Pre-Migration (30 min)
|
||||
|
||||
1. **Create full backup**
|
||||
```bash
|
||||
git stash # Save any uncommitted changes
|
||||
git checkout -b refactor/optimization-engine-reorganization
|
||||
cp -r optimization_engine optimization_engine_backup
|
||||
```
|
||||
|
||||
2. **Run baseline tests**
|
||||
```bash
|
||||
python -m pytest tests/ -v --tb=short > baseline_tests.log 2>&1
|
||||
```
|
||||
|
||||
3. **Document current working state**
|
||||
- Run one study end-to-end
|
||||
- Verify dashboard loads
|
||||
- Note any existing failures
|
||||
|
||||
### Phase 1: Create Directory Structure (15 min)
|
||||
|
||||
```bash
|
||||
# Create new directories
|
||||
mkdir -p optimization_engine/core
|
||||
mkdir -p optimization_engine/processors/surrogates
|
||||
mkdir -p optimization_engine/processors/dynamic_response
|
||||
mkdir -p optimization_engine/nx
|
||||
mkdir -p optimization_engine/study
|
||||
mkdir -p optimization_engine/reporting
|
||||
mkdir -p optimization_engine/config
|
||||
|
||||
# Create __init__.py files
|
||||
touch optimization_engine/core/__init__.py
|
||||
touch optimization_engine/processors/__init__.py
|
||||
touch optimization_engine/processors/surrogates/__init__.py
|
||||
touch optimization_engine/processors/dynamic_response/__init__.py
|
||||
touch optimization_engine/nx/__init__.py
|
||||
touch optimization_engine/study/__init__.py
|
||||
touch optimization_engine/reporting/__init__.py
|
||||
touch optimization_engine/config/__init__.py
|
||||
```
|
||||
|
||||
### Phase 2: Move Files (30 min)
|
||||
|
||||
**Execute in this order to minimize circular import issues:**
|
||||
|
||||
```bash
|
||||
# 1. UTILITIES FIRST (no dependencies)
|
||||
mv optimization_engine/logger.py optimization_engine/utils/
|
||||
mv optimization_engine/auto_doc.py optimization_engine/utils/
|
||||
mv optimization_engine/realtime_tracking.py optimization_engine/utils/
|
||||
mv optimization_engine/codebase_analyzer.py optimization_engine/utils/
|
||||
mv optimization_engine/pruning_logger.py optimization_engine/utils/
|
||||
|
||||
# 2. CONFIG (low dependencies)
|
||||
mv optimization_engine/config_manager.py optimization_engine/config/manager.py
|
||||
mv optimization_engine/optimization_config_builder.py optimization_engine/config/builder.py
|
||||
mv optimization_engine/optimization_setup_wizard.py optimization_engine/config/setup_wizard.py
|
||||
mv optimization_engine/capability_matcher.py optimization_engine/config/capability_matcher.py
|
||||
|
||||
# 3. NX INTEGRATION
|
||||
mv optimization_engine/nx_solver.py optimization_engine/nx/solver.py
|
||||
mv optimization_engine/nx_updater.py optimization_engine/nx/updater.py
|
||||
mv optimization_engine/nx_session_manager.py optimization_engine/nx/session_manager.py
|
||||
mv optimization_engine/solve_simulation.py optimization_engine/nx/
|
||||
mv optimization_engine/solve_simulation_simple.py optimization_engine/nx/
|
||||
mv optimization_engine/model_cleanup.py optimization_engine/nx/
|
||||
mv optimization_engine/export_expressions.py optimization_engine/nx/
|
||||
mv optimization_engine/import_expressions.py optimization_engine/nx/
|
||||
mv optimization_engine/mesh_converter.py optimization_engine/nx/
|
||||
|
||||
# 4. SURROGATES
|
||||
mv optimization_engine/neural_surrogate.py optimization_engine/processors/surrogates/
|
||||
mv optimization_engine/generic_surrogate.py optimization_engine/processors/surrogates/
|
||||
mv optimization_engine/adaptive_surrogate.py optimization_engine/processors/surrogates/
|
||||
mv optimization_engine/simple_mlp_surrogate.py optimization_engine/processors/surrogates/
|
||||
mv optimization_engine/active_learning_surrogate.py optimization_engine/processors/surrogates/
|
||||
mv optimization_engine/surrogate_tuner.py optimization_engine/processors/surrogates/
|
||||
mv optimization_engine/auto_trainer.py optimization_engine/processors/surrogates/
|
||||
mv optimization_engine/training_data_exporter.py optimization_engine/processors/surrogates/
|
||||
|
||||
# 5. STUDY MANAGEMENT
|
||||
mv optimization_engine/study_creator.py optimization_engine/study/creator.py
|
||||
mv optimization_engine/study_wizard.py optimization_engine/study/wizard.py
|
||||
mv optimization_engine/study_state.py optimization_engine/study/state.py
|
||||
mv optimization_engine/study_reset.py optimization_engine/study/reset.py
|
||||
mv optimization_engine/study_continuation.py optimization_engine/study/continuation.py
|
||||
mv optimization_engine/benchmarking_substudy.py optimization_engine/study/benchmarking.py
|
||||
mv optimization_engine/generate_history_from_trials.py optimization_engine/study/history_generator.py
|
||||
|
||||
# 6. REPORTING
|
||||
mv optimization_engine/generate_report.py optimization_engine/reporting/report_generator.py
|
||||
mv optimization_engine/generate_report_markdown.py optimization_engine/reporting/markdown_report.py
|
||||
mv optimization_engine/comprehensive_results_analyzer.py optimization_engine/reporting/results_analyzer.py
|
||||
mv optimization_engine/visualizer.py optimization_engine/reporting/
|
||||
mv optimization_engine/landscape_analyzer.py optimization_engine/reporting/
|
||||
|
||||
# 7. CORE (depends on many things, do last)
|
||||
mv optimization_engine/runner.py optimization_engine/core/
|
||||
mv optimization_engine/base_runner.py optimization_engine/core/
|
||||
mv optimization_engine/runner_with_neural.py optimization_engine/core/
|
||||
mv optimization_engine/intelligent_optimizer.py optimization_engine/core/
|
||||
mv optimization_engine/method_selector.py optimization_engine/core/
|
||||
mv optimization_engine/strategy_selector.py optimization_engine/core/
|
||||
mv optimization_engine/strategy_portfolio.py optimization_engine/core/
|
||||
|
||||
# 8. RESEARCH/FUTURE
|
||||
mv optimization_engine/research_agent.py optimization_engine/future/
|
||||
mv optimization_engine/pynastran_research_agent.py optimization_engine/future/
|
||||
mv optimization_engine/targeted_research_planner.py optimization_engine/future/
|
||||
mv optimization_engine/workflow_decomposer.py optimization_engine/future/
|
||||
mv optimization_engine/step_classifier.py optimization_engine/future/
|
||||
|
||||
# 9. REMAINING MISC
|
||||
mv optimization_engine/op2_extractor.py optimization_engine/extractors/
|
||||
mv optimization_engine/extractor_library.py optimization_engine/extractors/
|
||||
mv optimization_engine/simulation_validator.py optimization_engine/validators/
|
||||
mv optimization_engine/adaptive_characterization.py optimization_engine/processors/
|
||||
```
|
||||
|
||||
### Phase 3: Update Internal Imports (1-2 hours)
|
||||
|
||||
**Use sed/grep for bulk updates:**
|
||||
|
||||
```bash
|
||||
# Example sed commands (run from project root)
|
||||
|
||||
# NX imports
|
||||
find . -name "*.py" -exec sed -i 's/from optimization_engine\.nx_solver/from optimization_engine.nx.solver/g' {} +
|
||||
find . -name "*.py" -exec sed -i 's/from optimization_engine\.nx_updater/from optimization_engine.nx.updater/g' {} +
|
||||
find . -name "*.py" -exec sed -i 's/from optimization_engine\.nx_session_manager/from optimization_engine.nx.session_manager/g' {} +
|
||||
find . -name "*.py" -exec sed -i 's/from optimization_engine\.solve_simulation/from optimization_engine.nx.solve_simulation/g' {} +
|
||||
|
||||
# Study imports
|
||||
find . -name "*.py" -exec sed -i 's/from optimization_engine\.study_creator/from optimization_engine.study.creator/g' {} +
|
||||
find . -name "*.py" -exec sed -i 's/from optimization_engine\.study_wizard/from optimization_engine.study.wizard/g' {} +
|
||||
find . -name "*.py" -exec sed -i 's/from optimization_engine\.study_state/from optimization_engine.study.state/g' {} +
|
||||
|
||||
# Config imports
|
||||
find . -name "*.py" -exec sed -i 's/from optimization_engine\.config_manager/from optimization_engine.config.manager/g' {} +
|
||||
|
||||
# Core imports
|
||||
find . -name "*.py" -exec sed -i 's/from optimization_engine\.runner import/from optimization_engine.core.runner import/g' {} +
|
||||
find . -name "*.py" -exec sed -i 's/from optimization_engine\.base_runner/from optimization_engine.core.base_runner/g' {} +
|
||||
find . -name "*.py" -exec sed -i 's/from optimization_engine\.intelligent_optimizer/from optimization_engine.core.intelligent_optimizer/g' {} +
|
||||
|
||||
# Surrogate imports
|
||||
find . -name "*.py" -exec sed -i 's/from optimization_engine\.neural_surrogate/from optimization_engine.processors.surrogates.neural_surrogate/g' {} +
|
||||
find . -name "*.py" -exec sed -i 's/from optimization_engine\.generic_surrogate/from optimization_engine.processors.surrogates.generic_surrogate/g' {} +
|
||||
|
||||
# Logger
|
||||
find . -name "*.py" -exec sed -i 's/from optimization_engine\.logger/from optimization_engine.utils.logger/g' {} +
|
||||
```
|
||||
|
||||
**Handle edge cases manually:**
|
||||
- `import optimization_engine.nx_solver` (without `from`)
|
||||
- Dynamic imports using `importlib`
|
||||
- String references in configs
|
||||
|
||||
### Phase 4: Create __init__.py Files (30 min)
|
||||
|
||||
**Example: `optimization_engine/core/__init__.py`**
|
||||
|
||||
```python
|
||||
"""
|
||||
Optimization Engine Core
|
||||
|
||||
Main optimization runners and algorithm selection.
|
||||
"""
|
||||
|
||||
from .runner import OptimizationRunner
|
||||
from .base_runner import BaseRunner
|
||||
from .intelligent_optimizer import IntelligentOptimizer, IMSO
|
||||
from .method_selector import MethodSelector
|
||||
from .strategy_selector import StrategySelector
|
||||
from .strategy_portfolio import StrategyPortfolio
|
||||
|
||||
__all__ = [
|
||||
'OptimizationRunner',
|
||||
'BaseRunner',
|
||||
'IntelligentOptimizer',
|
||||
'IMSO',
|
||||
'MethodSelector',
|
||||
'StrategySelector',
|
||||
'StrategyPortfolio',
|
||||
]
|
||||
```
|
||||
|
||||
**Create similar for each new directory.**
|
||||
|
||||
### Phase 5: Add Backwards Compatibility (30 min)
|
||||
|
||||
Update `optimization_engine/__init__.py`:
|
||||
|
||||
```python
|
||||
"""
|
||||
Optimization Engine for Atomizer
|
||||
|
||||
Reorganized structure (v2.0):
|
||||
- core/ - Optimization runners
|
||||
- processors/ - Data processing (surrogates, dynamic response)
|
||||
- nx/ - NX/Nastran integration
|
||||
- study/ - Study management
|
||||
- reporting/ - Reports and analysis
|
||||
- config/ - Configuration
|
||||
- extractors/ - Physics extraction
|
||||
- insights/ - Visualizations
|
||||
- gnn/ - Graph neural networks
|
||||
- hooks/ - NX hooks
|
||||
"""
|
||||
|
||||
# Re-export commonly used items at top level for convenience
|
||||
from optimization_engine.core.runner import OptimizationRunner
|
||||
from optimization_engine.core.intelligent_optimizer import IMSO
|
||||
from optimization_engine.nx.solver import run_nx_simulation, NXSolver
|
||||
from optimization_engine.study.creator import create_study
|
||||
from optimization_engine.config.manager import ConfigManager
|
||||
|
||||
# Backwards compatibility aliases (deprecated)
|
||||
# These will be removed in a future version
|
||||
import warnings as _warnings
|
||||
|
||||
def _deprecated_import(old_name, new_location):
|
||||
_warnings.warn(
|
||||
f"Importing '{old_name}' directly from optimization_engine is deprecated. "
|
||||
f"Use '{new_location}' instead.",
|
||||
DeprecationWarning,
|
||||
stacklevel=3
|
||||
)
|
||||
|
||||
# Lazy loading for backwards compatibility
|
||||
def __getattr__(name):
|
||||
# Map old names to new locations
|
||||
_compat_map = {
|
||||
'nx_solver': ('optimization_engine.nx.solver', 'nx_solver'),
|
||||
'nx_updater': ('optimization_engine.nx.updater', 'nx_updater'),
|
||||
'study_creator': ('optimization_engine.study.creator', 'study_creator'),
|
||||
'config_manager': ('optimization_engine.config.manager', 'config_manager'),
|
||||
'runner': ('optimization_engine.core.runner', 'runner'),
|
||||
'neural_surrogate': ('optimization_engine.processors.surrogates.neural_surrogate', 'neural_surrogate'),
|
||||
}
|
||||
|
||||
if name in _compat_map:
|
||||
module_path, attr = _compat_map[name]
|
||||
_deprecated_import(name, module_path)
|
||||
import importlib
|
||||
module = importlib.import_module(module_path)
|
||||
return module
|
||||
|
||||
raise AttributeError(f"module 'optimization_engine' has no attribute '{name}'")
|
||||
```
|
||||
|
||||
### Phase 6: Update Documentation (1 hour)
|
||||
|
||||
**Use sed for bulk updates in markdown:**
|
||||
|
||||
```bash
|
||||
# Update protocol files
|
||||
find docs/protocols -name "*.md" -exec sed -i 's/optimization_engine\/nx_solver/optimization_engine\/nx\/solver/g' {} +
|
||||
find docs/protocols -name "*.md" -exec sed -i 's/optimization_engine\/study_creator/optimization_engine\/study\/creator/g' {} +
|
||||
# ... etc for all mappings
|
||||
|
||||
# Update .claude skills
|
||||
find .claude -name "*.md" -exec sed -i 's/optimization_engine\.nx_solver/optimization_engine.nx.solver/g' {} +
|
||||
# ... etc
|
||||
```
|
||||
|
||||
**Manual review needed for:**
|
||||
- Code examples in markdown
|
||||
- Directory structure diagrams
|
||||
- Command-line examples
|
||||
|
||||
### Phase 7: Update JSON Configs (30 min)
|
||||
|
||||
**feature_registry.json** - Use a Python script:
|
||||
|
||||
```python
|
||||
import json
|
||||
|
||||
with open('optimization_engine/feature_registry.json', 'r') as f:
|
||||
registry = json.load(f)
|
||||
|
||||
# Define path mappings
|
||||
path_map = {
|
||||
'optimization_engine/neural_surrogate.py': 'optimization_engine/processors/surrogates/neural_surrogate.py',
|
||||
'optimization_engine/nx_solver.py': 'optimization_engine/nx/solver.py',
|
||||
# ... all other mappings
|
||||
}
|
||||
|
||||
def update_paths(obj):
|
||||
if isinstance(obj, dict):
|
||||
for key, value in obj.items():
|
||||
if key == 'file_path' and value in path_map:
|
||||
obj[key] = path_map[value]
|
||||
else:
|
||||
update_paths(value)
|
||||
elif isinstance(obj, list):
|
||||
for item in obj:
|
||||
update_paths(item)
|
||||
|
||||
update_paths(registry)
|
||||
|
||||
with open('optimization_engine/feature_registry.json', 'w') as f:
|
||||
json.dump(registry, f, indent=2)
|
||||
```
|
||||
|
||||
### Phase 8: Testing & Validation (1-2 hours)
|
||||
|
||||
```bash
|
||||
# 1. Run Python import tests
|
||||
python -c "from optimization_engine.core.runner import OptimizationRunner"
|
||||
python -c "from optimization_engine.nx.solver import run_nx_simulation"
|
||||
python -c "from optimization_engine.study.creator import create_study"
|
||||
python -c "from optimization_engine.processors.surrogates.neural_surrogate import NeuralSurrogate"
|
||||
|
||||
# 2. Run backwards compatibility tests
|
||||
python -c "from optimization_engine import nx_solver" # Should work with deprecation warning
|
||||
|
||||
# 3. Run test suite
|
||||
python -m pytest tests/ -v --tb=short
|
||||
|
||||
# 4. Test a study end-to-end
|
||||
cd studies/Simple_Bracket/bracket_displacement_maximizing
|
||||
python run_optimization.py --trials 2 --dry-run
|
||||
|
||||
# 5. Test dashboard
|
||||
cd atomizer-dashboard
|
||||
python backend/api/main.py # Should start without import errors
|
||||
```
|
||||
|
||||
### Phase 9: Cleanup (15 min)
|
||||
|
||||
```bash
|
||||
# Remove backup after successful testing
|
||||
rm -rf optimization_engine_backup
|
||||
|
||||
# Remove any .pyc files that might cache old imports
|
||||
find . -name "*.pyc" -delete
|
||||
find . -name "__pycache__" -type d -exec rm -rf {} +
|
||||
|
||||
# Commit
|
||||
git add -A
|
||||
git commit -m "refactor: Reorganize optimization_engine into modular structure
|
||||
|
||||
- Move 50+ top-level files into logical subdirectories
|
||||
- Create core/, processors/, nx/, study/, reporting/, config/
|
||||
- Add backwards compatibility aliases with deprecation warnings
|
||||
- Update all imports across codebase (640+ changes)
|
||||
- Update documentation and protocols
|
||||
|
||||
BREAKING: Direct imports from optimization_engine.* are deprecated.
|
||||
Use new paths like optimization_engine.core.runner instead.
|
||||
|
||||
🤖 Generated with [Claude Code](https://claude.com/claude-code)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Part 6: Rollback Plan
|
||||
|
||||
If migration fails:
|
||||
|
||||
```bash
|
||||
# Option 1: Git reset (if not committed)
|
||||
git checkout -- .
|
||||
|
||||
# Option 2: Restore backup
|
||||
rm -rf optimization_engine
|
||||
mv optimization_engine_backup optimization_engine
|
||||
|
||||
# Option 3: Git revert (if committed)
|
||||
git revert HEAD
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Part 7: Post-Migration Tasks
|
||||
|
||||
### 7.1 Update LAC Knowledge Base
|
||||
|
||||
Record the migration in LAC:
|
||||
|
||||
```python
|
||||
from knowledge_base.lac import get_lac
|
||||
lac = get_lac()
|
||||
lac.record_insight(
|
||||
category="architecture",
|
||||
context="optimization_engine reorganization",
|
||||
insight="Migrated 50+ files into modular structure. New paths: "
|
||||
"core/, processors/, nx/, study/, reporting/, config/. "
|
||||
"Backwards compat aliases provided with deprecation warnings.",
|
||||
confidence=1.0,
|
||||
tags=["refactoring", "architecture", "breaking-change"]
|
||||
)
|
||||
```
|
||||
|
||||
### 7.2 Update CLAUDE.md
|
||||
|
||||
Add section about new structure.
|
||||
|
||||
### 7.3 Schedule Deprecation Removal
|
||||
|
||||
After 2-4 weeks of stable operation:
|
||||
- Remove backwards compatibility aliases
|
||||
- Update remaining old imports
|
||||
- Clean up __init__.py files
|
||||
|
||||
---
|
||||
|
||||
## Part 8: Time Estimate Summary
|
||||
|
||||
| Phase | Task | Time |
|
||||
|-------|------|------|
|
||||
| 0 | Pre-migration (backup, baseline) | 30 min |
|
||||
| 1 | Create directory structure | 15 min |
|
||||
| 2 | Move files | 30 min |
|
||||
| 3 | Update internal imports | 1-2 hours |
|
||||
| 4 | Create __init__.py files | 30 min |
|
||||
| 5 | Add backwards compatibility | 30 min |
|
||||
| 6 | Update documentation | 1 hour |
|
||||
| 7 | Update JSON configs | 30 min |
|
||||
| 8 | Testing & validation | 1-2 hours |
|
||||
| 9 | Cleanup & commit | 15 min |
|
||||
| **TOTAL** | | **6-8 hours** |
|
||||
|
||||
---
|
||||
|
||||
## Part 9: Decision Points
|
||||
|
||||
### Option A: Full Migration Now
|
||||
- **Pros**: Clean structure, enables dynamic_response cleanly
|
||||
- **Cons**: 6-8 hours work, risk of breakage
|
||||
- **When**: When you have a dedicated half-day
|
||||
|
||||
### Option B: Minimal Migration (processors/ only)
|
||||
- **Pros**: Low risk, enables dynamic_response
|
||||
- **Cons**: Leaves technical debt, inconsistent structure
|
||||
- **When**: If you need dynamic_response urgently
|
||||
|
||||
### Option C: Defer Migration
|
||||
- **Pros**: Zero risk now
|
||||
- **Cons**: Dynamic_response goes in awkward location
|
||||
- **When**: If stability is critical
|
||||
|
||||
---
|
||||
|
||||
## Appendix A: Automated Migration Script
|
||||
|
||||
A Python script to automate most of the migration:
|
||||
|
||||
```python
|
||||
#!/usr/bin/env python
|
||||
"""
|
||||
optimization_engine Migration Script
|
||||
|
||||
Run with: python migrate_optimization_engine.py --dry-run
|
||||
Then: python migrate_optimization_engine.py --execute
|
||||
"""
|
||||
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
|
||||
# ... full script would be ~200 lines
|
||||
# Handles: directory creation, file moves, import updates, __init__ generation
|
||||
```
|
||||
|
||||
Would you like me to create this full automation script?
|
||||
|
||||
---
|
||||
|
||||
*This plan ensures a safe, reversible migration with clear validation steps.*
|
||||
Reference in New Issue
Block a user