# SYS_15: Adaptive Method Selector ## Overview The **Adaptive Method Selector (AMS)** analyzes optimization problems and recommends the best method (turbo, hybrid_loop, pure_fea, etc.) based on: 1. **Static Analysis**: Problem characteristics from config (dimensionality, objectives, constraints) 2. **Dynamic Analysis**: Early FEA trial metrics (smoothness, correlations, feasibility) 3. **NN Quality Assessment**: Relative accuracy thresholds comparing NN error to problem variability 4. **Runtime Monitoring**: Continuous optimization performance assessment **Key Value**: Eliminates guesswork in choosing optimization strategies by providing data-driven recommendations with relative accuracy thresholds. --- ## When to Use | Trigger | Action | |---------|--------| | Starting a new optimization | Run method selector first | | "which method", "recommend" mentioned | Suggest method selector | | Unsure between turbo/hybrid/fea | Use method selector | | > 20 FEA trials completed | Re-run for updated recommendation | --- ## Quick Reference ### CLI Usage ```bash python -m optimization_engine.method_selector [db_path] ``` **Examples**: ```bash # Config-only analysis (before any FEA trials) python -m optimization_engine.method_selector 1_setup/optimization_config.json # Full analysis with FEA data python -m optimization_engine.method_selector 1_setup/optimization_config.json 2_results/study.db ``` ### Python API ```python from optimization_engine.method_selector import AdaptiveMethodSelector selector = AdaptiveMethodSelector() recommendation = selector.recommend("1_setup/optimization_config.json", "2_results/study.db") print(recommendation.method) # 'turbo', 'hybrid_loop', 'pure_fea', 'gnn_field' print(recommendation.confidence) # 0.0 - 1.0 print(recommendation.parameters) # {'nn_trials': 5000, 'batch_size': 100, ...} print(recommendation.reasoning) # Explanation string print(recommendation.alternatives) # Other methods with scores ``` --- ## Available Methods | Method | Description | Best For | |--------|-------------|----------| | **TURBO** | Aggressive NN exploration with single-best FEA validation | Low-dimensional, smooth responses | | **HYBRID_LOOP** | Iterative train→predict→validate→retrain cycle | Moderate complexity, uncertain landscape | | **PURE_FEA** | Traditional FEA-only optimization | High-dimensional, complex physics | | **GNN_FIELD** | Graph neural network for field prediction | Need full field visualization | --- ## Selection Criteria ### Static Factors (from config) | Factor | Favors TURBO | Favors HYBRID_LOOP | Favors PURE_FEA | |--------|--------------|---------------------|-----------------| | **n_variables** | ≤5 | 5-10 | >10 | | **n_objectives** | 1-3 | 2-4 | Any | | **n_constraints** | ≤3 | 3-5 | >5 | | **FEA budget** | >50 trials | 30-50 trials | <30 trials | ### Dynamic Factors (from FEA trials) | Factor | Measurement | Impact | |--------|-------------|--------| | **Response smoothness** | Lipschitz constant estimate | Smooth → NN works well | | **Variable sensitivity** | Correlation with objectives | High correlation → easier to learn | | **Feasibility rate** | % of valid designs | Low feasibility → need more exploration | | **Objective correlations** | Pairwise correlations | Strong correlations → simpler landscape | --- ## NN Quality Assessment The method selector uses **relative accuracy thresholds** to assess NN suitability. Instead of absolute error limits, it compares NN error to the problem's natural variability (coefficient of variation). ### Core Concept ``` NN Suitability = f(nn_error / coefficient_of_variation) If nn_error >> CV → NN is unreliable (not learning, just noise) If nn_error ≈ CV → NN captures the trend (hybrid recommended) If nn_error << CV → NN is excellent (turbo viable) ``` ### Physics-Based Classification Objectives are classified by their expected predictability: | Objective Type | Examples | Max Expected Error | CV Ratio Limit | |----------------|----------|-------------------|----------------| | **Linear** | mass, volume | 2% | 0.5 | | **Smooth** | frequency, avg stress | 5% | 1.0 | | **Nonlinear** | max stress, stiffness | 10% | 2.0 | | **Chaotic** | contact, buckling | 20% | 3.0 | ### CV Ratio Interpretation The **CV Ratio** = NN Error / (Coefficient of Variation × 100): | CV Ratio | Quality | Interpretation | |----------|---------|----------------| | < 0.5 | ✓ Great | NN captures physics much better than noise | | 0.5 - 1.0 | ✓ Good | NN adds significant value for exploration | | 1.0 - 2.0 | ~ OK | NN is marginal, use with validation | | > 2.0 | ✗ Poor | NN not learning effectively, use FEA | ### Method Recommendations Based on Quality | Turbo Suitability | Hybrid Suitability | Recommendation | |-------------------|--------------------|-----------------------| | > 80% | any | **TURBO** - trust NN fully | | 50-80% | > 50% | **TURBO** with monitoring | | < 50% | > 50% | **HYBRID_LOOP** - verify periodically | | < 30% | < 50% | **PURE_FEA** or retrain first | ### Data Sources NN quality metrics are collected from: 1. `validation_report.json` - FEA validation results 2. `turbo_report.json` - Turbo mode validation history 3. `study.db` - Trial `nn_error_percent` user attributes --- ## Architecture ``` ┌─────────────────────────────────────────────────────────────────────────┐ │ AdaptiveMethodSelector │ ├─────────────────────────────────────────────────────────────────────────┤ │ │ │ ┌─────────────────┐ ┌────────────────────┐ ┌───────────────────┐ │ │ │ ProblemProfiler │ │EarlyMetricsCollector│ │ NNQualityAssessor │ │ │ │(static analysis)│ │ (dynamic analysis) │ │ (NN accuracy) │ │ │ └───────┬─────────┘ └─────────┬──────────┘ └─────────┬─────────┘ │ │ │ │ │ │ │ ▼ ▼ ▼ │ │ ┌─────────────────────────────────────────────────────────────────┐ │ │ │ _score_methods() │ │ │ │ (rule-based scoring with static + dynamic + NN factors) │ │ │ └───────────────────────────────┬─────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────────────┐ │ │ │ MethodRecommendation │ │ │ │ method, confidence, parameters, reasoning, warnings │ │ │ └─────────────────────────────────────────────────────────────────┘ │ │ │ │ ┌──────────────────┐ │ │ │ RuntimeAdvisor │ ← Monitors during optimization │ │ │ (pivot advisor) │ │ │ └──────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────────────────┘ ``` --- ## Components ### 1. ProblemProfiler Extracts static problem characteristics from `optimization_config.json`: ```python @dataclass class ProblemProfile: n_variables: int variable_names: List[str] variable_bounds: Dict[str, Tuple[float, float]] n_objectives: int objective_names: List[str] n_constraints: int fea_time_estimate: float max_fea_trials: int is_multi_objective: bool has_constraints: bool ``` ### 2. EarlyMetricsCollector Computes metrics from first N FEA trials in `study.db`: ```python @dataclass class EarlyMetrics: n_trials_analyzed: int objective_means: Dict[str, float] objective_stds: Dict[str, float] coefficient_of_variation: Dict[str, float] objective_correlations: Dict[str, float] variable_objective_correlations: Dict[str, Dict[str, float]] feasibility_rate: float response_smoothness: float # 0-1, higher = better for NN variable_sensitivity: Dict[str, float] ``` ### 3. NNQualityAssessor Assesses NN surrogate quality relative to problem complexity: ```python @dataclass class NNQualityMetrics: has_nn_data: bool = False n_validations: int = 0 nn_errors: Dict[str, float] # Absolute % error per objective cv_ratios: Dict[str, float] # nn_error / (CV * 100) per objective expected_errors: Dict[str, float] # Physics-based threshold overall_quality: float # 0-1, based on absolute thresholds turbo_suitability: float # 0-1, based on CV ratios hybrid_suitability: float # 0-1, more lenient threshold objective_types: Dict[str, str] # 'linear', 'smooth', 'nonlinear', 'chaotic' ``` ### 4. AdaptiveMethodSelector Main entry point that combines static + dynamic + NN quality analysis: ```python selector = AdaptiveMethodSelector(min_trials=20) recommendation = selector.recommend(config_path, db_path, results_dir=results_dir) # Access last NN quality for display print(f"Turbo suitability: {selector.last_nn_quality.turbo_suitability:.0%}") ``` ### 5. RuntimeAdvisor Monitors optimization progress and suggests pivots: ```python advisor = RuntimeAdvisor() pivot_advice = advisor.assess(db_path, config_path, current_method="turbo") if pivot_advice.should_pivot: print(f"Consider switching to {pivot_advice.recommended_method}") print(f"Reason: {pivot_advice.reason}") ``` --- ## Example Output ``` ====================================================================== OPTIMIZATION METHOD ADVISOR ====================================================================== Problem Profile: Variables: 2 (support_angle, tip_thickness) Objectives: 3 (mass, stress, stiffness) Constraints: 1 Max FEA budget: ~72 trials NN Quality Assessment: Validations analyzed: 10 | Objective | NN Error | CV | Ratio | Type | Quality | |---------------|----------|--------|-------|------------|---------| | mass | 3.7% | 16.0% | 0.23 | linear | ✓ Great | | stress | 2.0% | 7.7% | 0.26 | nonlinear | ✓ Great | | stiffness | 7.8% | 38.9% | 0.20 | nonlinear | ✓ Great | Overall Quality: 22% Turbo Suitability: 77% Hybrid Suitability: 88% ---------------------------------------------------------------------- RECOMMENDED: TURBO Confidence: 100% Reason: low-dimensional design space; sufficient FEA budget; smooth landscape (79%); good NN quality (77%) Suggested parameters: --nn-trials: 5000 --batch-size: 100 --retrain-every: 10 --epochs: 150 Alternatives: - hybrid_loop (90%): uncertain landscape - hybrid adapts; NN adds value with periodic retraining - pure_fea (50%): default recommendation Warnings: ! mass: NN error (3.7%) above expected (2%) - consider retraining or using hybrid mode ====================================================================== ``` --- ## Parameter Recommendations The selector suggests optimal parameters based on problem characteristics: | Parameter | Low-D (≤3 vars) | Medium-D (4-6 vars) | High-D (>6 vars) | |-----------|-----------------|---------------------|------------------| | `--nn-trials` | 5000 | 10000 | 20000 | | `--batch-size` | 100 | 100 | 200 | | `--retrain-every` | 10 | 15 | 20 | | `--epochs` | 150 | 200 | 300 | --- ## Scoring Algorithm Each method receives a score based on weighted factors: ```python # TURBO scoring turbo_score = 50 # base score turbo_score += 30 if n_variables <= 5 else -20 # dimensionality turbo_score += 25 if smoothness > 0.7 else -10 # response smoothness turbo_score += 20 if fea_budget > 50 else -15 # budget turbo_score += 15 if feasibility > 0.8 else -5 # feasibility turbo_score = max(0, min(100, turbo_score)) # clamp 0-100 # Similar for HYBRID_LOOP, PURE_FEA, GNN_FIELD ``` --- ## Integration with run_optimization.py The method selector can be integrated into the optimization workflow: ```python # At start of optimization from optimization_engine.method_selector import recommend_method recommendation = recommend_method(config_path, db_path) print(f"Recommended method: {recommendation.method}") print(f"Parameters: {recommendation.parameters}") # Ask user confirmation if user_confirms: if recommendation.method == 'turbo': os.system(f"python run_nn_optimization.py --turbo " f"--nn-trials {recommendation.parameters['nn_trials']} " f"--batch-size {recommendation.parameters['batch_size']}") ``` --- ## Troubleshooting | Symptom | Cause | Solution | |---------|-------|----------| | "Insufficient trials" | < 20 FEA trials | Run more FEA trials first | | Low confidence score | Conflicting signals | Try hybrid_loop as safe default | | PURE_FEA recommended | High dimensionality | Consider dimension reduction | | GNN_FIELD recommended | Need field visualization | Set up atomizer-field | ### Config Format Compatibility The method selector supports multiple config JSON formats: | Old Format | New Format | Both Supported | |------------|------------|----------------| | `parameter` | `name` | Variable name | | `bounds: [min, max]` | `min`, `max` | Variable bounds | | `goal` | `direction` | Objective direction | **Example equivalent configs:** ```json // Old format (UAV study style) {"design_variables": [{"parameter": "angle", "bounds": [30, 60]}]} // New format (beam study style) {"design_variables": [{"name": "angle", "min": 30, "max": 60}]} ``` --- ## Cross-References - **Depends On**: - [SYS_10_IMSO](./SYS_10_IMSO.md) for optimization framework - [SYS_14_NEURAL_ACCELERATION](./SYS_14_NEURAL_ACCELERATION.md) for neural methods - **Used By**: [OP_02_RUN_OPTIMIZATION](../operations/OP_02_RUN_OPTIMIZATION.md) - **See Also**: [modules/method-selection.md](../../.claude/skills/modules/method-selection.md) --- ## Implementation Files ``` optimization_engine/ └── method_selector.py # Complete AMS implementation ├── ProblemProfiler # Static config analysis ├── EarlyMetricsCollector # Dynamic FEA metrics ├── NNQualityMetrics # NN accuracy dataclass ├── NNQualityAssessor # Relative accuracy assessment ├── AdaptiveMethodSelector # Main recommendation engine ├── RuntimeAdvisor # Mid-run pivot advisor ├── print_recommendation() # CLI output with NN quality table └── recommend_method() # Convenience function ``` --- ## Version History | Version | Date | Changes | |---------|------|---------| | 2.1 | 2025-12-07 | Added config format flexibility (parameter/name, bounds/min-max, goal/direction) | | 2.0 | 2025-12-07 | Added NNQualityAssessor with relative accuracy thresholds | | 1.0 | 2025-12-06 | Initial implementation with 4 methods |