Atomizer/docs/06_PROTOCOLS_DETAILED/protocol_10_imso.md

Protocol 10: Intelligent Multi-Strategy Optimization (IMSO)

Status: Active Version: 2.0 (Adaptive Two-Study Architecture) Last Updated: 2025-11-20

Overview

Protocol 10 implements intelligent, adaptive optimization that automatically:

1. Characterizes the optimization landscape
2. Selects the best optimization algorithm
3. Executes optimization with the ideal strategy

Key Innovation: Adaptive characterization phase that intelligently determines when enough landscape exploration has been done, then seamlessly transitions to the optimal algorithm.

Architecture

Two-Study Approach

Protocol 10 uses a two-study architecture to overcome Optuna's fixed-sampler limitation:

┌─────────────────────────────────────────────────────────────┐
│  PROTOCOL 10: INTELLIGENT MULTI-STRATEGY OPTIMIZATION       │
└─────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────┐
│  PHASE 1: ADAPTIVE CHARACTERIZATION STUDY                   │
│  ─────────────────────────────────────────────────────────  │
│  Sampler: Random/Sobol (unbiased exploration)               │
│  Trials: 10-30 (adapts to problem complexity)               │
│                                                              │
│  Every 5 trials:                                            │
│    → Analyze landscape metrics                              │
│    → Check metric convergence                               │
│    → Calculate characterization confidence                  │
│    → Decide if ready to stop                                │
│                                                              │
│  Stop when:                                                 │
│    ✓ Confidence ≥ 85%                                       │
│    ✓ OR max trials reached (30)                             │
│                                                              │
│  Simple problems (smooth, unimodal):                        │
│    Stop at ~10-15 trials                                    │
│                                                              │
│  Complex problems (multimodal, rugged):                     │
│    Continue to ~20-30 trials                                │
└─────────────────────────────────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────┐
│  TRANSITION: LANDSCAPE ANALYSIS & STRATEGY SELECTION        │
│  ─────────────────────────────────────────────────────────  │
│  Analyze final landscape:                                   │
│    - Smoothness (0-1)                                       │
│    - Multimodality (clusters of good solutions)             │
│    - Parameter correlation                                  │
│    - Noise level                                            │
│                                                              │
│  Classify landscape:                                        │
│    → smooth_unimodal                                        │
│    → smooth_multimodal                                      │
│    → rugged_unimodal                                        │
│    → rugged_multimodal                                      │
│    → noisy                                                  │
│                                                              │
│  Recommend strategy:                                        │
│    smooth_unimodal    → GP-BO (best) or CMA-ES             │
│    smooth_multimodal  → GP-BO                               │
│    rugged_multimodal  → TPE                                 │
│    rugged_unimodal    → TPE or CMA-ES                       │
│    noisy              → TPE (most robust)                   │
└─────────────────────────────────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────┐
│  PHASE 2: OPTIMIZATION STUDY                                │
│  ─────────────────────────────────────────────────────────  │
│  Sampler: Recommended from Phase 1                          │
│  Warm Start: Initialize from best characterization point    │
│  Trials: User-specified (default 50)                        │
│                                                              │
│  Optimizes efficiently using:                               │
│    - Right algorithm for the landscape                      │
│    - Knowledge from characterization phase                  │
│    - Focused exploitation around promising regions          │
└─────────────────────────────────────────────────────────────┘

Core Components

1. Adaptive Characterization (adaptive_characterization.py)

Purpose: Intelligently determine when enough landscape exploration has been done.

Key Features:

• Progressive landscape analysis (every 5 trials starting at trial 10)
• Metric convergence detection
• Complexity-aware sample adequacy
• Parameter space coverage assessment
• Confidence scoring (combines all factors)

Confidence Calculation (weighted sum):

confidence = (
    0.40 * metric_stability_score +      # Are metrics converging?
    0.30 * parameter_coverage_score +    # Explored enough space?
    0.20 * sample_adequacy_score +       # Enough samples for complexity?
    0.10 * landscape_clarity_score       # Clear classification?
)

Stopping Criteria:

• Minimum trials: 10 (always gather baseline data)
• Maximum trials: 30 (prevent over-characterization)
• Confidence threshold: 85% (high confidence in landscape understanding)
• Check interval: Every 5 trials

Adaptive Behavior:

# Simple problem (smooth, unimodal, low noise):
if smoothness > 0.6 and unimodal and noise < 0.3:
    required_samples = 10 + dimensionality
    # Stops at ~10-15 trials

# Complex problem (multimodal with N modes):
if multimodal and n_modes > 2:
    required_samples = 10 + 5 * n_modes + 2 * dimensionality
    # Continues to ~20-30 trials

2. Landscape Analyzer (landscape_analyzer.py)

Purpose: Characterize the optimization landscape from trial history.

Metrics Computed:

1. Smoothness (0-1):

   • Method: Spearman correlation between parameter distance and objective difference
   • High smoothness (>0.6): Nearby points have similar objectives (good for CMA-ES, GP-BO)
   • Low smoothness (<0.4): Rugged landscape (good for TPE)

2. Multimodality (boolean + n_modes):

   • Method: DBSCAN clustering on good trials (bottom 30%)
   • Detects multiple distinct regions of good solutions

3. Parameter Correlation:

   • Method: Spearman correlation between each parameter and objective
   • Identifies which parameters strongly affect objective

4. Noise Level (0-1):

   • Method: Local consistency check (nearby points should give similar outputs)
   • Important: Wide exploration range ≠ noise
   • Only true noise (simulation instability) is detected

Landscape Classification:

'smooth_unimodal'      # Single smooth bowl → GP-BO or CMA-ES
'smooth_multimodal'    # Multiple smooth regions → GP-BO
'rugged_unimodal'      # Single rugged region → TPE or CMA-ES
'rugged_multimodal'    # Multiple rugged regions → TPE
'noisy'                # High noise level → TPE (robust)

3. Strategy Selector (strategy_selector.py)

Purpose: Recommend the best optimization algorithm based on landscape.

Algorithm Recommendations:

Landscape Type	Primary Strategy	Fallback	Rationale
smooth_unimodal	GP-BO	CMA-ES	GP surrogate models smoothness explicitly
smooth_multimodal	GP-BO	TPE	GP handles multiple modes well
rugged_unimodal	TPE	CMA-ES	TPE robust to ruggedness
rugged_multimodal	TPE	-	TPE excellent for complex landscapes
noisy	TPE	-	TPE most robust to noise

Algorithm Characteristics:

GP-BO (Gaussian Process Bayesian Optimization):

• ✅ Best for: Smooth, expensive functions (like FEA)
• ✅ Explicit surrogate model (Gaussian Process)
• ✅ Models smoothness + uncertainty
• ✅ Acquisition function balances exploration/exploitation
• ❌ Less effective: Highly rugged landscapes

CMA-ES (Covariance Matrix Adaptation Evolution Strategy):

• ✅ Best for: Smooth unimodal problems
• ✅ Fast convergence to local optimum
• ✅ Adapts search distribution to landscape
• ❌ Can get stuck in local minima
• ❌ No explicit surrogate model

TPE (Tree-structured Parzen Estimator):

• ✅ Best for: Multimodal, rugged, or noisy problems
• ✅ Robust to noise and discontinuities
• ✅ Good global exploration
• ❌ Slower convergence than GP-BO/CMA-ES on smooth problems

4. Intelligent Optimizer (intelligent_optimizer.py)

Purpose: Orchestrate the entire Protocol 10 workflow.

Workflow:

1. Create characterization study (Random/Sobol sampler)
2. Run adaptive characterization with stopping criterion
3. Analyze final landscape
4. Select optimal strategy
5. Create optimization study with recommended sampler
6. Warm-start from best characterization point
7. Run optimization
8. Generate intelligence report

Usage

Basic Usage

from optimization_engine.intelligent_optimizer import IntelligentOptimizer

# Create optimizer
optimizer = IntelligentOptimizer(
    study_name="my_optimization",
    study_dir=results_dir,
    config=optimization_config,
    verbose=True
)

# Define design variables
design_vars = {
    'parameter1': (lower_bound, upper_bound),
    'parameter2': (lower_bound, upper_bound)
}

# Run Protocol 10
results = optimizer.optimize(
    objective_function=my_objective,
    design_variables=design_vars,
    n_trials=50,  # For optimization phase
    target_value=target,
    tolerance=0.1
)

Configuration

Add to optimization_config.json:

{
  "intelligent_optimization": {
    "enabled": true,
    "characterization": {
      "min_trials": 10,
      "max_trials": 30,
      "confidence_threshold": 0.85,
      "check_interval": 5
    },
    "landscape_analysis": {
      "min_trials_for_analysis": 10
    },
    "strategy_selection": {
      "allow_cmaes": true,
      "allow_gpbo": true,
      "allow_tpe": true
    }
  },
  "trials": {
    "n_trials": 50
  }
}

Intelligence Report

Protocol 10 generates comprehensive reports tracking:

1. Characterization Phase:

   • Metric evolution (smoothness, multimodality, noise)
   • Confidence progression
   • Stopping decision details

2. Landscape Analysis:

   • Final landscape classification
   • Parameter correlations
   • Objective statistics

3. Strategy Selection:

   • Recommended algorithm
   • Decision rationale
   • Alternative strategies considered

4. Optimization Performance:

   • Best solution found
   • Convergence history
   • Algorithm effectiveness

Benefits

Efficiency

• Simple problems: Stops characterization early (~10-15 trials)
• Complex problems: Extends characterization for adequate coverage (~20-30 trials)
• Right algorithm: Uses optimal strategy for the landscape type

Robustness

• Adaptive: Adjusts to problem complexity automatically
• Confidence-based: Only stops when confident in landscape understanding
• Fallback strategies: Handles edge cases gracefully

Transparency

• Detailed reports: Explains all decisions
• Metric tracking: Full history of landscape analysis
• Reproducibility: All decisions logged to JSON

Example: Circular Plate Frequency Tuning

Problem: Tune circular plate dimensions to achieve 115 Hz first natural frequency

Protocol 10 Behavior:

PHASE 1: CHARACTERIZATION (Trials 1-14)
  Trial 5:  Landscape = smooth_unimodal (preliminary)
  Trial 10: Landscape = smooth_unimodal (confidence 72%)
  Trial 14: Landscape = smooth_unimodal (confidence 87%)

  → CHARACTERIZATION COMPLETE
  → Confidence threshold met (87% ≥ 85%)
  → Recommended Strategy: GP-BO

PHASE 2: OPTIMIZATION (Trials 15-64)
  Sampler: GP-BO (warm-started from best characterization point)
  Trial 15: 0.325 Hz error (baseline from characterization)
  Trial 23: 0.142 Hz error
  Trial 31: 0.089 Hz error
  Trial 42: 0.047 Hz error
  Trial 56: 0.012 Hz error ← TARGET ACHIEVED!

  → Total Trials: 56 (14 characterization + 42 optimization)
  → Best Frequency: 115.012 Hz (error 0.012 Hz)

Comparison (without Protocol 10):

• TPE alone: ~95 trials to achieve target
• Random search: ~150+ trials
• Protocol 10: 56 trials (41% reduction vs TPE)

Limitations and Future Work

Current Limitations

1. Optuna Constraint: Cannot change sampler mid-study (necessitates two-study approach)
2. GP-BO Integration: Requires external GP-BO library (e.g., BoTorch, scikit-optimize)
3. Warm Start: Not all samplers support warm-starting equally well

Future Enhancements

1. Multi-Fidelity: Extend to support cheap/expensive function evaluations
2. Constraint Handling: Better support for constrained optimization
3. Transfer Learning: Use knowledge from previous similar problems
4. Active Learning: More sophisticated characterization sampling

References

• Landscape Analysis: Mersmann et al. "Exploratory Landscape Analysis" (2011)
• CMA-ES: Hansen & Ostermeier "Completely Derandomized Self-Adaptation" (2001)
• GP-BO: Snoek et al. "Practical Bayesian Optimization" (2012)
• TPE: Bergstra et al. "Algorithms for Hyper-Parameter Optimization" (2011)

Version History

Version 2.0 (2025-11-20)

• ✅ Added adaptive characterization with intelligent stopping
• ✅ Implemented two-study architecture (overcomes Optuna limitation)
• ✅ Fixed noise detection algorithm (local consistency instead of global CV)
• ✅ Added GP-BO as primary recommendation for smooth problems
• ✅ Comprehensive intelligence reporting

Version 1.0 (2025-11-19)

• Initial implementation with dynamic strategy switching
• Discovered Optuna sampler limitation
• Single-study architecture (non-functional)