386 lines
15 KiB
Markdown
386 lines
15 KiB
Markdown
|
# Protocol 10: Intelligent Multi-Strategy Optimization (IMSO)
|
||
|
|
|
||
|
|
**Status**: Active
|
||
|
|
**Version**: 2.0 (Adaptive Two-Study Architecture)
|
||
|
|
**Last Updated**: 2025-11-20
|
||
|
|
|
||
|
|
## Overview
|
||
|
|
|
||
|
|
Protocol 10 implements intelligent, adaptive optimization that automatically:
|
||
|
|
1. Characterizes the optimization landscape
|
||
|
|
2. Selects the best optimization algorithm
|
||
|
|
3. Executes optimization with the ideal strategy
|
||
|
|
|
||
|
|
**Key Innovation**: Adaptive characterization phase that intelligently determines when enough landscape exploration has been done, then seamlessly transitions to the optimal algorithm.
|
||
|
|
|
||
|
|
## Architecture
|
||
|
|
|
||
|
|
### Two-Study Approach
|
||
|
|
|
||
|
|
Protocol 10 uses a **two-study architecture** to overcome Optuna's fixed-sampler limitation:
|
||
|
|
|
||
|
|
```
|
||
|
|
┌─────────────────────────────────────────────────────────────┐
|
||
|
|
│ PROTOCOL 10: INTELLIGENT MULTI-STRATEGY OPTIMIZATION │
|
||
|
|
└─────────────────────────────────────────────────────────────┘
|
||
|
|
|
||
|
|
┌─────────────────────────────────────────────────────────────┐
|
||
|
|
│ PHASE 1: ADAPTIVE CHARACTERIZATION STUDY │
|
||
|
|
│ ───────────────────────────────────────────────────────── │
|
||
|
|
│ Sampler: Random/Sobol (unbiased exploration) │
|
||
|
|
│ Trials: 10-30 (adapts to problem complexity) │
|
||
|
|
│ │
|
||
|
|
│ Every 5 trials: │
|
||
|
|
│ → Analyze landscape metrics │
|
||
|
|
│ → Check metric convergence │
|
||
|
|
│ → Calculate characterization confidence │
|
||
|
|
│ → Decide if ready to stop │
|
||
|
|
│ │
|
||
|
|
│ Stop when: │
|
||
|
|
│ ✓ Confidence ≥ 85% │
|
||
|
|
│ ✓ OR max trials reached (30) │
|
||
|
|
│ │
|
||
|
|
│ Simple problems (smooth, unimodal): │
|
||
|
|
│ Stop at ~10-15 trials │
|
||
|
|
│ │
|
||
|
|
│ Complex problems (multimodal, rugged): │
|
||
|
|
│ Continue to ~20-30 trials │
|
||
|
|
└─────────────────────────────────────────────────────────────┘
|
||
|
|
│
|
||
|
|
▼
|
||
|
|
┌─────────────────────────────────────────────────────────────┐
|
||
|
|
│ TRANSITION: LANDSCAPE ANALYSIS & STRATEGY SELECTION │
|
||
|
|
│ ───────────────────────────────────────────────────────── │
|
||
|
|
│ Analyze final landscape: │
|
||
|
|
│ - Smoothness (0-1) │
|
||
|
|
│ - Multimodality (clusters of good solutions) │
|
||
|
|
│ - Parameter correlation │
|
||
|
|
│ - Noise level │
|
||
|
|
│ │
|
||
|
|
│ Classify landscape: │
|
||
|
|
│ → smooth_unimodal │
|
||
|
|
│ → smooth_multimodal │
|
||
|
|
│ → rugged_unimodal │
|
||
|
|
│ → rugged_multimodal │
|
||
|
|
│ → noisy │
|
||
|
|
│ │
|
||
|
|
│ Recommend strategy: │
|
||
|
|
│ smooth_unimodal → GP-BO (best) or CMA-ES │
|
||
|
|
│ smooth_multimodal → GP-BO │
|
||
|
|
│ rugged_multimodal → TPE │
|
||
|
|
│ rugged_unimodal → TPE or CMA-ES │
|
||
|
|
│ noisy → TPE (most robust) │
|
||
|
|
└─────────────────────────────────────────────────────────────┘
|
||
|
|
│
|
||
|
|
▼
|
||
|
|
┌─────────────────────────────────────────────────────────────┐
|
||
|
|
│ PHASE 2: OPTIMIZATION STUDY │
|
||
|
|
│ ───────────────────────────────────────────────────────── │
|
||
|
|
│ Sampler: Recommended from Phase 1 │
|
||
|
|
│ Warm Start: Initialize from best characterization point │
|
||
|
|
│ Trials: User-specified (default 50) │
|
||
|
|
│ │
|
||
|
|
│ Optimizes efficiently using: │
|
||
|
|
│ - Right algorithm for the landscape │
|
||
|
|
│ - Knowledge from characterization phase │
|
||
|
|
│ - Focused exploitation around promising regions │
|
||
|
|
└─────────────────────────────────────────────────────────────┘
|
||
|
|
```
|
||
|
|
|
||
|
|
## Core Components
|
||
|
|
|
||
|
|
### 1. Adaptive Characterization (`adaptive_characterization.py`)
|
||
|
|
|
||
|
|
**Purpose**: Intelligently determine when enough landscape exploration has been done.
|
||
|
|
|
||
|
|
**Key Features**:
|
||
|
|
- Progressive landscape analysis (every 5 trials starting at trial 10)
|
||
|
|
- Metric convergence detection
|
||
|
|
- Complexity-aware sample adequacy
|
||
|
|
- Parameter space coverage assessment
|
||
|
|
- Confidence scoring (combines all factors)
|
||
|
|
|
||
|
|
**Confidence Calculation** (weighted sum):
|
||
|
|
```python
|
||
|
|
confidence = (
|
||
|
|
0.40 * metric_stability_score + # Are metrics converging?
|
||
|
|
0.30 * parameter_coverage_score + # Explored enough space?
|
||
|
|
0.20 * sample_adequacy_score + # Enough samples for complexity?
|
||
|
|
0.10 * landscape_clarity_score # Clear classification?
|
||
|
|
)
|
||
|
|
```
|
||
|
|
|
||
|
|
**Stopping Criteria**:
|
||
|
|
- **Minimum trials**: 10 (always gather baseline data)
|
||
|
|
- **Maximum trials**: 30 (prevent over-characterization)
|
||
|
|
- **Confidence threshold**: 85% (high confidence in landscape understanding)
|
||
|
|
- **Check interval**: Every 5 trials
|
||
|
|
|
||
|
|
**Adaptive Behavior**:
|
||
|
|
```python
|
||
|
|
# Simple problem (smooth, unimodal, low noise):
|
||
|
|
if smoothness > 0.6 and unimodal and noise < 0.3:
|
||
|
|
required_samples = 10 + dimensionality
|
||
|
|
# Stops at ~10-15 trials
|
||
|
|
|
||
|
|
# Complex problem (multimodal with N modes):
|
||
|
|
if multimodal and n_modes > 2:
|
||
|
|
required_samples = 10 + 5 * n_modes + 2 * dimensionality
|
||
|
|
# Continues to ~20-30 trials
|
||
|
|
```
|
||
|
|
|
||
|
|
### 2. Landscape Analyzer (`landscape_analyzer.py`)
|
||
|
|
|
||
|
|
**Purpose**: Characterize the optimization landscape from trial history.
|
||
|
|
|
||
|
|
**Metrics Computed**:
|
||
|
|
|
||
|
|
1. **Smoothness** (0-1):
|
||
|
|
- Method: Spearman correlation between parameter distance and objective difference
|
||
|
|
- High smoothness (>0.6): Nearby points have similar objectives (good for CMA-ES, GP-BO)
|
||
|
|
- Low smoothness (<0.4): Rugged landscape (good for TPE)
|
||
|
|
|
||
|
|
2. **Multimodality** (boolean + n_modes):
|
||
|
|
- Method: DBSCAN clustering on good trials (bottom 30%)
|
||
|
|
- Detects multiple distinct regions of good solutions
|
||
|
|
|
||
|
|
3. **Parameter Correlation**:
|
||
|
|
- Method: Spearman correlation between each parameter and objective
|
||
|
|
- Identifies which parameters strongly affect objective
|
||
|
|
|
||
|
|
4. **Noise Level** (0-1):
|
||
|
|
- Method: Local consistency check (nearby points should give similar outputs)
|
||
|
|
- **Important**: Wide exploration range ≠ noise
|
||
|
|
- Only true noise (simulation instability) is detected
|
||
|
|
|
||
|
|
**Landscape Classification**:
|
||
|
|
```python
|
||
|
|
'smooth_unimodal' # Single smooth bowl → GP-BO or CMA-ES
|
||
|
|
'smooth_multimodal' # Multiple smooth regions → GP-BO
|
||
|
|
'rugged_unimodal' # Single rugged region → TPE or CMA-ES
|
||
|
|
'rugged_multimodal' # Multiple rugged regions → TPE
|
||
|
|
'noisy' # High noise level → TPE (robust)
|
||
|
|
```
|
||
|
|
|
||
|
|
### 3. Strategy Selector (`strategy_selector.py`)
|
||
|
|
|
||
|
|
**Purpose**: Recommend the best optimization algorithm based on landscape.
|
||
|
|
|
||
|
|
**Algorithm Recommendations**:
|
||
|
|
|
||
|
|
| Landscape Type | Primary Strategy | Fallback | Rationale |
|
||
|
|
|----------------|------------------|----------|-----------|
|
||
|
|
| smooth_unimodal | GP-BO | CMA-ES | GP surrogate models smoothness explicitly |
|
||
|
|
| smooth_multimodal | GP-BO | TPE | GP handles multiple modes well |
|
||
|
|
| rugged_unimodal | TPE | CMA-ES | TPE robust to ruggedness |
|
||
|
|
| rugged_multimodal | TPE | - | TPE excellent for complex landscapes |
|
||
|
|
| noisy | TPE | - | TPE most robust to noise |
|
||
|
|
|
||
|
|
**Algorithm Characteristics**:
|
||
|
|
|
||
|
|
**GP-BO (Gaussian Process Bayesian Optimization)**:
|
||
|
|
- ✅ Best for: Smooth, expensive functions (like FEA)
|
||
|
|
- ✅ Explicit surrogate model (Gaussian Process)
|
||
|
|
- ✅ Models smoothness + uncertainty
|
||
|
|
- ✅ Acquisition function balances exploration/exploitation
|
||
|
|
- ❌ Less effective: Highly rugged landscapes
|
||
|
|
|
||
|
|
**CMA-ES (Covariance Matrix Adaptation Evolution Strategy)**:
|
||
|
|
- ✅ Best for: Smooth unimodal problems
|
||
|
|
- ✅ Fast convergence to local optimum
|
||
|
|
- ✅ Adapts search distribution to landscape
|
||
|
|
- ❌ Can get stuck in local minima
|
||
|
|
- ❌ No explicit surrogate model
|
||
|
|
|
||
|
|
**TPE (Tree-structured Parzen Estimator)**:
|
||
|
|
- ✅ Best for: Multimodal, rugged, or noisy problems
|
||
|
|
- ✅ Robust to noise and discontinuities
|
||
|
|
- ✅ Good global exploration
|
||
|
|
- ❌ Slower convergence than GP-BO/CMA-ES on smooth problems
|
||
|
|
|
||
|
|
### 4. Intelligent Optimizer (`intelligent_optimizer.py`)
|
||
|
|
|
||
|
|
**Purpose**: Orchestrate the entire Protocol 10 workflow.
|
||
|
|
|
||
|
|
**Workflow**:
|
||
|
|
```python
|
||
|
|
1. Create characterization study (Random/Sobol sampler)
|
||
|
|
2. Run adaptive characterization with stopping criterion
|
||
|
|
3. Analyze final landscape
|
||
|
|
4. Select optimal strategy
|
||
|
|
5. Create optimization study with recommended sampler
|
||
|
|
6. Warm-start from best characterization point
|
||
|
|
7. Run optimization
|
||
|
|
8. Generate intelligence report
|
||
|
|
```
|
||
|
|
|
||
|
|
## Usage
|
||
|
|
|
||
|
|
### Basic Usage
|
||
|
|
|
||
|
|
```python
|
||
|
|
from optimization_engine.intelligent_optimizer import IntelligentOptimizer
|
||
|
|
|
||
|
|
# Create optimizer
|
||
|
|
optimizer = IntelligentOptimizer(
|
||
|
|
study_name="my_optimization",
|
||
|
|
study_dir=results_dir,
|
||
|
|
config=optimization_config,
|
||
|
|
verbose=True
|
||
|
|
)
|
||
|
|
|
||
|
|
# Define design variables
|
||
|
|
design_vars = {
|
||
|
|
'parameter1': (lower_bound, upper_bound),
|
||
|
|
'parameter2': (lower_bound, upper_bound)
|
||
|
|
}
|
||
|
|
|
||
|
|
# Run Protocol 10
|
||
|
|
results = optimizer.optimize(
|
||
|
|
objective_function=my_objective,
|
||
|
|
design_variables=design_vars,
|
||
|
|
n_trials=50, # For optimization phase
|
||
|
|
target_value=target,
|
||
|
|
tolerance=0.1
|
||
|
|
)
|
||
|
|
```
|
||
|
|
|
||
|
|
### Configuration
|
||
|
|
|
||
|
|
Add to `optimization_config.json`:
|
||
|
|
|
||
|
|
```json
|
||
|
|
{
|
||
|
|
"intelligent_optimization": {
|
||
|
|
"enabled": true,
|
||
|
|
"characterization": {
|
||
|
|
"min_trials": 10,
|
||
|
|
"max_trials": 30,
|
||
|
|
"confidence_threshold": 0.85,
|
||
|
|
"check_interval": 5
|
||
|
|
},
|
||
|
|
"landscape_analysis": {
|
||
|
|
"min_trials_for_analysis": 10
|
||
|
|
},
|
||
|
|
"strategy_selection": {
|
||
|
|
"allow_cmaes": true,
|
||
|
|
"allow_gpbo": true,
|
||
|
|
"allow_tpe": true
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"trials": {
|
||
|
|
"n_trials": 50
|
||
|
|
}
|
||
|
|
}
|
||
|
|
```
|
||
|
|
|
||
|
|
## Intelligence Report
|
||
|
|
|
||
|
|
Protocol 10 generates comprehensive reports tracking:
|
||
|
|
|
||
|
|
1. **Characterization Phase**:
|
||
|
|
- Metric evolution (smoothness, multimodality, noise)
|
||
|
|
- Confidence progression
|
||
|
|
- Stopping decision details
|
||
|
|
|
||
|
|
2. **Landscape Analysis**:
|
||
|
|
- Final landscape classification
|
||
|
|
- Parameter correlations
|
||
|
|
- Objective statistics
|
||
|
|
|
||
|
|
3. **Strategy Selection**:
|
||
|
|
- Recommended algorithm
|
||
|
|
- Decision rationale
|
||
|
|
- Alternative strategies considered
|
||
|
|
|
||
|
|
4. **Optimization Performance**:
|
||
|
|
- Best solution found
|
||
|
|
- Convergence history
|
||
|
|
- Algorithm effectiveness
|
||
|
|
|
||
|
|
## Benefits
|
||
|
|
|
||
|
|
### Efficiency
|
||
|
|
- **Simple problems**: Stops characterization early (~10-15 trials)
|
||
|
|
- **Complex problems**: Extends characterization for adequate coverage (~20-30 trials)
|
||
|
|
- **Right algorithm**: Uses optimal strategy for the landscape type
|
||
|
|
|
||
|
|
### Robustness
|
||
|
|
- **Adaptive**: Adjusts to problem complexity automatically
|
||
|
|
- **Confidence-based**: Only stops when confident in landscape understanding
|
||
|
|
- **Fallback strategies**: Handles edge cases gracefully
|
||
|
|
|
||
|
|
### Transparency
|
||
|
|
- **Detailed reports**: Explains all decisions
|
||
|
|
- **Metric tracking**: Full history of landscape analysis
|
||
|
|
- **Reproducibility**: All decisions logged to JSON
|
||
|
|
|
||
|
|
## Example: Circular Plate Frequency Tuning
|
||
|
|
|
||
|
|
**Problem**: Tune circular plate dimensions to achieve 115 Hz first natural frequency
|
||
|
|
|
||
|
|
**Protocol 10 Behavior**:
|
||
|
|
|
||
|
|
```
|
||
|
|
PHASE 1: CHARACTERIZATION (Trials 1-14)
|
||
|
|
Trial 5: Landscape = smooth_unimodal (preliminary)
|
||
|
|
Trial 10: Landscape = smooth_unimodal (confidence 72%)
|
||
|
|
Trial 14: Landscape = smooth_unimodal (confidence 87%)
|
||
|
|
|
||
|
|
→ CHARACTERIZATION COMPLETE
|
||
|
|
→ Confidence threshold met (87% ≥ 85%)
|
||
|
|
→ Recommended Strategy: GP-BO
|
||
|
|
|
||
|
|
PHASE 2: OPTIMIZATION (Trials 15-64)
|
||
|
|
Sampler: GP-BO (warm-started from best characterization point)
|
||
|
|
Trial 15: 0.325 Hz error (baseline from characterization)
|
||
|
|
Trial 23: 0.142 Hz error
|
||
|
|
Trial 31: 0.089 Hz error
|
||
|
|
Trial 42: 0.047 Hz error
|
||
|
|
Trial 56: 0.012 Hz error ← TARGET ACHIEVED!
|
||
|
|
|
||
|
|
→ Total Trials: 56 (14 characterization + 42 optimization)
|
||
|
|
→ Best Frequency: 115.012 Hz (error 0.012 Hz)
|
||
|
|
```
|
||
|
|
|
||
|
|
**Comparison** (without Protocol 10):
|
||
|
|
- TPE alone: ~95 trials to achieve target
|
||
|
|
- Random search: ~150+ trials
|
||
|
|
- **Protocol 10: 56 trials** (41% reduction vs TPE)
|
||
|
|
|
||
|
|
## Limitations and Future Work
|
||
|
|
|
||
|
|
### Current Limitations
|
||
|
|
|
||
|
|
1. **Optuna Constraint**: Cannot change sampler mid-study (necessitates two-study approach)
|
||
|
|
2. **GP-BO Integration**: Requires external GP-BO library (e.g., BoTorch, scikit-optimize)
|
||
|
|
3. **Warm Start**: Not all samplers support warm-starting equally well
|
||
|
|
|
||
|
|
### Future Enhancements
|
||
|
|
|
||
|
|
1. **Multi-Fidelity**: Extend to support cheap/expensive function evaluations
|
||
|
|
2. **Constraint Handling**: Better support for constrained optimization
|
||
|
|
3. **Transfer Learning**: Use knowledge from previous similar problems
|
||
|
|
4. **Active Learning**: More sophisticated characterization sampling
|
||
|
|
|
||
|
|
## References
|
||
|
|
|
||
|
|
- Landscape Analysis: Mersmann et al. "Exploratory Landscape Analysis" (2011)
|
||
|
|
- CMA-ES: Hansen & Ostermeier "Completely Derandomized Self-Adaptation" (2001)
|
||
|
|
- GP-BO: Snoek et al. "Practical Bayesian Optimization" (2012)
|
||
|
|
- TPE: Bergstra et al. "Algorithms for Hyper-Parameter Optimization" (2011)
|
||
|
|
|
||
|
|
## Version History
|
||
|
|
|
||
|
|
### Version 2.0 (2025-11-20)
|
||
|
|
- ✅ Added adaptive characterization with intelligent stopping
|
||
|
|
- ✅ Implemented two-study architecture (overcomes Optuna limitation)
|
||
|
|
- ✅ Fixed noise detection algorithm (local consistency instead of global CV)
|
||
|
|
- ✅ Added GP-BO as primary recommendation for smooth problems
|
||
|
|
- ✅ Comprehensive intelligence reporting
|
||
|
|
|
||
|
|
### Version 1.0 (2025-11-19)
|
||
|
|
- Initial implementation with dynamic strategy switching
|
||
|
|
- Discovered Optuna sampler limitation
|
||
|
|
- Single-study architecture (non-functional)
|