Atomizer/docs/protocols/system/SYS_10_IMSO.md

# SYS_10: Intelligent Multi-Strategy Optimization (IMSO)

<!--
PROTOCOL: Intelligent Multi-Strategy Optimization
LAYER: System
VERSION: 2.1
STATUS: Active
LAST_UPDATED: 2025-12-05
PRIVILEGE: user
LOAD_WITH: []
-->

## Overview

Protocol 10 implements adaptive optimization that automatically characterizes the problem landscape and selects the best optimization algorithm. This two-phase approach combines automated landscape analysis with algorithm-specific optimization.

**Key Innovation**: Adaptive characterization phase that intelligently determines when enough exploration has been done, then transitions to the optimal algorithm.

---

## When to Use

| Trigger | Action |
|---------|--------|
| Single-objective optimization | Use this protocol |
| "adaptive", "intelligent", "IMSO" mentioned | Load this protocol |
| User unsure which algorithm to use | Recommend this protocol |
| Complex landscape suspected | Use this protocol |

**Do NOT use when**: Multi-objective optimization needed (use SYS_11 instead)

---

## Quick Reference

| Parameter | Default | Range | Description |
|-----------|---------|-------|-------------|
| `min_trials` | 10 | 5-50 | Minimum characterization trials |
| `max_trials` | 30 | 10-100 | Maximum characterization trials |
| `confidence_threshold` | 0.85 | 0.0-1.0 | Stopping confidence level |
| `check_interval` | 5 | 1-10 | Trials between checks |

**Landscape → Algorithm Mapping**:

| Landscape Type | Primary Strategy | Fallback |
|----------------|------------------|----------|
| smooth_unimodal | GP-BO | CMA-ES |
| smooth_multimodal | GP-BO | TPE |
| rugged_unimodal | TPE | CMA-ES |
| rugged_multimodal | TPE | - |
| noisy | TPE | - |

---

## Architecture

```
┌─────────────────────────────────────────────────────────────┐
│  PHASE 1: ADAPTIVE CHARACTERIZATION STUDY                   │
│  ─────────────────────────────────────────────────────────  │
│  Sampler: Random/Sobol (unbiased exploration)               │
│  Trials: 10-30 (adapts to problem complexity)               │
│                                                              │
│  Every 5 trials:                                            │
│    → Analyze landscape metrics                              │
│    → Check metric convergence                               │
│    → Calculate characterization confidence                  │
│    → Decide if ready to stop                                │
│                                                              │
│  Stop when:                                                 │
│    ✓ Confidence ≥ 85%                                       │
│    ✓ OR max trials reached (30)                             │
└─────────────────────────────────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────┐
│  TRANSITION: LANDSCAPE ANALYSIS & STRATEGY SELECTION        │
│  ─────────────────────────────────────────────────────────  │
│  Analyze:                                                   │
│    - Smoothness (0-1)                                       │
│    - Multimodality (number of modes)                        │
│    - Parameter correlation                                  │
│    - Noise level                                            │
│                                                              │
│  Classify & Recommend:                                      │
│    smooth_unimodal    → GP-BO (best) or CMA-ES             │
│    smooth_multimodal  → GP-BO                               │
│    rugged_multimodal  → TPE                                 │
│    rugged_unimodal    → TPE or CMA-ES                       │
│    noisy              → TPE (most robust)                   │
└─────────────────────────────────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────┐
│  PHASE 2: OPTIMIZATION STUDY                                │
│  ─────────────────────────────────────────────────────────  │
│  Sampler: Recommended from Phase 1                          │
│  Warm Start: Initialize from best characterization point    │
│  Trials: User-specified (default 50)                        │
└─────────────────────────────────────────────────────────────┘
```

---

## Core Components

### 1. Adaptive Characterization (`adaptive_characterization.py`)

**Confidence Calculation**:
```python
confidence = (
    0.40 * metric_stability_score +      # Are metrics converging?
    0.30 * parameter_coverage_score +    # Explored enough space?
    0.20 * sample_adequacy_score +       # Enough samples for complexity?
    0.10 * landscape_clarity_score       # Clear classification?
)
```

**Stopping Criteria**:
- **Minimum trials**: 10 (baseline data requirement)
- **Maximum trials**: 30 (prevent over-characterization)
- **Confidence threshold**: 85% (high confidence required)
- **Check interval**: Every 5 trials

**Adaptive Behavior**:
```python
# Simple problem (smooth, unimodal, low noise):
if smoothness > 0.6 and unimodal and noise < 0.3:
    required_samples = 10 + dimensionality
    # Stops at ~10-15 trials

# Complex problem (multimodal with N modes):
if multimodal and n_modes > 2:
    required_samples = 10 + 5 * n_modes + 2 * dimensionality
    # Continues to ~20-30 trials
```

### 2. Landscape Analyzer (`landscape_analyzer.py`)

**Metrics Computed**:

| Metric | Method | Interpretation |
|--------|--------|----------------|
| Smoothness (0-1) | Spearman correlation | >0.6: Good for CMA-ES, GP-BO |
| Multimodality | DBSCAN clustering | Detects distinct good regions |
| Correlation | Parameter-objective correlation | Identifies influential params |
| Noise (0-1) | Local consistency check | True simulation instability |

**Landscape Classifications**:
- `smooth_unimodal`: Single smooth bowl
- `smooth_multimodal`: Multiple smooth regions
- `rugged_unimodal`: Single rugged region
- `rugged_multimodal`: Multiple rugged regions
- `noisy`: High noise level

### 3. Strategy Selector (`strategy_selector.py`)

**Algorithm Characteristics**:

**GP-BO (Gaussian Process Bayesian Optimization)**:
- Best for: Smooth, expensive functions (like FEA)
- Explicit surrogate model with uncertainty quantification
- Acquisition function balances exploration/exploitation

**CMA-ES (Covariance Matrix Adaptation Evolution Strategy)**:
- Best for: Smooth unimodal problems
- Fast convergence to local optimum
- Adapts search distribution to landscape

**TPE (Tree-structured Parzen Estimator)**:
- Best for: Multimodal, rugged, or noisy problems
- Robust to noise and discontinuities
- Good global exploration

### 4. Intelligent Optimizer (`intelligent_optimizer.py`)

**Workflow**:
1. Create characterization study (Random/Sobol sampler)
2. Run adaptive characterization with stopping criterion
3. Analyze final landscape
4. Select optimal strategy
5. Create optimization study with recommended sampler
6. Warm-start from best characterization point
7. Run optimization
8. Generate intelligence report

---

## Configuration

Add to `optimization_config.json`:

```json
{
  "intelligent_optimization": {
    "enabled": true,
    "characterization": {
      "min_trials": 10,
      "max_trials": 30,
      "confidence_threshold": 0.85,
      "check_interval": 5
    },
    "landscape_analysis": {
      "min_trials_for_analysis": 10
    },
    "strategy_selection": {
      "allow_cmaes": true,
      "allow_gpbo": true,
      "allow_tpe": true
    }
  },
  "trials": {
    "n_trials": 50
  }
}
```

---

## Usage Example

```python
from pathlib import Path
from optimization_engine.intelligent_optimizer import IntelligentOptimizer

# Create optimizer
optimizer = IntelligentOptimizer(
    study_name="my_optimization",
    study_dir=Path("studies/my_study/2_results"),
    config=optimization_config,
    verbose=True
)

# Define design variables
design_vars = {
    'parameter1': (lower_bound, upper_bound),
    'parameter2': (lower_bound, upper_bound)
}

# Run Protocol 10
results = optimizer.optimize(
    objective_function=my_objective,
    design_variables=design_vars,
    n_trials=50,
    target_value=target,
    tolerance=0.1
)
```

---

## Performance Benefits

**Efficiency**:
- **Simple problems**: Early stop at ~10-15 trials (33% reduction)
- **Complex problems**: Extended characterization at ~20-30 trials
- **Right algorithm**: Uses optimal strategy for landscape type

**Example Performance** (Circular Plate Frequency Tuning):
- TPE alone: ~95 trials to target
- Random search: ~150+ trials
- **Protocol 10**: ~56 trials (**41% reduction**)

---

## Intelligence Reports

Protocol 10 generates three tracking files:

| File | Purpose |
|------|---------|
| `characterization_progress.json` | Metric evolution, confidence progression, stopping decision |
| `intelligence_report.json` | Final landscape classification, parameter correlations, strategy recommendation |
| `strategy_transitions.json` | Phase transitions, algorithm switches, performance metrics |

**Location**: `studies/{study_name}/2_results/intelligent_optimizer/`

---

## Troubleshooting

| Symptom | Cause | Solution |
|---------|-------|----------|
| Characterization takes too long | Complex landscape | Increase `max_trials` or accept longer characterization |
| Wrong algorithm selected | Insufficient exploration | Lower `confidence_threshold` or increase `min_trials` |
| Poor convergence | Mismatch between landscape and algorithm | Review `intelligence_report.json`, consider manual override |
| "No characterization data" | Study not using Protocol 10 | Enable `intelligent_optimization.enabled: true` |

---

## Cross-References

- **Depends On**: None
- **Used By**: [OP_01_CREATE_STUDY](../operations/OP_01_CREATE_STUDY.md), [OP_02_RUN_OPTIMIZATION](../operations/OP_02_RUN_OPTIMIZATION.md)
- **Integrates With**: [SYS_13_DASHBOARD_TRACKING](./SYS_13_DASHBOARD_TRACKING.md)
- **See Also**: [SYS_11_MULTI_OBJECTIVE](./SYS_11_MULTI_OBJECTIVE.md) for multi-objective optimization

---

## Implementation Files

- `optimization_engine/intelligent_optimizer.py` - Main orchestrator
- `optimization_engine/adaptive_characterization.py` - Stopping criterion
- `optimization_engine/landscape_analyzer.py` - Landscape metrics
- `optimization_engine/strategy_selector.py` - Algorithm recommendation

---

## Version History

| Version | Date | Changes |
|---------|------|---------|
| 2.1 | 2025-11-20 | Fixed strategy selector timing, multimodality detection, added simulation validation |
| 2.0 | 2025-11-20 | Added adaptive characterization, two-study architecture |
| 1.0 | 2025-11-19 | Initial implementation |

### Version 2.1 Bug Fixes Detail

**Fix #1: Strategy Selector - Use Characterization Trial Count**

*Problem*: Strategy selector used total trial count (including pruned) instead of characterization trial count, causing wrong algorithm selection after characterization.

*Solution* (`strategy_selector.py`): Use `char_trials = landscape.get('total_trials', trials_completed)` for decisions.

**Fix #2: Improved Multimodality Detection**

*Problem*: False multimodality detected on smooth continuous surfaces (2 modes detected when problem was unimodal).

*Solution* (`landscape_analyzer.py`): Added heuristic - if only 2 modes with smoothness > 0.6 and noise < 0.2, reclassify as unimodal (smooth continuous manifold).

**Fix #3: Simulation Validation**

*Problem*: 20% pruning rate due to extreme parameters causing mesh/solver failures.

*Solution*: Created `simulation_validator.py` with:
- Hard limits (reject invalid parameters)
- Soft limits (warn about risky parameters)
- Aspect ratio checks
- Model-specific validation rules

*Impact*: Reduced pruning rate from 20% to ~5%.