Atomizer/docs/protocols/operations/OP_04_ANALYZE_RESULTS.md

# OP_04: Analyze Results

<!--
PROTOCOL: Analyze Optimization Results
LAYER: Operations
VERSION: 1.0
STATUS: Active
LAST_UPDATED: 2025-12-05
PRIVILEGE: user
LOAD_WITH: []
-->

## Overview

This protocol covers analyzing optimization results, including extracting best solutions, generating reports, comparing designs, and interpreting Pareto fronts.

---

## When to Use

| Trigger | Action |
|---------|--------|
| "results", "what did we find" | Follow this protocol |
| "best design" | Extract best trial |
| "compare", "trade-off" | Pareto analysis |
| "report" | Generate summary |
| Optimization complete | Analyze and document |

---

## Quick Reference

**Key Outputs**:
| Output | Location | Purpose |
|--------|----------|---------|
| Best parameters | `study.best_params` | Optimal design |
| Pareto front | `study.best_trials` | Trade-off solutions |
| Trial history | `study.trials` | Full exploration |
| Intelligence report | `intelligent_optimizer/` | Algorithm insights |

---

## Analysis Methods

### 1. Single-Objective Results

```python
import optuna

study = optuna.load_study(
    study_name='my_study',
    storage='sqlite:///2_results/study.db'
)

# Best result
print(f"Best value: {study.best_value}")
print(f"Best parameters: {study.best_params}")
print(f"Best trial: #{study.best_trial.number}")

# Get full best trial details
best = study.best_trial
print(f"User attributes: {best.user_attrs}")
```

### 2. Multi-Objective Results (Pareto Front)

```python
import optuna

study = optuna.load_study(
    study_name='my_study',
    storage='sqlite:///2_results/study.db'
)

# All Pareto-optimal solutions
pareto_trials = study.best_trials
print(f"Pareto front size: {len(pareto_trials)}")

# Print all Pareto solutions
for trial in pareto_trials:
    print(f"Trial {trial.number}: {trial.values} - {trial.params}")

# Find extremes
# Assuming objectives: [stiffness (max), mass (min)]
best_stiffness = max(pareto_trials, key=lambda t: t.values[0])
lightest = min(pareto_trials, key=lambda t: t.values[1])

print(f"Best stiffness: Trial {best_stiffness.number}")
print(f"Lightest: Trial {lightest.number}")
```

### 3. Parameter Importance

```python
import optuna

study = optuna.load_study(...)

# Parameter importance (which parameters matter most)
importance = optuna.importance.get_param_importances(study)
for param, score in importance.items():
    print(f"{param}: {score:.3f}")
```

### 4. Constraint Analysis

```python
# Find feasibility rate
completed = [t for t in study.trials if t.state == optuna.trial.TrialState.COMPLETE]
pruned = [t for t in study.trials if t.state == optuna.trial.TrialState.PRUNED]

feasibility_rate = len(completed) / (len(completed) + len(pruned))
print(f"Feasibility rate: {feasibility_rate:.1%}")

# Analyze why trials were pruned
for trial in pruned[:5]:  # First 5 pruned
    reason = trial.user_attrs.get('pruning_reason', 'Unknown')
    print(f"Trial {trial.number}: {reason}")
```

---

## Visualization

### Using Optuna Dashboard

```bash
optuna-dashboard sqlite:///2_results/study.db
# Open http://localhost:8080
```

**Available Plots**:
- Optimization history
- Parameter importance
- Slice plot (parameter vs objective)
- Parallel coordinates
- Contour plot (2D parameter interaction)

### Using Atomizer Dashboard

Navigate to `http://localhost:3000` and select study.

**Features**:
- Pareto front plot with normalization
- Parallel coordinates with selection
- Real-time convergence chart

### Custom Visualization

```python
import matplotlib.pyplot as plt
import optuna

study = optuna.load_study(...)

# Plot optimization history
fig = optuna.visualization.plot_optimization_history(study)
fig.show()

# Plot parameter importance
fig = optuna.visualization.plot_param_importances(study)
fig.show()

# Plot Pareto front (multi-objective)
if len(study.directions) > 1:
    fig = optuna.visualization.plot_pareto_front(study)
    fig.show()
```

---

## Generate Reports

### Update STUDY_REPORT.md

After analysis, fill in the template:

```markdown
# Study Report: bracket_optimization

## Executive Summary
- **Trials completed**: 50
- **Best mass**: 0.195 kg
- **Best parameters**: thickness=4.2mm, width=25.8mm
- **Constraint satisfaction**: All constraints met

## Optimization Progress
- Initial best: 0.342 kg (trial 1)
- Final best: 0.195 kg (trial 38)
- Improvement: 43%

## Best Designs Found

### Design 1 (Overall Best)
| Parameter | Value |
|-----------|-------|
| thickness | 4.2 mm |
| width | 25.8 mm |

| Metric | Value | Constraint |
|--------|-------|------------|
| Mass | 0.195 kg | - |
| Max stress | 238.5 MPa | < 250 MPa ✓ |

## Engineering Recommendations
1. Recommended design: Trial 38 parameters
2. Safety margin: 4.6% on stress constraint
3. Consider manufacturing tolerance analysis
```

### Export to CSV

```python
import pandas as pd

# All trials to DataFrame
trials_data = []
for trial in study.trials:
    if trial.state == optuna.trial.TrialState.COMPLETE:
        row = {'trial': trial.number, 'value': trial.value}
        row.update(trial.params)
        trials_data.append(row)

df = pd.DataFrame(trials_data)
df.to_csv('optimization_results.csv', index=False)
```

### Export Best Design for FEA Validation

```python
# Get best parameters
best_params = study.best_params

# Format for NX expression update
for name, value in best_params.items():
    print(f"{name} = {value}")

# Or save as JSON
import json
with open('best_design.json', 'w') as f:
    json.dump(best_params, f, indent=2)
```

---

## Intelligence Report (Protocol 10)

If using Protocol 10, check intelligence files:

```bash
# Landscape analysis
cat 2_results/intelligent_optimizer/intelligence_report.json

# Characterization progress
cat 2_results/intelligent_optimizer/characterization_progress.json
```

**Key Insights**:
- Landscape classification (smooth/rugged, unimodal/multimodal)
- Algorithm recommendation rationale
- Parameter correlations
- Confidence metrics

---

## Validation Checklist

Before finalizing results:

- [ ] Best solution satisfies all constraints
- [ ] Results are physically reasonable
- [ ] Parameter values within manufacturing limits
- [ ] Consider re-running FEA on best design to confirm
- [ ] Document any anomalies or surprises
- [ ] Update STUDY_REPORT.md

---

## Troubleshooting

| Symptom | Cause | Solution |
|---------|-------|----------|
| Best value seems wrong | Constraint not enforced | Check objective function |
| No Pareto solutions | All trials failed | Check constraints |
| Unexpected best params | Local minimum | Try different starting points |
| Can't load study | Wrong path | Verify database location |

---

## Cross-References

- **Preceded By**: [OP_02_RUN_OPTIMIZATION](./OP_02_RUN_OPTIMIZATION.md), [OP_03_MONITOR_PROGRESS](./OP_03_MONITOR_PROGRESS.md)
- **Related**: [SYS_11_MULTI_OBJECTIVE](../system/SYS_11_MULTI_OBJECTIVE.md) for Pareto analysis
- **Skill**: `.claude/skills/generate-report.md`

---

## Version History

| Version | Date | Changes |
|---------|------|---------|
| 1.0 | 2025-12-05 | Initial release |