Atomizer/docs/protocols/operations/OP_04_ANALYZE_RESULTS.md

OP_04: Analyze Results

Overview

This protocol covers analyzing optimization results, including extracting best solutions, generating reports, comparing designs, and interpreting Pareto fronts.

---

When to Use

Trigger	Action
"results", "what did we find"	Follow this protocol
"best design"	Extract best trial
"compare", "trade-off"	Pareto analysis
"report"	Generate summary
Optimization complete	Analyze and document

---

Quick Reference

Key Outputs:

Output	Location	Purpose
Best parameters	study.best_params	Optimal design
Pareto front	study.best_trials	Trade-off solutions
Trial history	study.trials	Full exploration
Intelligence report	intelligent_optimizer/	Algorithm insights

---

Analysis Methods

1. Single-Objective Results

import optuna

study = optuna.load_study(
    study_name='my_study',
    storage='sqlite:///2_results/study.db'
)

# Best result
print(f"Best value: {study.best_value}")
print(f"Best parameters: {study.best_params}")
print(f"Best trial: #{study.best_trial.number}")

# Get full best trial details
best = study.best_trial
print(f"User attributes: {best.user_attrs}")

2. Multi-Objective Results (Pareto Front)

import optuna

study = optuna.load_study(
    study_name='my_study',
    storage='sqlite:///2_results/study.db'
)

# All Pareto-optimal solutions
pareto_trials = study.best_trials
print(f"Pareto front size: {len(pareto_trials)}")

# Print all Pareto solutions
for trial in pareto_trials:
    print(f"Trial {trial.number}: {trial.values} - {trial.params}")

# Find extremes
# Assuming objectives: [stiffness (max), mass (min)]
best_stiffness = max(pareto_trials, key=lambda t: t.values[0])
lightest = min(pareto_trials, key=lambda t: t.values[1])

print(f"Best stiffness: Trial {best_stiffness.number}")
print(f"Lightest: Trial {lightest.number}")

3. Parameter Importance

import optuna

study = optuna.load_study(...)

# Parameter importance (which parameters matter most)
importance = optuna.importance.get_param_importances(study)
for param, score in importance.items():
    print(f"{param}: {score:.3f}")

4. Constraint Analysis

# Find feasibility rate
completed = [t for t in study.trials if t.state == optuna.trial.TrialState.COMPLETE]
pruned = [t for t in study.trials if t.state == optuna.trial.TrialState.PRUNED]

feasibility_rate = len(completed) / (len(completed) + len(pruned))
print(f"Feasibility rate: {feasibility_rate:.1%}")

# Analyze why trials were pruned
for trial in pruned[:5]:  # First 5 pruned
    reason = trial.user_attrs.get('pruning_reason', 'Unknown')
    print(f"Trial {trial.number}: {reason}")

---

Visualization

Using Optuna Dashboard

optuna-dashboard sqlite:///2_results/study.db
# Open http://localhost:8080

Available Plots:

• Optimization history
• Parameter importance
• Slice plot (parameter vs objective)
• Parallel coordinates
• Contour plot (2D parameter interaction)

Using Atomizer Dashboard

Navigate to http://localhost:3000 and select study.

Features:

• Pareto front plot with normalization
• Parallel coordinates with selection
• Real-time convergence chart

Custom Visualization

import matplotlib.pyplot as plt
import optuna

study = optuna.load_study(...)

# Plot optimization history
fig = optuna.visualization.plot_optimization_history(study)
fig.show()

# Plot parameter importance
fig = optuna.visualization.plot_param_importances(study)
fig.show()

# Plot Pareto front (multi-objective)
if len(study.directions) > 1:
    fig = optuna.visualization.plot_pareto_front(study)
    fig.show()

---

Generate Reports

Update STUDY_REPORT.md

After analysis, fill in the template:

# Study Report: bracket_optimization

## Executive Summary
- **Trials completed**: 50
- **Best mass**: 0.195 kg
- **Best parameters**: thickness=4.2mm, width=25.8mm
- **Constraint satisfaction**: All constraints met

## Optimization Progress
- Initial best: 0.342 kg (trial 1)
- Final best: 0.195 kg (trial 38)
- Improvement: 43%

## Best Designs Found

### Design 1 (Overall Best)
| Parameter | Value |
|-----------|-------|
| thickness | 4.2 mm |
| width | 25.8 mm |

| Metric | Value | Constraint |
|--------|-------|------------|
| Mass | 0.195 kg | - |
| Max stress | 238.5 MPa | < 250 MPa ✓ |

## Engineering Recommendations
1. Recommended design: Trial 38 parameters
2. Safety margin: 4.6% on stress constraint
3. Consider manufacturing tolerance analysis

Export to CSV

import pandas as pd

# All trials to DataFrame
trials_data = []
for trial in study.trials:
    if trial.state == optuna.trial.TrialState.COMPLETE:
        row = {'trial': trial.number, 'value': trial.value}
        row.update(trial.params)
        trials_data.append(row)

df = pd.DataFrame(trials_data)
df.to_csv('optimization_results.csv', index=False)

Export Best Design for FEA Validation

# Get best parameters
best_params = study.best_params

# Format for NX expression update
for name, value in best_params.items():
    print(f"{name} = {value}")

# Or save as JSON
import json
with open('best_design.json', 'w') as f:
    json.dump(best_params, f, indent=2)

---

Intelligence Report (Protocol 10)

If using Protocol 10, check intelligence files:

# Landscape analysis
cat 2_results/intelligent_optimizer/intelligence_report.json

# Characterization progress
cat 2_results/intelligent_optimizer/characterization_progress.json

Key Insights:

• Landscape classification (smooth/rugged, unimodal/multimodal)
• Algorithm recommendation rationale
• Parameter correlations
• Confidence metrics

---

Validation Checklist

Before finalizing results:

• Best solution satisfies all constraints
• Results are physically reasonable
• Parameter values within manufacturing limits
• Consider re-running FEA on best design to confirm
• Document any anomalies or surprises
• Update STUDY_REPORT.md

---

Troubleshooting

Symptom	Cause	Solution
Best value seems wrong	Constraint not enforced	Check objective function
No Pareto solutions	All trials failed	Check constraints
Unexpected best params	Local minimum	Try different starting points
Can't load study	Wrong path	Verify database location

---

Cross-References

• Preceded By: OP_02_RUN_OPTIMIZATION, OP_03_MONITOR_PROGRESS
• Related: SYS_11_MULTI_OBJECTIVE for Pareto analysis
• Skill: .claude/skills/generate-report.md

---

Version History

Version	Date	Changes
1.0	2025-12-05	Initial release