feat: Add MLP surrogate with Turbo Mode for 100x faster optimization
Neural Acceleration (MLP Surrogate): - Add run_nn_optimization.py with hybrid FEA/NN workflow - MLP architecture: 4-layer (64->128->128->64) with BatchNorm/Dropout - Three workflow modes: - --all: Sequential export->train->optimize->validate - --hybrid-loop: Iterative Train->NN->Validate->Retrain cycle - --turbo: Aggressive single-best validation (RECOMMENDED) - Turbo mode: 5000 NN trials + 50 FEA validations in ~12 minutes - Separate nn_study.db to avoid overloading dashboard Performance Results (bracket_pareto_3obj study): - NN prediction errors: mass 1-5%, stress 1-4%, stiffness 5-15% - Found minimum mass designs at boundary (angle~30deg, thick~30mm) - 100x speedup vs pure FEA exploration Protocol Operating System: - Add .claude/skills/ with Bootstrap, Cheatsheet, Context Loader - Add docs/protocols/ with operations (OP_01-06) and system (SYS_10-14) - Update SYS_14_NEURAL_ACCELERATION.md with MLP Turbo Mode docs NX Automation: - Add optimization_engine/hooks/ for NX CAD/CAE automation - Add study_wizard.py for guided study creation - Fix FEM mesh update: load idealized part before UpdateFemodel() New Study: - bracket_pareto_3obj: 3-objective Pareto (mass, stress, stiffness) - 167 FEA trials + 5000 NN trials completed - Demonstrates full hybrid workflow 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
Antoine
302
docs/protocols/operations/OP_04_ANALYZE_RESULTS.md
Normal file
302
docs/protocols/operations/OP_04_ANALYZE_RESULTS.md
Normal file
@@ -0,0 +1,302 @@
|
||||
# OP_04: Analyze Results
|
||||
|
||||
<!--
|
||||
PROTOCOL: Analyze Optimization Results
|
||||
LAYER: Operations
|
||||
VERSION: 1.0
|
||||
STATUS: Active
|
||||
LAST_UPDATED: 2025-12-05
|
||||
PRIVILEGE: user
|
||||
LOAD_WITH: []
|
||||
-->
|
||||
|
||||
## Overview
|
||||
|
||||
This protocol covers analyzing optimization results, including extracting best solutions, generating reports, comparing designs, and interpreting Pareto fronts.
|
||||
|
||||
---
|
||||
|
||||
## When to Use
|
||||
|
||||
| Trigger | Action |
|
||||
|---------|--------|
|
||||
| "results", "what did we find" | Follow this protocol |
|
||||
| "best design" | Extract best trial |
|
||||
| "compare", "trade-off" | Pareto analysis |
|
||||
| "report" | Generate summary |
|
||||
| Optimization complete | Analyze and document |
|
||||
|
||||
---
|
||||
|
||||
## Quick Reference
|
||||
|
||||
**Key Outputs**:
|
||||
| Output | Location | Purpose |
|
||||
|--------|----------|---------|
|
||||
| Best parameters | `study.best_params` | Optimal design |
|
||||
| Pareto front | `study.best_trials` | Trade-off solutions |
|
||||
| Trial history | `study.trials` | Full exploration |
|
||||
| Intelligence report | `intelligent_optimizer/` | Algorithm insights |
|
||||
|
||||
---
|
||||
|
||||
## Analysis Methods
|
||||
|
||||
### 1. Single-Objective Results
|
||||
|
||||
```python
|
||||
import optuna
|
||||
|
||||
study = optuna.load_study(
|
||||
study_name='my_study',
|
||||
storage='sqlite:///2_results/study.db'
|
||||
)
|
||||
|
||||
# Best result
|
||||
print(f"Best value: {study.best_value}")
|
||||
print(f"Best parameters: {study.best_params}")
|
||||
print(f"Best trial: #{study.best_trial.number}")
|
||||
|
||||
# Get full best trial details
|
||||
best = study.best_trial
|
||||
print(f"User attributes: {best.user_attrs}")
|
||||
```
|
||||
|
||||
### 2. Multi-Objective Results (Pareto Front)
|
||||
|
||||
```python
|
||||
import optuna
|
||||
|
||||
study = optuna.load_study(
|
||||
study_name='my_study',
|
||||
storage='sqlite:///2_results/study.db'
|
||||
)
|
||||
|
||||
# All Pareto-optimal solutions
|
||||
pareto_trials = study.best_trials
|
||||
print(f"Pareto front size: {len(pareto_trials)}")
|
||||
|
||||
# Print all Pareto solutions
|
||||
for trial in pareto_trials:
|
||||
print(f"Trial {trial.number}: {trial.values} - {trial.params}")
|
||||
|
||||
# Find extremes
|
||||
# Assuming objectives: [stiffness (max), mass (min)]
|
||||
best_stiffness = max(pareto_trials, key=lambda t: t.values[0])
|
||||
lightest = min(pareto_trials, key=lambda t: t.values[1])
|
||||
|
||||
print(f"Best stiffness: Trial {best_stiffness.number}")
|
||||
print(f"Lightest: Trial {lightest.number}")
|
||||
```
|
||||
|
||||
### 3. Parameter Importance
|
||||
|
||||
```python
|
||||
import optuna
|
||||
|
||||
study = optuna.load_study(...)
|
||||
|
||||
# Parameter importance (which parameters matter most)
|
||||
importance = optuna.importance.get_param_importances(study)
|
||||
for param, score in importance.items():
|
||||
print(f"{param}: {score:.3f}")
|
||||
```
|
||||
|
||||
### 4. Constraint Analysis
|
||||
|
||||
```python
|
||||
# Find feasibility rate
|
||||
completed = [t for t in study.trials if t.state == optuna.trial.TrialState.COMPLETE]
|
||||
pruned = [t for t in study.trials if t.state == optuna.trial.TrialState.PRUNED]
|
||||
|
||||
feasibility_rate = len(completed) / (len(completed) + len(pruned))
|
||||
print(f"Feasibility rate: {feasibility_rate:.1%}")
|
||||
|
||||
# Analyze why trials were pruned
|
||||
for trial in pruned[:5]: # First 5 pruned
|
||||
reason = trial.user_attrs.get('pruning_reason', 'Unknown')
|
||||
print(f"Trial {trial.number}: {reason}")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Visualization
|
||||
|
||||
### Using Optuna Dashboard
|
||||
|
||||
```bash
|
||||
optuna-dashboard sqlite:///2_results/study.db
|
||||
# Open http://localhost:8080
|
||||
```
|
||||
|
||||
**Available Plots**:
|
||||
- Optimization history
|
||||
- Parameter importance
|
||||
- Slice plot (parameter vs objective)
|
||||
- Parallel coordinates
|
||||
- Contour plot (2D parameter interaction)
|
||||
|
||||
### Using Atomizer Dashboard
|
||||
|
||||
Navigate to `http://localhost:3000` and select study.
|
||||
|
||||
**Features**:
|
||||
- Pareto front plot with normalization
|
||||
- Parallel coordinates with selection
|
||||
- Real-time convergence chart
|
||||
|
||||
### Custom Visualization
|
||||
|
||||
```python
|
||||
import matplotlib.pyplot as plt
|
||||
import optuna
|
||||
|
||||
study = optuna.load_study(...)
|
||||
|
||||
# Plot optimization history
|
||||
fig = optuna.visualization.plot_optimization_history(study)
|
||||
fig.show()
|
||||
|
||||
# Plot parameter importance
|
||||
fig = optuna.visualization.plot_param_importances(study)
|
||||
fig.show()
|
||||
|
||||
# Plot Pareto front (multi-objective)
|
||||
if len(study.directions) > 1:
|
||||
fig = optuna.visualization.plot_pareto_front(study)
|
||||
fig.show()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Generate Reports
|
||||
|
||||
### Update STUDY_REPORT.md
|
||||
|
||||
After analysis, fill in the template:
|
||||
|
||||
```markdown
|
||||
# Study Report: bracket_optimization
|
||||
|
||||
## Executive Summary
|
||||
- **Trials completed**: 50
|
||||
- **Best mass**: 0.195 kg
|
||||
- **Best parameters**: thickness=4.2mm, width=25.8mm
|
||||
- **Constraint satisfaction**: All constraints met
|
||||
|
||||
## Optimization Progress
|
||||
- Initial best: 0.342 kg (trial 1)
|
||||
- Final best: 0.195 kg (trial 38)
|
||||
- Improvement: 43%
|
||||
|
||||
## Best Designs Found
|
||||
|
||||
### Design 1 (Overall Best)
|
||||
| Parameter | Value |
|
||||
|-----------|-------|
|
||||
| thickness | 4.2 mm |
|
||||
| width | 25.8 mm |
|
||||
|
||||
| Metric | Value | Constraint |
|
||||
|--------|-------|------------|
|
||||
| Mass | 0.195 kg | - |
|
||||
| Max stress | 238.5 MPa | < 250 MPa ✓ |
|
||||
|
||||
## Engineering Recommendations
|
||||
1. Recommended design: Trial 38 parameters
|
||||
2. Safety margin: 4.6% on stress constraint
|
||||
3. Consider manufacturing tolerance analysis
|
||||
```
|
||||
|
||||
### Export to CSV
|
||||
|
||||
```python
|
||||
import pandas as pd
|
||||
|
||||
# All trials to DataFrame
|
||||
trials_data = []
|
||||
for trial in study.trials:
|
||||
if trial.state == optuna.trial.TrialState.COMPLETE:
|
||||
row = {'trial': trial.number, 'value': trial.value}
|
||||
row.update(trial.params)
|
||||
trials_data.append(row)
|
||||
|
||||
df = pd.DataFrame(trials_data)
|
||||
df.to_csv('optimization_results.csv', index=False)
|
||||
```
|
||||
|
||||
### Export Best Design for FEA Validation
|
||||
|
||||
```python
|
||||
# Get best parameters
|
||||
best_params = study.best_params
|
||||
|
||||
# Format for NX expression update
|
||||
for name, value in best_params.items():
|
||||
print(f"{name} = {value}")
|
||||
|
||||
# Or save as JSON
|
||||
import json
|
||||
with open('best_design.json', 'w') as f:
|
||||
json.dump(best_params, f, indent=2)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Intelligence Report (Protocol 10)
|
||||
|
||||
If using Protocol 10, check intelligence files:
|
||||
|
||||
```bash
|
||||
# Landscape analysis
|
||||
cat 2_results/intelligent_optimizer/intelligence_report.json
|
||||
|
||||
# Characterization progress
|
||||
cat 2_results/intelligent_optimizer/characterization_progress.json
|
||||
```
|
||||
|
||||
**Key Insights**:
|
||||
- Landscape classification (smooth/rugged, unimodal/multimodal)
|
||||
- Algorithm recommendation rationale
|
||||
- Parameter correlations
|
||||
- Confidence metrics
|
||||
|
||||
---
|
||||
|
||||
## Validation Checklist
|
||||
|
||||
Before finalizing results:
|
||||
|
||||
- [ ] Best solution satisfies all constraints
|
||||
- [ ] Results are physically reasonable
|
||||
- [ ] Parameter values within manufacturing limits
|
||||
- [ ] Consider re-running FEA on best design to confirm
|
||||
- [ ] Document any anomalies or surprises
|
||||
- [ ] Update STUDY_REPORT.md
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Cause | Solution |
|
||||
|---------|-------|----------|
|
||||
| Best value seems wrong | Constraint not enforced | Check objective function |
|
||||
| No Pareto solutions | All trials failed | Check constraints |
|
||||
| Unexpected best params | Local minimum | Try different starting points |
|
||||
| Can't load study | Wrong path | Verify database location |
|
||||
|
||||
---
|
||||
|
||||
## Cross-References
|
||||
|
||||
- **Preceded By**: [OP_02_RUN_OPTIMIZATION](./OP_02_RUN_OPTIMIZATION.md), [OP_03_MONITOR_PROGRESS](./OP_03_MONITOR_PROGRESS.md)
|
||||
- **Related**: [SYS_11_MULTI_OBJECTIVE](../system/SYS_11_MULTI_OBJECTIVE.md) for Pareto analysis
|
||||
- **Skill**: `.claude/skills/generate-report.md`
|
||||
|
||||
---
|
||||
|
||||
## Version History
|
||||
|
||||
| Version | Date | Changes |
|
||||
|---------|------|---------|
|
||||
| 1.0 | 2025-12-05 | Initial release |
|
||||
Reference in New Issue
Block a user