Atomizer/docs/archive/historical/LESSONS_LEARNED.md

# Lessons Learned - Atomizer Optimization System

This document captures lessons learned from optimization studies to continuously improve the system.

## Date: 2025-11-19 - Circular Plate Frequency Tuning Study

### What Worked Well

1. **Hybrid Study Creator** - Successfully auto-generated complete optimization workflow
   - Automatically detected design variables from NX expressions
   - Correctly matched objectives to available simulation results
   - Generated working extractor code for eigenvalue extraction
   - Created comprehensive configuration reports

2. **Modal Analysis Support** - System now handles eigenvalue extraction properly
   - Fixed nx_solver.py to select correct solution-specific OP2 files
   - Solution name parameter properly passed through solve pipeline
   - Eigenvalue extractor successfully reads LAMA tables from OP2

3. **Incremental History Tracking** - Added real-time progress monitoring
   - JSON file updated after each trial
   - Enables live monitoring of optimization progress
   - Provides backup if optimization is interrupted

### Critical Bugs Fixed

1. **nx_solver OP2 File Selection Bug**
   - **Problem**: nx_solver was hardcoded to return `-solution_1.op2` files
   - **Root Cause**: Missing solution_name parameter support in run_simulation()
   - **Solution**: Added solution_name parameter that dynamically constructs correct OP2 filename
   - **Location**: [nx_solver.py:181-197](../optimization_engine/nx_solver.py#L181-L197)
   - **Impact**: HIGH - Blocks all modal analysis optimizations

2. **Missing Incremental History Tracking**
   - **Problem**: Generated runners only saved to Optuna database, no live JSON file
   - **Root Cause**: hybrid_study_creator template didn't include history tracking
   - **Solution**: Added history initialization and per-trial saving to template
   - **Location**: [hybrid_study_creator.py:388-436](../optimization_engine/hybrid_study_creator.py#L388-L436)
   - **Impact**: MEDIUM - User experience issue, no technical blocker

3. **No Automatic Report Generation**
   - **Problem**: User had to manually request reports after optimization
   - **Root Cause**: System wasn't proactive about generating human-readable output
   - **Solution**: Created generate_report.py and integrated into hybrid runner template
   - **Location**: [generate_report.py](../optimization_engine/generate_report.py)
   - **Impact**: MEDIUM - User experience issue

### System Improvements Made

1. **Created Automatic Report Generator**
   - Location: `optimization_engine/generate_report.py`
   - Generates comprehensive human-readable reports
   - Includes statistics, top trials, success assessment
   - Automatically called at end of optimization

2. **Updated Hybrid Study Creator**
   - Now generates runners with automatic report generation
   - Includes incremental history tracking by default
   - Better documentation in generated code

3. **Created Lessons Learned Documentation**
   - This file! To track improvements over time
   - Should be updated after each study

### Proactive Behaviors to Add

1. **Automatic report generation** - DONE ✓
   - System should automatically generate reports after optimization completes
   - No need for user to request this

2. **Progress summaries during long runs**
   - Could periodically print best-so-far results
   - Show estimated time remaining
   - Alert if optimization appears stuck

3. **Automatic visualization**
   - Generate plots of design space exploration
   - Show convergence curves
   - Visualize parameter sensitivities

4. **Study validation before running**
   - Check if design variable bounds make physical sense
   - Verify baseline simulation runs successfully
   - Estimate total runtime based on trial time

### Technical Learnings

1. **NX Nastran OP2 File Naming**
   - When solving specific solutions via journal mode: `<base>-<solution_name_lowercase>.op2`
   - When solving all solutions: Files named `-solution_1`, `-solution_2`, etc.
   - Solution names must be converted to lowercase and spaces replaced with underscores
   - Example: "Solution_Normal_Modes" → "solution_normal_modes"

2. **pyNastran Eigenvalue Access**
   - Eigenvalues stored in `model.eigenvalues` dict (keyed by subcase)
   - Each subcase has a `RealEigenvalues` object
   - Access via `eigenvalues_obj.eigenvalues` (not `.eigrs` or `.data`)
   - Need to convert eigenvalues to frequencies: `f = sqrt(eigenvalue) / (2*pi)`

3. **Optuna Study Continuation**
   - Using `load_if_exists=True` allows resuming interrupted studies
   - Trial numbers continue from previous runs
   - History tracking needs to handle this gracefully

### Future Improvements Needed

1. **Better Objective Function Formulation**
   - Current: Minimize absolute error from target
   - Issue: Doesn't penalize being above vs below target differently
   - Suggestion: Add constraint handling for hard requirements

2. **Smarter Initial Sampling**
   - Current: Pure random sampling
   - Suggestion: Use Latin hypercube or Sobol sequences for better coverage

3. **Adaptive Trial Allocation**
   - Current: Fixed number of trials
   - Suggestion: Stop automatically when tolerance is met
   - Or: Increase trials if not converging

4. **Multi-Objective Support**
   - Current: Single objective only
   - Many real problems have multiple competing objectives
   - Need Pareto frontier visualization

5. **Sensitivity Analysis**
   - Automatically identify which design variables matter most
   - Could reduce dimensionality for faster optimization

### Template for Future Entries

```markdown
## Date: YYYY-MM-DD - Study Name

### What Worked Well
- ...

### Critical Bugs Fixed
1. **Bug Title**
   - **Problem**:
   - **Root Cause**:
   - **Solution**:
   - **Location**:
   - **Impact**:

### System Improvements Made
- ...

### Proactive Behaviors to Add
- ...

### Technical Learnings
- ...

### Future Improvements Needed
- ...
```

## Continuous Improvement Process

1. **After Each Study**:
   - Review what went wrong
   - Document bugs and fixes
   - Identify missing proactive behaviors
   - Update this document

2. **Monthly Review**:
   - Look for patterns in issues
   - Prioritize improvements
   - Update system architecture if needed

3. **Version Tracking**:
   - Tag major improvements with version numbers
   - Keep changelog synchronized
   - Document breaking changes