Atomizer/docs/archive/historical/LESSONS_LEARNED.md

Lessons Learned - Atomizer Optimization System

This document captures lessons learned from optimization studies to continuously improve the system.

Date: 2025-11-19 - Circular Plate Frequency Tuning Study

What Worked Well

1. Hybrid Study Creator - Successfully auto-generated complete optimization workflow

   • Automatically detected design variables from NX expressions
   • Correctly matched objectives to available simulation results
   • Generated working extractor code for eigenvalue extraction
   • Created comprehensive configuration reports

2. Modal Analysis Support - System now handles eigenvalue extraction properly

   • Fixed nx_solver.py to select correct solution-specific OP2 files
   • Solution name parameter properly passed through solve pipeline
   • Eigenvalue extractor successfully reads LAMA tables from OP2

3. Incremental History Tracking - Added real-time progress monitoring

   • JSON file updated after each trial
   • Enables live monitoring of optimization progress
   • Provides backup if optimization is interrupted

Critical Bugs Fixed

1. nx_solver OP2 File Selection Bug

   • Problem: nx_solver was hardcoded to return -solution_1.op2 files
   • Root Cause: Missing solution_name parameter support in run_simulation()
   • Solution: Added solution_name parameter that dynamically constructs correct OP2 filename
   • Location: nx_solver.py:181-197
   • Impact: HIGH - Blocks all modal analysis optimizations

2. Missing Incremental History Tracking

   • Problem: Generated runners only saved to Optuna database, no live JSON file
   • Root Cause: hybrid_study_creator template didn't include history tracking
   • Solution: Added history initialization and per-trial saving to template
   • Location: hybrid_study_creator.py:388-436
   • Impact: MEDIUM - User experience issue, no technical blocker

3. No Automatic Report Generation

   • Problem: User had to manually request reports after optimization
   • Root Cause: System wasn't proactive about generating human-readable output
   • Solution: Created generate_report.py and integrated into hybrid runner template
   • Location: generate_report.py
   • Impact: MEDIUM - User experience issue

System Improvements Made

1. Created Automatic Report Generator

   • Location: optimization_engine/generate_report.py
   • Generates comprehensive human-readable reports
   • Includes statistics, top trials, success assessment
   • Automatically called at end of optimization

2. Updated Hybrid Study Creator

   • Now generates runners with automatic report generation
   • Includes incremental history tracking by default
   • Better documentation in generated code

3. Created Lessons Learned Documentation

   • This file! To track improvements over time
   • Should be updated after each study

Proactive Behaviors to Add

1. Automatic report generation - DONE ✓

   • System should automatically generate reports after optimization completes
   • No need for user to request this

2. Progress summaries during long runs

   • Could periodically print best-so-far results
   • Show estimated time remaining
   • Alert if optimization appears stuck

3. Automatic visualization

   • Generate plots of design space exploration
   • Show convergence curves
   • Visualize parameter sensitivities

4. Study validation before running

   • Check if design variable bounds make physical sense
   • Verify baseline simulation runs successfully
   • Estimate total runtime based on trial time

Technical Learnings

1. NX Nastran OP2 File Naming

   • When solving specific solutions via journal mode: <base>-<solution_name_lowercase>.op2
   • When solving all solutions: Files named -solution_1, -solution_2, etc.
   • Solution names must be converted to lowercase and spaces replaced with underscores
   • Example: "Solution_Normal_Modes" → "solution_normal_modes"

2. pyNastran Eigenvalue Access

   • Eigenvalues stored in model.eigenvalues dict (keyed by subcase)
   • Each subcase has a RealEigenvalues object
   • Access via eigenvalues_obj.eigenvalues (not .eigrs or .data)
   • Need to convert eigenvalues to frequencies: f = sqrt(eigenvalue) / (2*pi)

3. Optuna Study Continuation

   • Using load_if_exists=True allows resuming interrupted studies
   • Trial numbers continue from previous runs
   • History tracking needs to handle this gracefully

Future Improvements Needed

1. Better Objective Function Formulation

   • Current: Minimize absolute error from target
   • Issue: Doesn't penalize being above vs below target differently
   • Suggestion: Add constraint handling for hard requirements

2. Smarter Initial Sampling

   • Current: Pure random sampling
   • Suggestion: Use Latin hypercube or Sobol sequences for better coverage

3. Adaptive Trial Allocation

   • Current: Fixed number of trials
   • Suggestion: Stop automatically when tolerance is met
   • Or: Increase trials if not converging

4. Multi-Objective Support

   • Current: Single objective only
   • Many real problems have multiple competing objectives
   • Need Pareto frontier visualization

5. Sensitivity Analysis

   • Automatically identify which design variables matter most
   • Could reduce dimensionality for faster optimization

Template for Future Entries

## Date: YYYY-MM-DD - Study Name

### What Worked Well
- ...

### Critical Bugs Fixed
1. **Bug Title**
   - **Problem**:
   - **Root Cause**:
   - **Solution**:
   - **Location**:
   - **Impact**:

### System Improvements Made
- ...

### Proactive Behaviors to Add
- ...

### Technical Learnings
- ...

### Future Improvements Needed
- ...

Continuous Improvement Process

1. After Each Study:

   • Review what went wrong
   • Document bugs and fixes
   • Identify missing proactive behaviors
   • Update this document

2. Monthly Review:

   • Look for patterns in issues
   • Prioritize improvements
   • Update system architecture if needed

3. Version Tracking:

   • Tag major improvements with version numbers
   • Keep changelog synchronized
   • Document breaking changes