Atomizer/docs/protocols/system/SYS_17_STUDY_INSIGHTS.md

# SYS_16: Study Insights

**Version**: 1.0.0
**Status**: Active
**Purpose**: Physics-focused visualizations for FEA optimization results

---

## Overview

Study Insights provide **physics understanding** of optimization results through interactive 3D visualizations. Unlike the Analysis page (which shows optimizer metrics like convergence and Pareto fronts), Insights answer the question: **"What does this design actually look like?"**

### Analysis vs Insights

| Aspect | **Analysis** | **Insights** |
|--------|--------------|--------------|
| Focus | Optimization performance | Physics understanding |
| Questions | "Is the optimizer converging?" | "What does the best design look like?" |
| Data Source | `study.db` (trials, objectives) | Simulation outputs (OP2, mesh, fields) |
| Typical Plots | Convergence, Pareto, parameters | 3D surfaces, stress contours, mode shapes |
| When Used | During/after optimization | After specific trial of interest |

---

## Available Insight Types

| Type ID | Name | Applicable To | Data Required |
|---------|------|---------------|---------------|
| `zernike_dashboard` | **Zernike Dashboard (RECOMMENDED)** | Mirror, optics | OP2 with displacement subcases |
| `zernike_wfe` | Zernike WFE Analysis | Mirror, optics | OP2 with displacement subcases |
| `zernike_opd_comparison` | Zernike OPD Method Comparison | Mirror, optics, lateral | OP2 with displacement subcases |
| `msf_zernike` | MSF Zernike Analysis | Mirror, optics | OP2 with displacement subcases |
| `stress_field` | Stress Distribution | Structural, bracket, beam | OP2 with stress results |
| `modal` | Modal Analysis | Vibration, dynamic | OP2 with eigenvalue/eigenvector |
| `thermal` | Thermal Analysis | Thermo-structural | OP2 with temperature results |
| `design_space` | Design Space Explorer | All optimization studies | study.db with 5+ trials |

### Zernike Method Comparison: Standard vs OPD

The Zernike insights now support **two WFE computation methods**:

| Method | Description | When to Use |
|--------|-------------|-------------|
| **Standard (Z-only)** | Uses only Z-displacement at original (x,y) coordinates | Quick analysis, negligible lateral displacement |
| **OPD (X,Y,Z)** ← RECOMMENDED | Accounts for lateral (X,Y) displacement via interpolation | Any surface with gravity loads, most rigorous |

**How OPD method works**:
1. Builds interpolator from undeformed BDF mesh geometry
2. For each deformed node at `(x+dx, y+dy, z+dz)`, interpolates `Z_ideal` at new XY position
3. Computes `WFE = z_deformed - Z_ideal(x_def, y_def)`
4. Fits Zernike polynomials to the surface error map

**Typical difference**: OPD method gives **8-11% higher** WFE values than Standard (more conservative/accurate).

---

## Architecture

### Module Structure

```
optimization_engine/insights/
├── __init__.py               # Registry and public API
├── base.py                   # StudyInsight base class, InsightConfig, InsightResult
├── zernike_wfe.py            # Mirror wavefront error visualization (50 modes)
├── zernike_opd_comparison.py # OPD vs Standard method comparison (lateral disp. analysis)
├── msf_zernike.py            # MSF band decomposition (100 modes, LSF/MSF/HSF)
├── stress_field.py           # Stress contour visualization
├── modal_analysis.py         # Mode shape visualization
├── thermal_field.py      # Temperature distribution
└── design_space.py       # Parameter-objective exploration
```

### Class Hierarchy

```python
StudyInsight (ABC)
├── ZernikeDashboardInsight     # RECOMMENDED: Unified dashboard with all views
├── ZernikeWFEInsight           # Standard 50-mode WFE analysis (with OPD toggle)
├── ZernikeOPDComparisonInsight # OPD method comparison (lateral displacement)
├── MSFZernikeInsight           # 100-mode MSF band analysis
├── StressFieldInsight
├── ModalInsight
├── ThermalInsight
└── DesignSpaceInsight
```

### Key Classes

#### StudyInsight (Base Class)

```python
class StudyInsight(ABC):
    insight_type: str       # Unique identifier (e.g., 'zernike_wfe')
    name: str               # Human-readable name
    description: str        # What this insight shows
    applicable_to: List[str]  # Study types this applies to

    def can_generate(self) -> bool:
        """Check if required data exists."""

    def generate(self, config: InsightConfig) -> InsightResult:
        """Generate visualization."""

    def generate_html(self, trial_id=None, **kwargs) -> Path:
        """Generate standalone HTML file."""

    def get_plotly_data(self, trial_id=None, **kwargs) -> dict:
        """Get Plotly figure for dashboard embedding."""
```

#### InsightConfig

```python
@dataclass
class InsightConfig:
    trial_id: Optional[int] = None     # Which trial to visualize
    colorscale: str = 'Turbo'          # Plotly colorscale
    amplification: float = 1.0         # Deformation scale factor
    lighting: bool = True              # 3D lighting effects
    output_dir: Optional[Path] = None  # Where to save HTML
    extra: Dict[str, Any] = {}         # Type-specific config
```

#### InsightResult

```python
@dataclass
class InsightResult:
    success: bool
    html_path: Optional[Path] = None        # Generated HTML file
    plotly_figure: Optional[dict] = None    # Figure for dashboard
    summary: Optional[dict] = None          # Key metrics
    error: Optional[str] = None             # Error message if failed
```

---

## Usage

### Python API

```python
from optimization_engine.insights import get_insight, list_available_insights
from pathlib import Path

study_path = Path("studies/my_mirror_study")

# List what's available
available = list_available_insights(study_path)
for info in available:
    print(f"{info['type']}: {info['name']}")

# Generate specific insight
insight = get_insight('zernike_wfe', study_path)
if insight and insight.can_generate():
    result = insight.generate()
    print(f"Generated: {result.html_path}")
    print(f"40-20 Filtered RMS: {result.summary['40_vs_20_filtered_rms']:.2f} nm")
```

### CLI

```bash
# List all insight types
python -m optimization_engine.insights list

# Generate all available insights for a study
python -m optimization_engine.insights generate studies/my_study

# Generate specific insight
python -m optimization_engine.insights generate studies/my_study --type zernike_wfe
```

### With Configuration

```python
from optimization_engine.insights import get_insight, InsightConfig

insight = get_insight('stress_field', study_path)
config = InsightConfig(
    colorscale='Hot',
    extra={
        'yield_stress': 250,  # MPa
        'stress_unit': 'MPa'
    }
)
result = insight.generate(config)
```

---

## Insight Type Details

### 0. Zernike Dashboard (`zernike_dashboard`) - RECOMMENDED

**Purpose**: Unified dashboard with all orientations (40°, 60°, 90°) and MSF band analysis on one page. Light theme, executive summary, and method comparison.

**Generates**: 1 comprehensive HTML file with:
- Executive summary with metric cards (40-20, 60-20, MFG workload)
- MSF band analysis (LSF/MSF/HSF decomposition)
- 3D surface plots for each orientation
- Zernike coefficient bar charts color-coded by band

**Configuration**:
```python
config = InsightConfig(
    extra={
        'n_modes': 50,
        'filter_low_orders': 4,
        'theme': 'light',  # Light theme for reports
    }
)
```

**Summary Output**:
```python
{
    '40_vs_20_filtered_rms': 6.53,   # nm (OPD method)
    '60_vs_20_filtered_rms': 14.21,  # nm (OPD method)
    '90_optician_workload': 26.34,   # nm (J1-J3 filtered)
    'msf_rss_40': 2.1,               # nm (MSF band contribution)
}
```

### 1. Zernike WFE Analysis (`zernike_wfe`)

**Purpose**: Visualize wavefront error for mirror optimization with Zernike polynomial decomposition. **Now includes Standard/OPD method toggle and lateral displacement maps**.

**Generates**: 6 HTML files
- `zernike_*_40_vs_20.html` - 40° vs 20° relative WFE (with method toggle)
- `zernike_*_40_lateral.html` - Lateral displacement map for 40°
- `zernike_*_60_vs_20.html` - 60° vs 20° relative WFE (with method toggle)
- `zernike_*_60_lateral.html` - Lateral displacement map for 60°
- `zernike_*_90_mfg.html` - 90° manufacturing (with method toggle)
- `zernike_*_90_mfg_lateral.html` - Lateral displacement map for 90°

**Features**:
- Toggle buttons to switch between **Standard (Z-only)** and **OPD (X,Y,Z)** methods
- Toggle between WFE view and **ΔX, ΔY, ΔZ displacement components**
- Metrics comparison table showing both methods side-by-side
- Lateral displacement statistics (Max, RMS in µm)

**Configuration**:
```python
config = InsightConfig(
    amplification=0.5,  # Reduce deformation scaling
    colorscale='Turbo',
    extra={
        'n_modes': 50,
        'filter_low_orders': 4,  # Remove piston, tip, tilt, defocus
        'disp_unit': 'mm',
    }
)
```

**Summary Output**:
```python
{
    '40_vs_20_filtered_rms_std': 6.01,   # nm (Standard method)
    '40_vs_20_filtered_rms_opd': 6.53,   # nm (OPD method)
    '60_vs_20_filtered_rms_std': 12.81,  # nm
    '60_vs_20_filtered_rms_opd': 14.21,  # nm
    '90_mfg_filtered_rms_std': 24.5,     # nm
    '90_mfg_filtered_rms_opd': 26.34,    # nm
    '90_optician_workload': 26.34,       # nm (J1-J3 filtered)
    'lateral_40_max_um': 0.234,          # µm max lateral displacement
    'lateral_60_max_um': 0.312,          # µm
    'lateral_90_max_um': 0.089,          # µm
}
```

### 2. MSF Zernike Analysis (`msf_zernike`)

**Purpose**: Detailed mid-spatial frequency analysis for telescope mirrors with gravity-induced support print-through.

**Generates**: 1 comprehensive HTML file with:
- Band decomposition table (LSF/MSF/HSF RSS metrics)
- MSF-only 3D surface visualization
- Coefficient bar chart color-coded by band
- Dominant MSF mode identification
- Mesh resolution analysis

**Band Definitions** (for 1.2m class mirror):

| Band | Zernike Order | Feature Size | Physical Meaning |
|------|---------------|--------------|------------------|
| LSF  | n ≤ 10        | > 120 mm     | M2 hexapod correctable |
| MSF  | n = 11-50     | 24-109 mm    | Support print-through |
| HSF  | n > 50        | < 24 mm      | Near mesh resolution limit |

**Configuration**:
```python
config = InsightConfig(
    extra={
        'n_modes': 100,      # Higher than zernike_wfe (100 vs 50)
        'lsf_max': 10,       # n ≤ 10 is LSF
        'msf_max': 50,       # n = 11-50 is MSF
        'disp_unit': 'mm',
    }
)
```

**Analyses Performed**:
- Absolute WFE at each orientation (40°, 60°, 90°)
- Relative to 20° (operational reference)
- Relative to 90° (manufacturing/polishing reference)

**Summary Output**:
```python
{
    'n_modes': 100,
    'lsf_max_order': 10,
    'msf_max_order': 50,
    'mesh_nodes': 78290,
    'mesh_spacing_mm': 4.1,
    'max_resolvable_order': 157,
    '40deg_vs_20deg_lsf_rss': 12.3,   # nm
    '40deg_vs_20deg_msf_rss': 8.7,    # nm - KEY METRIC
    '40deg_vs_20deg_total_rss': 15.2, # nm
    '40deg_vs_20deg_msf_pct': 33.0,   # % of total in MSF band
    # ... similar for 60deg, 90deg
}
```

**When to Use**:
- Analyzing support structure print-through
- Quantifying gravity-induced MSF content
- Comparing MSF at different orientations
- Validating mesh resolution is adequate for MSF capture

---

### 3. Stress Distribution (`stress_field`)

**Purpose**: Visualize Von Mises stress distribution with hot spot identification.

**Configuration**:
```python
config = InsightConfig(
    colorscale='Hot',
    extra={
        'yield_stress': 250,    # MPa - shows safety factor
        'stress_unit': 'MPa',
    }
)
```

**Summary Output**:
```python
{
    'max_stress': 187.5,      # MPa
    'mean_stress': 45.2,      # MPa
    'p95_stress': 120.3,      # 95th percentile
    'p99_stress': 165.8,      # 99th percentile
    'safety_factor': 1.33,    # If yield_stress provided
}
```

### 4. Modal Analysis (`modal`)

**Purpose**: Visualize natural frequencies and mode shapes.

**Configuration**:
```python
config = InsightConfig(
    amplification=50.0,  # Mode shape scale
    extra={
        'n_modes': 20,      # Number of modes to show
        'show_mode': 1,     # Which mode shape to display
    }
)
```

**Summary Output**:
```python
{
    'n_modes': 20,
    'first_frequency_hz': 125.4,
    'frequencies_hz': [125.4, 287.8, 312.5, ...],
}
```

### 5. Thermal Analysis (`thermal`)

**Purpose**: Visualize temperature distribution and gradients.

**Configuration**:
```python
config = InsightConfig(
    colorscale='Thermal',
    extra={
        'temp_unit': 'K',  # or 'C', 'F'
    }
)
```

**Summary Output**:
```python
{
    'max_temp': 423.5,    # K
    'min_temp': 293.0,    # K
    'mean_temp': 345.2,   # K
    'temp_range': 130.5,  # K
}
```

### 6. Design Space Explorer (`design_space`)

**Purpose**: Visualize parameter-objective relationships from optimization trials.

**Configuration**:
```python
config = InsightConfig(
    extra={
        'primary_objective': 'filtered_rms',  # Color by this objective
    }
)
```

**Summary Output**:
```python
{
    'n_trials': 100,
    'n_params': 4,
    'n_objectives': 2,
    'best_trial_id': 47,
    'best_params': {'p1': 0.5, 'p2': 1.2, ...},
    'best_values': {'filtered_rms': 45.2, 'mass': 2.34},
}
```

---

## Output Directory

Insights are saved to `{study}/3_insights/`:

```
studies/my_study/
├── 1_setup/
├── 2_results/
└── 3_insights/               # Created by insights module
    ├── zernike_20241220_143022_40_vs_20.html
    ├── zernike_20241220_143022_60_vs_20.html
    ├── zernike_20241220_143022_90_mfg.html
    ├── stress_20241220_143025.html
    └── design_space_20241220_143030.html
```

---

## Creating New Insight Types

To add a new insight type (power_user+):

### 1. Create the insight class

```python
# optimization_engine/insights/my_insight.py

from .base import StudyInsight, InsightConfig, InsightResult, register_insight

@register_insight
class MyInsight(StudyInsight):
    insight_type = "my_insight"
    name = "My Custom Insight"
    description = "Description of what it shows"
    applicable_to = ["structural", "all"]

    def can_generate(self) -> bool:
        # Check if required data exists
        return self.results_path.exists()

    def _generate(self, config: InsightConfig) -> InsightResult:
        # Generate visualization
        # ... build Plotly figure ...

        html_path = config.output_dir / f"my_insight_{timestamp}.html"
        html_path.write_text(fig.to_html(...))

        return InsightResult(
            success=True,
            html_path=html_path,
            summary={'key_metric': value}
        )
```

### 2. Register in `__init__.py`

```python
from .my_insight import MyInsight
```

### 3. Test

```bash
python -m optimization_engine.insights list
# Should show "my_insight" in the list
```

---

## Dashboard Integration

The Insights tab in the Atomizer Dashboard provides a 3-step workflow:

### Step 1: Select Iteration
- Lists all available iterations (iter1, iter2, etc.) and best_design_archive
- Shows OP2 file name and modification timestamp
- Auto-selects "Best Design (Recommended)" if available

### Step 2: Choose Insight Type
- Groups insights by category (Optical, Structural, Thermal, etc.)
- Shows insight name and description
- Click to select, then "Generate Insight"

### Step 3: View Result
- Displays summary metrics (RMS values, etc.)
- Embedded Plotly visualization (if available)
- "Open Full View" button for multi-file insights (like Zernike WFE)
- Fullscreen mode for detailed analysis

### API Endpoints

```
GET  /api/insights/studies/{id}/iterations      # List available iterations
GET  /api/insights/studies/{id}/available       # List available insight types
GET  /api/insights/studies/{id}/generated       # List previously generated files
POST /api/insights/studies/{id}/generate/{type} # Generate insight for iteration
GET  /api/insights/studies/{id}/view/{type}     # View generated HTML
```

### Generate Request Body

```json
{
  "iteration": "best_design_archive",  // or "iter5", etc.
  "trial_id": null,                    // Optional specific trial
  "config": {}                         // Insight-specific config
}
```

---

## Version History

| Version | Date | Changes |
|---------|------|---------|
| 1.3.0 | 2025-12-22 | Added ZernikeDashboardInsight (unified view), OPD method toggle, lateral displacement maps |
| 1.2.0 | 2024-12-22 | Dashboard overhaul: 3-step workflow, iteration selection, faster loading |
| 1.1.0 | 2024-12-21 | Added MSF Zernike Analysis insight (6 insight types) |
| 1.0.0 | 2024-12-20 | Initial release with 5 insight types |