Atomizer/docs/protocols/system/SYS_16_STUDY_INSIGHTS.md

# SYS_16: Study Insights

**Version**: 1.0.0
**Status**: Active
**Purpose**: Physics-focused visualizations for FEA optimization results

---

## Overview

Study Insights provide **physics understanding** of optimization results through interactive 3D visualizations. Unlike the Analysis page (which shows optimizer metrics like convergence and Pareto fronts), Insights answer the question: **"What does this design actually look like?"**

### Analysis vs Insights

| Aspect | **Analysis** | **Insights** |
|--------|--------------|--------------|
| Focus | Optimization performance | Physics understanding |
| Questions | "Is the optimizer converging?" | "What does the best design look like?" |
| Data Source | `study.db` (trials, objectives) | Simulation outputs (OP2, mesh, fields) |
| Typical Plots | Convergence, Pareto, parameters | 3D surfaces, stress contours, mode shapes |
| When Used | During/after optimization | After specific trial of interest |

---

## Available Insight Types

| Type ID | Name | Applicable To | Data Required |
|---------|------|---------------|---------------|
| `zernike_wfe` | Zernike WFE Analysis | Mirror, optics | OP2 with displacement subcases |
| `stress_field` | Stress Distribution | Structural, bracket, beam | OP2 with stress results |
| `modal` | Modal Analysis | Vibration, dynamic | OP2 with eigenvalue/eigenvector |
| `thermal` | Thermal Analysis | Thermo-structural | OP2 with temperature results |
| `design_space` | Design Space Explorer | All optimization studies | study.db with 5+ trials |

---

## Architecture

### Module Structure

```
optimization_engine/insights/
├── __init__.py           # Registry and public API
├── base.py               # StudyInsight base class, InsightConfig, InsightResult
├── zernike_wfe.py        # Mirror wavefront error visualization
├── stress_field.py       # Stress contour visualization
├── modal_analysis.py     # Mode shape visualization
├── thermal_field.py      # Temperature distribution
└── design_space.py       # Parameter-objective exploration
```

### Class Hierarchy

```python
StudyInsight (ABC)
├── ZernikeWFEInsight
├── StressFieldInsight
├── ModalInsight
├── ThermalInsight
└── DesignSpaceInsight
```

### Key Classes

#### StudyInsight (Base Class)

```python
class StudyInsight(ABC):
    insight_type: str       # Unique identifier (e.g., 'zernike_wfe')
    name: str               # Human-readable name
    description: str        # What this insight shows
    applicable_to: List[str]  # Study types this applies to

    def can_generate(self) -> bool:
        """Check if required data exists."""

    def generate(self, config: InsightConfig) -> InsightResult:
        """Generate visualization."""

    def generate_html(self, trial_id=None, **kwargs) -> Path:
        """Generate standalone HTML file."""

    def get_plotly_data(self, trial_id=None, **kwargs) -> dict:
        """Get Plotly figure for dashboard embedding."""
```

#### InsightConfig

```python
@dataclass
class InsightConfig:
    trial_id: Optional[int] = None     # Which trial to visualize
    colorscale: str = 'Turbo'          # Plotly colorscale
    amplification: float = 1.0         # Deformation scale factor
    lighting: bool = True              # 3D lighting effects
    output_dir: Optional[Path] = None  # Where to save HTML
    extra: Dict[str, Any] = {}         # Type-specific config
```

#### InsightResult

```python
@dataclass
class InsightResult:
    success: bool
    html_path: Optional[Path] = None        # Generated HTML file
    plotly_figure: Optional[dict] = None    # Figure for dashboard
    summary: Optional[dict] = None          # Key metrics
    error: Optional[str] = None             # Error message if failed
```

---

## Usage

### Python API

```python
from optimization_engine.insights import get_insight, list_available_insights
from pathlib import Path

study_path = Path("studies/my_mirror_study")

# List what's available
available = list_available_insights(study_path)
for info in available:
    print(f"{info['type']}: {info['name']}")

# Generate specific insight
insight = get_insight('zernike_wfe', study_path)
if insight and insight.can_generate():
    result = insight.generate()
    print(f"Generated: {result.html_path}")
    print(f"40-20 Filtered RMS: {result.summary['40_vs_20_filtered_rms']:.2f} nm")
```

### CLI

```bash
# List all insight types
python -m optimization_engine.insights list

# Generate all available insights for a study
python -m optimization_engine.insights generate studies/my_study

# Generate specific insight
python -m optimization_engine.insights generate studies/my_study --type zernike_wfe
```

### With Configuration

```python
from optimization_engine.insights import get_insight, InsightConfig

insight = get_insight('stress_field', study_path)
config = InsightConfig(
    colorscale='Hot',
    extra={
        'yield_stress': 250,  # MPa
        'stress_unit': 'MPa'
    }
)
result = insight.generate(config)
```

---

## Insight Type Details

### 1. Zernike WFE Analysis (`zernike_wfe`)

**Purpose**: Visualize wavefront error for mirror optimization with Zernike polynomial decomposition.

**Generates**: 3 HTML files
- `zernike_*_40_vs_20.html` - 40° vs 20° relative WFE
- `zernike_*_60_vs_20.html` - 60° vs 20° relative WFE
- `zernike_*_90_mfg.html` - 90° manufacturing (absolute)

**Configuration**:
```python
config = InsightConfig(
    amplification=0.5,  # Reduce deformation scaling
    colorscale='Turbo',
    extra={
        'n_modes': 50,
        'filter_low_orders': 4,  # Remove piston, tip, tilt, defocus
        'disp_unit': 'mm',
    }
)
```

**Summary Output**:
```python
{
    '40_vs_20_filtered_rms': 45.2,  # nm
    '60_vs_20_filtered_rms': 78.3,  # nm
    '90_mfg_filtered_rms': 120.5,   # nm
    '90_optician_workload': 89.4,   # nm (J1-J3 filtered)
}
```

### 2. Stress Distribution (`stress_field`)

**Purpose**: Visualize Von Mises stress distribution with hot spot identification.

**Configuration**:
```python
config = InsightConfig(
    colorscale='Hot',
    extra={
        'yield_stress': 250,    # MPa - shows safety factor
        'stress_unit': 'MPa',
    }
)
```

**Summary Output**:
```python
{
    'max_stress': 187.5,      # MPa
    'mean_stress': 45.2,      # MPa
    'p95_stress': 120.3,      # 95th percentile
    'p99_stress': 165.8,      # 99th percentile
    'safety_factor': 1.33,    # If yield_stress provided
}
```

### 3. Modal Analysis (`modal`)

**Purpose**: Visualize natural frequencies and mode shapes.

**Configuration**:
```python
config = InsightConfig(
    amplification=50.0,  # Mode shape scale
    extra={
        'n_modes': 20,      # Number of modes to show
        'show_mode': 1,     # Which mode shape to display
    }
)
```

**Summary Output**:
```python
{
    'n_modes': 20,
    'first_frequency_hz': 125.4,
    'frequencies_hz': [125.4, 287.8, 312.5, ...],
}
```

### 4. Thermal Analysis (`thermal`)

**Purpose**: Visualize temperature distribution and gradients.

**Configuration**:
```python
config = InsightConfig(
    colorscale='Thermal',
    extra={
        'temp_unit': 'K',  # or 'C', 'F'
    }
)
```

**Summary Output**:
```python
{
    'max_temp': 423.5,    # K
    'min_temp': 293.0,    # K
    'mean_temp': 345.2,   # K
    'temp_range': 130.5,  # K
}
```

### 5. Design Space Explorer (`design_space`)

**Purpose**: Visualize parameter-objective relationships from optimization trials.

**Configuration**:
```python
config = InsightConfig(
    extra={
        'primary_objective': 'filtered_rms',  # Color by this objective
    }
)
```

**Summary Output**:
```python
{
    'n_trials': 100,
    'n_params': 4,
    'n_objectives': 2,
    'best_trial_id': 47,
    'best_params': {'p1': 0.5, 'p2': 1.2, ...},
    'best_values': {'filtered_rms': 45.2, 'mass': 2.34},
}
```

---

## Output Directory

Insights are saved to `{study}/3_insights/`:

```
studies/my_study/
├── 1_setup/
├── 2_results/
└── 3_insights/               # Created by insights module
    ├── zernike_20241220_143022_40_vs_20.html
    ├── zernike_20241220_143022_60_vs_20.html
    ├── zernike_20241220_143022_90_mfg.html
    ├── stress_20241220_143025.html
    └── design_space_20241220_143030.html
```

---

## Creating New Insight Types

To add a new insight type (power_user+):

### 1. Create the insight class

```python
# optimization_engine/insights/my_insight.py

from .base import StudyInsight, InsightConfig, InsightResult, register_insight

@register_insight
class MyInsight(StudyInsight):
    insight_type = "my_insight"
    name = "My Custom Insight"
    description = "Description of what it shows"
    applicable_to = ["structural", "all"]

    def can_generate(self) -> bool:
        # Check if required data exists
        return self.results_path.exists()

    def _generate(self, config: InsightConfig) -> InsightResult:
        # Generate visualization
        # ... build Plotly figure ...

        html_path = config.output_dir / f"my_insight_{timestamp}.html"
        html_path.write_text(fig.to_html(...))

        return InsightResult(
            success=True,
            html_path=html_path,
            summary={'key_metric': value}
        )
```

### 2. Register in `__init__.py`

```python
from .my_insight import MyInsight
```

### 3. Test

```bash
python -m optimization_engine.insights list
# Should show "my_insight" in the list
```

---

## Dashboard Integration (Future)

The insights module is designed for future dashboard integration:

```python
# Backend API endpoint
@app.get("/api/study/{study_name}/insights")
def list_study_insights(study_name: str):
    study_path = Path(f"studies/{study_name}")
    return list_available_insights(study_path)

@app.post("/api/study/{study_name}/insights/{type}/generate")
def generate_insight(study_name: str, type: str, config: dict = {}):
    insight = get_insight(type, Path(f"studies/{study_name}"))
    result = insight.generate(InsightConfig(**config))
    return {
        'success': result.success,
        'html_path': str(result.html_path),
        'summary': result.summary
    }

@app.get("/api/study/{study_name}/insights/{type}/plotly")
def get_insight_plotly(study_name: str, type: str):
    insight = get_insight(type, Path(f"studies/{study_name}"))
    return insight.get_plotly_data()
```

---

## Version History

| Version | Date | Changes |
|---------|------|---------|
| 1.0.0 | 2024-12-20 | Initial release with 5 insight types |