feat: Add OPD method support to Zernike visualization with Standard/OPD toggle

Major improvements to Zernike WFE visualization: - Add ZernikeDashboardInsight: Unified dashboard with all orientations (40°, 60°, 90°) on one page with light theme and executive summary - Add OPD method toggle: Switch between Standard (Z-only) and OPD (X,Y,Z) methods in ZernikeWFEInsight with interactive buttons - Add lateral displacement maps: Visualize X,Y displacement for each orientation - Add displacement component views: Toggle between WFE, ΔX, ΔY, ΔZ in relative views - Add metrics comparison table showing both methods side-by-side New extractors: - extract_zernike_figure.py: ZernikeOPDExtractor using BDF geometry interpolation - extract_zernike_opd.py: Parabola-based OPD with focal length Key finding: OPD method gives 8-11% higher WFE values than Standard method (more conservative/accurate for surfaces with lateral displacement under gravity) Documentation updates: - SYS_12: Added E22 ZernikeOPD as recommended method - SYS_16: Added ZernikeDashboard, updated ZernikeWFE with OPD features - Cheatsheet: Added Zernike method comparison table 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-22 21:03:19 -05:00
parent d089003ced
commit d19fc39a2a
19 changed files with 8117 additions and 396 deletions
--- a/docs/protocols/system/SYS_12_EXTRACTOR_LIBRARY.md
+++ b/docs/protocols/system/SYS_12_EXTRACTOR_LIBRARY.md
@@ -57,6 +57,10 @@ The Extractor Library provides centralized, reusable functions for extracting ph
 | E18 | Modal Mass | `extract_modal_mass()` | .f06 | kg |
 | **Phase 4 (2025-12-19)** | | | | |
 | E19 | Part Introspection | `introspect_part()` | .prt | dict |
+| **Phase 5 (2025-12-22)** | | | | |
+| E20 | Zernike Analytic (Parabola) | `extract_zernike_analytic()` | .op2 + .bdf | nm |
+| E21 | Zernike Method Comparison | `compare_zernike_methods()` | .op2 + .bdf | dict |
+| E22 | **Zernike OPD (RECOMMENDED)** | `extract_zernike_opd()` | .op2 + .bdf | nm |

 ---

@@ -326,6 +330,161 @@ results = builder.evaluate_all()
 # Returns: {'rel_40_vs_20': 4.2, 'rel_60_vs_20': 8.7, 'rms_90': 15.3}
 ```

+### E20: Zernike Analytic (Parabola-Based with Lateral Correction)
+
+**Module**: `optimization_engine.extractors.extract_zernike_opd`
+
+Uses an analytical parabola formula to account for lateral (X, Y) displacements. Requires knowing the focal length.
+
+**Use when**: You know the optical prescription and want to compare against theoretical parabola.
+
+```python
+from optimization_engine.extractors import extract_zernike_analytic, ZernikeAnalyticExtractor
+
+# Full extraction with lateral displacement diagnostics
+result = extract_zernike_analytic(
+    op2_file,
+    subcase="20",
+    focal_length=5000.0,  # Required for analytic method
+)
+
+# Class-based usage
+extractor = ZernikeAnalyticExtractor(op2_file, focal_length=5000.0)
+result = extractor.extract_subcase('20')
+```
+
+### E21: Zernike Method Comparison
+
+**Module**: `optimization_engine.extractors.extract_zernike_opd`
+
+Compare standard (Z-only) vs analytic (parabola) methods.
+
+```python
+from optimization_engine.extractors import compare_zernike_methods
+
+comparison = compare_zernike_methods(op2_file, subcase="20", focal_length=5000.0)
+print(comparison['recommendation'])
+```
+
+### E22: Zernike OPD (RECOMMENDED - Most Rigorous)
+
+**Module**: `optimization_engine.extractors.extract_zernike_figure`
+
+**MOST RIGOROUS METHOD** for computing WFE. Uses the actual BDF geometry (filtered to OP2 nodes) as the reference surface instead of assuming a parabolic shape.
+
+**Advantages over E20 (Analytic)**:
+- No need to know focal length or optical prescription
+- Works with **any surface shape**: parabola, hyperbola, asphere, freeform
+- Uses the actual mesh geometry as the "ideal" surface reference
+- Interpolates `z_figure` at deformed `(x+dx, y+dy)` position for true OPD
+
+**How it works**:
+1. Load BDF geometry for nodes present in OP2 (figure surface nodes)
+2. Build 2D interpolator `z_figure(x, y)` from undeformed coordinates
+3. For each deformed node at `(x0+dx, y0+dy, z0+dz)`:
+   - Interpolate `z_figure` at the deformed (x,y) position
+   - Surface error = `(z0 + dz) - z_interpolated`
+4. Fit Zernike polynomials to the surface error map
+
+```python
+from optimization_engine.extractors import (
+    ZernikeOPDExtractor,
+    extract_zernike_opd,
+    extract_zernike_opd_filtered_rms,
+)
+
+# Full extraction with diagnostics
+result = extract_zernike_opd(op2_file, subcase="20")
+# Returns: {
+#     'global_rms_nm': float,
+#     'filtered_rms_nm': float,
+#     'max_lateral_displacement_um': float,
+#     'rms_lateral_displacement_um': float,
+#     'coefficients': list,           # 50 Zernike coefficients
+#     'method': 'opd',
+#     'figure_file': 'BDF (filtered to OP2)',
+#     ...
+# }
+
+# Simple usage for optimization objective
+rms = extract_zernike_opd_filtered_rms(op2_file, subcase="20")
+
+# Class-based for multi-subcase analysis
+extractor = ZernikeOPDExtractor(op2_file)
+results = extractor.extract_all_subcases()
+```
+
+#### Relative WFE (CRITICAL for Optimization)
+
+**Use `extract_relative()` for computing relative WFE between subcases!**
+
+> **BUG WARNING (V10 Fix - 2025-12-22)**: The WRONG way to compute relative WFE is:
+> ```python
+> # ❌ WRONG: Difference of RMS values
+> result_40 = extractor.extract_subcase("3")
+> result_ref = extractor.extract_subcase("2")
+> rel_40 = abs(result_40['filtered_rms_nm'] - result_ref['filtered_rms_nm'])  # WRONG!
+> ```
+>
+> This computes `|RMS(WFE_40) - RMS(WFE_20)|`, which is NOT the same as `RMS(WFE_40 - WFE_20)`.
+> The difference can be **3-4x lower** than the correct value, leading to false "too good to be true" results.
+
+**The CORRECT approach uses `extract_relative()`:**
+
+```python
+# ✅ CORRECT: Computes node-by-node WFE difference, then fits Zernike, then RMS
+extractor = ZernikeOPDExtractor(op2_file)
+
+rel_40 = extractor.extract_relative("3", "2")  # 40 deg vs 20 deg
+rel_60 = extractor.extract_relative("4", "2")  # 60 deg vs 20 deg
+rel_90 = extractor.extract_relative("1", "2")  # 90 deg vs 20 deg
+
+# Returns: {
+#     'target_subcase': '3',
+#     'reference_subcase': '2',
+#     'method': 'figure_opd_relative',
+#     'relative_global_rms_nm': float,     # RMS of the difference field
+#     'relative_filtered_rms_nm': float,   # Use this for optimization!
+#     'relative_rms_filter_j1to3': float,  # For manufacturing/optician workload
+#     'max_lateral_displacement_um': float,
+#     'rms_lateral_displacement_um': float,
+#     'delta_coefficients': list,          # Zernike coeffs of difference
+# }
+
+# Use in optimization objectives:
+objectives = {
+    'rel_filtered_rms_40_vs_20': rel_40['relative_filtered_rms_nm'],
+    'rel_filtered_rms_60_vs_20': rel_60['relative_filtered_rms_nm'],
+    'mfg_90_optician_workload': rel_90['relative_rms_filter_j1to3'],
+}
+```
+
+**Mathematical Difference**:
+```
+WRONG:  |RMS(WFE_40) - RMS(WFE_20)| = |6.14 - 8.13| = 1.99 nm  ← FALSE!
+CORRECT: RMS(WFE_40 - WFE_20)      = RMS(diff_field) = 6.59 nm ← TRUE!
+```
+
+The Standard `ZernikeExtractor` also has `extract_relative()` if you don't need the OPD method:
+```python
+from optimization_engine.extractors import ZernikeExtractor
+
+extractor = ZernikeExtractor(op2_file, n_modes=50, filter_orders=4)
+rel_40 = extractor.extract_relative("3", "2")  # Z-only method
+```
+
+**Backwards compatibility**: The old names (`ZernikeFigureExtractor`, `extract_zernike_figure`, `extract_zernike_figure_rms`) still work but are deprecated.
+
+**When to use which Zernike method**:
+
+| Method | Class | When to Use | Assumptions |
+|--------|-------|-------------|-------------|
+| Standard (E8) | `ZernikeExtractor` | Quick analysis, negligible lateral displacement | Z-only at original (x,y) |
+| Analytic (E20) | `ZernikeAnalyticExtractor` | Known focal length, parabolic surface | Parabola shape |
+| **OPD (E22)** | `ZernikeOPDExtractor` | **Any surface, most rigorous** | None - uses actual geometry |
+
+**IMPORTANT**: Do NOT provide a figure.dat file unless you're certain it matches your BDF geometry exactly. The default behavior (using BDF geometry filtered to OP2 nodes) is the safest option.
+
 ---

 ## Code Reuse Protocol
@@ -698,7 +857,9 @@ optimization_engine/extractors/
 ├── extract_mass_from_expression.py # E5
 ├── field_data_extractor.py         # E6
 ├── stiffness_calculator.py         # E7
-├── extract_zernike.py              # E8, E9
+├── extract_zernike.py              # E8, E9 (Standard Z-only)
+├── extract_zernike_opd.py          # E20, E21 (Parabola OPD)
+├── extract_zernike_figure.py       # E22 (Figure OPD - most rigorous)
 ├── zernike_helpers.py              # E10
 ├── extract_part_mass_material.py   # E11 (Part mass & material)
 ├── extract_zernike_surface.py      # Surface utilities
@@ -728,3 +889,4 @@ nx_journals/
 | 1.2 | 2025-12-06 | Added Phase 3: E15-E17 (thermal), E18 (modal mass) |
 | 1.3 | 2025-12-07 | Added Element Type Selection Guide; documented shell vs solid stress columns |
 | 1.4 | 2025-12-19 | Added Phase 4: E19 (comprehensive part introspection) |
+| 1.5 | 2025-12-22 | Added Phase 5: E20 (Parabola OPD), E21 (comparison), E22 (Figure OPD - most rigorous) |
--- a/docs/protocols/system/SYS_16_STUDY_INSIGHTS.md
+++ b/docs/protocols/system/SYS_16_STUDY_INSIGHTS.md
@@ -26,12 +26,32 @@ Study Insights provide **physics understanding** of optimization results through

 | Type ID | Name | Applicable To | Data Required |
 |---------|------|---------------|---------------|
+| `zernike_dashboard` | **Zernike Dashboard (RECOMMENDED)** | Mirror, optics | OP2 with displacement subcases |
 | `zernike_wfe` | Zernike WFE Analysis | Mirror, optics | OP2 with displacement subcases |
+| `zernike_opd_comparison` | Zernike OPD Method Comparison | Mirror, optics, lateral | OP2 with displacement subcases |
+| `msf_zernike` | MSF Zernike Analysis | Mirror, optics | OP2 with displacement subcases |
 | `stress_field` | Stress Distribution | Structural, bracket, beam | OP2 with stress results |
 | `modal` | Modal Analysis | Vibration, dynamic | OP2 with eigenvalue/eigenvector |
 | `thermal` | Thermal Analysis | Thermo-structural | OP2 with temperature results |
 | `design_space` | Design Space Explorer | All optimization studies | study.db with 5+ trials |

+### Zernike Method Comparison: Standard vs OPD
+
+The Zernike insights now support **two WFE computation methods**:
+
+| Method | Description | When to Use |
+|--------|-------------|-------------|
+| **Standard (Z-only)** | Uses only Z-displacement at original (x,y) coordinates | Quick analysis, negligible lateral displacement |
+| **OPD (X,Y,Z)** ← RECOMMENDED | Accounts for lateral (X,Y) displacement via interpolation | Any surface with gravity loads, most rigorous |
+
+**How OPD method works**:
+1. Builds interpolator from undeformed BDF mesh geometry
+2. For each deformed node at `(x+dx, y+dy, z+dz)`, interpolates `Z_ideal` at new XY position
+3. Computes `WFE = z_deformed - Z_ideal(x_def, y_def)`
+4. Fits Zernike polynomials to the surface error map
+
+**Typical difference**: OPD method gives **8-11% higher** WFE values than Standard (more conservative/accurate).
+
 ---

 ## Architecture
@@ -40,11 +60,13 @@ Study Insights provide **physics understanding** of optimization results through

 ```
 optimization_engine/insights/
-├── __init__.py           # Registry and public API
-├── base.py               # StudyInsight base class, InsightConfig, InsightResult
-├── zernike_wfe.py        # Mirror wavefront error visualization
-├── stress_field.py       # Stress contour visualization
-├── modal_analysis.py     # Mode shape visualization
+├── __init__.py               # Registry and public API
+├── base.py                   # StudyInsight base class, InsightConfig, InsightResult
+├── zernike_wfe.py            # Mirror wavefront error visualization (50 modes)
+├── zernike_opd_comparison.py # OPD vs Standard method comparison (lateral disp. analysis)
+├── msf_zernike.py            # MSF band decomposition (100 modes, LSF/MSF/HSF)
+├── stress_field.py           # Stress contour visualization
+├── modal_analysis.py         # Mode shape visualization
 ├── thermal_field.py      # Temperature distribution
 └── design_space.py       # Parameter-objective exploration
 ```
@@ -53,7 +75,10 @@ optimization_engine/insights/

 ```python
 StudyInsight (ABC)
-├── ZernikeWFEInsight
+├── ZernikeDashboardInsight     # RECOMMENDED: Unified dashboard with all views
+├── ZernikeWFEInsight           # Standard 50-mode WFE analysis (with OPD toggle)
+├── ZernikeOPDComparisonInsight # OPD method comparison (lateral displacement)
+├── MSFZernikeInsight           # 100-mode MSF band analysis
 ├── StressFieldInsight
 ├── ModalInsight
 ├── ThermalInsight
@@ -167,14 +192,54 @@ result = insight.generate(config)

 ## Insight Type Details

+### 0. Zernike Dashboard (`zernike_dashboard`) - RECOMMENDED
+
+**Purpose**: Unified dashboard with all orientations (40°, 60°, 90°) and MSF band analysis on one page. Light theme, executive summary, and method comparison.
+
+**Generates**: 1 comprehensive HTML file with:
+- Executive summary with metric cards (40-20, 60-20, MFG workload)
+- MSF band analysis (LSF/MSF/HSF decomposition)
+- 3D surface plots for each orientation
+- Zernike coefficient bar charts color-coded by band
+
+**Configuration**:
+```python
+config = InsightConfig(
+    extra={
+        'n_modes': 50,
+        'filter_low_orders': 4,
+        'theme': 'light',  # Light theme for reports
+    }
+)
+```
+
+**Summary Output**:
+```python
+{
+    '40_vs_20_filtered_rms': 6.53,   # nm (OPD method)
+    '60_vs_20_filtered_rms': 14.21,  # nm (OPD method)
+    '90_optician_workload': 26.34,   # nm (J1-J3 filtered)
+    'msf_rss_40': 2.1,               # nm (MSF band contribution)
+}
+```
+
 ### 1. Zernike WFE Analysis (`zernike_wfe`)

-**Purpose**: Visualize wavefront error for mirror optimization with Zernike polynomial decomposition.
+**Purpose**: Visualize wavefront error for mirror optimization with Zernike polynomial decomposition. **Now includes Standard/OPD method toggle and lateral displacement maps**.

-**Generates**: 3 HTML files
- `zernike_*_40_vs_20.html` - 40° vs 20° relative WFE
- `zernike_*_60_vs_20.html` - 60° vs 20° relative WFE
- `zernike_*_90_mfg.html` - 90° manufacturing (absolute)
+**Generates**: 6 HTML files
+- `zernike_*_40_vs_20.html` - 40° vs 20° relative WFE (with method toggle)
+- `zernike_*_40_lateral.html` - Lateral displacement map for 40°
+- `zernike_*_60_vs_20.html` - 60° vs 20° relative WFE (with method toggle)
+- `zernike_*_60_lateral.html` - Lateral displacement map for 60°
+- `zernike_*_90_mfg.html` - 90° manufacturing (with method toggle)
+- `zernike_*_90_mfg_lateral.html` - Lateral displacement map for 90°
+
+**Features**:
+- Toggle buttons to switch between **Standard (Z-only)** and **OPD (X,Y,Z)** methods
+- Toggle between WFE view and **ΔX, ΔY, ΔZ displacement components**
+- Metrics comparison table showing both methods side-by-side
+- Lateral displacement statistics (Max, RMS in µm)

 **Configuration**:
 ```python
@@ -192,14 +257,81 @@ config = InsightConfig(
 **Summary Output**:
 ```python
 {
-    '40_vs_20_filtered_rms': 45.2,  # nm
-    '60_vs_20_filtered_rms': 78.3,  # nm
-    '90_mfg_filtered_rms': 120.5,   # nm
-    '90_optician_workload': 89.4,   # nm (J1-J3 filtered)
+    '40_vs_20_filtered_rms_std': 6.01,   # nm (Standard method)
+    '40_vs_20_filtered_rms_opd': 6.53,   # nm (OPD method)
+    '60_vs_20_filtered_rms_std': 12.81,  # nm
+    '60_vs_20_filtered_rms_opd': 14.21,  # nm
+    '90_mfg_filtered_rms_std': 24.5,     # nm
+    '90_mfg_filtered_rms_opd': 26.34,    # nm
+    '90_optician_workload': 26.34,       # nm (J1-J3 filtered)
+    'lateral_40_max_um': 0.234,          # µm max lateral displacement
+    'lateral_60_max_um': 0.312,          # µm
+    'lateral_90_max_um': 0.089,          # µm
 }
 ```

-### 2. Stress Distribution (`stress_field`)
+### 2. MSF Zernike Analysis (`msf_zernike`)
+
+**Purpose**: Detailed mid-spatial frequency analysis for telescope mirrors with gravity-induced support print-through.
+
+**Generates**: 1 comprehensive HTML file with:
+- Band decomposition table (LSF/MSF/HSF RSS metrics)
+- MSF-only 3D surface visualization
+- Coefficient bar chart color-coded by band
+- Dominant MSF mode identification
+- Mesh resolution analysis
+
+**Band Definitions** (for 1.2m class mirror):
+
+| Band | Zernike Order | Feature Size | Physical Meaning |
+|------|---------------|--------------|------------------|
+| LSF  | n ≤ 10        | > 120 mm     | M2 hexapod correctable |
+| MSF  | n = 11-50     | 24-109 mm    | Support print-through |
+| HSF  | n > 50        | < 24 mm      | Near mesh resolution limit |
+
+**Configuration**:
+```python
+config = InsightConfig(
+    extra={
+        'n_modes': 100,      # Higher than zernike_wfe (100 vs 50)
+        'lsf_max': 10,       # n ≤ 10 is LSF
+        'msf_max': 50,       # n = 11-50 is MSF
+        'disp_unit': 'mm',
+    }
+)
+```
+
+**Analyses Performed**:
+- Absolute WFE at each orientation (40°, 60°, 90°)
+- Relative to 20° (operational reference)
+- Relative to 90° (manufacturing/polishing reference)
+
+**Summary Output**:
+```python
+{
+    'n_modes': 100,
+    'lsf_max_order': 10,
+    'msf_max_order': 50,
+    'mesh_nodes': 78290,
+    'mesh_spacing_mm': 4.1,
+    'max_resolvable_order': 157,
+    '40deg_vs_20deg_lsf_rss': 12.3,   # nm
+    '40deg_vs_20deg_msf_rss': 8.7,    # nm - KEY METRIC
+    '40deg_vs_20deg_total_rss': 15.2, # nm
+    '40deg_vs_20deg_msf_pct': 33.0,   # % of total in MSF band
+    # ... similar for 60deg, 90deg
+}
+```
+
+**When to Use**:
+- Analyzing support structure print-through
+- Quantifying gravity-induced MSF content
+- Comparing MSF at different orientations
+- Validating mesh resolution is adequate for MSF capture
+
+---
+
+### 3. Stress Distribution (`stress_field`)

 **Purpose**: Visualize Von Mises stress distribution with hot spot identification.

@@ -225,7 +357,7 @@ config = InsightConfig(
 }
 ```

-### 3. Modal Analysis (`modal`)
+### 4. Modal Analysis (`modal`)

 **Purpose**: Visualize natural frequencies and mode shapes.

@@ -249,7 +381,7 @@ config = InsightConfig(
 }
 ```

-### 4. Thermal Analysis (`thermal`)
+### 5. Thermal Analysis (`thermal`)

 **Purpose**: Visualize temperature distribution and gradients.

@@ -273,7 +405,7 @@ config = InsightConfig(
 }
 ```

-### 5. Design Space Explorer (`design_space`)
+### 6. Design Space Explorer (`design_space`)

 **Purpose**: Visualize parameter-objective relationships from optimization trials.

@@ -369,31 +501,44 @@ python -m optimization_engine.insights list

 ---

-## Dashboard Integration (Future)
+## Dashboard Integration

-The insights module is designed for future dashboard integration:
+The Insights tab in the Atomizer Dashboard provides a 3-step workflow:

-```python
-# Backend API endpoint
-@app.get("/api/study/{study_name}/insights")
-def list_study_insights(study_name: str):
-    study_path = Path(f"studies/{study_name}")
-    return list_available_insights(study_path)
+### Step 1: Select Iteration
+- Lists all available iterations (iter1, iter2, etc.) and best_design_archive
+- Shows OP2 file name and modification timestamp
+- Auto-selects "Best Design (Recommended)" if available

-@app.post("/api/study/{study_name}/insights/{type}/generate")
-def generate_insight(study_name: str, type: str, config: dict = {}):
-    insight = get_insight(type, Path(f"studies/{study_name}"))
-    result = insight.generate(InsightConfig(**config))
-    return {
-        'success': result.success,
-        'html_path': str(result.html_path),
-        'summary': result.summary
-    }
+### Step 2: Choose Insight Type
+- Groups insights by category (Optical, Structural, Thermal, etc.)
+- Shows insight name and description
+- Click to select, then "Generate Insight"

-@app.get("/api/study/{study_name}/insights/{type}/plotly")
-def get_insight_plotly(study_name: str, type: str):
-    insight = get_insight(type, Path(f"studies/{study_name}"))
-    return insight.get_plotly_data()
+### Step 3: View Result
+- Displays summary metrics (RMS values, etc.)
+- Embedded Plotly visualization (if available)
+- "Open Full View" button for multi-file insights (like Zernike WFE)
+- Fullscreen mode for detailed analysis
+
+### API Endpoints
+
+```
+GET  /api/insights/studies/{id}/iterations      # List available iterations
+GET  /api/insights/studies/{id}/available       # List available insight types
+GET  /api/insights/studies/{id}/generated       # List previously generated files
+POST /api/insights/studies/{id}/generate/{type} # Generate insight for iteration
+GET  /api/insights/studies/{id}/view/{type}     # View generated HTML
+```
+
+### Generate Request Body
+
+```json
+{
+  "iteration": "best_design_archive",  // or "iter5", etc.
+  "trial_id": null,                    // Optional specific trial
+  "config": {}                         // Insight-specific config
+}
 ```

 ---
@@ -402,4 +547,7 @@ def get_insight_plotly(study_name: str, type: str):

 | Version | Date | Changes |
 |---------|------|---------|
+| 1.3.0 | 2025-12-22 | Added ZernikeDashboardInsight (unified view), OPD method toggle, lateral displacement maps |
+| 1.2.0 | 2024-12-22 | Dashboard overhaul: 3-step workflow, iteration selection, faster loading |
+| 1.1.0 | 2024-12-21 | Added MSF Zernike Analysis insight (6 insight types) |
 | 1.0.0 | 2024-12-20 | Initial release with 5 insight types |