Atomizer/docs/06_PHYSICS/ZERNIKE_OPD_METHOD.md

# Rigorous OPD-Based Zernike Analysis for Mirror Optimization

**Document Version**: 1.0
**Created**: 2024-12-22
**Author**: Atomizer Framework
**Status**: Active

---

## Executive Summary

This document describes a **rigorous Optical Path Difference (OPD)** method for computing Zernike wavefront error that correctly accounts for **lateral (X, Y) displacements** in addition to axial (Z) displacements.

**The Problem**: Standard Zernike analysis uses only Z-displacement at the original (x, y) node positions. When supports pinch the mirror or lateral forces cause in-plane deformation, nodes shift in X and Y. The standard method is **blind to this**, potentially leading to:
- Optimized designs that appear good but have poor actual optical performance
- Optimizer convergence to non-optimal solutions that "cheat" by distorting laterally

**The Solution**: The OPD method computes the true surface error by accounting for the fact that a laterally-displaced node should be compared against the parabola height **at its new (x+dx, y+dy) position**, not its original position.

---

## Table of Contents

1. [The Optical Physics Problem](#1-the-optical-physics-problem)
2. [Mathematical Formulation](#2-mathematical-formulation)
3. [When This Matters](#3-when-this-matters)
4. [Implementation Details](#4-implementation-details)
5. [Usage Guide](#5-usage-guide)
6. [Validation and Testing](#6-validation-and-testing)
7. [Migration Guide](#7-migration-guide)

---

## 1. The Optical Physics Problem

### 1.1 What Zernike Analysis Does

Zernike polynomials decompose a wavefront error surface into orthogonal modes:

```
W(r, θ) = Σ cⱼ Zⱼ(r, θ)
```

Where:
- `W` = wavefront error (nm)
- `cⱼ` = Zernike coefficient for mode j
- `Zⱼ` = Zernike polynomial (Noll indexing)

For a reflective mirror, the wavefront error is **twice** the surface error:
```
WFE = 2 × surface_error
```

### 1.2 Standard Method (Z-Only)

The standard approach:
1. Read node original positions `(x₀, y₀, z₀)` from BDF/DAT
2. Read displacement vector `(Δx, Δy, Δz)` from OP2
3. Compute surface error = `Δz` (Z-displacement only)
4. Compute WFE = `2 × Δz × nm_scale`
5. Fit Zernike at original coordinates `(x₀, y₀)`

```python
# Standard method (simplified)
for nid, (dx, dy, dz) in displacements:
    x, y, z = original_coords[nid]
    wfe = dz * 2 * nm_scale  # ONLY uses Z-displacement
    X.append(x)              # Original X
    Y.append(y)              # Original Y
    WFE.append(wfe)

coeffs = fit_zernike(X, Y, WFE)
```

### 1.3 The Problem: Lateral Displacement is Ignored

Consider a node on a parabolic mirror:
- **Original position**: `(x₀, y₀, z₀)` where `z₀ = -r₀²/(4f)` on the parabola
- **Deformed position**: `(x₀+Δx, y₀+Δy, z₀+Δz)`

**Question**: What is the true surface error?

**Standard method says**: surface_error = `Δz`

**But this is wrong!** If the node moved laterally to a new `(x, y)`, the ideal parabola has a **different** Z at that location. The node should be compared against:

```
z_expected = parabola(x₀+Δx, y₀+Δy) = -(x₀+Δx)² + (y₀+Δy)² / (4f)
```

Not against `z₀ = parabola(x₀, y₀)`.

### 1.4 Visual Example

```
                    Original parabola
                         ___
                      _/     \_
                    /           \
                   /      *A     \      A = original node at (x₀, y₀, z₀)
                  /     ↗  ↘      \     B = deformed position (x₀+Δx, y₀+Δy, z₀+Δz)
                 /     B    C      \    C = where node SHOULD be if staying on parabola
                /                   \
               /_____________________\

Standard method: error = z_B - z_A = Δz
                 (compares B to A vertically)

OPD method: error = z_B - z_C = Δz - Δz_parabola
            (compares B to where parabola is at B's (x,y))
```

---

## 2. Mathematical Formulation

### 2.1 Differential OPD Formulation

For a paraboloid with optical axis along Z:

```
z = -r² / (4f)    [concave mirror, vertex at origin]
```

Where:
- `r² = x² + y²`
- `f` = focal length

**Key Insight**: We can compute the **change** in parabola Z due to lateral movement:

```
Δz_parabola = z(x₀+Δx, y₀+Δy) - z(x₀, y₀)
            = -[(x₀+Δx)² + (y₀+Δy)²] / (4f) - [-( x₀² + y₀²) / (4f)]
            = -[r_def² - r₀²] / (4f)
            = -Δr² / (4f)
```

Where:
```
Δr² = r_def² - r₀² = (x₀+Δx)² + (y₀+Δy)² - x₀² - y₀²
    = 2·x₀·Δx + Δx² + 2·y₀·Δy + Δy²
```

### 2.2 True Surface Error

The true surface error is:

```
surface_error = Δz - Δz_parabola
              = Δz - (-Δr² / 4f)
              = Δz + Δr² / (4f)
```

**Interpretation**:
- If a node moves **outward** (larger r), it should also move in **-Z** to stay on the concave parabola
- If the FEA says it moved by `Δz`, but staying on the parabola requires `Δz_parabola`, the difference is the true error
- This corrects for the "false error" that the standard method counts when nodes shift laterally

### 2.3 Wavefront Error

```
WFE = 2 × surface_error × nm_scale
    = 2 × (Δz - Δz_parabola) × nm_scale
```

### 2.4 Zernike Fitting Coordinates

Another subtlety: the Zernike fit should use the **deformed** coordinates `(x₀+Δx, y₀+Δy)` rather than the original coordinates. This is because the WFE surface represents the error at the positions where the nodes **actually are** after deformation.

```python
# OPD method
X_fit = x0 + dx  # Deformed X
Y_fit = y0 + dy  # Deformed Y
WFE = surface_error * 2 * nm_scale

coeffs = fit_zernike(X_fit, Y_fit, WFE)
```

---

## 3. When This Matters

### 3.1 Magnitude Analysis

The correction term is:

```
Δz_parabola = -Δr² / (4f) ≈ -(2·x₀·Δx + 2·y₀·Δy) / (4f)    [ignoring Δx², Δy²]
            ≈ -(x₀·Δx + y₀·Δy) / (2f)
```

For a node at radius `r₀` with tangential displacement `Δ_tangential`:
- The correction is approximately: `r₀ · Δ_lateral / (2f)`

**Example**: Mirror with f = 5000 mm, outer radius = 400 mm
- Node at r = 400 mm shifts laterally by Δx = 0.001 mm (1 µm)
- Correction: `400 × 0.001 / (2 × 5000) = 0.00004 mm = 40 nm`

This is **significant** when typical WFE is in the 10-100 nm range!

### 3.2 Classification by Load Case

| Load Case | Lateral Disp. | Method Impact |
|-----------|--------------|---------------|
| **Axial support** (gravity in Z) | Very small | Minimal - both methods similar |
| **Lateral support** (gravity in X/Y) | **Large** | **Significant** - OPD method required |
| **Clamp/fixture forces** | Can be large locally | May be significant at pinch points |
| **Thermal** | Variable | Depends on thermal gradients |
| **Mirror cell deflection** | Variable | Check lateral displacement magnitude |

### 3.3 Diagnostic Thresholds

The `ZernikeOPDExtractor` provides lateral displacement statistics:

| Max Lateral Disp. | Recommendation |
|-------------------|----------------|
| > 10 µm | **CRITICAL**: OPD method required |
| 1 - 10 µm | **RECOMMENDED**: OPD method provides meaningful improvement |
| 0.1 - 1 µm | **OPTIONAL**: OPD method provides minor improvement |
| < 0.1 µm | **EQUIVALENT**: Both methods give essentially identical results |

---

## 4. Implementation Details

### 4.1 Module: `extract_zernike_opd.py`

Location: `optimization_engine/extractors/extract_zernike_opd.py`

**Key Functions**:

```python
def compute_true_opd(x0, y0, z0, dx, dy, dz, focal_length, concave=True):
    """
    Compute true surface error accounting for lateral displacement.

    Returns:
        x_def: Deformed X coordinates
        y_def: Deformed Y coordinates
        surface_error: True surface error (not just Δz)
        lateral_magnitude: |Δx, Δy| for diagnostics
    """
```

```python
def estimate_focal_length_from_geometry(x, y, z, concave=True):
    """
    Estimate parabola focal length by fitting z = a·r² + b.
    Focal length = 1 / (4·|a|)
    """
```

**Main Class**:

```python
class ZernikeOPDExtractor:
    """
    Rigorous OPD-based Zernike extractor.

    Key differences from ZernikeExtractor:
    - Uses deformed (x, y) coordinates for fitting
    - Computes surface error relative to parabola at deformed position
    - Provides lateral displacement diagnostics
    """
```

### 4.2 Algorithm Flow

```
1. Load geometry (BDF) and displacements (OP2)

2. For each node:
   a. Get original position: (x₀, y₀, z₀)
   b. Get displacement: (Δx, Δy, Δz)
   c. Compute deformed position: (x_def, y_def) = (x₀+Δx, y₀+Δy)
   d. Compute Δr² = r_def² - r₀²
   e. Compute Δz_parabola = -Δr² / (4f)   [for concave]
   f. Compute surface_error = Δz - Δz_parabola
   g. Store lateral_disp = √(Δx² + Δy²)

3. Convert to WFE: WFE = 2 × surface_error × nm_scale

4. Fit Zernike coefficients using (x_def, y_def, WFE)

5. Compute RMS metrics:
   - Global RMS = √(mean(WFE²))
   - Filtered RMS = √(mean((WFE - low_order_fit)²))
```

### 4.3 Focal Length Handling

The extractor can:
1. Use a **provided** focal length (most accurate)
2. **Auto-estimate** from geometry by fitting `z = a·r² + b`

Auto-estimation works well for clean parabolic meshes but may need manual override for:
- Off-axis parabolas
- Aspheric surfaces
- Meshes with significant manufacturing errors

```python
# Explicit focal length
extractor = ZernikeOPDExtractor(op2_file, focal_length=5000.0)

# Auto-estimate (default)
extractor = ZernikeOPDExtractor(op2_file, auto_estimate_focal=True)
```

---

## 5. Usage Guide

### 5.1 Quick Comparison Test

Run the test script to see how much the methods differ for your data:

```bash
conda activate atomizer
python test_zernike_opd_comparison.py
```

Output example:
```
--- Standard Method (Z-only) ---
  Global RMS:   171.65 nm
  Filtered RMS: 28.72 nm

--- Rigorous OPD Method ---
  Global RMS:   171.89 nm
  Filtered RMS: 29.15 nm

--- Difference (OPD - Standard) ---
  Filtered RMS: +0.43 nm (+1.5%)

--- Lateral Displacement ---
  Max:  0.156 µm
  RMS:  0.111 µm

>>> OPTIONAL: Small lateral displacements. OPD method provides minor improvement.
```

### 5.2 Using in Optimization

**For new studies**, use the OPD extractor:

```python
from optimization_engine.extractors import extract_zernike_opd_filtered_rms

def objective(trial):
    # ... parameter suggestion and FEA solve ...

    # Use OPD method instead of standard
    rms = extract_zernike_opd_filtered_rms(
        op2_file,
        subcase='20',
        focal_length=5000.0  # Optional: specify or let it auto-estimate
    )
    return rms
```

**In optimization config** (future enhancement):

```json
{
  "objectives": [
    {
      "name": "filtered_rms",
      "extractor": "zernike_opd",
      "extractor_config": {
        "subcase": "20",
        "metric": "filtered_rms_nm",
        "focal_length": 5000.0
      }
    }
  ]
}
```

### 5.3 Visualization with Insights

Generate the comparison insight for a study:

```bash
python -m optimization_engine.insights generate studies/my_study --type zernike_opd_comparison
```

This creates an HTML visualization showing:
1. **Lateral displacement map** - Where pinching/lateral deformation occurs
2. **WFE surface** - Using the rigorous OPD method
3. **Comparison table** - Quantitative difference between methods
4. **Recommendation** - Whether OPD method is needed for your study

### 5.4 API Reference

```python
from optimization_engine.extractors import (
    # Main extractor class
    ZernikeOPDExtractor,

    # Convenience functions
    extract_zernike_opd,           # Full metrics dict
    extract_zernike_opd_filtered_rms,  # Just the filtered RMS (float)
    compare_zernike_methods,       # Compare standard vs OPD
)

# Full extraction with all metrics
result = extract_zernike_opd(op2_file, subcase='20')
# Returns: {
#   'filtered_rms_nm': float,
#   'global_rms_nm': float,
#   'max_lateral_disp_um': float,
#   'rms_lateral_disp_um': float,
#   'focal_length_used': float,
#   'astigmatism_rms_nm': float,
#   'coma_rms_nm': float,
#   ...
# }

# Just the primary metric for optimization
rms = extract_zernike_opd_filtered_rms(op2_file, subcase='20')

# Compare both methods
comparison = compare_zernike_methods(op2_file, subcase='20')
# Returns: {
#   'standard_method': {'filtered_rms_nm': ...},
#   'opd_method': {'filtered_rms_nm': ...},
#   'delta': {'filtered_rms_nm': ..., 'percent_difference_filtered': ...},
#   'lateral_displacement': {'max_um': ..., 'rms_um': ...},
#   'recommendation': str
# }
```

---

## 6. Validation and Testing

### 6.1 Analytical Test Case

For a simple test: apply a known lateral displacement and verify the correction.

**Setup**:
- Parabola: f = 5000 mm
- Node at (x₀, y₀) = (400, 0) mm, so r₀ = 400 mm
- Apply uniform X-displacement: Δx = 0.01 mm, Δy = 0, Δz = 0

**Expected correction**:
```
Δr² = (400.01)² + 0² - 400² - 0² = 8.0001 mm²
Δz_parabola = -8.0001 / (4 × 5000) = -0.0004 mm = -400 nm (surface)
WFE_correction = 2 × 400 nm = 800 nm
```

**Standard method**: WFE = 2 × Δz × 1e6 = 0 nm
**OPD method**: WFE = 2 × (0 - (-0.0004)) × 1e6 = 800 nm

The OPD method correctly identifies that a purely lateral displacement **does** affect the wavefront!

### 6.2 Sanity Checks

The OPD method should:
1. Give **identical** results to standard method when Δx = Δy = 0 everywhere
2. Show **larger** WFE when nodes move outward laterally (positive Δr²)
3. Show **smaller** WFE when nodes move inward laterally (negative Δr²)
4. Scale with 1/f (larger effect for faster mirrors)

### 6.3 Running the Test

```bash
conda activate atomizer
python test_zernike_opd_comparison.py
```

---

## 7. Migration Guide

### 7.1 For Existing Studies

1. **Run comparison test** on a few representative iterations
2. **Check the difference** - if > 5%, consider re-optimizing
3. **For lateral support studies** - strongly recommend re-optimization with OPD method

### 7.2 For New Studies

1. **Use OPD method by default** - it's never worse than standard
2. **Specify focal length** if known (more accurate than auto-estimate)
3. **Monitor lateral displacement** in the insight reports

### 7.3 Code Changes

**Before** (standard method):
```python
from optimization_engine.extractors import extract_zernike_filtered_rms

rms = extract_zernike_filtered_rms(op2_file, subcase='20')
```

**After** (OPD method):
```python
from optimization_engine.extractors import extract_zernike_opd_filtered_rms

rms = extract_zernike_opd_filtered_rms(op2_file, subcase='20', focal_length=5000.0)
```

---

## Appendix A: Derivation Details

### A.1 Full Derivation of Δz_parabola

For a concave paraboloid: `z = -r²/(4f) = -(x² + y²)/(4f)`

Original position: `z₀ = -(x₀² + y₀²)/(4f)`
Deformed position: `z_expected = -((x₀+Δx)² + (y₀+Δy)²)/(4f)`

```
z_expected - z₀ = -[(x₀+Δx)² + (y₀+Δy)² - x₀² - y₀²] / (4f)
                = -[x₀² + 2x₀Δx + Δx² + y₀² + 2y₀Δy + Δy² - x₀² - y₀²] / (4f)
                = -[2x₀Δx + Δx² + 2y₀Δy + Δy²] / (4f)
                = -[r_def² - r₀²] / (4f)
                = -Δr² / (4f)
```

This is `Δz_parabola` - the Z change required to stay on the ideal parabola.

### A.2 Sign Convention

For a **concave** mirror (typical telescope primary):
- Surface curves toward -Z (vertex is the highest point)
- `z = -r²/(4f)` (negative coefficient)
- Moving outward (Δr² > 0) requires moving in -Z direction
- `Δz_parabola = -Δr²/(4f)` is negative for outward movement

For a **convex** mirror:
- Surface curves toward +Z
- `z = +r²/(4f)` (positive coefficient)
- `Δz_parabola = +Δr²/(4f)` is positive for outward movement

The `concave` parameter in the code handles this sign flip.

---

## Appendix B: Files Reference

| File | Purpose |
|------|---------|
| `optimization_engine/extractors/extract_zernike_opd.py` | Main OPD extractor implementation |
| `optimization_engine/extractors/extract_zernike.py` | Standard (Z-only) extractor |
| `optimization_engine/insights/zernike_opd_comparison.py` | Visualization insight |
| `test_zernike_opd_comparison.py` | Quick test script |

### Related Documentation

| Document | Purpose |
|----------|---------|
| [ZERNIKE_FUNDAMENTALS.md](ZERNIKE_FUNDAMENTALS.md) | General Zernike usage, RMS calculation, multi-subcase analysis |
| [00_INDEX.md](00_INDEX.md) | Physics documentation index |
| `.claude/skills/modules/extractors-catalog.md` | Quick extractor lookup |
| `.claude/skills/modules/insights-catalog.md` | Quick insight lookup |
| `docs/protocols/system/SYS_12_EXTRACTOR_LIBRARY.md` | Extractor specifications (E8-E10, E20-E21) |
| `docs/protocols/system/SYS_17_STUDY_INSIGHTS.md` | Insight specifications |

---

## Appendix C: Glossary

| Term | Definition |
|------|------------|
| **OPD** | Optical Path Difference - the path length difference experienced by light rays |
| **WFE** | Wavefront Error - deviation of actual wavefront from ideal (WFE = 2 × surface error for reflection) |
| **Zernike polynomials** | Orthogonal basis functions for representing wavefronts over a circular aperture |
| **Noll index** | Standard optical indexing scheme for Zernike modes (j=1 is piston, j=4 is defocus, etc.) |
| **Filtered RMS** | RMS after removing low-order modes (piston, tip, tilt, defocus) that can be corrected by alignment |
| **Lateral displacement** | In-plane (X, Y) movement of nodes, as opposed to axial (Z) movement |
| **Focal length** | Distance from vertex to focus for a parabola; f = R/(2) where R is vertex radius of curvature |

---

*Document maintained by Atomizer Framework. Last updated: 2024-12-22*