docs: Consolidate documentation and fix protocol numbering (partial)

Phase 2 of restructuring plan: - Rename SYS_16_STUDY_INSIGHTS -> SYS_17_STUDY_INSIGHTS - Rename SYS_17_CONTEXT_ENGINEERING -> SYS_18_CONTEXT_ENGINEERING - Promote Bootstrap V3.0 (Context Engineering) as default - Archive old Bootstrap V2.0 - Create knowledge_base/playbook.json for ACE framework - Add OP_08 (Generate Report) to routing tables - Add SYS_16-18 to protocol tables - Update docs/protocols/README.md to version 1.1 - Update CLAUDE.md with new protocols - Create docs/plans/RESTRUCTURING_PLAN.md for continuation Remaining: Phase 2.8 (Cheatsheet), Phases 3-6 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-07 08:52:07 -05:00
parent 18c221a218
commit b8a04c62b8
17 changed files with 1568 additions and 623 deletions
--- a/docs/protocols/system/SYS_17_STUDY_INSIGHTS.md
+++ b/docs/protocols/system/SYS_17_STUDY_INSIGHTS.md
@@ -0,0 +1,553 @@
+# SYS_16: Study Insights
+
+**Version**: 1.0.0
+**Status**: Active
+**Purpose**: Physics-focused visualizations for FEA optimization results
+
+---
+
+## Overview
+
+Study Insights provide **physics understanding** of optimization results through interactive 3D visualizations. Unlike the Analysis page (which shows optimizer metrics like convergence and Pareto fronts), Insights answer the question: **"What does this design actually look like?"**
+
+### Analysis vs Insights
+
+| Aspect | **Analysis** | **Insights** |
+|--------|--------------|--------------|
+| Focus | Optimization performance | Physics understanding |
+| Questions | "Is the optimizer converging?" | "What does the best design look like?" |
+| Data Source | `study.db` (trials, objectives) | Simulation outputs (OP2, mesh, fields) |
+| Typical Plots | Convergence, Pareto, parameters | 3D surfaces, stress contours, mode shapes |
+| When Used | During/after optimization | After specific trial of interest |
+
+---
+
+## Available Insight Types
+
+| Type ID | Name | Applicable To | Data Required |
+|---------|------|---------------|---------------|
+| `zernike_dashboard` | **Zernike Dashboard (RECOMMENDED)** | Mirror, optics | OP2 with displacement subcases |
+| `zernike_wfe` | Zernike WFE Analysis | Mirror, optics | OP2 with displacement subcases |
+| `zernike_opd_comparison` | Zernike OPD Method Comparison | Mirror, optics, lateral | OP2 with displacement subcases |
+| `msf_zernike` | MSF Zernike Analysis | Mirror, optics | OP2 with displacement subcases |
+| `stress_field` | Stress Distribution | Structural, bracket, beam | OP2 with stress results |
+| `modal` | Modal Analysis | Vibration, dynamic | OP2 with eigenvalue/eigenvector |
+| `thermal` | Thermal Analysis | Thermo-structural | OP2 with temperature results |
+| `design_space` | Design Space Explorer | All optimization studies | study.db with 5+ trials |
+
+### Zernike Method Comparison: Standard vs OPD
+
+The Zernike insights now support **two WFE computation methods**:
+
+| Method | Description | When to Use |
+|--------|-------------|-------------|
+| **Standard (Z-only)** | Uses only Z-displacement at original (x,y) coordinates | Quick analysis, negligible lateral displacement |
+| **OPD (X,Y,Z)** ← RECOMMENDED | Accounts for lateral (X,Y) displacement via interpolation | Any surface with gravity loads, most rigorous |
+
+**How OPD method works**:
+1. Builds interpolator from undeformed BDF mesh geometry
+2. For each deformed node at `(x+dx, y+dy, z+dz)`, interpolates `Z_ideal` at new XY position
+3. Computes `WFE = z_deformed - Z_ideal(x_def, y_def)`
+4. Fits Zernike polynomials to the surface error map
+
+**Typical difference**: OPD method gives **8-11% higher** WFE values than Standard (more conservative/accurate).
+
+---
+
+## Architecture
+
+### Module Structure
+
+```
+optimization_engine/insights/
+├── __init__.py               # Registry and public API
+├── base.py                   # StudyInsight base class, InsightConfig, InsightResult
+├── zernike_wfe.py            # Mirror wavefront error visualization (50 modes)
+├── zernike_opd_comparison.py # OPD vs Standard method comparison (lateral disp. analysis)
+├── msf_zernike.py            # MSF band decomposition (100 modes, LSF/MSF/HSF)
+├── stress_field.py           # Stress contour visualization
+├── modal_analysis.py         # Mode shape visualization
+├── thermal_field.py      # Temperature distribution
+└── design_space.py       # Parameter-objective exploration
+```
+
+### Class Hierarchy
+
+```python
+StudyInsight (ABC)
+├── ZernikeDashboardInsight     # RECOMMENDED: Unified dashboard with all views
+├── ZernikeWFEInsight           # Standard 50-mode WFE analysis (with OPD toggle)
+├── ZernikeOPDComparisonInsight # OPD method comparison (lateral displacement)
+├── MSFZernikeInsight           # 100-mode MSF band analysis
+├── StressFieldInsight
+├── ModalInsight
+├── ThermalInsight
+└── DesignSpaceInsight
+```
+
+### Key Classes
+
+#### StudyInsight (Base Class)
+
+```python
+class StudyInsight(ABC):
+    insight_type: str       # Unique identifier (e.g., 'zernike_wfe')
+    name: str               # Human-readable name
+    description: str        # What this insight shows
+    applicable_to: List[str]  # Study types this applies to
+
+    def can_generate(self) -> bool:
+        """Check if required data exists."""
+
+    def generate(self, config: InsightConfig) -> InsightResult:
+        """Generate visualization."""
+
+    def generate_html(self, trial_id=None, **kwargs) -> Path:
+        """Generate standalone HTML file."""
+
+    def get_plotly_data(self, trial_id=None, **kwargs) -> dict:
+        """Get Plotly figure for dashboard embedding."""
+```
+
+#### InsightConfig
+
+```python
+@dataclass
+class InsightConfig:
+    trial_id: Optional[int] = None     # Which trial to visualize
+    colorscale: str = 'Turbo'          # Plotly colorscale
+    amplification: float = 1.0         # Deformation scale factor
+    lighting: bool = True              # 3D lighting effects
+    output_dir: Optional[Path] = None  # Where to save HTML
+    extra: Dict[str, Any] = {}         # Type-specific config
+```
+
+#### InsightResult
+
+```python
+@dataclass
+class InsightResult:
+    success: bool
+    html_path: Optional[Path] = None        # Generated HTML file
+    plotly_figure: Optional[dict] = None    # Figure for dashboard
+    summary: Optional[dict] = None          # Key metrics
+    error: Optional[str] = None             # Error message if failed
+```
+
+---
+
+## Usage
+
+### Python API
+
+```python
+from optimization_engine.insights import get_insight, list_available_insights
+from pathlib import Path
+
+study_path = Path("studies/my_mirror_study")
+
+# List what's available
+available = list_available_insights(study_path)
+for info in available:
+    print(f"{info['type']}: {info['name']}")
+
+# Generate specific insight
+insight = get_insight('zernike_wfe', study_path)
+if insight and insight.can_generate():
+    result = insight.generate()
+    print(f"Generated: {result.html_path}")
+    print(f"40-20 Filtered RMS: {result.summary['40_vs_20_filtered_rms']:.2f} nm")
+```
+
+### CLI
+
+```bash
+# List all insight types
+python -m optimization_engine.insights list
+
+# Generate all available insights for a study
+python -m optimization_engine.insights generate studies/my_study
+
+# Generate specific insight
+python -m optimization_engine.insights generate studies/my_study --type zernike_wfe
+```
+
+### With Configuration
+
+```python
+from optimization_engine.insights import get_insight, InsightConfig
+
+insight = get_insight('stress_field', study_path)
+config = InsightConfig(
+    colorscale='Hot',
+    extra={
+        'yield_stress': 250,  # MPa
+        'stress_unit': 'MPa'
+    }
+)
+result = insight.generate(config)
+```
+
+---
+
+## Insight Type Details
+
+### 0. Zernike Dashboard (`zernike_dashboard`) - RECOMMENDED
+
+**Purpose**: Unified dashboard with all orientations (40°, 60°, 90°) and MSF band analysis on one page. Light theme, executive summary, and method comparison.
+
+**Generates**: 1 comprehensive HTML file with:
+- Executive summary with metric cards (40-20, 60-20, MFG workload)
+- MSF band analysis (LSF/MSF/HSF decomposition)
+- 3D surface plots for each orientation
+- Zernike coefficient bar charts color-coded by band
+
+**Configuration**:
+```python
+config = InsightConfig(
+    extra={
+        'n_modes': 50,
+        'filter_low_orders': 4,
+        'theme': 'light',  # Light theme for reports
+    }
+)
+```
+
+**Summary Output**:
+```python
+{
+    '40_vs_20_filtered_rms': 6.53,   # nm (OPD method)
+    '60_vs_20_filtered_rms': 14.21,  # nm (OPD method)
+    '90_optician_workload': 26.34,   # nm (J1-J3 filtered)
+    'msf_rss_40': 2.1,               # nm (MSF band contribution)
+}
+```
+
+### 1. Zernike WFE Analysis (`zernike_wfe`)
+
+**Purpose**: Visualize wavefront error for mirror optimization with Zernike polynomial decomposition. **Now includes Standard/OPD method toggle and lateral displacement maps**.
+
+**Generates**: 6 HTML files
+- `zernike_*_40_vs_20.html` - 40° vs 20° relative WFE (with method toggle)
+- `zernike_*_40_lateral.html` - Lateral displacement map for 40°
+- `zernike_*_60_vs_20.html` - 60° vs 20° relative WFE (with method toggle)
+- `zernike_*_60_lateral.html` - Lateral displacement map for 60°
+- `zernike_*_90_mfg.html` - 90° manufacturing (with method toggle)
+- `zernike_*_90_mfg_lateral.html` - Lateral displacement map for 90°
+
+**Features**:
+- Toggle buttons to switch between **Standard (Z-only)** and **OPD (X,Y,Z)** methods
+- Toggle between WFE view and **ΔX, ΔY, ΔZ displacement components**
+- Metrics comparison table showing both methods side-by-side
+- Lateral displacement statistics (Max, RMS in µm)
+
+**Configuration**:
+```python
+config = InsightConfig(
+    amplification=0.5,  # Reduce deformation scaling
+    colorscale='Turbo',
+    extra={
+        'n_modes': 50,
+        'filter_low_orders': 4,  # Remove piston, tip, tilt, defocus
+        'disp_unit': 'mm',
+    }
+)
+```
+
+**Summary Output**:
+```python
+{
+    '40_vs_20_filtered_rms_std': 6.01,   # nm (Standard method)
+    '40_vs_20_filtered_rms_opd': 6.53,   # nm (OPD method)
+    '60_vs_20_filtered_rms_std': 12.81,  # nm
+    '60_vs_20_filtered_rms_opd': 14.21,  # nm
+    '90_mfg_filtered_rms_std': 24.5,     # nm
+    '90_mfg_filtered_rms_opd': 26.34,    # nm
+    '90_optician_workload': 26.34,       # nm (J1-J3 filtered)
+    'lateral_40_max_um': 0.234,          # µm max lateral displacement
+    'lateral_60_max_um': 0.312,          # µm
+    'lateral_90_max_um': 0.089,          # µm
+}
+```
+
+### 2. MSF Zernike Analysis (`msf_zernike`)
+
+**Purpose**: Detailed mid-spatial frequency analysis for telescope mirrors with gravity-induced support print-through.
+
+**Generates**: 1 comprehensive HTML file with:
+- Band decomposition table (LSF/MSF/HSF RSS metrics)
+- MSF-only 3D surface visualization
+- Coefficient bar chart color-coded by band
+- Dominant MSF mode identification
+- Mesh resolution analysis
+
+**Band Definitions** (for 1.2m class mirror):
+
+| Band | Zernike Order | Feature Size | Physical Meaning |
+|------|---------------|--------------|------------------|
+| LSF  | n ≤ 10        | > 120 mm     | M2 hexapod correctable |
+| MSF  | n = 11-50     | 24-109 mm    | Support print-through |
+| HSF  | n > 50        | < 24 mm      | Near mesh resolution limit |
+
+**Configuration**:
+```python
+config = InsightConfig(
+    extra={
+        'n_modes': 100,      # Higher than zernike_wfe (100 vs 50)
+        'lsf_max': 10,       # n ≤ 10 is LSF
+        'msf_max': 50,       # n = 11-50 is MSF
+        'disp_unit': 'mm',
+    }
+)
+```
+
+**Analyses Performed**:
+- Absolute WFE at each orientation (40°, 60°, 90°)
+- Relative to 20° (operational reference)
+- Relative to 90° (manufacturing/polishing reference)
+
+**Summary Output**:
+```python
+{
+    'n_modes': 100,
+    'lsf_max_order': 10,
+    'msf_max_order': 50,
+    'mesh_nodes': 78290,
+    'mesh_spacing_mm': 4.1,
+    'max_resolvable_order': 157,
+    '40deg_vs_20deg_lsf_rss': 12.3,   # nm
+    '40deg_vs_20deg_msf_rss': 8.7,    # nm - KEY METRIC
+    '40deg_vs_20deg_total_rss': 15.2, # nm
+    '40deg_vs_20deg_msf_pct': 33.0,   # % of total in MSF band
+    # ... similar for 60deg, 90deg
+}
+```
+
+**When to Use**:
+- Analyzing support structure print-through
+- Quantifying gravity-induced MSF content
+- Comparing MSF at different orientations
+- Validating mesh resolution is adequate for MSF capture
+
+---
+
+### 3. Stress Distribution (`stress_field`)
+
+**Purpose**: Visualize Von Mises stress distribution with hot spot identification.
+
+**Configuration**:
+```python
+config = InsightConfig(
+    colorscale='Hot',
+    extra={
+        'yield_stress': 250,    # MPa - shows safety factor
+        'stress_unit': 'MPa',
+    }
+)
+```
+
+**Summary Output**:
+```python
+{
+    'max_stress': 187.5,      # MPa
+    'mean_stress': 45.2,      # MPa
+    'p95_stress': 120.3,      # 95th percentile
+    'p99_stress': 165.8,      # 99th percentile
+    'safety_factor': 1.33,    # If yield_stress provided
+}
+```
+
+### 4. Modal Analysis (`modal`)
+
+**Purpose**: Visualize natural frequencies and mode shapes.
+
+**Configuration**:
+```python
+config = InsightConfig(
+    amplification=50.0,  # Mode shape scale
+    extra={
+        'n_modes': 20,      # Number of modes to show
+        'show_mode': 1,     # Which mode shape to display
+    }
+)
+```
+
+**Summary Output**:
+```python
+{
+    'n_modes': 20,
+    'first_frequency_hz': 125.4,
+    'frequencies_hz': [125.4, 287.8, 312.5, ...],
+}
+```
+
+### 5. Thermal Analysis (`thermal`)
+
+**Purpose**: Visualize temperature distribution and gradients.
+
+**Configuration**:
+```python
+config = InsightConfig(
+    colorscale='Thermal',
+    extra={
+        'temp_unit': 'K',  # or 'C', 'F'
+    }
+)
+```
+
+**Summary Output**:
+```python
+{
+    'max_temp': 423.5,    # K
+    'min_temp': 293.0,    # K
+    'mean_temp': 345.2,   # K
+    'temp_range': 130.5,  # K
+}
+```
+
+### 6. Design Space Explorer (`design_space`)
+
+**Purpose**: Visualize parameter-objective relationships from optimization trials.
+
+**Configuration**:
+```python
+config = InsightConfig(
+    extra={
+        'primary_objective': 'filtered_rms',  # Color by this objective
+    }
+)
+```
+
+**Summary Output**:
+```python
+{
+    'n_trials': 100,
+    'n_params': 4,
+    'n_objectives': 2,
+    'best_trial_id': 47,
+    'best_params': {'p1': 0.5, 'p2': 1.2, ...},
+    'best_values': {'filtered_rms': 45.2, 'mass': 2.34},
+}
+```
+
+---
+
+## Output Directory
+
+Insights are saved to `{study}/3_insights/`:
+
+```
+studies/my_study/
+├── 1_setup/
+├── 2_results/
+└── 3_insights/               # Created by insights module
+    ├── zernike_20241220_143022_40_vs_20.html
+    ├── zernike_20241220_143022_60_vs_20.html
+    ├── zernike_20241220_143022_90_mfg.html
+    ├── stress_20241220_143025.html
+    └── design_space_20241220_143030.html
+```
+
+---
+
+## Creating New Insight Types
+
+To add a new insight type (power_user+):
+
+### 1. Create the insight class
+
+```python
+# optimization_engine/insights/my_insight.py
+
+from .base import StudyInsight, InsightConfig, InsightResult, register_insight
+
+@register_insight
+class MyInsight(StudyInsight):
+    insight_type = "my_insight"
+    name = "My Custom Insight"
+    description = "Description of what it shows"
+    applicable_to = ["structural", "all"]
+
+    def can_generate(self) -> bool:
+        # Check if required data exists
+        return self.results_path.exists()
+
+    def _generate(self, config: InsightConfig) -> InsightResult:
+        # Generate visualization
+        # ... build Plotly figure ...
+
+        html_path = config.output_dir / f"my_insight_{timestamp}.html"
+        html_path.write_text(fig.to_html(...))
+
+        return InsightResult(
+            success=True,
+            html_path=html_path,
+            summary={'key_metric': value}
+        )
+```
+
+### 2. Register in `__init__.py`
+
+```python
+from .my_insight import MyInsight
+```
+
+### 3. Test
+
+```bash
+python -m optimization_engine.insights list
+# Should show "my_insight" in the list
+```
+
+---
+
+## Dashboard Integration
+
+The Insights tab in the Atomizer Dashboard provides a 3-step workflow:
+
+### Step 1: Select Iteration
+- Lists all available iterations (iter1, iter2, etc.) and best_design_archive
+- Shows OP2 file name and modification timestamp
+- Auto-selects "Best Design (Recommended)" if available
+
+### Step 2: Choose Insight Type
+- Groups insights by category (Optical, Structural, Thermal, etc.)
+- Shows insight name and description
+- Click to select, then "Generate Insight"
+
+### Step 3: View Result
+- Displays summary metrics (RMS values, etc.)
+- Embedded Plotly visualization (if available)
+- "Open Full View" button for multi-file insights (like Zernike WFE)
+- Fullscreen mode for detailed analysis
+
+### API Endpoints
+
+```
+GET  /api/insights/studies/{id}/iterations      # List available iterations
+GET  /api/insights/studies/{id}/available       # List available insight types
+GET  /api/insights/studies/{id}/generated       # List previously generated files
+POST /api/insights/studies/{id}/generate/{type} # Generate insight for iteration
+GET  /api/insights/studies/{id}/view/{type}     # View generated HTML
+```
+
+### Generate Request Body
+
+```json
+{
+  "iteration": "best_design_archive",  // or "iter5", etc.
+  "trial_id": null,                    // Optional specific trial
+  "config": {}                         // Insight-specific config
+}
+```
+
+---
+
+## Version History
+
+| Version | Date | Changes |
+|---------|------|---------|
+| 1.3.0 | 2025-12-22 | Added ZernikeDashboardInsight (unified view), OPD method toggle, lateral displacement maps |
+| 1.2.0 | 2024-12-22 | Dashboard overhaul: 3-step workflow, iteration selection, faster loading |
+| 1.1.0 | 2024-12-21 | Added MSF Zernike Analysis insight (6 insight types) |
+| 1.0.0 | 2024-12-20 | Initial release with 5 insight types |