context: Organized M1 Mirror documentation with parent-child README hierarchy

insight: DOCUMENTATION PATTERN: Studies use a two-level README hierarchy. Parent README at studies/{geometry_type}/README.md contains project-wide context (optical specs, design variables catalog, objectives catalog, campaign history, sub-studies index). Child README at studies/{geometry_type}/{study_name}/README.md references parent and contains study-specific details (active variables, algorithm config, results). This eliminates duplication, maintains single source of truth for specs, and makes sub-study docs concise. Pattern documented in OP_01_CREATE_STUDY.md and study-creation-core.md.

rule: When creating studies for a geometry type: (1) Create parent README with project context if first study, (2) Add reference banner to child README: '> See [../README.md](../README.md) for project overview', (3) Update parent's sub-studies index table when adding new sub-studies.


context: Created universal mirror optical specs extraction tool

insight: TOOL PATTERN: Mirror optical specs (focal length, f-number, diameter) can be auto-estimated from FEA mesh geometry by fitting z = a*r² + b to node coordinates. Focal length = 1/(4*|a|). Tool at tools/extract_mirror_optical_specs.py works with any mirror study - just point it at an OP2 file or study directory. Reports fit quality to indicate if explicit focal length should be used instead. Use: python tools/extract_mirror_optical_specs.py path/to/study

rule: For mirror optimization: (1) Run extract_mirror_optical_specs.py to estimate optical prescription from mesh, (2) Validate against design specs, (3) Document in parent README, (4) Use explicit focal_length in ZernikeOPDExtractor if fit quality is poor.


context: Implemented OPD-based Zernike method for lateral support optimization

insight: PHYSICS PATTERN: Standard Zernike WFE analysis uses Z-displacement at original (x,y) coordinates. This is INCORRECT for lateral support optimization where nodes shift in X,Y. The rigorous OPD method computes: surface_error = dz - delta_z_parabola where delta_z_parabola = -delta_r²/(4f) for concave mirrors. This accounts for the fact that laterally displaced nodes should be compared against parabola height at their NEW position. Implemented in extract_zernike_opd.py with ZernikeOPDExtractor class. Use extract_comparison() to see method difference. Threshold: >10µm lateral displacement = CRITICAL to use OPD.

rule: For mirror optimization with lateral supports or any case where X,Y displacement may be significant: (1) Use ZernikeOPDExtractor instead of ZernikeExtractor, (2) Run zernike_opd_comparison insight to check lateral displacement magnitude, (3) If max lateral >10µm, OPD method is CRITICAL.


context: Neural surrogate turbo optimization with FEA validation

insight: For surrogate-based optimization, log FEA validation trials to a SEPARATE Optuna study.db for dashboard visibility. The surrogate exploration runs internally (not logged), but every FEA validation gets logged to Optuna using study.ask()/tell() pattern. This allows dashboard monitoring of FEA progress while keeping surrogate trials private.


context: Unified trial management with TrialManager and DashboardDB

insight: TRIAL MANAGEMENT PATTERN: Use TrialManager for consistent trial_NNNN naming across all optimization methods (Optuna, Turbo, GNN, manual). Key principles: (1) Trial numbers NEVER reset (monotonic), (2) Folders NEVER get overwritten, (3) Database always synced with filesystem, (4) Surrogate predictions are NOT trials - only FEA results. DashboardDB provides Optuna-compatible schema for dashboard integration. Path: optimization_engine/utils/trial_manager.py


context: GNN Turbo training data loading from multiple studies

insight: MULTI-STUDY TRAINING: When loading training data from multiple prior studies for GNN surrogate training, param names may have unit prefixes like '[mm]rib_thickness' or '[Degrees]angle'. Strip prefixes: if ']' in name: name = name.split(']', 1)[1]. Also, objective attribute names vary between studies (rel_filtered_rms_40_vs_20 vs obj_rel_filtered_rms_40_vs_20) - use fallback chain with 'or'. V5 successfully trained on 316 samples (V3: 297, V4: 19) with R²=[0.94, 0.94, 0.89, 0.95].


context: Implemented L-BFGS gradient optimizer for surrogate polish phase

insight: L-BFGS on trained MLP surrogates provides 100-1000x faster convergence than derivative-free methods (TPE, CMA-ES) for local refinement. Key: use multi-start from top FEA candidates, not random initialization. Integration: GradientOptimizer class in optimization_engine/gradient_optimizer.py.


context: V6 pure TPE outperformed V5 surrogate+L-BFGS by 22%

insight: SIMPLE BEATS COMPLEX: V6 Pure TPE achieved WS=225.41 vs V5's WS=290.18 (22.3% better). Key insight: surrogates fail when gradient methods descend to OOD regions. Fix: EnsembleSurrogate with (1) N=5 MLPs for disagreement-based uncertainty, (2) OODDetector with KNN+z-score, (3) acquisition_score balancing exploitation+exploration, (4) trust-region L-BFGS that stays in training distribution. Never trust point predictions - always require uncertainty bounds. Protocol: SYS_16_SELF_AWARE_TURBO.md. Code: optimization_engine/surrogates/ensemble_surrogate.py


context: Disk space optimization for FEA studies

insight: Per-trial FEA files are ~150MB but only OP2+JSON (~70MB) are essential. PRT/FEM/SIM/DAT are copies of master files and can be deleted after study completion. Archive to dalidou server for long-term storage.