Atomizer/docs/archive/historical/PRUNING_DIAGNOSTICS.md

Pruning Diagnostics - Comprehensive Trial Failure Tracking

Created: November 20, 2025 Status: ✅ Production Ready

---

Overview

The pruning diagnostics system provides detailed logging and analysis of failed optimization trials. It helps identify:

• Why trials are failing (validation, simulation, or extraction)
• Which parameters cause failures
• False positives from pyNastran OP2 reader
• Patterns that can improve validation rules

---

Components

1. Pruning Logger

Module: optimization_engine/pruning_logger.py

Logs every pruned trial with full details:

• Parameters that failed
• Failure cause (validation, simulation, OP2 extraction)
• Error messages and stack traces
• F06 file analysis (for OP2 failures)

2. Robust OP2 Extractor

Module: optimization_engine/op2_extractor.py

Handles pyNastran issues gracefully:

• Tries multiple extraction strategies
• Ignores benign FATAL flags
• Falls back to F06 parsing
• Prevents false positive failures

---

Usage in Optimization Scripts

Basic Integration

from pathlib import Path
from optimization_engine.pruning_logger import PruningLogger
from optimization_engine.op2_extractor import robust_extract_first_frequency
from optimization_engine.simulation_validator import SimulationValidator

# Initialize pruning logger
results_dir = Path("studies/my_study/2_results")
pruning_logger = PruningLogger(results_dir, verbose=True)

# Initialize validator
validator = SimulationValidator(model_type='circular_plate', verbose=True)

def objective(trial):
    """Objective function with comprehensive pruning logging."""

    # Sample parameters
    params = {
        'inner_diameter': trial.suggest_float('inner_diameter', 50, 150),
        'plate_thickness': trial.suggest_float('plate_thickness', 2, 10)
    }

    # VALIDATION
    is_valid, warnings = validator.validate(params)
    if not is_valid:
        # Log validation failure
        pruning_logger.log_validation_failure(
            trial_number=trial.number,
            design_variables=params,
            validation_warnings=warnings
        )
        raise optuna.TrialPruned()

    # Update CAD and run simulation
    updater.update_expressions(params)
    result = solver.run_simulation(str(sim_file), solution_name="Solution_Normal_Modes")

    # SIMULATION FAILURE
    if not result['success']:
        pruning_logger.log_simulation_failure(
            trial_number=trial.number,
            design_variables=params,
            error_message=result.get('error', 'Unknown error'),
            return_code=result.get('return_code'),
            solver_errors=result.get('errors')
        )
        raise optuna.TrialPruned()

    # OP2 EXTRACTION (robust method)
    op2_file = result['op2_file']
    f06_file = result.get('f06_file')

    try:
        frequency = robust_extract_first_frequency(
            op2_file=op2_file,
            mode_number=1,
            f06_file=f06_file,
            verbose=True
        )
    except Exception as e:
        # Log OP2 extraction failure
        pruning_logger.log_op2_extraction_failure(
            trial_number=trial.number,
            design_variables=params,
            exception=e,
            op2_file=op2_file,
            f06_file=f06_file
        )
        raise optuna.TrialPruned()

    # Success - calculate objective
    return abs(frequency - 115.0)

# After optimization completes
pruning_logger.save_summary()

---

Output Files

Pruning History (Detailed Log)

File: 2_results/pruning_history.json

Contains every pruned trial with full details:

[
  {
    "trial_number": 0,
    "timestamp": "2025-11-20T19:09:45.123456",
    "pruning_cause": "op2_extraction_failure",
    "design_variables": {
      "inner_diameter": 126.56,
      "plate_thickness": 9.17
    },
    "exception_type": "ValueError",
    "exception_message": "There was a Nastran FATAL Error. Check the F06.",
    "stack_trace": "Traceback (most recent call last)...",
    "details": {
      "op2_file": "studies/.../circular_plate_sim1-solution_normal_modes.op2",
      "op2_exists": true,
      "op2_size_bytes": 245760,
      "f06_file": "studies/.../circular_plate_sim1-solution_normal_modes.f06",
      "is_pynastran_fatal_flag": true,
      "f06_has_fatal_errors": false,
      "f06_errors": []
    }
  },
  {
    "trial_number": 5,
    "timestamp": "2025-11-20T19:11:23.456789",
    "pruning_cause": "simulation_failure",
    "design_variables": {
      "inner_diameter": 95.2,
      "plate_thickness": 3.8
    },
    "error_message": "Mesh generation failed - element quality below threshold",
    "details": {
      "return_code": 1,
      "solver_errors": ["FATAL: Mesh quality check failed"]
    }
  }
]

Pruning Summary (Analysis Report)

File: 2_results/pruning_summary.json

Statistical analysis and recommendations:

{
  "generated": "2025-11-20T19:15:30.123456",
  "total_pruned_trials": 9,
  "breakdown": {
    "validation_failures": 2,
    "simulation_failures": 1,
    "op2_extraction_failures": 6
  },
  "validation_failure_reasons": {},
  "simulation_failure_types": {
    "Mesh generation failed": 1
  },
  "op2_extraction_analysis": {
    "total_op2_failures": 6,
    "likely_false_positives": 6,
    "description": "False positives are OP2 extraction failures where pyNastran detected FATAL flag but F06 has no errors"
  },
  "recommendations": [
    "CRITICAL: 6 trials failed due to pyNastran OP2 reader being overly strict. Use robust_extract_first_frequency() to ignore benign FATAL flags and extract valid results."
  ]
}

---

Robust OP2 Extraction

Problem: pyNastran False Positives

pyNastran's OP2 reader can be overly strict - it throws exceptions when it sees a FATAL flag in the OP2 header, even if:

• The F06 file shows no errors
• The simulation completed successfully
• The eigenvalue data is valid and extractable

Solution: Multi-Strategy Extraction

The robust_extract_first_frequency() function tries multiple strategies:

from optimization_engine.op2_extractor import robust_extract_first_frequency

frequency = robust_extract_first_frequency(
    op2_file=Path("results.op2"),
    mode_number=1,
    f06_file=Path("results.f06"),  # Optional fallback
    verbose=True
)

Strategies (in order):

1. Standard OP2 read - Normal pyNastran reading
2. Lenient OP2 read - debug=False, skip_undefined_matrices=True
3. F06 fallback - Parse text file if OP2 fails

Output (verbose mode):

[OP2 EXTRACT] Attempting standard read: circular_plate_sim1-solution_normal_modes.op2
[OP2 EXTRACT] ✗ Standard read failed: There was a Nastran FATAL Error
[OP2 EXTRACT] Detected pyNastran FATAL flag issue
[OP2 EXTRACT] Attempting partial extraction...
[OP2 EXTRACT] ✓ Success (lenient mode): 125.1234 Hz
[OP2 EXTRACT] Note: pyNastran reported FATAL but data is valid!

---

Analyzing Pruning Patterns

View Summary

import json
from pathlib import Path

# Load pruning summary
with open('studies/my_study/2_results/pruning_summary.json') as f:
    summary = json.load(f)

print(f"Total pruned: {summary['total_pruned_trials']}")
print(f"False positives: {summary['op2_extraction_analysis']['likely_false_positives']}")
print("\nRecommendations:")
for rec in summary['recommendations']:
    print(f"  - {rec}")

Find Specific Failures

import json

# Load detailed history
with open('studies/my_study/2_results/pruning_history.json') as f:
    history = json.load(f)

# Find all OP2 false positives
false_positives = [
    event for event in history
    if event['pruning_cause'] == 'op2_extraction_failure'
    and event['details']['is_pynastran_fatal_flag']
    and not event['details']['f06_has_fatal_errors']
]

print(f"Found {len(false_positives)} false positives:")
for fp in false_positives:
    params = fp['design_variables']
    print(f"  Trial #{fp['trial_number']}: {params}")

Parameter Analysis

# Find which parameter ranges cause failures
import numpy as np

validation_failures = [e for e in history if e['pruning_cause'] == 'validation_failure']

diameters = [e['design_variables']['inner_diameter'] for e in validation_failures]
thicknesses = [e['design_variables']['plate_thickness'] for e in validation_failures]

print(f"Validation failures occur at:")
print(f"  Diameter range: {min(diameters):.1f} - {max(diameters):.1f} mm")
print(f"  Thickness range: {min(thicknesses):.1f} - {max(thicknesses):.1f} mm")

---

Expected Impact

Before Robust Extraction

• Pruning rate: 18-20%
• False positives: ~6-10 per 50 trials
• Wasted time: ~5 minutes per study

After Robust Extraction

• Pruning rate: <2% (only genuine failures)
• False positives: 0
• Time saved: ~4-5 minutes per study
• Better optimization: More valid trials = better convergence

---

Testing

Test the robust extractor on a known "failed" OP2 file:

python -c "
from pathlib import Path
from optimization_engine.op2_extractor import robust_extract_first_frequency

# Use an OP2 file that pyNastran rejects
op2_file = Path('studies/circular_plate_protocol10_v2_2_test/1_setup/model/circular_plate_sim1-solution_normal_modes.op2')
f06_file = op2_file.with_suffix('.f06')

try:
    freq = robust_extract_first_frequency(op2_file, f06_file=f06_file, verbose=True)
    print(f'\n✓ Successfully extracted: {freq:.6f} Hz')
except Exception as e:
    print(f'\n✗ Extraction failed: {e}')
"

Expected output:

[OP2 EXTRACT] Attempting standard read: circular_plate_sim1-solution_normal_modes.op2
[OP2 EXTRACT] ✗ Standard read failed: There was a Nastran FATAL Error
[OP2 EXTRACT] Detected pyNastran FATAL flag issue
[OP2 EXTRACT] Attempting partial extraction...
[OP2 EXTRACT] ✓ Success (lenient mode): 115.0442 Hz
[OP2 EXTRACT] Note: pyNastran reported FATAL but data is valid!

✓ Successfully extracted: 115.044200 Hz

---

Summary

Feature	Description	File
Pruning Logger	Comprehensive failure tracking	pruning_logger.py
Robust OP2 Extractor	Handles pyNastran issues	op2_extractor.py
Pruning History	Detailed JSON log	2_results/pruning_history.json
Pruning Summary	Analysis and recommendations	2_results/pruning_summary.json

Status: ✅ Ready for production use

Benefits:

• Zero false positive failures
• Detailed diagnostics for genuine failures
• Pattern analysis for validation improvements
• ~5 minutes saved per 50-trial study