Atomizer/docs/04_USER_GUIDES/NEURAL_WORKFLOW_TUTORIAL.md

# Neural Workflow Tutorial

**End-to-End Guide: From FEA Data to Neural-Accelerated Optimization**

This tutorial walks you through the complete workflow of setting up neural network acceleration for your optimization studies.

---

## Prerequisites

Before starting, ensure you have:

- [ ] Atomizer installed and working
- [ ] An NX Nastran model with parametric geometry
- [ ] Python environment with PyTorch and PyTorch Geometric
- [ ] GPU recommended (CUDA) but not required

### Install Neural Dependencies

```bash
# Install PyTorch (with CUDA support)
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

# Install PyTorch Geometric
pip install torch-geometric

# Install other dependencies
pip install h5py pyNastran
```

---

## Overview

The workflow consists of 5 phases:

```
Phase 1: Initial FEA Study → Collect Training Data
                ↓
Phase 2: Parse Data → Convert BDF/OP2 to Neural Format
                ↓
Phase 3: Train Model → Train GNN on Collected Data
                ↓
Phase 4: Validate → Verify Model Accuracy
                ↓
Phase 5: Deploy → Run Neural-Accelerated Optimization
```

**Time Investment**:
- Phase 1: 4-8 hours (initial FEA runs)
- Phase 2: 30 minutes (parsing)
- Phase 3: 30-60 minutes (training)
- Phase 4: 10 minutes (validation)
- Phase 5: Minutes instead of hours!

---

## Phase 1: Collect Training Data

### Step 1.1: Configure Training Data Export

Edit your `workflow_config.json` to enable training data export:

```json
{
  "study_name": "uav_arm_optimization",

  "design_variables": [
    {
      "name": "beam_half_core_thickness",
      "expression_name": "beam_half_core_thickness",
      "min": 5.0,
      "max": 15.0,
      "units": "mm"
    },
    {
      "name": "beam_face_thickness",
      "expression_name": "beam_face_thickness",
      "min": 1.0,
      "max": 5.0,
      "units": "mm"
    },
    {
      "name": "holes_diameter",
      "expression_name": "holes_diameter",
      "min": 20.0,
      "max": 50.0,
      "units": "mm"
    },
    {
      "name": "hole_count",
      "expression_name": "hole_count",
      "min": 5,
      "max": 15,
      "units": ""
    }
  ],

  "objectives": [
    {"name": "mass", "direction": "minimize"},
    {"name": "frequency", "direction": "maximize"},
    {"name": "max_displacement", "direction": "minimize"},
    {"name": "max_stress", "direction": "minimize"}
  ],

  "training_data_export": {
    "enabled": true,
    "export_dir": "atomizer_field_training_data/uav_arm"
  },

  "optimization_settings": {
    "n_trials": 50,
    "sampler": "TPE"
  }
}
```

### Step 1.2: Run Initial Optimization

```bash
cd studies/uav_arm_optimization
python run_optimization.py --trials 50
```

This will:
1. Run 50 FEA simulations
2. Export each trial's BDF and OP2 files
3. Save design parameters and objectives

**Expected output**:
```
Trial 1/50: beam_half_core_thickness=10.2, beam_face_thickness=2.8...
  → Exporting training data to atomizer_field_training_data/uav_arm/trial_0001/
Trial 2/50: beam_half_core_thickness=7.5, beam_face_thickness=3.1...
  → Exporting training data to atomizer_field_training_data/uav_arm/trial_0002/
...
```

### Step 1.3: Verify Exported Data

Check the exported data structure:

```bash
ls atomizer_field_training_data/uav_arm/
```

Expected:
```
trial_0001/
trial_0002/
...
trial_0050/
study_summary.json
README.md
```

Each trial folder contains:
```
trial_0001/
├── input/
│   └── model.bdf      # Nastran input deck
├── output/
│   └── model.op2      # Binary results
└── metadata.json      # Design variables and objectives
```

---

## Phase 2: Parse Data

### Step 2.1: Navigate to AtomizerField

```bash
cd atomizer-field
```

### Step 2.2: Parse All Cases

```bash
python batch_parser.py ../atomizer_field_training_data/uav_arm
```

**What this does**:
1. Reads each BDF file (mesh, materials, BCs, loads)
2. Reads each OP2 file (displacement, stress, strain fields)
3. Converts to HDF5 + JSON format
4. Validates physics consistency

**Expected output**:
```
Processing 50 cases...
[1/50] trial_0001: ✓ Parsed successfully (2.3s)
[2/50] trial_0002: ✓ Parsed successfully (2.1s)
...
[50/50] trial_0050: ✓ Parsed successfully (2.4s)

Summary:
├── Successful: 50/50
├── Failed: 0
└── Total time: 115.2s
```

### Step 2.3: Validate Parsed Data

Run validation on a few cases:

```bash
python validate_parsed_data.py ../atomizer_field_training_data/uav_arm/trial_0001
```

**Expected output**:
```
Validation Results for trial_0001:
├── File Structure: ✓ Valid
├── Mesh Quality: ✓ Valid (15,432 nodes, 8,765 elements)
├── Material Properties: ✓ Valid (E=70 GPa, nu=0.33)
├── Boundary Conditions: ✓ Valid (12 fixed nodes)
├── Load Data: ✓ Valid (1 gravity load)
├── Displacement Field: ✓ Valid (max: 0.042 mm)
├── Stress Field: ✓ Valid (max: 125.3 MPa)
└── Overall: ✓ VALID
```

---

## Phase 3: Train Model

### Step 3.1: Split Data

Create train/validation split:

```bash
# Create directories
mkdir -p ../atomizer_field_training_data/uav_arm_train
mkdir -p ../atomizer_field_training_data/uav_arm_val

# Move 80% to train, 20% to validation
# (You can write a script or do this manually)
```

### Step 3.2: Train Parametric Model

```bash
python train_parametric.py \
  --train_dir ../atomizer_field_training_data/uav_arm_train \
  --val_dir ../atomizer_field_training_data/uav_arm_val \
  --epochs 200 \
  --hidden_channels 128 \
  --num_layers 4 \
  --learning_rate 0.001 \
  --output_dir runs/my_uav_model
```

**What this does**:
1. Loads parsed training data
2. Builds design-conditioned GNN
3. Trains with physics-informed loss
4. Saves best checkpoint based on validation loss

**Expected output**:
```
Training Parametric GNN
├── Training samples: 40
├── Validation samples: 10
├── Model parameters: 523,412

Epoch [1/200]:
├── Train Loss: 0.3421
├── Val Loss: 0.2987
└── Best model saved!

Epoch [50/200]:
├── Train Loss: 0.0234
├── Val Loss: 0.0312

Epoch [200/200]:
├── Train Loss: 0.0089
├── Val Loss: 0.0156
└── Training complete!

Best validation loss: 0.0142 (epoch 187)
Model saved to: runs/my_uav_model/checkpoint_best.pt
```

### Step 3.3: Monitor Training (Optional)

If TensorBoard is installed:

```bash
tensorboard --logdir runs/my_uav_model/logs
```

Open http://localhost:6006 to view:
- Loss curves
- Learning rate schedule
- Validation metrics

---

## Phase 4: Validate Model

### Step 4.1: Run Validation Script

```bash
python validate.py --checkpoint runs/my_uav_model/checkpoint_best.pt
```

**Expected output**:
```
Model Validation Results
========================

Per-Objective Metrics:
├── mass:
│   ├── MAE: 0.52 g
│   ├── MAPE: 0.8%
│   └── R²: 0.998
├── frequency:
│   ├── MAE: 2.1 Hz
│   ├── MAPE: 1.2%
│   └── R²: 0.995
├── max_displacement:
│   ├── MAE: 0.001 mm
│   ├── MAPE: 2.8%
│   └── R²: 0.987
└── max_stress:
    ├── MAE: 3.2 MPa
    ├── MAPE: 3.5%
    └── R²: 0.981

Performance:
├── Inference time: 4.5 ms ± 0.8 ms
├── GPU memory: 512 MB
└── Throughput: 220 predictions/sec

✓ Model validation passed!
```

### Step 4.2: Test on New Designs

```python
# test_model.py
import torch
from atomizer_field.neural_models.parametric_predictor import ParametricFieldPredictor

# Load model
checkpoint = torch.load('runs/my_uav_model/checkpoint_best.pt')
model = ParametricFieldPredictor(**checkpoint['config'])
model.load_state_dict(checkpoint['model_state_dict'])
model.eval()

# Test prediction
design = {
    'beam_half_core_thickness': 7.0,
    'beam_face_thickness': 2.5,
    'holes_diameter': 35.0,
    'hole_count': 10.0
}

# Convert to tensor
design_tensor = torch.tensor([[
    design['beam_half_core_thickness'],
    design['beam_face_thickness'],
    design['holes_diameter'],
    design['hole_count']
]])

# Predict
with torch.no_grad():
    predictions = model(design_tensor)

print(f"Mass: {predictions[0, 0]:.2f} g")
print(f"Frequency: {predictions[0, 1]:.2f} Hz")
print(f"Displacement: {predictions[0, 2]:.6f} mm")
print(f"Stress: {predictions[0, 3]:.2f} MPa")
```

---

## Phase 5: Deploy Neural-Accelerated Optimization

### Step 5.1: Update Configuration

Edit `workflow_config.json` to enable neural acceleration:

```json
{
  "study_name": "uav_arm_optimization_neural",

  "neural_surrogate": {
    "enabled": true,
    "model_checkpoint": "atomizer-field/runs/my_uav_model/checkpoint_best.pt",
    "confidence_threshold": 0.85,
    "device": "cuda"
  },

  "hybrid_optimization": {
    "enabled": true,
    "exploration_trials": 20,
    "validation_frequency": 50
  },

  "optimization_settings": {
    "n_trials": 5000
  }
}
```

### Step 5.2: Run Neural-Accelerated Optimization

```bash
python run_optimization.py --trials 5000 --use-neural
```

**Expected output**:
```
Neural-Accelerated Optimization
===============================

Loading neural model from: atomizer-field/runs/my_uav_model/checkpoint_best.pt
Model loaded successfully (4.5 ms inference time)

Phase 1: Exploration (FEA)
Trial [1/5000]: Using FEA (exploration phase)
Trial [2/5000]: Using FEA (exploration phase)
...
Trial [20/5000]: Using FEA (exploration phase)

Phase 2: Exploitation (Neural)
Trial [21/5000]: Using Neural (conf: 94.2%, time: 4.8 ms)
Trial [22/5000]: Using Neural (conf: 91.8%, time: 4.3 ms)
...
Trial [5000/5000]: Using Neural (conf: 93.1%, time: 4.6 ms)

============================================================
OPTIMIZATION COMPLETE
============================================================
Total trials: 5,000
├── FEA trials: 120 (2.4%)
├── Neural trials: 4,880 (97.6%)
├── Total time: 8.3 minutes
├── Equivalent FEA time: 14.2 hours
└── Speedup: 103x

Best Design Found:
├── beam_half_core_thickness: 6.8 mm
├── beam_face_thickness: 2.3 mm
├── holes_diameter: 32.5 mm
├── hole_count: 12

Objectives:
├── mass: 45.2 g (minimized)
├── frequency: 312.5 Hz (maximized)
├── max_displacement: 0.028 mm
└── max_stress: 89.3 MPa
============================================================
```

### Step 5.3: Validate Best Designs

Run FEA validation on top designs:

```python
# validate_best_designs.py
from optimization_engine.runner import OptimizationRunner

runner = OptimizationRunner(config_path="workflow_config.json")

# Get top 10 designs from neural optimization
top_designs = runner.get_best_trials(10)

print("Validating top 10 designs with FEA...")
for i, design in enumerate(top_designs):
    # Run actual FEA
    fea_result = runner.run_fea_simulation(design.params)
    nn_result = design.values

    # Compare
    mass_error = abs(fea_result['mass'] - nn_result['mass']) / fea_result['mass'] * 100
    freq_error = abs(fea_result['frequency'] - nn_result['frequency']) / fea_result['frequency'] * 100

    print(f"Design {i+1}: Mass error={mass_error:.1f}%, Freq error={freq_error:.1f}%")
```

---

## Troubleshooting

### Common Issues

**Issue: Low confidence predictions**
```
WARNING: Neural confidence below threshold (65.3% < 85%)
```

**Solution**:
- Collect more diverse training data
- Train for more epochs
- Reduce confidence threshold
- Check if design is outside training distribution

**Issue: Training loss not decreasing**
```
Epoch [100/200]: Train Loss: 0.3421 (same as epoch 1)
```

**Solution**:
- Reduce learning rate
- Check data preprocessing
- Increase hidden channels
- Add more training data

**Issue: Large validation error**
```
Val MAE: 15.2% (expected < 5%)
```

**Solution**:
- Check for data leakage
- Add regularization (dropout)
- Use physics-informed loss
- Collect more training data

---

## Best Practices

### Data Collection

1. **Diverse sampling**: Use Latin Hypercube or Sobol sequences
2. **Sufficient quantity**: Aim for 10-20x the number of design variables
3. **Full range coverage**: Ensure designs span the entire design space
4. **Quality control**: Validate all FEA results before training

### Training

1. **Start simple**: Begin with smaller models, increase if needed
2. **Use validation**: Always monitor validation loss
3. **Early stopping**: Stop training when validation loss plateaus
4. **Save checkpoints**: Keep intermediate models

### Deployment

1. **Conservative thresholds**: Start with high confidence (0.9)
2. **Periodic validation**: Always validate with FEA periodically
3. **Monitor drift**: Track prediction accuracy over time
4. **Retrain**: Update model when drift is detected

---

## Next Steps

After completing this tutorial, explore:

1. **[Neural Features Complete](NEURAL_FEATURES_COMPLETE.md)** - Advanced features
2. **[GNN Architecture](GNN_ARCHITECTURE.md)** - Technical deep-dive
3. **[Physics Loss Guide](PHYSICS_LOSS_GUIDE.md)** - Loss function selection

---

## Summary

You've learned how to:

- [x] Configure training data export
- [x] Collect training data from FEA
- [x] Parse BDF/OP2 to neural format
- [x] Train a parametric GNN
- [x] Validate model accuracy
- [x] Deploy neural-accelerated optimization

**Result**: 1000x faster optimization with <5% prediction error!

---

*Questions? See the [troubleshooting section](#troubleshooting) or check the [main documentation](../README.md).*