feat: Major update with validators, skills, dashboard, and docs reorganization

- Add validation framework (config, model, results, study validators) - Add Claude Code skills (create-study, run-optimization, generate-report, troubleshoot, analyze-model) - Add Atomizer Dashboard (React frontend + FastAPI backend) - Reorganize docs into structured directories (00-09) - Add neural surrogate modules and training infrastructure - Add multi-objective optimization support 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-25 19:23:58 -05:00
parent 74a92803b7
commit e3bdb08a22
155 changed files with 52729 additions and 37 deletions
--- a/docs/08_ARCHIVE/historical/BRACKET_STUDY_ISSUES_LOG.md
+++ b/docs/08_ARCHIVE/historical/BRACKET_STUDY_ISSUES_LOG.md
@@ -0,0 +1,599 @@
+# Bracket Stiffness Optimization - Issues Log
+**Date**: November 21, 2025
+**Study**: bracket_stiffness_optimization
+**Protocol**: Protocol 10 (IMSO)
+
+## Executive Summary
+Attempted to create a new bracket stiffness optimization study using Protocol 10. Encountered **8 critical issues** that prevented the study from running successfully. All issues are protocol violations that should be prevented by better templates, validation, and documentation.
+
+---
+
+## Issue #1: Unicode/Emoji Characters Breaking Windows Console
+**Severity**: CRITICAL
+**Category**: Output Formatting
+**Protocol Violation**: Using non-ASCII characters in code output
+
+### What Happened
+Code contained unicode symbols (≤, ✓, ✗, 🎯, 📊, ⚠) in print statements, causing:
+```
+UnicodeEncodeError: 'charmap' codec can't encode character '\u2264' in position 17
+```
+
+### Root Cause
+- Windows cmd uses cp1252 encoding by default
+- Unicode symbols not in cp1252 cause crashes
+- User explicitly requested NO emojis/unicode in previous sessions
+
+### Files Affected
+- `run_optimization.py` (multiple print statements)
+- `bracket_stiffness_extractor.py` (print statements)
+- `export_displacement_field.py` (success messages)
+
+### Fix Applied
+Replace ALL unicode with ASCII equivalents:
+- `≤` → `<=`
+- `✓` → `[OK]`
+- `✗` → `[X]`
+- `⚠` → `[!]`
+- `🎯` → `[BEST]`
+- etc.
+
+### Protocol Fix Required
+**MANDATORY RULE**: Never use unicode symbols or emojis in any Python code that prints to console.
+
+Create `atomizer/utils/safe_print.py`:
+```python
+"""Windows-safe printing utilities - ASCII only"""
+
+def print_success(msg):
+    print(f"[OK] {msg}")
+
+def print_error(msg):
+    print(f"[X] {msg}")
+
+def print_warning(msg):
+    print(f"[!] {msg}")
+```
+
+---
+
+## Issue #2: Hardcoded NX Version Instead of Using config.py
+**Severity**: CRITICAL
+**Category**: Configuration Management
+**Protocol Violation**: Not using central configuration
+
+### What Happened
+Code hardcoded `nastran_version="2306"` but user has NX 2412 installed:
+```
+FileNotFoundError: Could not auto-detect NX 2306 installation
+```
+
+User explicitly asked: "isn't it in the protocole to use the actual config in config.py????"
+
+### Root Cause
+- Ignored `config.py` which has `NX_VERSION = "2412"`
+- Hardcoded old version number
+- Same issue in bracket_stiffness_extractor.py line 152
+
+### Files Affected
+- `run_optimization.py` line 85
+- `bracket_stiffness_extractor.py` line 152
+
+### Fix Applied
+```python
+import config as atomizer_config
+
+nx_solver = NXSolver(
+    nastran_version=atomizer_config.NX_VERSION,  # Use central config
+    timeout=atomizer_config.NASTRAN_TIMEOUT,
+)
+```
+
+### Protocol Fix Required
+**MANDATORY RULE**: ALWAYS import and use `config.py` for ALL system paths and versions.
+
+Add validation check in all study templates:
+```python
+# Validate using central config
+assert 'atomizer_config' in dir(), "Must import config as atomizer_config"
+```
+
+---
+
+## Issue #3: Module Name Collision (config vs config parameter)
+**Severity**: HIGH
+**Category**: Code Quality
+**Protocol Violation**: Poor naming conventions
+
+### What Happened
+```python
+import config  # Module named 'config'
+
+def create_objective_function(config: dict, ...):  # Parameter named 'config'
+    # Inside function:
+    nastran_version=config.NX_VERSION  # ERROR: config is the dict, not the module!
+```
+
+Error: `AttributeError: 'dict' object has no attribute 'NX_VERSION'`
+
+### Root Cause
+Variable shadowing - parameter `config` shadows imported module `config`
+
+### Fix Applied
+```python
+import config as atomizer_config  # Unique name
+
+def create_objective_function(config: dict, ...):
+    nastran_version=atomizer_config.NX_VERSION  # Now unambiguous
+```
+
+### Protocol Fix Required
+**MANDATORY RULE**: Always import config as `atomizer_config` to prevent collisions.
+
+Update all templates and examples to use:
+```python
+import config as atomizer_config
+```
+
+---
+
+## Issue #4: Protocol 10 Didn't Support Multi-Objective Optimization
+**Severity**: CRITICAL
+**Category**: Feature Gap
+**Protocol Violation**: Protocol 10 documentation claims multi-objective support but doesn't implement it
+
+### What Happened
+Protocol 10 (`IntelligentOptimizer`) hardcoded `direction='minimize'` for single-objective only.
+Multi-objective problems (like bracket: maximize stiffness, minimize mass) couldn't use Protocol 10.
+
+### Root Cause
+`IntelligentOptimizer.optimize()` didn't accept `directions` parameter
+`_create_study()` always created single-objective studies
+
+### Fix Applied
+Enhanced `intelligent_optimizer.py`:
+```python
+def optimize(self, ..., directions: Optional[list] = None):
+    self.directions = directions
+
+def _create_study(self):
+    if self.directions is not None:
+        # Multi-objective
+        study = optuna.create_study(directions=self.directions, ...)
+    else:
+        # Single-objective (backward compatible)
+        study = optuna.create_study(direction='minimize', ...)
+```
+
+### Protocol Fix Required
+**PROTOCOL 10 UPDATE**: Document and test multi-objective support.
+
+Add to Protocol 10 documentation:
+- Single-objective: `directions=None` or `directions=["minimize"]`
+- Multi-objective: `directions=["minimize", "maximize", ...]`
+- Update all examples to show both cases
+
+---
+
+## Issue #5: Wrong Solution Name Parameter to NX Solver
+**Severity**: HIGH
+**Category**: NX API Usage
+**Protocol Violation**: Incorrect understanding of NX solution naming
+
+### What Happened
+Passed `solution_name="Bracket_sim1"` to NX solver, causing:
+```
+NXOpen.NXException: No object found with this name: Solution[Bracket_sim1]
+```
+
+All trials pruned because solver couldn't find solution.
+
+### Root Cause
+- NX solver looks for "Solution[<name>]" object
+- Solution name should be "Solution 1", not the sim file name
+- Passing `None` solves all solutions in .sim file (correct for most cases)
+
+### Fix Applied
+```python
+result = nx_solver.run_simulation(
+    sim_file=sim_file,
+    solution_name=None  # Solve all solutions
+)
+```
+
+### Protocol Fix Required
+**DOCUMENTATION**: Clarify `solution_name` parameter in NX solver docs.
+
+Default should be `None` (solve all solutions). Only specify when you need to solve a specific solution from a multi-solution .sim file.
+
+---
+
+## Issue #6: NX Journal Needs to Open Simulation File
+**Severity**: HIGH
+**Category**: NX Journal Design
+**Protocol Violation**: Journal assumes file is already open
+
+### What Happened
+`export_displacement_field.py` expected a simulation to already be open:
+```python
+workSimPart = theSession.Parts.BaseWork
+if workSimPart is None:
+    print("ERROR: No work part loaded")
+    return 1
+```
+
+When called via `run_journal.exe`, NX starts with no files open.
+
+### Root Cause
+Journal template didn't handle opening the sim file
+
+### Fix Applied
+Enhanced journal to open sim file:
+```python
+def main(args):
+    # Accept sim file path as argument
+    if len(args) > 0:
+        sim_file = Path(args[0])
+    else:
+        sim_file = Path(__file__).parent / "Bracket_sim1.sim"
+
+    # Open the simulation
+    basePart1, partLoadStatus1 = theSession.Parts.OpenBaseDisplay(str(sim_file))
+    partLoadStatus1.Dispose()
+```
+
+### Protocol Fix Required
+**JOURNAL TEMPLATE**: All NX journals should handle opening required files.
+
+Create standard journal template that:
+1. Accepts file paths as arguments
+2. Opens required files (part, sim, fem)
+3. Performs operation
+4. Closes gracefully
+
+---
+
+## Issue #7: Subprocess Check Fails on NX sys.exit(0)
+**Severity**: MEDIUM
+**Category**: NX Integration
+**Protocol Violation**: Incorrect error handling for NX journals
+
+### What Happened
+```python
+subprocess.run([nx_exe, journal], check=True)  # Raises exception even on success!
+```
+
+NX's `run_journal.exe` returns non-zero exit code even when journal exits with `sys.exit(0)`.
+The stderr shows:
+```
+SystemExit: 0  <-- Success!
+```
+
+But subprocess.run with `check=True` raises `CalledProcessError`.
+
+### Root Cause
+NX wraps Python journals and reports `sys.exit()` as a "Syntax error" in stderr, even for exit code 0.
+
+### Fix Applied
+Don't use `check=True`. Instead, verify output file was created:
+```python
+result = subprocess.run([nx_exe, journal], capture_output=True, text=True)
+if not output_file.exists():
+    raise RuntimeError(f"Journal completed but output file not created")
+```
+
+### Protocol Fix Required
+**NX SOLVER WRAPPER**: Never use `check=True` for NX journal execution.
+
+Create `nx_utils.run_journal_safe()`:
+```python
+def run_journal_safe(journal_path, expected_outputs=[]):
+    """Run NX journal and verify outputs, ignoring exit code"""
+    result = subprocess.run([NX_RUN_JOURNAL, journal_path],
+                          capture_output=True, text=True)
+
+    for output_file in expected_outputs:
+        if not Path(output_file).exists():
+            raise RuntimeError(f"Journal failed: {output_file} not created")
+
+    return result
+```
+
+---
+
+## Issue #8: OP2 File Naming Mismatch
+**Severity**: HIGH
+**Category**: File Path Management
+**Protocol Violation**: Assumed file naming instead of detecting actual names
+
+### What Happened
+Extractor looked for `Bracket_sim1.op2` but NX created `bracket_sim1-solution_1.op2`:
+```
+ERROR: OP2 file not found: Bracket_sim1.op2
+```
+
+### Root Cause
+- NX creates OP2 with lowercase sim base name
+- NX adds `-solution_1` suffix
+- Extractor hardcoded expected name without checking
+
+### Fix Applied
+```python
+self.sim_base = Path(sim_file).stem
+self.op2_file = self.model_dir / f"{self.sim_base.lower()}-solution_1.op2"
+```
+
+### Protocol Fix Required
+**FILE DETECTION**: Never hardcode output file names. Always detect or construct from input names.
+
+Create `nx_utils.find_op2_file()`:
+```python
+def find_op2_file(sim_file: Path, working_dir: Path) -> Path:
+    """Find OP2 file generated by NX simulation"""
+    sim_base = sim_file.stem.lower()
+
+    # Try common patterns
+    patterns = [
+        f"{sim_base}-solution_1.op2",
+        f"{sim_base}.op2",
+        f"{sim_base}-*.op2",
+    ]
+
+    for pattern in patterns:
+        matches = list(working_dir.glob(pattern))
+        if matches:
+            return matches[0]  # Return first match
+
+    raise FileNotFoundError(f"No OP2 file found for {sim_file}")
+```
+
+---
+
+## Issue #9: Field Data Extractor Expects CSV, NX Exports Custom Format
+**Severity**: CRITICAL
+**Category**: Data Format Mismatch
+**Protocol Violation**: Generic extractor not actually generic
+
+### What Happened
+```
+ERROR: No valid data found in column 'z(mm)'
+```
+
+### Root Cause
+NX field export format:
+```
+FIELD: [ResultProbe] : [TABLE]
+INDEP VAR: [step] : [] : [] : [0]
+INDEP VAR: [node_id] : [] : [] : [5]
+DEP VAR: [x] : [Length] : [mm] : [0]
+START DATA
+0, 396, -0.086716040968895
+0, 397, -0.087386816740036
+...
+END DATA
+```
+
+This is NOT a CSV with headers! But `FieldDataExtractor` uses:
+```python
+reader = csv.DictReader(f)  # Expects CSV headers!
+value = float(row[self.result_column])  # Looks for column 'z(mm)'
+```
+
+### Fix Required
+`FieldDataExtractor` needs complete rewrite to handle NX field format:
+
+```python
+def _parse_nx_field_file(self, file_path: Path) -> np.ndarray:
+    """Parse NX field export format (.fld)"""
+    values = []
+    in_data_section = False
+
+    with open(file_path, 'r') as f:
+        for line in f:
+            if line.startswith('START DATA'):
+                in_data_section = True
+                continue
+            if line.startswith('END DATA'):
+                break
+
+            if in_data_section:
+                parts = line.strip().split(',')
+                if len(parts) >= 3:
+                    try:
+                        value = float(parts[2].strip())  # Third column is value
+                        values.append(value)
+                    except ValueError:
+                        continue
+
+    return np.array(values)
+```
+
+### Protocol Fix Required
+**CRITICAL**: Fix `FieldDataExtractor` to actually parse NX field format.
+
+The extractor claims to be "generic" and "reusable" but only works with CSV files, not NX field exports!
+
+---
+
+## Issue #10: Grid Point Forces Not Requested in OP2 Output
+**Severity**: CRITICAL - BLOCKING ALL TRIALS
+**Category**: NX Simulation Configuration
+**Protocol Violation**: Missing output request validation
+
+### What Happened
+ALL trials (44-74+) are being pruned with the same error:
+```
+ERROR: Extraction failed: No grid point forces found in OP2 file
+```
+
+Simulation completes successfully:
+- NX solver runs without errors
+- OP2 file is generated and regenerated with fresh timestamps
+- Displacement field is exported successfully
+- Field data is parsed correctly
+
+But stiffness calculation fails because applied force cannot be extracted from OP2.
+
+### Root Cause
+The NX simulation is not configured to output grid point forces to the OP2 file.
+
+Nastran requires explicit output requests in the Case Control section. The bracket simulation likely only requests:
+- Displacement results
+- Stress results (maybe)
+
+But does NOT request:
+- Grid point forces (GPFORCE)
+
+Without this output request, the OP2 file contains nodal displacements but not reaction forces at grid points.
+
+### Evidence
+From stiffness_calculator.py (optimization_engine/extractors/stiffness_calculator.py):
+```python
+# Extract applied force from OP2
+force_results = self.op2_extractor.extract_force(component=self.force_component)
+# Raises: ValueError("No grid point forces found in OP2 file")
+```
+
+The OP2Extractor tries to read `op2.grid_point_forces` which is empty because NX didn't request this output.
+
+### Fix Required
+**Option A: Modify NX Simulation Configuration (Recommended)**
+
+Open `Bracket_sim1.sim` in NX and add grid point forces output request:
+1. Edit Solution 1
+2. Go to "Solution Control" or "Output Requests"
+3. Add "Grid Point Forces" to output requests
+4. Save simulation
+
+This will add to the Nastran deck:
+```
+GPFORCE = ALL
+```
+
+**Option B: Extract Forces from Load Definition (Alternative)**
+
+If the applied load is constant and defined in the model, extract it from the .sim file or model expressions instead of relying on OP2:
+```python
+# In bracket_stiffness_extractor.py
+def _get_applied_force_from_model(self):
+    """Extract applied force magnitude from model definition"""
+    # Load is 1000N in Z-direction based on model setup
+    return 1000.0  # N
+```
+
+This is less robust but works if the load is constant.
+
+**Option C: Enhance OP2Extractor to Read from F06 File**
+
+Nastran always writes grid point forces to the F06 text file. Add F06 parsing as fallback:
+```python
+def extract_force(self, component='fz'):
+    # Try OP2 first
+    if self.op2.grid_point_forces:
+        return self._extract_from_op2(component)
+
+    # Fallback to F06 file
+    f06_file = self.op2_file.with_suffix('.f06')
+    if f06_file.exists():
+        return self._extract_from_f06(f06_file, component)
+
+    raise ValueError("No grid point forces found in OP2 or F06 file")
+```
+
+### Protocol Fix Required
+**MANDATORY VALIDATION**: Add pre-flight check for required output requests.
+
+Create `nx_utils.validate_simulation_outputs()`:
+```python
+def validate_simulation_outputs(sim_file: Path, required_outputs: list):
+    """
+    Validate that NX simulation has required output requests configured.
+
+    Args:
+        sim_file: Path to .sim file
+        required_outputs: List of required outputs, e.g.,
+                         ['displacement', 'stress', 'grid_point_forces']
+
+    Raises:
+        ValueError: If required outputs are not configured
+    """
+    # Parse .sim file or generated .dat file to check output requests
+    # Provide helpful error message with instructions to add missing outputs
+    pass
+```
+
+Call this validation BEFORE starting optimization:
+```python
+# In run_optimization.py, before optimizer.optimize()
+validate_simulation_outputs(
+    sim_file=sim_file,
+    required_outputs=['displacement', 'grid_point_forces']
+)
+```
+
+### Immediate Action
+**For bracket study**: Open Bracket_sim1.sim in NX and add Grid Point Forces output request.
+
+---
+
+## Summary of Protocol Fixes Needed
+
+### HIGH PRIORITY (Blocking)
+1. ✅ Fix `FieldDataExtractor` to parse NX field format
+2. ✅ Create "no unicode" rule and safe_print utilities
+3. ✅ Enforce config.py usage in all templates
+4. ✅ Update Protocol 10 for multi-objective support
+5. ❌ **CURRENT BLOCKER**: Fix grid point forces extraction (Issue #10)
+
+### MEDIUM PRIORITY (Quality)
+5. ✅ Create NX journal template with file opening
+6. ✅ Create nx_utils.run_journal_safe() wrapper
+7. ✅ Create nx_utils.find_op2_file() detection
+8. ✅ Add naming convention (import config as atomizer_config)
+
+### DOCUMENTATION
+9. ✅ Document solution_name parameter behavior
+10. ✅ Update Protocol 10 docs with multi-objective examples
+11. ✅ Create "Windows Compatibility Guide"
+12. ✅ Add field file format documentation
+
+---
+
+## Lessons Learned
+
+### What Went Wrong
+1. **Generic tools weren't actually generic** - FieldDataExtractor only worked for CSV
+2. **No validation of central config usage** - Easy to forget to import
+3. **Unicode symbols slip in during development** - Need linter check
+4. **Subprocess error handling assumed standard behavior** - NX is non-standard
+5. **File naming assumptions instead of detection** - Brittle
+6. **Protocol 10 feature gap** - Claims multi-objective but didn't implement it
+7. **Journal templates incomplete** - Didn't handle file opening
+
+### What Should Have Been Caught
+- Pre-flight validation script should check:
+  - ✅ No unicode in any .py files
+  - ✅ All studies import config.py
+  - ✅ All output files use detected names, not hardcoded
+  - ✅ All journals can run standalone (no assumptions about open files)
+
+### Time Lost
+- Approximately 60+ minutes debugging issues that should have been prevented
+- Would have been 5 minutes to run successfully with proper templates
+
+---
+
+## Action Items
+
+1. [ ] Rewrite FieldDataExtractor to handle NX format
+2. [ ] Create pre-flight validation script
+3. [ ] Update all study templates
+4. [ ] Add linter rules for unicode detection
+5. [ ] Create nx_utils module with safe wrappers
+6. [ ] Update Protocol 10 documentation
+7. [ ] Create Windows compatibility guide
+8. [ ] Add integration tests for NX file formats
+
+---
+
+**Next Step**: Fix FieldDataExtractor and test complete workflow end-to-end.