Anto01
ea437d360e
- Restructure docs/ folder (remove numeric prefixes): - 04_USER_GUIDES -> guides/ - 05_API_REFERENCE -> api/ - 06_PHYSICS -> physics/ - 07_DEVELOPMENT -> development/ - 08_ARCHIVE -> archive/ - 09_DIAGRAMS -> diagrams/ - Replace tagline 'Talk, don't click' with 'LLM-driven optimization framework' in 9 files - Create comprehensive docs/GETTING_STARTED.md: - Prerequisites and quick setup - Project structure overview - First study tutorial (Claude or manual) - Dashboard usage guide - Neural acceleration introduction - Rewrite docs/00_INDEX.md with correct paths and modern structure - Archive obsolete files: - 01_PROTOCOLS.md -> archive/historical/01_PROTOCOLS_legacy.md - 03_GETTING_STARTED.md -> archive/historical/ - ATOMIZER_PODCAST_BRIEFING.md -> archive/marketing/ - Update timestamps to 2026-01-20 across all key files - Update .gitignore to exclude docs/generated/ - Version bump: ATOMIZER_CONTEXT v1.8 -> v2.0
114 lines
3.9 KiB
Markdown
114 lines
3.9 KiB
Markdown
# Validator Pruning Investigation - November 20, 2025
|
|
|
|
## DEPRECATED - This document is retained for historical reference only.
|
|
|
|
**Status**: Investigation completed. Aspect ratio validation approach was abandoned.
|
|
|
|
---
|
|
|
|
## Original Problem
|
|
|
|
The v2.1 and v2.2 tests showed 18-20% pruning rate. Investigation revealed two separate issues:
|
|
|
|
### Issue 1: Validator Not Enforcing Rules (FIXED, then REMOVED)
|
|
|
|
The `_validate_circular_plate_aspect_ratio()` method initially returned only **warnings**, not **rejections**.
|
|
|
|
**Fix Applied**: Changed to return hard rejections for aspect ratio violations.
|
|
|
|
**Result**: All pruned trials in v2.2 still had VALID aspect ratios (5.0-50.0 range).
|
|
|
|
**Conclusion**: Aspect ratio violations were NOT the cause of pruning.
|
|
|
|
### Issue 2: pyNastran False Positives (ROOT CAUSE)
|
|
|
|
All pruned trials failed due to pyNastran FATAL flag sensitivity:
|
|
- ✅ Nastran simulations succeeded (F06 files have no errors)
|
|
- ⚠️ FATAL flag in OP2 header (benign warning)
|
|
- ❌ pyNastran throws exception when reading OP2
|
|
- ❌ Valid trials incorrectly marked as failed
|
|
|
|
**Evidence**: All 9 pruned trials in v2.2 had:
|
|
- `is_pynastran_fatal_flag: true`
|
|
- `f06_has_fatal_errors: false`
|
|
- Valid aspect ratios within bounds
|
|
|
|
---
|
|
|
|
## Final Solution (Post-v2.3)
|
|
|
|
### Aspect Ratio Validation REMOVED
|
|
|
|
After deploying v2.3 with aspect ratio validation, user feedback revealed:
|
|
|
|
**User Requirement**: "I never asked for this check, where does that come from?"
|
|
|
|
**Issue**: Arbitrary aspect ratio limits (5.0-50.0) without:
|
|
- User approval
|
|
- Physical justification for circular plate modal analysis
|
|
- Visibility in optimization_config.json
|
|
|
|
**Fix Applied**:
|
|
- Removed ALL aspect ratio validation from circular_plate model type
|
|
- Validator now returns empty rules `{}`
|
|
- Relies solely on Optuna parameter bounds (50-150mm diameter, 2-10mm thickness)
|
|
|
|
**User Requirements Established**:
|
|
1. **No arbitrary checks** - validation rules must be proposed, not automatic
|
|
2. **Configurable validation** - rules should be visible in optimization_config.json
|
|
3. **Parameter bounds suffice** - ranges already define feasibility
|
|
4. **Physical justification required** - any constraint needs clear reasoning
|
|
|
|
### Real Solution: Robust OP2 Extraction
|
|
|
|
**Module**: [optimization_engine/op2_extractor.py](../optimization_engine/op2_extractor.py)
|
|
|
|
Multi-strategy extraction that handles pyNastran issues:
|
|
1. Standard OP2 read
|
|
2. Lenient read (debug=False, skip benign flags)
|
|
3. F06 fallback parsing
|
|
|
|
See [PRUNING_DIAGNOSTICS.md](PRUNING_DIAGNOSTICS.md) for details.
|
|
|
|
---
|
|
|
|
## Lessons Learned
|
|
|
|
1. **Validator is for simulation failures, not arbitrary physics assumptions**
|
|
- Parameter bounds already define feasible ranges
|
|
- Don't add validation rules without user approval
|
|
|
|
2. **18% pruning was pyNastran false positives, not validation issues**
|
|
- All pruned trials had valid parameters
|
|
- Robust extraction eliminates these false positives
|
|
|
|
3. **Transparency is critical**
|
|
- Validation rules must be visible in optimization_config.json
|
|
- Arbitrary constraints confuse users and reject valid designs
|
|
|
|
---
|
|
|
|
## Current State
|
|
|
|
**File**: [simulation_validator.py](../optimization_engine/simulation_validator.py:41-45)
|
|
|
|
```python
|
|
if model_type == 'circular_plate':
|
|
# NOTE: Only use parameter bounds for validation
|
|
# No arbitrary aspect ratio checks - let Optuna explore the full parameter space
|
|
# Modal analysis is robust and doesn't need strict aspect ratio limits
|
|
return {}
|
|
```
|
|
|
|
**Impact**: Clean separation of concerns
|
|
- **Parameter bounds** = Feasibility (user-defined ranges)
|
|
- **Validator** = Genuine simulation failures (e.g., mesh errors, solver crashes)
|
|
|
|
---
|
|
|
|
## References
|
|
|
|
- [SESSION_SUMMARY_NOV20.md](SESSION_SUMMARY_NOV20.md) - Complete session documentation
|
|
- [PRUNING_DIAGNOSTICS.md](PRUNING_DIAGNOSTICS.md) - Robust extraction solution
|
|
- [optimization_engine/simulation_validator.py](../optimization_engine/simulation_validator.py) - Current validator implementation
|