feat: Add configuration validation system for MVP stability (Phase 1.2)
Implements JSON Schema validation for optimization configurations to ensure
consistency across all studies and prevent configuration errors.
Added:
- optimization_engine/schemas/optimization_config_schema.json
- Comprehensive schema for Protocol 10 & 11 configurations
- Validates objectives, constraints, design variables, simulation settings
- Enforces standard field names (goal, bounds, parameter, threshold)
- optimization_engine/config_manager.py
- ConfigManager class with schema validation
- CLI tool: python config_manager.py <config.json>
- Type-safe accessor methods for config elements
- Custom validations: bounds check, multi-objective consistency, location check
- optimization_engine/schemas/README.md
- Complete documentation of standard configuration format
- Validation examples and common error fixes
- Migration guidance for legacy configs
- docs/07_DEVELOPMENT/Phase_1_2_Implementation_Plan.md
- Detailed implementation plan for remaining Phase 1.2 tasks
- Migration tool design, integration guide, testing plan
Testing:
- Validated drone_gimbal_arm_optimization config successfully
- ConfigManager works with drone_gimbal format (new standard)
- Identifies legacy format issues in bracket studies
Standards Established:
- Configuration location: studies/{name}/1_setup/
- Objective direction: "goal" not "type"
- Design var bounds: "bounds": [min, max] not "min"/"max"
- Design var name: "parameter" not "name"
- Constraint threshold: "threshold" not "value"
Next Steps (Phase 1.2.1+):
- Config migration tool for legacy studies
- Integration with run_optimization.py
- Update create-study Claude skill with schema reference
- Migrate bracket studies to new format
Relates to: Phase 1.2 MVP Development Plan
🤖 Generated with Claude Code
Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
334
docs/07_DEVELOPMENT/Phase_1_2_Implementation_Plan.md
Normal file
334
docs/07_DEVELOPMENT/Phase_1_2_Implementation_Plan.md
Normal file
@@ -0,0 +1,334 @@
|
||||
# Phase 1.2: Configuration Management Overhaul - Implementation Plan
|
||||
|
||||
**Status**: In Progress
|
||||
**Started**: January 2025
|
||||
**Target Completion**: 2 days
|
||||
|
||||
---
|
||||
|
||||
## ✅ Completed (January 24, 2025)
|
||||
|
||||
### 1. Configuration Inventory
|
||||
- Found 4 `optimization_config.json` files
|
||||
- Found 5 `workflow_config.json` files
|
||||
- Analyzed bracket_V3 (old format) vs drone_gimbal (new format)
|
||||
|
||||
### 2. Schema Analysis
|
||||
Documented critical inconsistencies:
|
||||
- Objectives: `"goal"` (new) vs `"type"` (old)
|
||||
- Design vars: `"parameter"` + `"bounds": [min, max]` (new) vs `"name"` + `"min"/"max"` (old)
|
||||
- Constraints: `"threshold"` (new) vs `"value"` (old)
|
||||
- Location: `1_setup/` (correct) vs root directory (incorrect)
|
||||
|
||||
### 3. JSON Schema Design
|
||||
Created [`optimization_engine/schemas/optimization_config_schema.json`](../../optimization_engine/schemas/optimization_config_schema.json):
|
||||
- Based on drone_gimbal format (cleaner, matches create-study skill)
|
||||
- Validates all required fields
|
||||
- Supports Protocol 10 (single-objective) and Protocol 11 (multi-objective)
|
||||
- Includes extraction spec validation
|
||||
|
||||
---
|
||||
|
||||
## 🔨 Remaining Implementation Tasks
|
||||
|
||||
### Task 1: Implement ConfigManager Class
|
||||
**Priority**: HIGH
|
||||
**File**: `optimization_engine/config_manager.py`
|
||||
|
||||
```python
|
||||
"""Configuration validation and management for Atomizer studies."""
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Dict, List, Any, Optional
|
||||
import jsonschema
|
||||
|
||||
class ConfigValidationError(Exception):
|
||||
"""Raised when configuration validation fails."""
|
||||
pass
|
||||
|
||||
class ConfigManager:
|
||||
"""Manages and validates optimization configuration files."""
|
||||
|
||||
def __init__(self, config_path: Path):
|
||||
"""
|
||||
Initialize ConfigManager with path to optimization_config.json.
|
||||
|
||||
Args:
|
||||
config_path: Path to optimization_config.json file
|
||||
"""
|
||||
self.config_path = Path(config_path)
|
||||
self.schema_path = Path(__file__).parent / "schemas" / "optimization_config_schema.json"
|
||||
self.config: Optional[Dict[str, Any]] = None
|
||||
self.validation_errors: List[str] = []
|
||||
|
||||
def load_schema(self) -> Dict[str, Any]:
|
||||
"""Load JSON schema for validation."""
|
||||
with open(self.schema_path, 'r') as f:
|
||||
return json.load(f)
|
||||
|
||||
def load_config(self) -> Dict[str, Any]:
|
||||
"""Load configuration file."""
|
||||
if not self.config_path.exists():
|
||||
raise FileNotFoundError(f"Config file not found: {self.config_path}")
|
||||
|
||||
with open(self.config_path, 'r') as f:
|
||||
self.config = json.load(f)
|
||||
return self.config
|
||||
|
||||
def validate(self) -> bool:
|
||||
"""
|
||||
Validate configuration against schema.
|
||||
|
||||
Returns:
|
||||
True if valid, False otherwise
|
||||
"""
|
||||
if self.config is None:
|
||||
self.load_config()
|
||||
|
||||
schema = self.load_schema()
|
||||
self.validation_errors = []
|
||||
|
||||
try:
|
||||
jsonschema.validate(instance=self.config, schema=schema)
|
||||
# Additional custom validations
|
||||
self._validate_design_variable_bounds()
|
||||
self._validate_multi_objective_consistency()
|
||||
self._validate_file_locations()
|
||||
return True
|
||||
except jsonschema.ValidationError as e:
|
||||
self.validation_errors.append(str(e))
|
||||
return False
|
||||
|
||||
def _validate_design_variable_bounds(self):
|
||||
"""Ensure bounds are valid (min < max)."""
|
||||
for dv in self.config.get("design_variables", []):
|
||||
bounds = dv.get("bounds", [])
|
||||
if len(bounds) == 2 and bounds[0] >= bounds[1]:
|
||||
self.validation_errors.append(
|
||||
f"Design variable '{dv['parameter']}': min ({bounds[0]}) must be < max ({bounds[1]})"
|
||||
)
|
||||
|
||||
def _validate_multi_objective_consistency(self):
|
||||
"""Validate multi-objective settings consistency."""
|
||||
n_objectives = len(self.config.get("objectives", []))
|
||||
protocol = self.config.get("optimization_settings", {}).get("protocol")
|
||||
sampler = self.config.get("optimization_settings", {}).get("sampler")
|
||||
|
||||
if n_objectives > 1:
|
||||
# Multi-objective must use protocol_11 and NSGA-II
|
||||
if protocol != "protocol_11_multi_objective":
|
||||
self.validation_errors.append(
|
||||
f"Multi-objective optimization ({n_objectives} objectives) requires protocol_11_multi_objective"
|
||||
)
|
||||
if sampler != "NSGAIISampler":
|
||||
self.validation_errors.append(
|
||||
f"Multi-objective optimization requires NSGAIISampler (got {sampler})"
|
||||
)
|
||||
|
||||
def _validate_file_locations(self):
|
||||
"""Check if config is in correct location (1_setup/)."""
|
||||
if "1_setup" not in str(self.config_path.parent):
|
||||
self.validation_errors.append(
|
||||
f"Config should be in '1_setup/' directory, found in {self.config_path.parent}"
|
||||
)
|
||||
|
||||
def get_validation_report(self) -> str:
|
||||
"""Get human-readable validation report."""
|
||||
if not self.validation_errors:
|
||||
return "✓ Configuration is valid"
|
||||
|
||||
report = "✗ Configuration validation failed:\n"
|
||||
for i, error in enumerate(self.validation_errors, 1):
|
||||
report += f" {i}. {error}\n"
|
||||
return report
|
||||
|
||||
# Type-safe accessor methods
|
||||
|
||||
def get_design_variables(self) -> List[Dict[str, Any]]:
|
||||
"""Get design variables with validated structure."""
|
||||
if self.config is None:
|
||||
self.load_config()
|
||||
return self.config.get("design_variables", [])
|
||||
|
||||
def get_objectives(self) -> List[Dict[str, Any]]:
|
||||
"""Get objectives with validated structure."""
|
||||
if self.config is None:
|
||||
self.load_config()
|
||||
return self.config.get("objectives", [])
|
||||
|
||||
def get_constraints(self) -> List[Dict[str, Any]]:
|
||||
"""Get constraints with validated structure."""
|
||||
if self.config is None:
|
||||
self.load_config()
|
||||
return self.config.get("constraints", [])
|
||||
|
||||
def get_simulation_settings(self) -> Dict[str, Any]:
|
||||
"""Get simulation settings."""
|
||||
if self.config is None:
|
||||
self.load_config()
|
||||
return self.config.get("simulation", {})
|
||||
|
||||
|
||||
# CLI tool for validation
|
||||
if __name__ == "__main__":
|
||||
import sys
|
||||
|
||||
if len(sys.argv) < 2:
|
||||
print("Usage: python config_manager.py <path_to_optimization_config.json>")
|
||||
sys.exit(1)
|
||||
|
||||
config_path = Path(sys.argv[1])
|
||||
manager = ConfigManager(config_path)
|
||||
|
||||
try:
|
||||
manager.load_config()
|
||||
is_valid = manager.validate()
|
||||
print(manager.get_validation_report())
|
||||
sys.exit(0 if is_valid else 1)
|
||||
except Exception as e:
|
||||
print(f"Error: {e}")
|
||||
sys.exit(1)
|
||||
```
|
||||
|
||||
**Dependencies**: Add to requirements.txt:
|
||||
```
|
||||
jsonschema>=4.17.0
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 2: Create Configuration Migration Tool
|
||||
**Priority**: MEDIUM
|
||||
**File**: `optimization_engine/config_migrator.py`
|
||||
|
||||
Tool to automatically migrate old-format configs to new format:
|
||||
- Convert `"type"` → `"goal"` in objectives
|
||||
- Convert `"min"/"max"` → `"bounds": [min, max]` in design variables
|
||||
- Convert `"name"` → `"parameter"` in design variables
|
||||
- Convert `"value"` → `"threshold"` in constraints
|
||||
- Move config files to `1_setup/` if in wrong location
|
||||
|
||||
---
|
||||
|
||||
### Task 3: Integration with run_optimization.py
|
||||
**Priority**: HIGH
|
||||
|
||||
Add validation to optimization runners:
|
||||
```python
|
||||
# At start of run_optimization.py
|
||||
from optimization_engine.config_manager import ConfigManager
|
||||
|
||||
# Load and validate config
|
||||
config_manager = ConfigManager(Path(__file__).parent / "1_setup" / "optimization_config.json")
|
||||
config_manager.load_config()
|
||||
|
||||
if not config_manager.validate():
|
||||
print(config_manager.get_validation_report())
|
||||
sys.exit(1)
|
||||
|
||||
print("✓ Configuration validated successfully")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 4: Update create-study Claude Skill
|
||||
**Priority**: HIGH
|
||||
**File**: `.claude/skills/create-study.md`
|
||||
|
||||
Update skill to reference the JSON schema:
|
||||
- Add link to schema documentation
|
||||
- Emphasize validation after generation
|
||||
- Include validation command in "Next Steps"
|
||||
|
||||
---
|
||||
|
||||
### Task 5: Create Configuration Documentation
|
||||
**Priority**: HIGH
|
||||
**File**: `docs/CONFIGURATION_GUIDE.md`
|
||||
|
||||
Comprehensive documentation covering:
|
||||
1. Standard configuration format (with drone_gimbal example)
|
||||
2. Field-by-field descriptions
|
||||
3. Validation rules and how to run validation
|
||||
4. Common validation errors and fixes
|
||||
5. Migration guide for old configs
|
||||
6. Protocol selection (10 vs 11)
|
||||
7. Extractor mapping table
|
||||
|
||||
---
|
||||
|
||||
### Task 6: Validate All Existing Studies
|
||||
**Priority**: MEDIUM
|
||||
|
||||
Run validation on all existing studies:
|
||||
```bash
|
||||
# Test validation tool
|
||||
python optimization_engine/config_manager.py studies/drone_gimbal_arm_optimization/1_setup/optimization_config.json
|
||||
python optimization_engine/config_manager.py studies/bracket_stiffness_optimization_V3/optimization_config.json
|
||||
|
||||
# Expected: drone passes, bracket_V3 fails with specific errors
|
||||
```
|
||||
|
||||
Create migration plan for failing configs.
|
||||
|
||||
---
|
||||
|
||||
### Task 7: Migrate Legacy Configs
|
||||
**Priority**: LOW (can defer to Phase 1.3)
|
||||
|
||||
Migrate all legacy configs to new format:
|
||||
- bracket_stiffness_optimization_V3
|
||||
- bracket_stiffness_optimization_V2
|
||||
- bracket_stiffness_optimization
|
||||
|
||||
Keep old versions in `archive/` for reference.
|
||||
|
||||
---
|
||||
|
||||
## Success Criteria
|
||||
|
||||
Phase 1.2 is complete when:
|
||||
- [x] JSON schema created and comprehensive
|
||||
- [ ] ConfigManager class implemented with all validation methods
|
||||
- [ ] Validation integrated into at least 1 study (drone_gimbal)
|
||||
- [ ] Configuration documentation written
|
||||
- [ ] create-study skill updated with schema reference
|
||||
- [ ] Migration tool created (basic version)
|
||||
- [ ] All tests pass on drone_gimbal study
|
||||
- [ ] Phase 1.2 changes committed with clear message
|
||||
|
||||
---
|
||||
|
||||
## Testing Plan
|
||||
|
||||
1. **Schema Validation Test**:
|
||||
- Valid config passes ✓
|
||||
- Invalid configs fail with clear errors ✓
|
||||
|
||||
2. **ConfigManager Test**:
|
||||
- Load valid config
|
||||
- Validate and get clean report
|
||||
- Load invalid config
|
||||
- Validate and get error details
|
||||
|
||||
3. **Integration Test**:
|
||||
- Run drone_gimbal study with validation enabled
|
||||
- Verify no performance impact
|
||||
- Check validation messages appear correctly
|
||||
|
||||
4. **Migration Test**:
|
||||
- Migrate bracket_V3 config
|
||||
- Validate migrated config
|
||||
- Compare before/after
|
||||
|
||||
---
|
||||
|
||||
## Next Phase Preview
|
||||
|
||||
**Phase 1.3: Error Handling & Logging** will build on this by:
|
||||
- Adding structured logging with configuration context
|
||||
- Error recovery using validated configurations
|
||||
- Checkpoint system that validates config before saving
|
||||
|
||||
The clean configuration management from Phase 1.2 enables reliable error handling in Phase 1.3.
|
||||
Reference in New Issue
Block a user