docs: Major documentation overhaul - restructure folders, update tagline, add Getting Started guide
- Restructure docs/ folder (remove numeric prefixes): - 04_USER_GUIDES -> guides/ - 05_API_REFERENCE -> api/ - 06_PHYSICS -> physics/ - 07_DEVELOPMENT -> development/ - 08_ARCHIVE -> archive/ - 09_DIAGRAMS -> diagrams/ - Replace tagline 'Talk, don't click' with 'LLM-driven optimization framework' in 9 files - Create comprehensive docs/GETTING_STARTED.md: - Prerequisites and quick setup - Project structure overview - First study tutorial (Claude or manual) - Dashboard usage guide - Neural acceleration introduction - Rewrite docs/00_INDEX.md with correct paths and modern structure - Archive obsolete files: - 01_PROTOCOLS.md -> archive/historical/01_PROTOCOLS_legacy.md - 03_GETTING_STARTED.md -> archive/historical/ - ATOMIZER_PODCAST_BRIEFING.md -> archive/marketing/ - Update timestamps to 2026-01-20 across all key files - Update .gitignore to exclude docs/generated/ - Version bump: ATOMIZER_CONTEXT v1.8 -> v2.0
This commit is contained in:
347
docs/development/LOGGING_MIGRATION_GUIDE.md
Normal file
347
docs/development/LOGGING_MIGRATION_GUIDE.md
Normal file
@@ -0,0 +1,347 @@
|
||||
# Logging Migration Guide
|
||||
|
||||
**How to migrate existing studies to use the new structured logging system**
|
||||
|
||||
## Overview
|
||||
|
||||
The new `optimization_engine.logger` module provides production-ready logging with:
|
||||
- Color-coded console output
|
||||
- Automatic file logging with rotation
|
||||
- Structured trial logging for dashboard integration
|
||||
- Zero external dependencies
|
||||
|
||||
## Migration Steps
|
||||
|
||||
### Step 1: Import the Logger
|
||||
|
||||
**Before:**
|
||||
```python
|
||||
import sys
|
||||
import json
|
||||
from pathlib import Path
|
||||
```
|
||||
|
||||
**After:**
|
||||
```python
|
||||
import sys
|
||||
import json
|
||||
from pathlib import Path
|
||||
from optimization_engine.logger import get_logger
|
||||
```
|
||||
|
||||
### Step 2: Initialize Logger in main()
|
||||
|
||||
**Before:**
|
||||
```python
|
||||
def main():
|
||||
study_dir = Path(__file__).parent
|
||||
results_dir = study_dir / "2_results"
|
||||
results_dir.mkdir(exist_ok=True)
|
||||
|
||||
print("=" * 80)
|
||||
print("MY OPTIMIZATION STUDY")
|
||||
print("=" * 80)
|
||||
```
|
||||
|
||||
**After:**
|
||||
```python
|
||||
def main():
|
||||
study_dir = Path(__file__).parent
|
||||
results_dir = study_dir / "2_results"
|
||||
results_dir.mkdir(exist_ok=True)
|
||||
|
||||
# Initialize logger with file logging
|
||||
logger = get_logger(
|
||||
"my_study_name",
|
||||
study_dir=results_dir
|
||||
)
|
||||
|
||||
logger.info("=" * 80)
|
||||
logger.info("MY OPTIMIZATION STUDY")
|
||||
logger.info("=" * 80)
|
||||
```
|
||||
|
||||
### Step 3: Replace print() with logger calls
|
||||
|
||||
**Basic Replacements:**
|
||||
```python
|
||||
# Before
|
||||
print("Starting optimization...")
|
||||
print(f"[ERROR] Simulation failed")
|
||||
print(f"[WARNING] Constraint violated")
|
||||
|
||||
# After
|
||||
logger.info("Starting optimization...")
|
||||
logger.error("Simulation failed")
|
||||
logger.warning("Constraint violated")
|
||||
```
|
||||
|
||||
### Step 4: Use Structured Trial Logging
|
||||
|
||||
**Trial Start - Before:**
|
||||
```python
|
||||
print(f"\n{'='*60}")
|
||||
print(f"Trial #{trial.number}")
|
||||
print(f"{'='*60}")
|
||||
print(f"Design Variables:")
|
||||
for name, value in design_vars.items():
|
||||
print(f" {name}: {value:.3f}")
|
||||
```
|
||||
|
||||
**Trial Start - After:**
|
||||
```python
|
||||
logger.trial_start(trial.number, design_vars)
|
||||
```
|
||||
|
||||
**Trial Complete - Before:**
|
||||
```python
|
||||
print(f"\nTrial #{trial.number} COMPLETE")
|
||||
print("Objectives:")
|
||||
for name, value in objectives.items():
|
||||
print(f" {name}: {value:.4f}")
|
||||
print("Constraints:")
|
||||
for name, value in constraints.items():
|
||||
print(f" {name}: {value:.4f}")
|
||||
print("[OK] Feasible" if feasible else "[WARNING] Infeasible")
|
||||
```
|
||||
|
||||
**Trial Complete - After:**
|
||||
```python
|
||||
logger.trial_complete(
|
||||
trial.number,
|
||||
objectives=objectives,
|
||||
constraints=constraints,
|
||||
feasible=feasible
|
||||
)
|
||||
```
|
||||
|
||||
**Trial Failed - Before:**
|
||||
```python
|
||||
print(f"\n[ERROR] Trial #{trial.number} FAILED")
|
||||
print(f"Error: {error_message}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
```
|
||||
|
||||
**Trial Failed - After:**
|
||||
```python
|
||||
logger.trial_failed(trial.number, error_message)
|
||||
logger.error("Full traceback:", exc_info=True)
|
||||
```
|
||||
|
||||
### Step 5: Use Study Lifecycle Methods
|
||||
|
||||
**Study Start - Before:**
|
||||
```python
|
||||
print("=" * 80)
|
||||
print(f"OPTIMIZATION STUDY: {study_name}")
|
||||
print("=" * 80)
|
||||
print(f"Trials: {n_trials}")
|
||||
print(f"Sampler: {sampler}")
|
||||
print("=" * 80)
|
||||
```
|
||||
|
||||
**Study Start - After:**
|
||||
```python
|
||||
logger.study_start(study_name, n_trials=n_trials, sampler=sampler)
|
||||
```
|
||||
|
||||
**Study Complete - Before:**
|
||||
```python
|
||||
print("=" * 80)
|
||||
print(f"STUDY COMPLETE: {study_name}")
|
||||
print("=" * 80)
|
||||
print(f"Total trials: {n_trials}")
|
||||
print(f"Successful: {n_successful}")
|
||||
print(f"Failed/Pruned: {n_trials - n_successful}")
|
||||
print("=" * 80)
|
||||
```
|
||||
|
||||
**Study Complete - After:**
|
||||
```python
|
||||
logger.study_complete(study_name, n_trials=n_trials, n_successful=n_successful)
|
||||
```
|
||||
|
||||
## Complete Example
|
||||
|
||||
### Before (Old Style)
|
||||
|
||||
```python
|
||||
import sys
|
||||
from pathlib import Path
|
||||
import optuna
|
||||
|
||||
def main():
|
||||
study_dir = Path(__file__).parent
|
||||
results_dir = study_dir / "2_results"
|
||||
results_dir.mkdir(exist_ok=True)
|
||||
|
||||
print("=" * 80)
|
||||
print("MY OPTIMIZATION STUDY")
|
||||
print("=" * 80)
|
||||
|
||||
def objective(trial):
|
||||
x = trial.suggest_float("x", -10, 10)
|
||||
|
||||
print(f"\nTrial #{trial.number}")
|
||||
print(f"x = {x:.4f}")
|
||||
|
||||
try:
|
||||
result = x ** 2
|
||||
print(f"Result: {result:.4f}")
|
||||
return result
|
||||
except Exception as e:
|
||||
print(f"[ERROR] Trial failed: {e}")
|
||||
raise
|
||||
|
||||
study = optuna.create_study()
|
||||
study.optimize(objective, n_trials=10)
|
||||
|
||||
print("\nOptimization complete!")
|
||||
print(f"Best value: {study.best_value:.4f}")
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
```
|
||||
|
||||
### After (New Style with Logger)
|
||||
|
||||
```python
|
||||
import sys
|
||||
from pathlib import Path
|
||||
import optuna
|
||||
from optimization_engine.logger import get_logger
|
||||
|
||||
def main():
|
||||
study_dir = Path(__file__).parent
|
||||
results_dir = study_dir / "2_results"
|
||||
results_dir.mkdir(exist_ok=True)
|
||||
|
||||
# Initialize logger with file logging
|
||||
logger = get_logger("my_study", study_dir=results_dir)
|
||||
|
||||
logger.study_start("my_study", n_trials=10, sampler="TPESampler")
|
||||
|
||||
def objective(trial):
|
||||
x = trial.suggest_float("x", -10, 10)
|
||||
|
||||
logger.trial_start(trial.number, {"x": x})
|
||||
|
||||
try:
|
||||
result = x ** 2
|
||||
logger.trial_complete(
|
||||
trial.number,
|
||||
objectives={"f(x)": result},
|
||||
feasible=True
|
||||
)
|
||||
return result
|
||||
except Exception as e:
|
||||
logger.trial_failed(trial.number, str(e))
|
||||
logger.error("Full traceback:", exc_info=True)
|
||||
raise
|
||||
|
||||
study = optuna.create_study()
|
||||
study.optimize(objective, n_trials=10)
|
||||
|
||||
logger.study_complete("my_study", n_trials=10, n_successful=len(study.trials))
|
||||
logger.info(f"Best value: {study.best_value:.4f}")
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
```
|
||||
|
||||
## Benefits
|
||||
|
||||
After migration, you'll get:
|
||||
|
||||
1. **Color-coded console output** - Green for INFO, Yellow for WARNING, Red for ERROR
|
||||
2. **Automatic file logging** - All output saved to `2_results/optimization.log`
|
||||
3. **Log rotation** - Automatic rotation at 50MB with 3 backups
|
||||
4. **Structured format** - Timestamps and module names in file logs
|
||||
5. **Dashboard integration** - Trial logs in structured format for future parsing
|
||||
|
||||
## Log File Location
|
||||
|
||||
After migration, logs will be automatically saved to:
|
||||
```
|
||||
studies/your_study/
|
||||
└── 2_results/
|
||||
├── optimization.log # Current log file
|
||||
├── optimization.log.1 # Backup 1 (most recent)
|
||||
├── optimization.log.2 # Backup 2
|
||||
└── optimization.log.3 # Backup 3 (oldest)
|
||||
```
|
||||
|
||||
## Testing Your Migration
|
||||
|
||||
Run your migrated study with a single trial:
|
||||
```bash
|
||||
cd studies/your_study
|
||||
python run_optimization.py --trials 1
|
||||
```
|
||||
|
||||
Check:
|
||||
- ✅ Console output is color-coded
|
||||
- ✅ File created at `2_results/optimization.log`
|
||||
- ✅ Trial start/complete messages format correctly
|
||||
- ✅ No errors about missing imports
|
||||
|
||||
## Common Patterns
|
||||
|
||||
### Error Handling with Context
|
||||
|
||||
**Before:**
|
||||
```python
|
||||
try:
|
||||
result = run_simulation()
|
||||
except Exception as e:
|
||||
print(f"[ERROR] Simulation failed: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
raise
|
||||
```
|
||||
|
||||
**After:**
|
||||
```python
|
||||
try:
|
||||
result = run_simulation()
|
||||
except Exception as e:
|
||||
logger.error(f"Simulation failed: {e}", exc_info=True)
|
||||
raise
|
||||
```
|
||||
|
||||
### Conditional Logging
|
||||
|
||||
```python
|
||||
# Use log levels appropriately
|
||||
logger.debug("Detailed debugging info") # Only when debugging
|
||||
logger.info("Starting optimization...") # General progress
|
||||
logger.warning("Design var out of bounds") # Potential issues
|
||||
logger.error("Simulation failed") # Actual errors
|
||||
```
|
||||
|
||||
### Progress Messages
|
||||
|
||||
```python
|
||||
# Before
|
||||
print(f"Processing trial {i}/{total}...")
|
||||
|
||||
# After
|
||||
logger.info(f"Processing trial {i}/{total}...")
|
||||
```
|
||||
|
||||
## Next Steps
|
||||
|
||||
After migrating your study:
|
||||
|
||||
1. Test with a few trials
|
||||
2. Check the log file in `2_results/optimization.log`
|
||||
3. Verify color output in console
|
||||
4. Update study documentation if needed
|
||||
|
||||
## Questions?
|
||||
|
||||
See:
|
||||
- [Phase 1.3 Implementation Plan](docs/07_DEVELOPMENT/Phase_1_3_Implementation_Plan.md)
|
||||
- [optimization_engine/logger.py](optimization_engine/logger.py) - Full API documentation
|
||||
- [drone_gimbal_arm_optimization](studies/drone_gimbal_arm_optimization/) - Reference implementation
|
||||
Reference in New Issue
Block a user