Atomizer/docs/development/LOGGING_MIGRATION_GUIDE.md

# Logging Migration Guide

**How to migrate existing studies to use the new structured logging system**

## Overview

The new `optimization_engine.logger` module provides production-ready logging with:
- Color-coded console output
- Automatic file logging with rotation
- Structured trial logging for dashboard integration
- Zero external dependencies

## Migration Steps

### Step 1: Import the Logger

**Before:**
```python
import sys
import json
from pathlib import Path
```

**After:**
```python
import sys
import json
from pathlib import Path
from optimization_engine.logger import get_logger
```

### Step 2: Initialize Logger in main()

**Before:**
```python
def main():
    study_dir = Path(__file__).parent
    results_dir = study_dir / "2_results"
    results_dir.mkdir(exist_ok=True)

    print("=" * 80)
    print("MY OPTIMIZATION STUDY")
    print("=" * 80)
```

**After:**
```python
def main():
    study_dir = Path(__file__).parent
    results_dir = study_dir / "2_results"
    results_dir.mkdir(exist_ok=True)

    # Initialize logger with file logging
    logger = get_logger(
        "my_study_name",
        study_dir=results_dir
    )

    logger.info("=" * 80)
    logger.info("MY OPTIMIZATION STUDY")
    logger.info("=" * 80)
```

### Step 3: Replace print() with logger calls

**Basic Replacements:**
```python
# Before
print("Starting optimization...")
print(f"[ERROR] Simulation failed")
print(f"[WARNING] Constraint violated")

# After
logger.info("Starting optimization...")
logger.error("Simulation failed")
logger.warning("Constraint violated")
```

### Step 4: Use Structured Trial Logging

**Trial Start - Before:**
```python
print(f"\n{'='*60}")
print(f"Trial #{trial.number}")
print(f"{'='*60}")
print(f"Design Variables:")
for name, value in design_vars.items():
    print(f"  {name}: {value:.3f}")
```

**Trial Start - After:**
```python
logger.trial_start(trial.number, design_vars)
```

**Trial Complete - Before:**
```python
print(f"\nTrial #{trial.number} COMPLETE")
print("Objectives:")
for name, value in objectives.items():
    print(f"  {name}: {value:.4f}")
print("Constraints:")
for name, value in constraints.items():
    print(f"  {name}: {value:.4f}")
print("[OK] Feasible" if feasible else "[WARNING] Infeasible")
```

**Trial Complete - After:**
```python
logger.trial_complete(
    trial.number,
    objectives=objectives,
    constraints=constraints,
    feasible=feasible
)
```

**Trial Failed - Before:**
```python
print(f"\n[ERROR] Trial #{trial.number} FAILED")
print(f"Error: {error_message}")
import traceback
traceback.print_exc()
```

**Trial Failed - After:**
```python
logger.trial_failed(trial.number, error_message)
logger.error("Full traceback:", exc_info=True)
```

### Step 5: Use Study Lifecycle Methods

**Study Start - Before:**
```python
print("=" * 80)
print(f"OPTIMIZATION STUDY: {study_name}")
print("=" * 80)
print(f"Trials: {n_trials}")
print(f"Sampler: {sampler}")
print("=" * 80)
```

**Study Start - After:**
```python
logger.study_start(study_name, n_trials=n_trials, sampler=sampler)
```

**Study Complete - Before:**
```python
print("=" * 80)
print(f"STUDY COMPLETE: {study_name}")
print("=" * 80)
print(f"Total trials: {n_trials}")
print(f"Successful: {n_successful}")
print(f"Failed/Pruned: {n_trials - n_successful}")
print("=" * 80)
```

**Study Complete - After:**
```python
logger.study_complete(study_name, n_trials=n_trials, n_successful=n_successful)
```

## Complete Example

### Before (Old Style)

```python
import sys
from pathlib import Path
import optuna

def main():
    study_dir = Path(__file__).parent
    results_dir = study_dir / "2_results"
    results_dir.mkdir(exist_ok=True)

    print("=" * 80)
    print("MY OPTIMIZATION STUDY")
    print("=" * 80)

    def objective(trial):
        x = trial.suggest_float("x", -10, 10)

        print(f"\nTrial #{trial.number}")
        print(f"x = {x:.4f}")

        try:
            result = x ** 2
            print(f"Result: {result:.4f}")
            return result
        except Exception as e:
            print(f"[ERROR] Trial failed: {e}")
            raise

    study = optuna.create_study()
    study.optimize(objective, n_trials=10)

    print("\nOptimization complete!")
    print(f"Best value: {study.best_value:.4f}")

if __name__ == "__main__":
    main()
```

### After (New Style with Logger)

```python
import sys
from pathlib import Path
import optuna
from optimization_engine.logger import get_logger

def main():
    study_dir = Path(__file__).parent
    results_dir = study_dir / "2_results"
    results_dir.mkdir(exist_ok=True)

    # Initialize logger with file logging
    logger = get_logger("my_study", study_dir=results_dir)

    logger.study_start("my_study", n_trials=10, sampler="TPESampler")

    def objective(trial):
        x = trial.suggest_float("x", -10, 10)

        logger.trial_start(trial.number, {"x": x})

        try:
            result = x ** 2
            logger.trial_complete(
                trial.number,
                objectives={"f(x)": result},
                feasible=True
            )
            return result
        except Exception as e:
            logger.trial_failed(trial.number, str(e))
            logger.error("Full traceback:", exc_info=True)
            raise

    study = optuna.create_study()
    study.optimize(objective, n_trials=10)

    logger.study_complete("my_study", n_trials=10, n_successful=len(study.trials))
    logger.info(f"Best value: {study.best_value:.4f}")

if __name__ == "__main__":
    main()
```

## Benefits

After migration, you'll get:

1. **Color-coded console output** - Green for INFO, Yellow for WARNING, Red for ERROR
2. **Automatic file logging** - All output saved to `2_results/optimization.log`
3. **Log rotation** - Automatic rotation at 50MB with 3 backups
4. **Structured format** - Timestamps and module names in file logs
5. **Dashboard integration** - Trial logs in structured format for future parsing

## Log File Location

After migration, logs will be automatically saved to:
```
studies/your_study/
└── 2_results/
    ├── optimization.log        # Current log file
    ├── optimization.log.1      # Backup 1 (most recent)
    ├── optimization.log.2      # Backup 2
    └── optimization.log.3      # Backup 3 (oldest)
```

## Testing Your Migration

Run your migrated study with a single trial:
```bash
cd studies/your_study
python run_optimization.py --trials 1
```

Check:
- ✅ Console output is color-coded
- ✅ File created at `2_results/optimization.log`
- ✅ Trial start/complete messages format correctly
- ✅ No errors about missing imports

## Common Patterns

### Error Handling with Context

**Before:**
```python
try:
    result = run_simulation()
except Exception as e:
    print(f"[ERROR] Simulation failed: {e}")
    import traceback
    traceback.print_exc()
    raise
```

**After:**
```python
try:
    result = run_simulation()
except Exception as e:
    logger.error(f"Simulation failed: {e}", exc_info=True)
    raise
```

### Conditional Logging

```python
# Use log levels appropriately
logger.debug("Detailed debugging info")      # Only when debugging
logger.info("Starting optimization...")       # General progress
logger.warning("Design var out of bounds")    # Potential issues
logger.error("Simulation failed")              # Actual errors
```

### Progress Messages

```python
# Before
print(f"Processing trial {i}/{total}...")

# After
logger.info(f"Processing trial {i}/{total}...")
```

## Next Steps

After migrating your study:

1. Test with a few trials
2. Check the log file in `2_results/optimization.log`
3. Verify color output in console
4. Update study documentation if needed

## Questions?

See:
- [Phase 1.3 Implementation Plan](docs/07_DEVELOPMENT/Phase_1_3_Implementation_Plan.md)
- [optimization_engine/logger.py](optimization_engine/logger.py) - Full API documentation
- [drone_gimbal_arm_optimization](studies/drone_gimbal_arm_optimization/) - Reference implementation