Atomizer/docs/07_DEVELOPMENT/LOGGING_MIGRATION_GUIDE.md

Logging Migration Guide

How to migrate existing studies to use the new structured logging system

Overview

The new optimization_engine.logger module provides production-ready logging with:

• Color-coded console output
• Automatic file logging with rotation
• Structured trial logging for dashboard integration
• Zero external dependencies

Migration Steps

Step 1: Import the Logger

Before:

import sys
import json
from pathlib import Path

After:

import sys
import json
from pathlib import Path
from optimization_engine.logger import get_logger

Step 2: Initialize Logger in main()

Before:

def main():
    study_dir = Path(__file__).parent
    results_dir = study_dir / "2_results"
    results_dir.mkdir(exist_ok=True)

    print("=" * 80)
    print("MY OPTIMIZATION STUDY")
    print("=" * 80)

After:

def main():
    study_dir = Path(__file__).parent
    results_dir = study_dir / "2_results"
    results_dir.mkdir(exist_ok=True)

    # Initialize logger with file logging
    logger = get_logger(
        "my_study_name",
        study_dir=results_dir
    )

    logger.info("=" * 80)
    logger.info("MY OPTIMIZATION STUDY")
    logger.info("=" * 80)

Step 3: Replace print() with logger calls

Basic Replacements:

# Before
print("Starting optimization...")
print(f"[ERROR] Simulation failed")
print(f"[WARNING] Constraint violated")

# After
logger.info("Starting optimization...")
logger.error("Simulation failed")
logger.warning("Constraint violated")

Step 4: Use Structured Trial Logging

Trial Start - Before:

print(f"\n{'='*60}")
print(f"Trial #{trial.number}")
print(f"{'='*60}")
print(f"Design Variables:")
for name, value in design_vars.items():
    print(f"  {name}: {value:.3f}")

Trial Start - After:

logger.trial_start(trial.number, design_vars)

Trial Complete - Before:

print(f"\nTrial #{trial.number} COMPLETE")
print("Objectives:")
for name, value in objectives.items():
    print(f"  {name}: {value:.4f}")
print("Constraints:")
for name, value in constraints.items():
    print(f"  {name}: {value:.4f}")
print("[OK] Feasible" if feasible else "[WARNING] Infeasible")

Trial Complete - After:

logger.trial_complete(
    trial.number,
    objectives=objectives,
    constraints=constraints,
    feasible=feasible
)

Trial Failed - Before:

print(f"\n[ERROR] Trial #{trial.number} FAILED")
print(f"Error: {error_message}")
import traceback
traceback.print_exc()

Trial Failed - After:

logger.trial_failed(trial.number, error_message)
logger.error("Full traceback:", exc_info=True)

Step 5: Use Study Lifecycle Methods

Study Start - Before:

print("=" * 80)
print(f"OPTIMIZATION STUDY: {study_name}")
print("=" * 80)
print(f"Trials: {n_trials}")
print(f"Sampler: {sampler}")
print("=" * 80)

Study Start - After:

logger.study_start(study_name, n_trials=n_trials, sampler=sampler)

Study Complete - Before:

print("=" * 80)
print(f"STUDY COMPLETE: {study_name}")
print("=" * 80)
print(f"Total trials: {n_trials}")
print(f"Successful: {n_successful}")
print(f"Failed/Pruned: {n_trials - n_successful}")
print("=" * 80)

Study Complete - After:

logger.study_complete(study_name, n_trials=n_trials, n_successful=n_successful)

Complete Example

Before (Old Style)

import sys
from pathlib import Path
import optuna

def main():
    study_dir = Path(__file__).parent
    results_dir = study_dir / "2_results"
    results_dir.mkdir(exist_ok=True)

    print("=" * 80)
    print("MY OPTIMIZATION STUDY")
    print("=" * 80)

    def objective(trial):
        x = trial.suggest_float("x", -10, 10)

        print(f"\nTrial #{trial.number}")
        print(f"x = {x:.4f}")

        try:
            result = x ** 2
            print(f"Result: {result:.4f}")
            return result
        except Exception as e:
            print(f"[ERROR] Trial failed: {e}")
            raise

    study = optuna.create_study()
    study.optimize(objective, n_trials=10)

    print("\nOptimization complete!")
    print(f"Best value: {study.best_value:.4f}")

if __name__ == "__main__":
    main()

After (New Style with Logger)

import sys
from pathlib import Path
import optuna
from optimization_engine.logger import get_logger

def main():
    study_dir = Path(__file__).parent
    results_dir = study_dir / "2_results"
    results_dir.mkdir(exist_ok=True)

    # Initialize logger with file logging
    logger = get_logger("my_study", study_dir=results_dir)

    logger.study_start("my_study", n_trials=10, sampler="TPESampler")

    def objective(trial):
        x = trial.suggest_float("x", -10, 10)

        logger.trial_start(trial.number, {"x": x})

        try:
            result = x ** 2
            logger.trial_complete(
                trial.number,
                objectives={"f(x)": result},
                feasible=True
            )
            return result
        except Exception as e:
            logger.trial_failed(trial.number, str(e))
            logger.error("Full traceback:", exc_info=True)
            raise

    study = optuna.create_study()
    study.optimize(objective, n_trials=10)

    logger.study_complete("my_study", n_trials=10, n_successful=len(study.trials))
    logger.info(f"Best value: {study.best_value:.4f}")

if __name__ == "__main__":
    main()

Benefits

After migration, you'll get:

1. Color-coded console output - Green for INFO, Yellow for WARNING, Red for ERROR
2. Automatic file logging - All output saved to 2_results/optimization.log
3. Log rotation - Automatic rotation at 50MB with 3 backups
4. Structured format - Timestamps and module names in file logs
5. Dashboard integration - Trial logs in structured format for future parsing

Log File Location

After migration, logs will be automatically saved to:

studies/your_study/
└── 2_results/
    ├── optimization.log        # Current log file
    ├── optimization.log.1      # Backup 1 (most recent)
    ├── optimization.log.2      # Backup 2
    └── optimization.log.3      # Backup 3 (oldest)

Testing Your Migration

Run your migrated study with a single trial:

cd studies/your_study
python run_optimization.py --trials 1

Check:

• ✅ Console output is color-coded
• ✅ File created at 2_results/optimization.log
• ✅ Trial start/complete messages format correctly
• ✅ No errors about missing imports

Common Patterns

Error Handling with Context

Before:

try:
    result = run_simulation()
except Exception as e:
    print(f"[ERROR] Simulation failed: {e}")
    import traceback
    traceback.print_exc()
    raise

After:

try:
    result = run_simulation()
except Exception as e:
    logger.error(f"Simulation failed: {e}", exc_info=True)
    raise

Conditional Logging

# Use log levels appropriately
logger.debug("Detailed debugging info")      # Only when debugging
logger.info("Starting optimization...")       # General progress
logger.warning("Design var out of bounds")    # Potential issues
logger.error("Simulation failed")              # Actual errors

Progress Messages

# Before
print(f"Processing trial {i}/{total}...")

# After
logger.info(f"Processing trial {i}/{total}...")

Next Steps

After migrating your study:

1. Test with a few trials
2. Check the log file in 2_results/optimization.log
3. Verify color output in console
4. Update study documentation if needed

Questions?

See:

• Phase 1.3 Implementation Plan
• optimization_engine/logger.py - Full API documentation
• drone_gimbal_arm_optimization - Reference implementation