Atomizer/docs/archive/historical/CRITICAL_ISSUES_ROADMAP.md

CRITICAL ISSUES - IMMEDIATE ACTION REQUIRED

Date: 2025-11-21 Status: 🚨 BLOCKING PRODUCTION USE

Issue 1: Real-Time Tracking Files - MANDATORY EVERY ITERATION

Current State ❌

• Intelligent optimizer only writes tracking files at END of optimization
• Dashboard cannot show real-time progress
• No visibility into optimizer state during execution

Required Behavior ✅

AFTER EVERY SINGLE TRIAL:
1. Write optimizer_state.json (current strategy, confidence, phase)
2. Write strategy_history.json (append new recommendation)
3. Write landscape_snapshot.json (current analysis if available)
4. Write trial_log.json (append trial result with timestamp)

Implementation Plan

1. Create RealtimeCallback class that triggers after each trial
2. Hook into study.optimize(..., callbacks=[realtime_callback])
3. Write incremental JSON files to intelligent_optimizer/ folder
4. Files must be atomic writes (temp file + rename)

Files to Modify

• optimization_engine/intelligent_optimizer.py - Add callback system
• New file: optimization_engine/realtime_tracking.py - Callback implementation

---

Issue 2: Dashboard - Complete Overhaul Required

###Current Problems ❌

1. No Pareto front plot for multi-objective
2. No parallel coordinates for high-dimensional visualization
3. Units hardcoded/wrong - should read from optimization_config.json
4. Convergence plot backwards - X-axis should be trial number (already is, but user reports issue)
5. No objective normalization - raw values make comparison difficult
6. Missing intelligent optimizer panel - no real-time strategy display
7. Poor UX - not professional looking

Required Features ✅

A. Intelligent Optimizer Panel (NEW)

<OptimizerPanel>
  - Current Phase: "Characterization" | "Optimization" | "Refinement"
  - Current Strategy: "TPE" | "CMA-ES" | "Random" | "GP-BO"
  - Confidence: 0.95 (progress bar)
  - Trials in Phase: 15/30
  - Strategy Transitions: Timeline view
  - Landscape Type: "Smooth Unimodal" | "Rugged Multi-modal" | etc.
</OptimizerPanel>

B. Pareto Front Plot (Multi-Objective)

<ParetoPlot objectives={study.objectives}>
  - 2D scatter: objective1 vs objective2
  - Color by constraint satisfaction
  - Interactive: click to see design variables
  - Dominance regions shaded
</ParetoPlot>

C. Parallel Coordinates (Multi-Objective)

<ParallelCoordinates>
  - One axis per design variable + objectives
  - Lines colored by Pareto front membership
  - Interactive brushing to filter solutions
</ParallelCoordinates>

D. Dynamic Units & Metadata

// Read from optimization_config.json
interface StudyMetadata {
  objectives: Array<{name: string, type: 'minimize'|'maximize', unit?: string}>
  design_variables: Array<{name: string, unit?: string, min: number, max: number}>
  constraints: Array<{name: string, type: string, value: number}>
}

E. Normalized Objectives

// Option 1: Min-Max normalization (0-1 scale)
normalized = (value - min) / (max - min)

// Option 2: Z-score normalization
normalized = (value - mean) / stddev

Implementation Plan

1. Backend: Add /api/studies/{id}/metadata endpoint (read config)
2. Backend: Add /api/studies/{id}/optimizer-state endpoint (read real-time JSON)
3. Frontend: Create <OptimizerPanel> component
4. Frontend: Create <ParetoPlot> component (use Recharts)
5. Frontend: Create <ParallelCoordinates> component (use D3.js or Plotly)
6. Frontend: Refactor Dashboard.tsx with new layout

---

Issue 3: Multi-Objective Strategy Selection (FIXED ✅)

Status: Completed - Protocol 12 implemented

• Multi-objective now uses: Random (8 trials) → TPE with multivariate
• No longer stuck on random for entire optimization

---

Issue 4: Missing Tracking Files in V2 Study

Root Cause

V2 study ran with OLD code (before Protocol 12). All 30 trials used random strategy.

Solution

Re-run V2 study with fixed optimizer:

cd studies/bracket_stiffness_optimization_V2
# Clear old results
del /Q 2_results\study.db
rd /S /Q 2_results\intelligent_optimizer
# Run with new code
python run_optimization.py --trials 50

---

Priority Order

P0 - CRITICAL (Do Immediately)

1. ✅ Fix multi-objective strategy selector (DONE - Protocol 12)
2. 🚧 Implement per-trial tracking callback
3. 🚧 Add intelligent optimizer panel to dashboard
4. 🚧 Add Pareto front plot

P1 - HIGH (Do Today)

5. Add parallel coordinates plot
6. Implement dynamic units (read from config)
7. Add objective normalization toggle

P2 - MEDIUM (Do This Week)

8. Improve dashboard UX/layout
9. Add hypervolume indicator for multi-objective
10. Create optimization report generator

---

Testing Protocol

After implementing each fix:

1. Per-Trial Tracking Test

   # Run optimization and check files appear immediately
   python run_optimization.py --trials 10
   # Verify: intelligent_optimizer/*.json files update EVERY trial
   

2. Dashboard Test

   # Start backend + frontend
   # Navigate to http://localhost:3001
   # Verify: All panels update in real-time
   # Verify: Pareto front appears for multi-objective
   # Verify: Units match optimization_config.json
   

3. Multi-Objective Test

   # Re-run bracket_stiffness_optimization_V2
   # Verify: Strategy switches from random → TPE after 8 trials
   # Verify: Tracking files generated every trial
   # Verify: Pareto front has 10+ solutions
   

---

Code Architecture

Realtime Tracking System

intelligent_optimizer/
├── optimizer_state.json          # Updated every trial
├── strategy_history.json         # Append-only log
├── landscape_snapshots.json      # Updated when landscape analyzed
├── trial_log.json                # Append-only with timestamps
├── confidence_history.json       # Confidence over time
└── strategy_transitions.json     # When/why strategy changed

Dashboard Data Flow

Trial Complete
  ↓
Optuna Callback
  ↓
Write JSON Files (atomic)
  ↓
Backend API detects file change
  ↓
WebSocket broadcast to frontend
  ↓
Dashboard components update

---

Estimated Effort

• Per-Trial Tracking: 2-3 hours
• Dashboard Overhaul: 6-8 hours
  • Optimizer Panel: 1 hour
  • Pareto Plot: 2 hours
  • Parallel Coordinates: 2 hours
  • Dynamic Units: 1 hour
  • Layout/UX: 2 hours

Total: 8-11 hours for production-ready system

---

Success Criteria

✅ After implementation:

1. User can see optimizer strategy change in real-time
2. Intelligent optimizer folder updates EVERY trial (not batched)
3. Dashboard shows Pareto front for multi-objective studies
4. Dashboard units are dynamic (read from config)
5. Dashboard is professional quality (like Optuna Dashboard or Weights & Biases)
6. No hardcoded assumptions (Hz, single-objective, etc.)