Atomizer/docs/09_DIAGRAMS/architecture_overview.md

# Atomizer System Architecture

**Last Updated**: 2025-11-21
**Version**: 1.0
**Status**: ✅ Complete

---

## Overview

This document provides comprehensive visual documentation of Atomizer's architecture, showing how components interact to provide intelligent, multi-strategy structural optimization powered by NX Nastran.

---

## High-Level System Architecture

```mermaid
graph TB
    subgraph User["👤 User Interface"]
        CLI[Command Line Interface]
        Dashboard[React Dashboard<br/>Real-time Monitoring]
        Config[JSON Configuration<br/>optimization_config.json]
    end

    subgraph Core["🧠 Atomizer Core (Python)"]
        direction TB

        subgraph Engine["Optimization Engine"]
            IntelOpt[IntelligentOptimizer<br/>Protocol 10: IMSO]
            Optuna[Optuna Study<br/>Multi-objective Support]
            Strategy[Strategy Portfolio<br/>TPE/NSGA-II/CMA-ES/...]
            Landscape[Landscape Analyzer<br/>Adaptive Characterization]
        end

        subgraph Tracking["Real-time Tracking"]
            RTTracker[RealtimeTracker<br/>Protocol 13]
            TrialLog[trial_log.json]
            OptimizerState[optimizer_state.json]
            Intelligence[intelligence_report.json]
        end

        subgraph Extractors["Extractor Library<br/>Protocol 12"]
            FreqExt[Frequency Extractor]
            MassExt[Mass Extractor]
            StressExt[Stress Extractor]
            CustomExt[Custom Extractors]
        end
    end

    subgraph NX["⚙️ NX Nastran Integration"]
        SessionMgr[NX Session Manager<br/>Session Pooling]
        ModelUpdate[Model Updater<br/>Design Variable Injection]
        Solver[NX Solver<br/>FEA Execution]
        Results[Result Extraction<br/>Extractor Execution]
    end

    subgraph Storage["💾 Storage Layer"]
        StudyDB[(Optuna SQLite DB<br/>study.db)]
        ModelFiles[(CAD Models<br/>.prt, .sim, .fem)]
        Reports[(Reports & Logs<br/>Markdown, JSON)]
    end

    %% User interactions
    CLI --> Config
    Config --> IntelOpt
    Dashboard -.->|WebSocket Updates| RTTracker

    %% Core optimization flow
    IntelOpt --> Landscape
    Landscape --> Strategy
    Strategy --> Optuna
    IntelOpt --> RTTracker

    %% Optimization loop
    Optuna -->|Trial Parameters| ModelUpdate
    ModelUpdate --> SessionMgr
    SessionMgr --> Solver
    Solver --> Results
    Results --> Extractors
    Extractors -->|Objectives| Optuna

    %% Tracking
    RTTracker --> TrialLog
    RTTracker --> OptimizerState
    Landscape --> Intelligence

    %% Storage
    Optuna --> StudyDB
    SessionMgr --> ModelFiles
    IntelOpt --> Reports

    %% Dashboard reads
    Dashboard -.->|REST API| StudyDB
    Dashboard -.->|File Polling| TrialLog

    %% Styling
    classDef userClass fill:#E3F2FD,stroke:#1976D2,stroke-width:2px
    classDef coreClass fill:#E8F5E9,stroke:#388E3C,stroke-width:2px
    classDef nxClass fill:#FFF3E0,stroke:#F57C00,stroke-width:2px
    classDef storageClass fill:#F3E5F5,stroke:#7B1FA2,stroke-width:2px

    class CLI,Dashboard,Config userClass
    class IntelOpt,Optuna,Strategy,Landscape,RTTracker,FreqExt,MassExt,StressExt,CustomExt coreClass
    class SessionMgr,ModelUpdate,Solver,Results nxClass
    class StudyDB,ModelFiles,Reports storageClass
```

---

## Component Details

### 1. Optimization Engine Core

```mermaid
graph LR
    subgraph IntelligentOptimizer["IntelligentOptimizer (Protocol 10)"]
        Init[Initialize Study]
        Config[Load Configuration]
        Phase[Determine Phase]
        Execute[Execute Trials]
        Finalize[Generate Report]
    end

    subgraph Protocols["Protocol Support"]
        P10[Protocol 10: IMSO<br/>Adaptive Strategy]
        P11[Protocol 11: Multi-Obj<br/>NSGA-II Support]
        P13[Protocol 13: Tracking<br/>Real-time Updates]
    end

    Init --> Config
    Config --> Phase
    Phase -->|Single-Objective| P10
    Phase -->|Multi-Objective| P11
    P10 --> Execute
    P11 --> Execute
    Execute --> P13
    P13 --> Finalize

    classDef protocolClass fill:#E1BEE7,stroke:#6A1B9A,stroke-width:2px
    class P10,P11,P13 protocolClass
```

### 2. NX Integration Architecture

```mermaid
graph TB
    subgraph SessionManagement["NX Session Manager"]
        Pool[Session Pool<br/>Reusable Sessions]
        Create[Create Session]
        Reuse[Reuse Session]
        Cleanup[Cleanup on Error]
    end

    subgraph ModelOperation["Model Operations"]
        Load[Load CAD Model<br/>.prt file]
        Update[Update Parameters<br/>Design Variables]
        Solve[Run FEA Solver<br/>.sim file]
        Extract[Extract Results<br/>Custom Extractors]
    end

    subgraph ErrorHandling["Error Handling"]
        Timeout[Timeout Detection]
        Retry[Retry Logic]
        FallbackCreate[Fallback: New Session]
    end

    Pool --> Reuse
    Pool -->|No Available| Create
    Reuse --> Load
    Create --> Load
    Load --> Update
    Update --> Solve
    Solve --> Extract

    Solve -.->|Error| Timeout
    Timeout --> Retry
    Retry -->|Fail| FallbackCreate
    FallbackCreate --> Cleanup

    classDef sessionClass fill:#FFEBEE,stroke:#C62828,stroke-width:2px
    classDef modelClass fill:#E0F2F1,stroke:#00695C,stroke-width:2px

    class Pool,Create,Reuse,Cleanup sessionClass
    class Load,Update,Solve,Extract modelClass
```

### 3. Data Flow Architecture

```mermaid
sequenceDiagram
    participant User
    participant IntelOpt as IntelligentOptimizer
    participant Optuna
    participant NX as NX Solver
    participant Extractor
    participant Tracker as RealtimeTracker
    participant Dashboard

    User->>IntelOpt: Start Optimization
    IntelOpt->>Optuna: Create Study

    loop For each trial
        Optuna->>Optuna: Sample Design Variables
        Optuna->>NX: Trial Parameters
        NX->>NX: Update CAD Model
        NX->>NX: Run FEA Solver
        NX->>Extractor: Simulation Results
        Extractor->>Extractor: Calculate Objectives
        Extractor->>Optuna: Objective Values

        Optuna->>Tracker: Trial Complete
        Tracker->>Tracker: Write trial_log.json
        Tracker->>Tracker: Write optimizer_state.json
        Tracker-->>Dashboard: File Update Signal
        Dashboard-->>User: Live UI Update
    end

    Optuna->>IntelOpt: Study Complete
    IntelOpt->>IntelOpt: Generate Report
    IntelOpt->>User: Optimization Results
```

---

## Protocol Integration

### Protocol 10: Intelligent Multi-Strategy Optimization (IMSO)

```mermaid
stateDiagram-v2
    [*] --> Initialization

    Initialization --> TypeCheck: Load Config

    TypeCheck --> SingleObj: Single Objective
    TypeCheck --> MultiObj: Multi-Objective

    SingleObj --> Characterization: Protocol 10 Active
    Characterization --> LandscapeAnalysis: Min trials reached
    LandscapeAnalysis --> StrategySelection: Confidence ≥ 0.85
    StrategySelection --> Optimization: Strategy Selected

    MultiObj --> NSGAIISetup: Protocol 11 Active
    NSGAIISetup --> Optimization: NSGA-II Configured

    Optimization --> TrialExecution: Run Trial
    TrialExecution --> RealtimeTracking: Protocol 13
    RealtimeTracking --> TrialExecution: Next Trial
    TrialExecution --> Finalization: All trials complete

    Finalization --> [*]: Report Generated

    note right of Characterization
        Adaptive phase
        10-30 trials
        Random/Sobol sampling
    end note

    note right of LandscapeAnalysis
        Analyzes:
        - Smoothness
        - Multimodality
        - Separability
        - Noise level
    end note

    note right of NSGAIISetup
        Multi-objective:
        - Skips characterization
        - Direct NSGA-II
        - Pareto front optimization
    end note
```

### Protocol 11: Multi-Objective Support

```mermaid
graph TB
    subgraph Detection["Multi-Objective Detection"]
        CheckObj[Check len objectives]
        Single{objectives == 1?}
        Multi{objectives > 1?}
    end

    subgraph SinglePath["Single-Objective Path"]
        BestValue[study.best_value]
        BestTrial[study.best_trial]
        TrialValue[trial.value]
    end

    subgraph MultiPath["Multi-Objective Path"]
        BestTrials[study.best_trials<br/>Pareto Front]
        TrialValues[trial.values<br/>Multiple objectives]
        NSGAII[NSGA-II Sampler]
    end

    subgraph Validation["Protocol 11 Compliance"]
        CheckAPI{API call type?}
        UseSingle[Use .value]
        UseMulti[Use .values]
        SkipChar[Skip Characterization]
        SkipStag[Skip Stagnation]
    end

    CheckObj --> Single
    CheckObj --> Multi

    Single --> SinglePath
    Multi --> MultiPath

    SinglePath --> CheckAPI
    MultiPath --> CheckAPI

    CheckAPI -->|Single| UseSingle
    CheckAPI -->|Multi| UseMulti

    UseMulti --> SkipChar
    UseMulti --> SkipStag

    classDef errorClass fill:#FFCDD2,stroke:#C62828,stroke-width:3px
    classDef safeClass fill:#C8E6C9,stroke:#2E7D32,stroke-width:3px

    class UseSingle,UseMulti safeClass
```

### Protocol 13: Real-Time Dashboard Tracking

```mermaid
sequenceDiagram
    participant Trial as Optuna Trial
    participant Callback as Optuna Callback
    participant Tracker as RealtimeTracker
    participant FS as File System
    participant Dashboard

    Trial->>Callback: Trial Complete
    Callback->>Tracker: after_trial()

    activate Tracker
    Tracker->>Tracker: Prepare Trial Data
    Tracker->>FS: Write trial_log.json
    Tracker->>FS: Write optimizer_state.json

    alt Characterization Phase
        Tracker->>Tracker: Update Progress
        Tracker->>FS: Write characterization_progress.json
    end

    alt Landscape Analysis
        Tracker->>Tracker: Analyze Landscape
        Tracker->>FS: Write intelligence_report.json
    end

    alt Strategy Transition
        Tracker->>Tracker: Log Transition
        Tracker->>FS: Write strategy_transitions.json
    end

    deactivate Tracker

    FS-->>Dashboard: File Watch Event
    Dashboard->>FS: Read JSON Files
    Dashboard->>Dashboard: Update UI
    Dashboard-->>Dashboard: Render Charts
```

---

## Extractor Library (Protocol 12)

```mermaid
graph TB
    subgraph ExtractorLibrary["Extractor Library (optimization_engine/extractors/)"]
        Base[base_extractor.py<br/>BaseExtractor ABC]

        subgraph Builtin["Built-in Extractors"]
            Freq[frequency_extractor.py<br/>FrequencyExtractor]
            Mass[mass_extractor.py<br/>MassExtractor]
            Stress[stress_extractor.py<br/>StressExtractor]
            Disp[displacement_extractor.py<br/>DisplacementExtractor]
        end

        Custom[Custom User Extractors<br/>Extend BaseExtractor]
    end

    subgraph Usage["Extractor Usage Flow"]
        Registry[Extractor Registry<br/>Map name -> class]
        Load[Load by Name]
        Execute[Execute Extraction]
        Return[Return Value]
    end

    Base --> Freq
    Base --> Mass
    Base --> Stress
    Base --> Disp
    Base --> Custom

    Freq --> Registry
    Mass --> Registry
    Stress --> Registry
    Disp --> Registry
    Custom --> Registry

    Registry --> Load
    Load --> Execute
    Execute --> Return

    classDef extractorClass fill:#FFF9C4,stroke:#F57F17,stroke-width:2px
    class Freq,Mass,Stress,Disp,Custom extractorClass
```

---

## Study Directory Structure

```mermaid
graph TB
    subgraph StudyRoot["studies/my_optimization/"]
        Setup["1_setup/"]
        Results["2_results/"]
    end

    subgraph SetupContents["1_setup/"]
        Model["model/<br/>.prt, .sim, .fem files"]
        OptConfig["optimization_config.json"]
        RunScript["run_optimization.py"]
    end

    subgraph ResultsContents["2_results/"]
        StudyDB["study.db<br/>Optuna SQLite"]
        Summary["optimization_summary.json"]

        subgraph IntelFolder["intelligent_optimizer/"]
            TrialLog["trial_log.json"]
            OptimizerState["optimizer_state.json"]
            CharProgress["characterization_progress.json"]
            IntelReport["intelligence_report.json"]
            StratTrans["strategy_transitions.json"]
        end

        Report["final_report.md<br/>Markdown summary"]
    end

    StudyRoot --> Setup
    StudyRoot --> Results

    Setup --> Model
    Setup --> OptConfig
    Setup --> RunScript

    Results --> StudyDB
    Results --> Summary
    Results --> IntelFolder
    Results --> Report

    IntelFolder --> TrialLog
    IntelFolder --> OptimizerState
    IntelFolder --> CharProgress
    IntelFolder --> IntelReport
    IntelFolder --> StratTrans
```

---

## Philosophy & Design Principles

### 1. Separation of Concerns

```mermaid
graph LR
    subgraph Philosophy["Atomizer Philosophy"]
        Engine[Optimization Engine<br/>Pure algorithmic logic]
        NX[NX Integration<br/>CAD/FEA operations]
        Track[Tracking System<br/>Observability]
        User[User Interface<br/>Configuration & Monitoring]
    end

    Engine -.->|Clean API| NX
    Engine -.->|Events| Track
    Track -.->|Data| User
    User -.->|Config| Engine

    note1[No tight coupling<br/>Each layer independent]
    note2[Swap NX for Ansys?<br/>Just change integration layer]
    note3[Add new optimizer?<br/>Implement strategy interface]

    Engine -.-> note2
    NX -.-> note2
    Track -.-> note3
```

### 2. Protocol-Based Architecture

```mermaid
mindmap
    root((Protocols))
        Protocol 10: IMSO
            Adaptive Characterization
            Landscape Analysis
            Strategy Selection
            Dynamic Switching
        Protocol 11: Multi-Obj
            Pareto Front
            NSGA-II
            API Consistency
            Backward Compatible
        Protocol 13: Tracking
            Real-time Updates
            JSON File Writes
            Dashboard Integration
            Observability
        Future Protocols
            Protocol 14: ?
            Protocol 15: ?
            Extensible System
```

### 3. Extensibility Points

```mermaid
graph TB
    subgraph Core["Atomizer Core<br/>(Unchangeable)"]
        CoreEngine[Optimization Engine]
        CoreProtocols[Protocol System]
    end

    subgraph Extension["Extension Points"]
        CustomExtractor[Custom Extractors<br/>BaseExtractor ABC]
        CustomStrategy[Custom Strategies<br/>Optuna Sampler]
        CustomSolver[Custom Solvers<br/>Solver Interface]
        CustomCallback[Custom Callbacks<br/>Optuna Callback]
    end

    subgraph User["User Code"]
        MyExtractor[MyCustomExtractor]
        MyStrategy[MyCustomSampler]
        MySolver[MyFEASolver]
    end

    CoreEngine --> CustomExtractor
    CoreEngine --> CustomStrategy
    CoreEngine --> CustomSolver
    CoreEngine --> CustomCallback

    CustomExtractor -.->|Implement| MyExtractor
    CustomStrategy -.->|Implement| MyStrategy
    CustomSolver -.->|Implement| MySolver

    classDef coreClass fill:#BBDEFB,stroke:#1565C0,stroke-width:3px
    classDef extClass fill:#C8E6C9,stroke:#2E7D32,stroke-width:2px
    classDef userClass fill:#FFE082,stroke:#F57F00,stroke-width:2px

    class CoreEngine,CoreProtocols coreClass
    class CustomExtractor,CustomStrategy,CustomSolver,CustomCallback extClass
    class MyExtractor,MyStrategy,MySolver userClass
```

---

## Technology Stack

```mermaid
graph TB
    subgraph Frontend["Frontend Layer"]
        React[React 18<br/>TypeScript]
        Vite[Vite<br/>Build Tool]
        Recharts[Recharts<br/>Visualization]
    end

    subgraph Backend["Backend Layer"]
        FastAPI[FastAPI<br/>REST API]
        Uvicorn[Uvicorn<br/>ASGI Server]
    end

    subgraph Core["Core Engine"]
        Python[Python 3.8+]
        Optuna[Optuna 3.x<br/>Hyperparameter Optimization]
        NumPy[NumPy/SciPy<br/>Numerical Computing]
    end

    subgraph Integration["Integration"]
        NXOpen[NX Open API<br/>Python Journals]
        SQLite[SQLite<br/>Study Storage]
    end

    React --> FastAPI
    FastAPI --> Optuna
    Optuna --> SQLite
    Optuna --> NXOpen

    classDef frontClass fill:#E1F5FE,stroke:#0277BD
    classDef backClass fill:#F3E5F5,stroke:#6A1B9A
    classDef coreClass fill:#E8F5E9,stroke:#2E7D32
    classDef integClass fill:#FFF3E0,stroke:#E65100

    class React,Vite,Recharts frontClass
    class FastAPI,Uvicorn backClass
    class Python,Optuna,NumPy coreClass
    class NXOpen,SQLite integClass
```

---

## Performance Characteristics

```mermaid
graph LR
    subgraph Timing["Typical Trial Timing (100 trials)"]
        T1[Trial Overhead<br/>~0.5s]
        T2[NX Model Update<br/>~1s]
        T3[FEA Solve<br/>~5-10s]
        T4[Result Extraction<br/>~0.5s]
        T5[Optuna Update<br/>~0.1s]
    end

    T1 --> T2 --> T3 --> T4 --> T5

    Total[Total: ~7-12s per trial<br/>100 trials = 12-20 minutes]

    T5 -.-> Total

    note1[Parallelization possible<br/>Multiple NX sessions]
    note2[Session reuse saves<br/>~2s per trial]

    T3 -.-> note1
    T2 -.-> note2
```

---

## Next Steps

For protocol-specific workflows, see:
- [Protocol Workflows](protocol_workflows.md) - Detailed protocol execution diagrams
- [Optimization Lifecycle](optimization_lifecycle.md) - Trial-by-trial flow

For implementation details, see:
- [Main Documentation Index](../00_INDEX.md)
- [Protocol Specifications](../01_PROTOCOLS.md)