docs: Update documentation for AtomizerSpec v2.0 unified configuration

Phase 4 documentation updates: - CANVAS.md: Add AtomizerSpec v2.0 schema, custom extractors section, spec REST API endpoints, and updated file structure - DASHBOARD.md: Add V3.1 release notes, AtomizerSpec API endpoints, and unified configuration features - CLAUDE.md: Add complete AtomizerSpec v2.0 section with code examples, MCP tools reference, API endpoints, and updated directory structure Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-17 10:57:58 -05:00
parent ac5e9b4054
commit 37f73cc2be
3 changed files with 572 additions and 28 deletions
--- a/docs/04_USER_GUIDES/CANVAS.md
+++ b/docs/04_USER_GUIDES/CANVAS.md
@@ -1,7 +1,7 @@
 # Atomizer Canvas - Visual Workflow Builder

-**Last Updated**: January 16, 2026
-**Version**: 3.0
+**Last Updated**: January 17, 2026
+**Version**: 3.1 (AtomizerSpec v2.0)
 **Status**: Production

 ---
@@ -10,19 +10,34 @@

 The Atomizer Canvas is a visual, node-based workflow builder for designing optimization studies. It provides a drag-and-drop interface for configuring FEA optimizations that integrates with Claude to validate and execute workflows.

+**New in v3.1**: The Canvas now uses **AtomizerSpec v2.0** as the unified configuration format. All studies are defined by a single `atomizer_spec.json` file that serves as the single source of truth for Canvas, Backend, Claude, and the Optimization Engine.
+
 ### Key Features

 - **Visual Workflow Design**: Drag-and-drop nodes to build optimization pipelines
 - **Professional Lucide Icons**: Clean, consistent iconography throughout the interface
- **Auto-Load from Studies**: Import existing optimization_config.json files
+- **AtomizerSpec v2.0**: Unified JSON spec format (`atomizer_spec.json`) for all configuration
+- **Auto-Load from Studies**: Import existing studies (supports both v2.0 specs and legacy configs)
 - **NX Model Introspection**: Automatically extract expressions from .prt/.sim/.fem files
 - **File Browser**: Browse and select model files with type filtering
 - **Expression Search**: Searchable dropdown for design variable configuration
 - **One-Click Add**: Add discovered expressions as design variables instantly
+- **Custom Extractors**: Define custom Python extraction functions directly in the spec
 - **Claude Integration**: "Process with Claude" button for AI-assisted study creation
+- **Real-time Sync**: WebSocket-based synchronization between clients
 - **Responsive Layout**: Full-screen canvas that adapts to window size

-### What's New in V3
+### What's New in V3.1
+
+| Feature | Description |
+|---------|-------------|
+| **AtomizerSpec v2.0** | Unified configuration format replacing `optimization_config.json` |
+| **Spec REST API** | Full CRUD operations on spec via `/api/studies/{id}/spec` |
+| **Custom Extractors** | Define custom Python functions as extractors in the spec |
+| **WebSocket Sync** | Real-time spec synchronization between clients |
+| **Legacy Migration** | Automatic migration of old `optimization_config.json` files |
+
+### What's New in V3.0

 | Feature | Description |
 |---------|-------------|
@@ -75,7 +90,8 @@ atomizer-dashboard/frontend/src/
 │   │   ├── ObjectiveNode.tsx     # Objective node
 │   │   ├── ConstraintNode.tsx    # Constraint node
 │   │   ├── AlgorithmNode.tsx     # Algorithm node
-│   │   └── SurrogateNode.tsx     # Surrogate node
+│   │   ├── SurrogateNode.tsx     # Surrogate node
+│   │   └── CustomExtractorNode.tsx # Custom extractor node (V3.1)
 │   ├── panels/
 │   │   ├── NodeConfigPanel.tsx   # Node configuration sidebar
 │   │   ├── ValidationPanel.tsx   # Validation toast display
@@ -85,21 +101,52 @@ atomizer-dashboard/frontend/src/
 │   │   ├── TemplateSelector.tsx  # Workflow template chooser
 │   │   ├── FileBrowser.tsx       # File picker modal (V3)
 │   │   ├── IntrospectionPanel.tsx # Model introspection results (V3)
-│   │   └── ExpressionSelector.tsx # Expression search dropdown (V3)
+│   │   ├── ExpressionSelector.tsx # Expression search dropdown (V3)
+│   │   └── CustomExtractorPanel.tsx # Code editor for custom extractors (V3.1)
 │   └── palette/
 │       └── NodePalette.tsx       # Draggable node palette
 ├── hooks/
 │   ├── useCanvasStore.ts         # Zustand store for canvas state
+│   ├── useSpecStore.ts           # Zustand store for AtomizerSpec (V3.1)
+│   ├── useSpecSync.ts            # WebSocket spec sync (V3.1)
 │   └── useCanvasChat.ts          # Claude chat integration
-├── lib/canvas/
-│   ├── schema.ts                 # TypeScript type definitions
-│   ├── intent.ts                 # Intent serialization (174 lines)
-│   ├── validation.ts             # Graph validation logic
-│   └── templates.ts              # Workflow templates
+├── lib/
+│   ├── canvas/
+│   │   ├── schema.ts             # TypeScript type definitions
+│   │   ├── intent.ts             # Intent serialization (legacy)
+│   │   ├── validation.ts         # Graph validation logic
+│   │   └── templates.ts          # Workflow templates
+│   └── spec/
+│       └── converter.ts          # AtomizerSpec ↔ ReactFlow converter (V3.1)
+├── types/
+│   └── atomizer-spec.ts          # AtomizerSpec TypeScript types (V3.1)
 └── pages/
    └── CanvasView.tsx            # Canvas page (/canvas route)
 ```

+### Backend File Structure (V3.1)
+
+```
+atomizer-dashboard/backend/api/
+├── services/
+│   ├── spec_manager.py           # SpecManager - load/save/validate specs
+│   ├── claude_agent.py           # Claude API integration
+│   └── context_builder.py        # Context assembly with spec awareness
+└── routes/
+    ├── spec.py                   # AtomizerSpec REST API endpoints
+    └── optimization.py           # Optimization endpoints
+
+optimization_engine/
+├── config/
+│   ├── spec_models.py            # Pydantic models for AtomizerSpec
+│   ├── spec_validator.py         # Semantic validation
+│   └── migrator.py               # Legacy config migration
+├── extractors/
+│   └── custom_extractor_loader.py # Runtime custom function loader
+└── schemas/
+    └── atomizer_spec_v2.json     # JSON Schema definition
+```
+
 ---

 ## User Interface
@@ -275,6 +322,92 @@ When loading a `.sim` file, the system introspects to find:

 ---

+## Custom Extractors (V3.1)
+
+Custom extractors allow you to define arbitrary Python extraction functions directly in the AtomizerSpec. These functions are validated for safety and executed during optimization.
+
+### Creating a Custom Extractor
+
+1. Add a **Custom Extractor** node from the palette
+2. Open the **Code Editor** in the config panel
+3. Write your extraction function following the required signature
+4. The code is validated in real-time for:
+   - Correct function signature
+   - Allowed imports only (numpy, pyNastran)
+   - No dangerous operations (os.system, exec, eval, etc.)
+
+### Function Signature
+
+```python
+def extract(op2_path, bdf_path=None, params=None, working_dir=None):
+    """
+    Custom extraction function.
+
+    Args:
+        op2_path: Path to the .op2 results file
+        bdf_path: Path to the .bdf mesh file (optional)
+        params: Dict of current design variable values (optional)
+        working_dir: Path to the trial working directory (optional)
+
+    Returns:
+        Dict with output values, e.g., {"custom_metric": 42.0}
+    """
+    import numpy as np
+    from pyNastran.op2.op2 import OP2
+
+    op2 = OP2()
+    op2.read_op2(op2_path)
+
+    # Your extraction logic here
+    result = np.max(op2.displacements[1].data)
+
+    return {"custom_metric": result}
+```
+
+### Allowed Imports
+
+| Module | Usage |
+|--------|-------|
+| `numpy` | Numerical operations |
+| `pyNastran` | OP2/BDF file parsing |
+| `math` | Basic math functions |
+| `pathlib` | Path manipulation |
+
+### Security Restrictions
+
+The following are **NOT allowed** in custom extractor code:
+- `import os`, `import subprocess`, `import sys`
+- `exec()`, `eval()`, `compile()`
+- `__import__()`, `open()` with write mode
+- Network operations
+- File system modifications outside working_dir
+
+### Example: Custom Stress Ratio Extractor
+
+```python
+def extract(op2_path, bdf_path=None, params=None, working_dir=None):
+    """Extract stress ratio (max_stress / allowable)."""
+    import numpy as np
+    from pyNastran.op2.op2 import OP2
+
+    op2 = OP2()
+    op2.read_op2(op2_path)
+
+    # Get von Mises stress from solid elements
+    stress_data = op2.ctetra_stress[1].data
+    max_von_mises = np.max(stress_data[:, :, 7])  # Column 7 is von Mises
+
+    allowable = 250.0  # MPa
+    stress_ratio = max_von_mises / allowable
+
+    return {
+        "stress_ratio": stress_ratio,
+        "max_stress_mpa": max_von_mises
+    }
+```
+
+---
+
 ## File Browser (V3)

 The File Browser allows you to navigate the studies directory to select model files.
@@ -330,11 +463,47 @@ Model Introspection analyzes NX model files to discover expressions, solver type

 ### Backend Endpoints

+#### AtomizerSpec REST API (V3.1)
+
+The spec API provides full CRUD operations on the unified AtomizerSpec:
+
+```
+GET    /api/studies/{study_id}/spec              # Get full AtomizerSpec
+  Returns: AtomizerSpec JSON with meta, model, design_variables, extractors, objectives, constraints, optimization, canvas
+
+PUT    /api/studies/{study_id}/spec              # Replace entire spec
+  Body: AtomizerSpec JSON
+  Returns: { hash: string, warnings: [] }
+
+PATCH  /api/studies/{study_id}/spec              # Partial update (JSONPath)
+  Body: { path: "design_variables[0].bounds.max", value: 15.0, modified_by: "canvas" }
+  Returns: { hash: string }
+
+POST   /api/studies/{study_id}/spec/validate     # Validate spec
+  Returns: { valid: boolean, errors: [], warnings: [] }
+
+POST   /api/studies/{study_id}/spec/nodes        # Add node (design var, extractor, etc.)
+  Body: { type: "designVar", data: {...}, modified_by: "canvas" }
+  Returns: { node_id: "dv_002", hash: string }
+
+PATCH  /api/studies/{study_id}/spec/nodes/{id}   # Update node
+  Body: { updates: {...}, modified_by: "canvas" }
+  Returns: { hash: string }
+
+DELETE /api/studies/{study_id}/spec/nodes/{id}   # Remove node
+  Query: modified_by=canvas
+  Returns: { hash: string }
+
+POST   /api/studies/{study_id}/spec/edges        # Add edge connection
+  Query: source=ext_001&target=obj_001&modified_by=canvas
+  Returns: { hash: string }
+```
+
 #### Study Configuration

 ```
 GET  /api/studies/                    # List all studies
-GET  /api/studies/{path}/config       # Get optimization_config.json
+GET  /api/studies/{path}/config       # Get optimization_config.json (legacy)
 ```

 #### File Browser (V3)
@@ -415,9 +584,169 @@ Analyzes a Canvas intent and provides recommendations.

 ---

-## OptimizationIntent Schema
+## AtomizerSpec v2.0 Schema

-The Canvas serializes workflows to the `OptimizationIntent` JSON format:
+The Canvas uses **AtomizerSpec v2.0** as the unified configuration format. This replaces the legacy `optimization_config.json` and `OptimizationIntent` with a single schema.
+
+### Core Structure
+
+```typescript
+interface AtomizerSpec {
+  meta: {
+    version: "2.0";
+    created: string;               // ISO timestamp
+    modified: string;              // ISO timestamp
+    created_by: "canvas" | "claude" | "api" | "migration" | "manual";
+    modified_by: "canvas" | "claude" | "api" | "migration" | "manual";
+    study_name: string;
+    description?: string;
+    tags?: string[];
+  };
+  model: {
+    sim: { path: string; solver: string };
+    prt?: { path: string };
+    fem?: { path: string };
+  };
+  design_variables: DesignVariable[];
+  extractors: Extractor[];
+  objectives: Objective[];
+  constraints: Constraint[];
+  optimization: OptimizationConfig;
+  canvas: CanvasLayout;
+}
+```
+
+### Design Variable
+
+```typescript
+interface DesignVariable {
+  id: string;                      // "dv_001", "dv_002", etc.
+  name: string;                    // Display name
+  expression_name: string;         // NX expression name
+  type: "continuous" | "integer" | "categorical";
+  bounds: { min: number; max: number };
+  baseline?: number;
+  unit?: string;
+  enabled: boolean;
+  canvas_position: { x: number; y: number };
+}
+```
+
+### Extractor
+
+```typescript
+interface Extractor {
+  id: string;                      // "ext_001", etc.
+  name: string;                    // Display name
+  type: string;                    // "mass", "displacement", "zernike_opd", "custom"
+  builtin: boolean;                // true for E1-E10, false for custom
+  outputs: Array<{ name: string; units?: string }>;
+  config?: Record<string, unknown>;
+  custom_function?: {              // For custom extractors only
+    code: string;                  // Python function code
+    entry_point: string;           // Function name (default: "extract")
+  };
+  canvas_position: { x: number; y: number };
+}
+```
+
+### Objective
+
+```typescript
+interface Objective {
+  id: string;                      // "obj_001", etc.
+  name: string;                    // Display name
+  direction: "minimize" | "maximize";
+  source: {
+    extractor_id: string;          // Reference to extractor
+    output_name: string;           // Which output from the extractor
+  };
+  weight?: number;                 // For weighted sum multi-objective
+  enabled: boolean;
+  canvas_position: { x: number; y: number };
+}
+```
+
+### Canvas Layout
+
+```typescript
+interface CanvasLayout {
+  edges: Array<{ source: string; target: string }>;
+  layout_version: "2.0";
+  viewport?: { x: number; y: number; zoom: number };
+}
+```
+
+### Example AtomizerSpec
+
+```json
+{
+  "meta": {
+    "version": "2.0",
+    "created": "2026-01-17T10:00:00Z",
+    "modified": "2026-01-17T10:30:00Z",
+    "created_by": "canvas",
+    "modified_by": "canvas",
+    "study_name": "bracket_mass_optimization",
+    "description": "Optimize bracket mass while maintaining stress limits"
+  },
+  "model": {
+    "sim": { "path": "bracket_sim1.sim", "solver": "nastran" }
+  },
+  "design_variables": [
+    {
+      "id": "dv_001",
+      "name": "Thickness",
+      "expression_name": "web_thickness",
+      "type": "continuous",
+      "bounds": { "min": 2.0, "max": 10.0 },
+      "baseline": 5.0,
+      "unit": "mm",
+      "enabled": true,
+      "canvas_position": { "x": 50, "y": 100 }
+    }
+  ],
+  "extractors": [
+    {
+      "id": "ext_001",
+      "name": "Mass",
+      "type": "mass",
+      "builtin": true,
+      "outputs": [{ "name": "mass", "units": "kg" }],
+      "canvas_position": { "x": 740, "y": 100 }
+    }
+  ],
+  "objectives": [
+    {
+      "id": "obj_001",
+      "name": "mass",
+      "direction": "minimize",
+      "source": { "extractor_id": "ext_001", "output_name": "mass" },
+      "enabled": true,
+      "canvas_position": { "x": 1020, "y": 100 }
+    }
+  ],
+  "constraints": [],
+  "optimization": {
+    "algorithm": { "type": "TPE" },
+    "budget": { "max_trials": 100 }
+  },
+  "canvas": {
+    "edges": [
+      { "source": "dv_001", "target": "model" },
+      { "source": "model", "target": "solver" },
+      { "source": "solver", "target": "ext_001" },
+      { "source": "ext_001", "target": "obj_001" },
+      { "source": "obj_001", "target": "optimization" }
+    ],
+    "layout_version": "2.0"
+  }
+}
+```
+
+### Legacy OptimizationIntent (Deprecated)
+
+The `OptimizationIntent` format is still supported for backwards compatibility but will be automatically converted to AtomizerSpec v2.0 when saved.

 ```typescript
 interface OptimizationIntent {
@@ -637,6 +966,8 @@ npm run build

 ## References

+- **AtomizerSpec v2.0 Architecture**: See `docs/plans/UNIFIED_CONFIGURATION_ARCHITECTURE.md`
+- **AtomizerSpec JSON Schema**: See `optimization_engine/schemas/atomizer_spec_v2.json`
 - **React Flow Documentation**: https://reactflow.dev/
 - **Lucide Icons**: https://lucide.dev/icons/
 - **Zustand**: https://github.com/pmndrs/zustand
@@ -647,3 +978,4 @@ npm run build
 ---

 *Canvas Builder: Design optimizations visually, execute with AI.*
+*Powered by AtomizerSpec v2.0 - the unified configuration format.*
--- a/docs/04_USER_GUIDES/DASHBOARD.md
+++ b/docs/04_USER_GUIDES/DASHBOARD.md
@@ -1,7 +1,7 @@
 # Atomizer Dashboard

-**Last Updated**: January 16, 2026
-**Version**: 3.0
+**Last Updated**: January 17, 2026
+**Version**: 3.1 (AtomizerSpec v2.0)

 ---

@@ -9,17 +9,19 @@

 The Atomizer Dashboard is a real-time web-based interface for monitoring and analyzing multi-objective optimization studies. Built with React, TypeScript, and Tailwind CSS, it provides comprehensive visualization and interaction capabilities for NSGA-II based structural optimization.

+**New in V3.1**: All studies now use **AtomizerSpec v2.0** as the unified configuration format. The `atomizer_spec.json` file serves as the single source of truth for Canvas, Backend, Claude, and the Optimization Engine.
+
 ### Major Features

 | Feature | Route | Description |
 |---------|-------|-------------|
-| **Canvas Builder** | `/canvas` | Visual node-based workflow designer |
+| **Canvas Builder** | `/canvas` | Visual node-based workflow designer (AtomizerSpec v2.0) |
 | **Live Dashboard** | `/dashboard` | Real-time optimization monitoring |
 | **Results Viewer** | `/results` | Markdown reports with charts |
 | **Analytics** | `/analytics` | Cross-study comparison |
-| **Claude Chat** | Global | AI-powered study creation |
+| **Claude Chat** | Global | AI-powered study creation with spec tools |

-> **New in V2**: The [Canvas Builder](CANVAS.md) provides a visual, drag-and-drop interface for designing optimization workflows with Claude AI integration.
+> **New in V3.1**: The [Canvas Builder](CANVAS.md) now uses AtomizerSpec v2.0 as the unified configuration format, with support for custom extractors and real-time WebSocket synchronization.

 ---

@@ -237,6 +239,43 @@ Reusable markdown rendering component:

 ## API Endpoints

+### AtomizerSpec (V3.1)
+
+#### GET `/api/studies/{study_id}/spec`
+Get the full AtomizerSpec for a study.
+
+**Response**:
+```json
+{
+  "meta": { "version": "2.0", "study_name": "..." },
+  "model": { "sim": { "path": "...", "solver": "nastran" } },
+  "design_variables": [...],
+  "extractors": [...],
+  "objectives": [...],
+  "constraints": [...],
+  "optimization": { "algorithm": { "type": "TPE" }, "budget": { "max_trials": 100 } },
+  "canvas": { "edges": [...], "layout_version": "2.0" }
+}
+```
+
+#### PUT `/api/studies/{study_id}/spec`
+Replace the entire spec with validation.
+
+#### PATCH `/api/studies/{study_id}/spec`
+Partial update using JSONPath. Body: `{ "path": "design_variables[0].bounds.max", "value": 15.0 }`
+
+#### POST `/api/studies/{study_id}/spec/validate`
+Validate spec and return detailed report. Returns: `{ "valid": true, "errors": [], "warnings": [] }`
+
+#### POST `/api/studies/{study_id}/spec/nodes`
+Add a new node (design variable, extractor, objective, constraint).
+
+#### PATCH `/api/studies/{study_id}/spec/nodes/{node_id}`
+Update an existing node's properties.
+
+#### DELETE `/api/studies/{study_id}/spec/nodes/{node_id}`
+Remove a node and clean up related edges.
+
 ### Studies

 #### GET `/api/optimization/studies`
@@ -576,6 +615,31 @@ if (!objectives || !designVariables) return <EmptyState />;

 ## Recent Updates

+### January 2026 (V3.1) - AtomizerSpec v2.0
+
+- [x] **AtomizerSpec v2.0**: Unified configuration architecture
+  - **Single Source of Truth**: One `atomizer_spec.json` file for Canvas, Backend, Claude, and Optimization Engine
+  - **Spec REST API**: Full CRUD operations at `/api/studies/{id}/spec`
+  - **Pydantic Validation**: Backend validation with detailed error messages
+  - **Legacy Migration**: Automatic migration from `optimization_config.json`
+  - **29 Studies Migrated**: All existing studies converted to v2.0 format
+
+- [x] **Custom Extractors**: Define custom Python extraction functions in the spec
+  - **Security Validation**: Code checked for dangerous patterns
+  - **Runtime Loading**: Functions executed during optimization
+  - **Canvas UI**: Code editor panel for custom extractor nodes
+
+- [x] **WebSocket Sync**: Real-time spec synchronization
+  - Multiple clients stay synchronized
+  - Optimistic updates with rollback on error
+  - Conflict detection with hash comparison
+
+- [x] **Comprehensive Testing**: Full test coverage for Phase 4
+  - Unit tests for SpecManager and Pydantic models
+  - API integration tests for all spec endpoints
+  - Migration tests for legacy config formats
+  - E2E tests for complete workflow
+
 ### January 2026 (V3.0)

 - [x] **Canvas Builder V3**: Major upgrade with model introspection and Claude fixes