Atomizer/docs/plans/UNIFIED_CONFIG_EXECUTION_PLAN.md

# Unified Configuration Architecture - Execution Plan

**Project**: AtomizerSpec v2.0 Implementation
**Reference Document**: `docs/plans/UNIFIED_CONFIGURATION_ARCHITECTURE.md`
**Schema Definition**: `optimization_engine/schemas/atomizer_spec_v2.json`
**Status**: Ready for Implementation

---

## Project Overview

Transform Atomizer's fragmented configuration system into a unified architecture where:
- One JSON schema (AtomizerSpec v2.0) is the single source of truth
- Canvas, Backend, Claude, and Optimization Engine all use the same spec
- Real-time WebSocket sync keeps all clients updated
- Claude can dynamically modify specs and add custom functions

---

## Phase Structure

| Phase | Name | Duration | Focus |
|-------|------|----------|-------|
| 1 | Foundation | Week 1-3 | Backend SpecManager, REST API, Migration |
| 2 | Frontend | Week 4-6 | SpecRenderer, WebSocket Sync, Store |
| 3 | Claude Integration | Week 7-9 | MCP Tools, Custom Functions |
| 4 | Polish & Testing | Week 10-12 | Migration, Testing, Documentation |

---

## Implementation Order (Critical Path)

```
1. Schema & Types (P1.1-P1.3)
   └── 2. SpecManager Service (P1.4-P1.7)
       └── 3. REST Endpoints (P1.8-P1.12)
           ├── 4. Migration Script (P1.13-P1.16)
           └── 5. Frontend Store (P2.1-P2.4)
               └── 6. SpecRenderer (P2.5-P2.10)
                   └── 7. WebSocket Sync (P2.11-P2.15)
                       └── 8. MCP Tools (P3.1-P3.8)
                           └── 9. Custom Functions (P3.9-P3.14)
                               └── 10. Testing & Polish (P4.1-P4.12)
```

---

## PHASE 1: Foundation (Backend)

### P1.1 - Create TypeScript types from JSON Schema
- **File**: `atomizer-dashboard/frontend/src/types/atomizer-spec.ts`
- **Action**: Generate TypeScript interfaces matching `atomizer_spec_v2.json`
- **Reference**: Schema at `optimization_engine/schemas/atomizer_spec_v2.json`
- **Acceptance**: Types compile, cover all schema definitions

### P1.2 - Create Python Pydantic models from JSON Schema
- **File**: `optimization_engine/config/spec_models.py`
- **Action**: Create Pydantic models for AtomizerSpec validation
- **Reference**: Schema at `optimization_engine/schemas/atomizer_spec_v2.json`
- **Acceptance**: Models validate example specs correctly

### P1.3 - Create spec validation utility
- **File**: `optimization_engine/config/spec_validator.py`
- **Action**: JSON Schema validation + semantic validation (bounds, references)
- **Dependencies**: P1.2
- **Acceptance**: Validates good specs, rejects invalid with clear errors

### P1.4 - Create SpecManager core class
- **File**: `atomizer-dashboard/backend/api/services/spec_manager.py`
- **Action**: Implement `load()`, `save()`, `_validate()`, `_compute_hash()`
- **Reference**: Design in UNIFIED_CONFIGURATION_ARCHITECTURE.md Section 5.1
- **Dependencies**: P1.3
- **Acceptance**: Can load/save/validate atomizer_spec.json files

### P1.5 - Add SpecManager patch functionality
- **File**: `atomizer-dashboard/backend/api/services/spec_manager.py`
- **Action**: Implement `patch()` method for JSONPath-style updates
- **Dependencies**: P1.4
- **Acceptance**: Can update nested fields with conflict detection

### P1.6 - Add SpecManager node operations
- **File**: `atomizer-dashboard/backend/api/services/spec_manager.py`
- **Action**: Implement `add_node()`, `remove_node()`, `_generate_id()`, `_auto_position()`
- **Dependencies**: P1.5
- **Acceptance**: Can add/remove design vars, extractors, objectives, constraints

### P1.7 - Add SpecManager custom function support
- **File**: `atomizer-dashboard/backend/api/services/spec_manager.py`
- **Action**: Implement `add_custom_function()` with Python syntax validation
- **Dependencies**: P1.6
- **Acceptance**: Can add custom extractors with validated Python code

### P1.8 - Create spec REST router
- **File**: `atomizer-dashboard/backend/api/routes/spec.py`
- **Action**: Create FastAPI router for spec endpoints
- **Dependencies**: P1.7
- **Acceptance**: Router imports and mounts correctly

### P1.9 - Implement GET /studies/{study_id}/spec
- **File**: `atomizer-dashboard/backend/api/routes/spec.py`
- **Action**: Return full AtomizerSpec for a study
- **Dependencies**: P1.8
- **Acceptance**: Returns valid spec JSON, 404 for missing studies

### P1.10 - Implement PUT /studies/{study_id}/spec
- **File**: `atomizer-dashboard/backend/api/routes/spec.py`
- **Action**: Replace entire spec with validation
- **Dependencies**: P1.9
- **Acceptance**: Validates, saves, returns new hash

### P1.11 - Implement PATCH /studies/{study_id}/spec
- **File**: `atomizer-dashboard/backend/api/routes/spec.py`
- **Action**: Partial update with JSONPath
- **Dependencies**: P1.10
- **Acceptance**: Updates specific fields, broadcasts change

### P1.12 - Implement POST /studies/{study_id}/spec/validate
- **File**: `atomizer-dashboard/backend/api/routes/spec.py`
- **Action**: Validate spec and return detailed report
- **Dependencies**: P1.11
- **Acceptance**: Returns errors, warnings, summary

### P1.13 - Create config migration base
- **File**: `optimization_engine/config/migrator.py`
- **Action**: Create SpecMigrator class with field mapping
- **Reference**: Design in UNIFIED_CONFIGURATION_ARCHITECTURE.md Section 8
- **Acceptance**: Class structure ready for migration logic

### P1.14 - Implement design variable migration
- **File**: `optimization_engine/config/migrator.py`
- **Action**: Migrate `bounds[]` → `bounds.min/max`, `parameter` → `expression_name`
- **Dependencies**: P1.13
- **Acceptance**: All DV formats convert correctly

### P1.15 - Implement objective/constraint migration
- **File**: `optimization_engine/config/migrator.py`
- **Action**: Migrate `goal` → `direction`, extraction configs to new format
- **Dependencies**: P1.14
- **Acceptance**: Objectives and constraints convert correctly

### P1.16 - Implement full config migration
- **File**: `optimization_engine/config/migrator.py`
- **Action**: Complete migration including canvas positions, extractors inference
- **Dependencies**: P1.15
- **Acceptance**: Can migrate any existing optimization_config.json to AtomizerSpec

### P1.17 - Create migration CLI tool
- **File**: `tools/migrate_to_spec_v2.py`
- **Action**: CLI for batch migration with dry-run support
- **Dependencies**: P1.16
- **Acceptance**: `python tools/migrate_to_spec_v2.py --dry-run studies/*`

---

## PHASE 2: Frontend Integration

### P2.1 - Create useSpecStore hook
- **File**: `atomizer-dashboard/frontend/src/hooks/useSpecStore.ts`
- **Action**: Zustand store for AtomizerSpec state management
- **Dependencies**: P1.1
- **Acceptance**: Store holds spec, provides typed accessors

### P2.2 - Add spec loading to useSpecStore
- **File**: `atomizer-dashboard/frontend/src/hooks/useSpecStore.ts`
- **Action**: Implement `loadSpec(studyId)` fetching from API
- **Dependencies**: P2.1, P1.9
- **Acceptance**: Loads spec from backend, updates store

### P2.3 - Add spec modification to useSpecStore
- **File**: `atomizer-dashboard/frontend/src/hooks/useSpecStore.ts`
- **Action**: Implement `patchSpec()`, `addNode()`, `removeNode()` calling API
- **Dependencies**: P2.2, P1.11
- **Acceptance**: Modifications persist to backend

### P2.4 - Add optimistic updates to useSpecStore
- **File**: `atomizer-dashboard/frontend/src/hooks/useSpecStore.ts`
- **Action**: Update local state immediately, rollback on error
- **Dependencies**: P2.3
- **Acceptance**: UI feels instant, handles errors gracefully

### P2.5 - Create specToNodes converter
- **File**: `atomizer-dashboard/frontend/src/lib/spec/converter.ts`
- **Action**: Convert AtomizerSpec to ReactFlow nodes array
- **Reference**: Design in UNIFIED_CONFIGURATION_ARCHITECTURE.md Section 5.2
- **Dependencies**: P1.1
- **Acceptance**: All node types render correctly

### P2.6 - Create specToEdges converter
- **File**: `atomizer-dashboard/frontend/src/lib/spec/converter.ts`
- **Action**: Convert spec.canvas.edges to ReactFlow edges
- **Dependencies**: P2.5
- **Acceptance**: All connections render correctly

### P2.7 - Create SpecRenderer component
- **File**: `atomizer-dashboard/frontend/src/components/canvas/SpecRenderer.tsx`
- **Action**: ReactFlow component that renders from useSpecStore
- **Dependencies**: P2.5, P2.6, P2.4
- **Acceptance**: Canvas displays spec correctly

### P2.8 - Wire node editing to spec updates
- **File**: `atomizer-dashboard/frontend/src/components/canvas/SpecRenderer.tsx`
- **Action**: Node data changes call useSpecStore.patchSpec()
- **Dependencies**: P2.7
- **Acceptance**: Editing node properties persists to spec

### P2.9 - Wire node position to spec updates
- **File**: `atomizer-dashboard/frontend/src/components/canvas/SpecRenderer.tsx`
- **Action**: Drag-drop updates canvas_position in spec
- **Dependencies**: P2.8
- **Acceptance**: Layout persists across reloads

### P2.10 - Update node panels for full spec fields
- **Files**: `atomizer-dashboard/frontend/src/components/canvas/panels/*.tsx`
- **Action**: Update all node config panels to show/edit full spec fields
- **Dependencies**: P2.8
- **Acceptance**: All spec fields are editable in UI

### P2.11 - Create WebSocket connection hook
- **File**: `atomizer-dashboard/frontend/src/hooks/useSpecSync.ts`
- **Action**: WebSocket connection to `/api/studies/{id}/sync`
- **Dependencies**: P2.4
- **Acceptance**: Connects, handles reconnection

### P2.12 - Create WebSocket backend endpoint
- **File**: `atomizer-dashboard/backend/api/routes/spec.py`
- **Action**: WebSocket endpoint for spec sync
- **Dependencies**: P1.12
- **Acceptance**: Accepts connections, tracks subscribers

### P2.13 - Implement spec_updated broadcast
- **File**: `atomizer-dashboard/backend/api/services/spec_manager.py`
- **Action**: SpecManager broadcasts to all subscribers on save
- **Dependencies**: P2.12
- **Acceptance**: All connected clients receive updates

### P2.14 - Handle spec_updated in frontend
- **File**: `atomizer-dashboard/frontend/src/hooks/useSpecSync.ts`
- **Action**: On spec_updated message, refresh spec from store
- **Dependencies**: P2.11, P2.13
- **Acceptance**: Changes from other clients appear in real-time

### P2.15 - Add conflict detection
- **File**: `atomizer-dashboard/frontend/src/hooks/useSpecStore.ts`
- **Action**: Compare hashes, warn on conflict, offer merge/overwrite
- **Dependencies**: P2.14
- **Acceptance**: Concurrent edits don't silently overwrite

### P2.16 - Replace CanvasView with SpecRenderer
- **File**: `atomizer-dashboard/frontend/src/pages/CanvasView.tsx`
- **Action**: Switch from useCanvasStore to useSpecStore + SpecRenderer
- **Dependencies**: P2.10, P2.15
- **Acceptance**: Canvas page uses new spec-based system

---

## PHASE 3: Claude Integration

### P3.1 - Create spec_get MCP tool
- **File**: `mcp-server/atomizer-tools/src/tools/spec_tools.ts`
- **Action**: Tool to retrieve full AtomizerSpec
- **Reference**: Design in UNIFIED_CONFIGURATION_ARCHITECTURE.md Section 5.3
- **Dependencies**: P1.9
- **Acceptance**: Claude can read spec via MCP

### P3.2 - Create spec_modify MCP tool
- **File**: `mcp-server/atomizer-tools/src/tools/spec_tools.ts`
- **Action**: Tool to apply modifications (set, add, remove operations)
- **Dependencies**: P3.1, P1.11
- **Acceptance**: Claude can modify spec fields

### P3.3 - Create spec_add_node MCP tool
- **File**: `mcp-server/atomizer-tools/src/tools/spec_tools.ts`
- **Action**: Tool to add design vars, extractors, objectives, constraints
- **Dependencies**: P3.2
- **Acceptance**: Claude can add nodes to canvas

### P3.4 - Create spec_remove_node MCP tool
- **File**: `mcp-server/atomizer-tools/src/tools/spec_tools.ts`
- **Action**: Tool to remove nodes (with edge cleanup)
- **Dependencies**: P3.3
- **Acceptance**: Claude can remove nodes

### P3.5 - Create spec_validate MCP tool
- **File**: `mcp-server/atomizer-tools/src/tools/spec_tools.ts`
- **Action**: Tool to validate spec and return report
- **Dependencies**: P1.12
- **Acceptance**: Claude can check spec validity

### P3.6 - Create spec_add_custom_extractor MCP tool
- **File**: `mcp-server/atomizer-tools/src/tools/spec_tools.ts`
- **Action**: Tool to add custom Python functions as extractors
- **Dependencies**: P1.7, P3.3
- **Acceptance**: Claude can add custom extraction logic

### P3.7 - Register spec tools in MCP server
- **File**: `mcp-server/atomizer-tools/src/index.ts`
- **Action**: Import and register all spec_* tools
- **Dependencies**: P3.1-P3.6
- **Acceptance**: Tools appear in MCP tool list

### P3.8 - Update ContextBuilder for spec awareness
- **File**: `atomizer-dashboard/backend/api/services/context_builder.py`
- **Action**: Include spec summary in Claude context, mention spec tools
- **Dependencies**: P3.7
- **Acceptance**: Claude knows about spec tools in context

### P3.9 - Create custom extractor runtime loader
- **File**: `optimization_engine/extractors/custom_loader.py`
- **Action**: Dynamically load and execute custom functions from spec
- **Dependencies**: P1.7
- **Acceptance**: Custom functions execute during optimization

### P3.10 - Integrate custom extractors into optimization runner
- **File**: `optimization_engine/core/runner.py`
- **Action**: Check spec for custom extractors, load and use them
- **Dependencies**: P3.9
- **Acceptance**: Optimization uses custom extractors defined in spec

### P3.11 - Add custom extractor node type to canvas
- **File**: `atomizer-dashboard/frontend/src/components/canvas/nodes/CustomExtractorNode.tsx`
- **Action**: New node type showing custom function with code preview
- **Dependencies**: P2.10
- **Acceptance**: Custom extractors display distinctly in canvas

### P3.12 - Add code editor for custom extractors
- **File**: `atomizer-dashboard/frontend/src/components/canvas/panels/CustomExtractorPanel.tsx`
- **Action**: Monaco editor for viewing/editing custom Python code
- **Dependencies**: P3.11
- **Acceptance**: Users can view and edit custom function code

### P3.13 - Create spec_create_from_description MCP tool
- **File**: `mcp-server/atomizer-tools/src/tools/spec_tools.ts`
- **Action**: Tool to create new spec from natural language + model path
- **Dependencies**: P3.6, P1.16
- **Acceptance**: Claude can create studies from descriptions

### P3.14 - Update Claude prompts for spec workflow
- **File**: `atomizer-dashboard/backend/api/services/context_builder.py`
- **Action**: Update system prompts to guide Claude on using spec tools
- **Dependencies**: P3.13
- **Acceptance**: Claude naturally uses spec tools for modifications

---

## PHASE 4: Polish & Testing

### P4.1 - Migrate m1_mirror studies
- **Action**: Run migration on all m1_mirror_* studies
- **Dependencies**: P1.17
- **Acceptance**: All studies have valid atomizer_spec.json

### P4.2 - Migrate drone_gimbal study
- **Action**: Run migration on drone_gimbal study
- **Dependencies**: P1.17
- **Acceptance**: Study has valid atomizer_spec.json

### P4.3 - Migrate all remaining studies
- **Action**: Run migration on all studies in studies/
- **Dependencies**: P4.1, P4.2
- **Acceptance**: All studies migrated, validated

### P4.4 - Create spec unit tests
- **File**: `tests/test_spec_manager.py`
- **Action**: Unit tests for SpecManager operations
- **Dependencies**: P1.7
- **Acceptance**: All SpecManager methods tested

### P4.5 - Create spec API integration tests
- **File**: `tests/test_spec_api.py`
- **Action**: Integration tests for REST endpoints
- **Dependencies**: P1.12
- **Acceptance**: All endpoints tested

### P4.6 - Create migration tests
- **File**: `tests/test_migrator.py`
- **Action**: Test migration with various config formats
- **Dependencies**: P1.16
- **Acceptance**: All config variants migrate correctly

### P4.7 - Create frontend component tests
- **File**: `atomizer-dashboard/frontend/src/__tests__/SpecRenderer.test.tsx`
- **Action**: Test SpecRenderer with various specs
- **Dependencies**: P2.16
- **Acceptance**: Canvas renders correctly for all spec types

### P4.8 - Create WebSocket sync tests
- **File**: `tests/test_spec_sync.py`
- **Action**: Test real-time sync between multiple clients
- **Dependencies**: P2.15
- **Acceptance**: Changes propagate correctly

### P4.9 - Create MCP tools tests
- **File**: `mcp-server/atomizer-tools/src/__tests__/spec_tools.test.ts`
- **Action**: Test all spec_* MCP tools
- **Dependencies**: P3.7
- **Acceptance**: All tools work correctly

### P4.10 - End-to-end testing
- **Action**: Full workflow test: create study in canvas, modify via Claude, run optimization
- **Dependencies**: P4.1-P4.9
- **Acceptance**: Complete workflow works

### P4.11 - Update documentation
- **Files**: `docs/04_USER_GUIDES/CANVAS.md`, `docs/04_USER_GUIDES/DASHBOARD.md`
- **Action**: Document new spec-based workflow
- **Dependencies**: P4.10
- **Acceptance**: Documentation reflects new system

### P4.12 - Update CLAUDE.md
- **File**: `CLAUDE.md`
- **Action**: Add spec tools documentation, update context loading
- **Dependencies**: P4.11
- **Acceptance**: Claude Code sessions know about spec system

---

## File Summary

### New Files to Create

| File | Phase | Purpose |
|------|-------|---------|
| `atomizer-dashboard/frontend/src/types/atomizer-spec.ts` | P1 | TypeScript types |
| `optimization_engine/config/spec_models.py` | P1 | Pydantic models |
| `optimization_engine/config/spec_validator.py` | P1 | Validation logic |
| `atomizer-dashboard/backend/api/services/spec_manager.py` | P1 | Core spec service |
| `atomizer-dashboard/backend/api/routes/spec.py` | P1 | REST endpoints |
| `optimization_engine/config/migrator.py` | P1 | Config migration |
| `tools/migrate_to_spec_v2.py` | P1 | Migration CLI |
| `atomizer-dashboard/frontend/src/hooks/useSpecStore.ts` | P2 | Spec state store |
| `atomizer-dashboard/frontend/src/hooks/useSpecSync.ts` | P2 | WebSocket sync |
| `atomizer-dashboard/frontend/src/lib/spec/converter.ts` | P2 | Spec ↔ ReactFlow |
| `atomizer-dashboard/frontend/src/components/canvas/SpecRenderer.tsx` | P2 | New canvas component |
| `mcp-server/atomizer-tools/src/tools/spec_tools.ts` | P3 | MCP tools |
| `optimization_engine/extractors/custom_loader.py` | P3 | Custom function loader |
| `atomizer-dashboard/frontend/src/components/canvas/nodes/CustomExtractorNode.tsx` | P3 | Custom node type |
| `atomizer-dashboard/frontend/src/components/canvas/panels/CustomExtractorPanel.tsx` | P3 | Code editor panel |

### Files to Modify

| File | Phase | Changes |
|------|-------|---------|
| `atomizer-dashboard/backend/api/main.py` | P1 | Mount spec router |
| `mcp-server/atomizer-tools/src/index.ts` | P3 | Register spec tools |
| `atomizer-dashboard/backend/api/services/context_builder.py` | P3 | Update Claude context |
| `optimization_engine/core/runner.py` | P3 | Custom extractor support |
| `atomizer-dashboard/frontend/src/pages/CanvasView.tsx` | P2 | Use SpecRenderer |
| `CLAUDE.md` | P4 | Document spec system |

---

## Success Criteria

### Phase 1 Complete When:
- [ ] SpecManager can load/save/validate specs
- [ ] All REST endpoints return correct responses
- [ ] Migration tool converts existing configs

### Phase 2 Complete When:
- [ ] Canvas renders from AtomizerSpec
- [ ] Edits persist to spec file
- [ ] WebSocket sync works between clients

### Phase 3 Complete When:
- [ ] Claude can read and modify specs via MCP
- [ ] Custom extractors work in optimization
- [ ] Claude can create studies from descriptions

### Phase 4 Complete When:
- [ ] All existing studies migrated
- [ ] All tests pass
- [ ] Documentation updated

---

## Risk Mitigation

| Risk | Mitigation |
|------|------------|
| Existing studies break | Migration tool with dry-run, keep old configs as backup |
| WebSocket complexity | Start with polling, add WebSocket as enhancement |
| Custom code security | Sandbox execution, syntax validation, no imports |
| Performance with large specs | Lazy loading, incremental updates |

---

## Quick Reference

**Master Design**: `docs/plans/UNIFIED_CONFIGURATION_ARCHITECTURE.md`
**Schema**: `optimization_engine/schemas/atomizer_spec_v2.json`
**This Plan**: `docs/plans/UNIFIED_CONFIG_EXECUTION_PLAN.md`

---

*Execute tasks in order. Each task has clear acceptance criteria. Reference the master design document for detailed specifications.*