Antoine/Atomizer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
a26914bbe8a44c1584da886d39566aa199ed934b
Atomizer/docs/plans/UNIFIED_CONFIG_EXECUTION_PLAN.md
Anto01 ea437d360e docs: Major documentation overhaul - restructure folders, update tagline, add Getting Started guide 
- Restructure docs/ folder (remove numeric prefixes):
  - 04_USER_GUIDES -> guides/
  - 05_API_REFERENCE -> api/
  - 06_PHYSICS -> physics/
  - 07_DEVELOPMENT -> development/
  - 08_ARCHIVE -> archive/
  - 09_DIAGRAMS -> diagrams/

- Replace tagline 'Talk, don't click' with 'LLM-driven optimization framework' in 9 files

- Create comprehensive docs/GETTING_STARTED.md:
  - Prerequisites and quick setup
  - Project structure overview
  - First study tutorial (Claude or manual)
  - Dashboard usage guide
  - Neural acceleration introduction

- Rewrite docs/00_INDEX.md with correct paths and modern structure

- Archive obsolete files:
  - 01_PROTOCOLS.md -> archive/historical/01_PROTOCOLS_legacy.md
  - 03_GETTING_STARTED.md -> archive/historical/
  - ATOMIZER_PODCAST_BRIEFING.md -> archive/marketing/

- Update timestamps to 2026-01-20 across all key files

- Update .gitignore to exclude docs/generated/

- Version bump: ATOMIZER_CONTEXT v1.8 -> v2.0
2026-01-20 10:03:45 -05:00

20 KiB
Raw Blame History

Unified Configuration Architecture - Execution Plan

Project: AtomizerSpec v2.0 Implementation Reference Document: docs/plans/UNIFIED_CONFIGURATION_ARCHITECTURE.md Schema Definition: optimization_engine/schemas/atomizer_spec_v2.json Status: Ready for Implementation

Project Overview

Transform Atomizer's fragmented configuration system into a unified architecture where:

  • One JSON schema (AtomizerSpec v2.0) is the single source of truth
  • Canvas, Backend, Claude, and Optimization Engine all use the same spec
  • Real-time WebSocket sync keeps all clients updated
  • Claude can dynamically modify specs and add custom functions

Phase Structure

Phase Name Duration Focus
1 Foundation Week 1-3 Backend SpecManager, REST API, Migration
2 Frontend Week 4-6 SpecRenderer, WebSocket Sync, Store
3 Claude Integration Week 7-9 MCP Tools, Custom Functions
4 Polish & Testing Week 10-12 Migration, Testing, Documentation

Implementation Order (Critical Path)

1. Schema & Types (P1.1-P1.3)
   └── 2. SpecManager Service (P1.4-P1.7)
       └── 3. REST Endpoints (P1.8-P1.12)
           ├── 4. Migration Script (P1.13-P1.16)
           └── 5. Frontend Store (P2.1-P2.4)
               └── 6. SpecRenderer (P2.5-P2.10)
                   └── 7. WebSocket Sync (P2.11-P2.15)
                       └── 8. MCP Tools (P3.1-P3.8)
                           └── 9. Custom Functions (P3.9-P3.14)
                               └── 10. Testing & Polish (P4.1-P4.12)

PHASE 1: Foundation (Backend)

P1.1 - Create TypeScript types from JSON Schema

  • File: atomizer-dashboard/frontend/src/types/atomizer-spec.ts
  • Action: Generate TypeScript interfaces matching atomizer_spec_v2.json
  • Reference: Schema at optimization_engine/schemas/atomizer_spec_v2.json
  • Acceptance: Types compile, cover all schema definitions

P1.2 - Create Python Pydantic models from JSON Schema

  • File: optimization_engine/config/spec_models.py
  • Action: Create Pydantic models for AtomizerSpec validation
  • Reference: Schema at optimization_engine/schemas/atomizer_spec_v2.json
  • Acceptance: Models validate example specs correctly

P1.3 - Create spec validation utility

  • File: optimization_engine/config/spec_validator.py
  • Action: JSON Schema validation + semantic validation (bounds, references)
  • Dependencies: P1.2
  • Acceptance: Validates good specs, rejects invalid with clear errors

P1.4 - Create SpecManager core class

  • File: atomizer-dashboard/backend/api/services/spec_manager.py
  • Action: Implement load(), save(), _validate(), _compute_hash()
  • Reference: Design in UNIFIED_CONFIGURATION_ARCHITECTURE.md Section 5.1
  • Dependencies: P1.3
  • Acceptance: Can load/save/validate atomizer_spec.json files

P1.5 - Add SpecManager patch functionality

  • File: atomizer-dashboard/backend/api/services/spec_manager.py
  • Action: Implement patch() method for JSONPath-style updates
  • Dependencies: P1.4
  • Acceptance: Can update nested fields with conflict detection

P1.6 - Add SpecManager node operations

  • File: atomizer-dashboard/backend/api/services/spec_manager.py
  • Action: Implement add_node(), remove_node(), _generate_id(), _auto_position()
  • Dependencies: P1.5
  • Acceptance: Can add/remove design vars, extractors, objectives, constraints

P1.7 - Add SpecManager custom function support

  • File: atomizer-dashboard/backend/api/services/spec_manager.py
  • Action: Implement add_custom_function() with Python syntax validation
  • Dependencies: P1.6
  • Acceptance: Can add custom extractors with validated Python code

P1.8 - Create spec REST router

  • File: atomizer-dashboard/backend/api/routes/spec.py
  • Action: Create FastAPI router for spec endpoints
  • Dependencies: P1.7
  • Acceptance: Router imports and mounts correctly

P1.9 - Implement GET /studies/{study_id}/spec

  • File: atomizer-dashboard/backend/api/routes/spec.py
  • Action: Return full AtomizerSpec for a study
  • Dependencies: P1.8
  • Acceptance: Returns valid spec JSON, 404 for missing studies

P1.10 - Implement PUT /studies/{study_id}/spec

  • File: atomizer-dashboard/backend/api/routes/spec.py
  • Action: Replace entire spec with validation
  • Dependencies: P1.9
  • Acceptance: Validates, saves, returns new hash

P1.11 - Implement PATCH /studies/{study_id}/spec

  • File: atomizer-dashboard/backend/api/routes/spec.py
  • Action: Partial update with JSONPath
  • Dependencies: P1.10
  • Acceptance: Updates specific fields, broadcasts change

P1.12 - Implement POST /studies/{study_id}/spec/validate

  • File: atomizer-dashboard/backend/api/routes/spec.py
  • Action: Validate spec and return detailed report
  • Dependencies: P1.11
  • Acceptance: Returns errors, warnings, summary

P1.13 - Create config migration base

  • File: optimization_engine/config/migrator.py
  • Action: Create SpecMigrator class with field mapping
  • Reference: Design in UNIFIED_CONFIGURATION_ARCHITECTURE.md Section 8
  • Acceptance: Class structure ready for migration logic

P1.14 - Implement design variable migration

  • File: optimization_engine/config/migrator.py
  • Action: Migrate bounds[] → bounds.min/max, parameter → expression_name
  • Dependencies: P1.13
  • Acceptance: All DV formats convert correctly

P1.15 - Implement objective/constraint migration

  • File: optimization_engine/config/migrator.py
  • Action: Migrate goal → direction, extraction configs to new format
  • Dependencies: P1.14
  • Acceptance: Objectives and constraints convert correctly

P1.16 - Implement full config migration

  • File: optimization_engine/config/migrator.py
  • Action: Complete migration including canvas positions, extractors inference
  • Dependencies: P1.15
  • Acceptance: Can migrate any existing optimization_config.json to AtomizerSpec

P1.17 - Create migration CLI tool

  • File: tools/migrate_to_spec_v2.py
  • Action: CLI for batch migration with dry-run support
  • Dependencies: P1.16
  • Acceptance: python tools/migrate_to_spec_v2.py --dry-run studies/*

PHASE 2: Frontend Integration

P2.1 - Create useSpecStore hook

  • File: atomizer-dashboard/frontend/src/hooks/useSpecStore.ts
  • Action: Zustand store for AtomizerSpec state management
  • Dependencies: P1.1
  • Acceptance: Store holds spec, provides typed accessors

P2.2 - Add spec loading to useSpecStore

  • File: atomizer-dashboard/frontend/src/hooks/useSpecStore.ts
  • Action: Implement loadSpec(studyId) fetching from API
  • Dependencies: P2.1, P1.9
  • Acceptance: Loads spec from backend, updates store

P2.3 - Add spec modification to useSpecStore

  • File: atomizer-dashboard/frontend/src/hooks/useSpecStore.ts
  • Action: Implement patchSpec(), addNode(), removeNode() calling API
  • Dependencies: P2.2, P1.11
  • Acceptance: Modifications persist to backend

P2.4 - Add optimistic updates to useSpecStore

  • File: atomizer-dashboard/frontend/src/hooks/useSpecStore.ts
  • Action: Update local state immediately, rollback on error
  • Dependencies: P2.3
  • Acceptance: UI feels instant, handles errors gracefully

P2.5 - Create specToNodes converter

  • File: atomizer-dashboard/frontend/src/lib/spec/converter.ts
  • Action: Convert AtomizerSpec to ReactFlow nodes array
  • Reference: Design in UNIFIED_CONFIGURATION_ARCHITECTURE.md Section 5.2
  • Dependencies: P1.1
  • Acceptance: All node types render correctly

P2.6 - Create specToEdges converter

  • File: atomizer-dashboard/frontend/src/lib/spec/converter.ts
  • Action: Convert spec.canvas.edges to ReactFlow edges
  • Dependencies: P2.5
  • Acceptance: All connections render correctly

P2.7 - Create SpecRenderer component

  • File: atomizer-dashboard/frontend/src/components/canvas/SpecRenderer.tsx
  • Action: ReactFlow component that renders from useSpecStore
  • Dependencies: P2.5, P2.6, P2.4
  • Acceptance: Canvas displays spec correctly

P2.8 - Wire node editing to spec updates

  • File: atomizer-dashboard/frontend/src/components/canvas/SpecRenderer.tsx
  • Action: Node data changes call useSpecStore.patchSpec()
  • Dependencies: P2.7
  • Acceptance: Editing node properties persists to spec

P2.9 - Wire node position to spec updates

  • File: atomizer-dashboard/frontend/src/components/canvas/SpecRenderer.tsx
  • Action: Drag-drop updates canvas_position in spec
  • Dependencies: P2.8
  • Acceptance: Layout persists across reloads

P2.10 - Update node panels for full spec fields

  • Files: atomizer-dashboard/frontend/src/components/canvas/panels/*.tsx
  • Action: Update all node config panels to show/edit full spec fields
  • Dependencies: P2.8
  • Acceptance: All spec fields are editable in UI

P2.11 - Create WebSocket connection hook

  • File: atomizer-dashboard/frontend/src/hooks/useSpecSync.ts
  • Action: WebSocket connection to /api/studies/{id}/sync
  • Dependencies: P2.4
  • Acceptance: Connects, handles reconnection

P2.12 - Create WebSocket backend endpoint

  • File: atomizer-dashboard/backend/api/routes/spec.py
  • Action: WebSocket endpoint for spec sync
  • Dependencies: P1.12
  • Acceptance: Accepts connections, tracks subscribers

P2.13 - Implement spec_updated broadcast

  • File: atomizer-dashboard/backend/api/services/spec_manager.py
  • Action: SpecManager broadcasts to all subscribers on save
  • Dependencies: P2.12
  • Acceptance: All connected clients receive updates

P2.14 - Handle spec_updated in frontend

  • File: atomizer-dashboard/frontend/src/hooks/useSpecSync.ts
  • Action: On spec_updated message, refresh spec from store
  • Dependencies: P2.11, P2.13
  • Acceptance: Changes from other clients appear in real-time

P2.15 - Add conflict detection

  • File: atomizer-dashboard/frontend/src/hooks/useSpecStore.ts
  • Action: Compare hashes, warn on conflict, offer merge/overwrite
  • Dependencies: P2.14
  • Acceptance: Concurrent edits don't silently overwrite

P2.16 - Replace CanvasView with SpecRenderer

  • File: atomizer-dashboard/frontend/src/pages/CanvasView.tsx
  • Action: Switch from useCanvasStore to useSpecStore + SpecRenderer
  • Dependencies: P2.10, P2.15
  • Acceptance: Canvas page uses new spec-based system

PHASE 3: Claude Integration

P3.1 - Create spec_get MCP tool

  • File: mcp-server/atomizer-tools/src/tools/spec_tools.ts
  • Action: Tool to retrieve full AtomizerSpec
  • Reference: Design in UNIFIED_CONFIGURATION_ARCHITECTURE.md Section 5.3
  • Dependencies: P1.9
  • Acceptance: Claude can read spec via MCP

P3.2 - Create spec_modify MCP tool

  • File: mcp-server/atomizer-tools/src/tools/spec_tools.ts
  • Action: Tool to apply modifications (set, add, remove operations)
  • Dependencies: P3.1, P1.11
  • Acceptance: Claude can modify spec fields

P3.3 - Create spec_add_node MCP tool

  • File: mcp-server/atomizer-tools/src/tools/spec_tools.ts
  • Action: Tool to add design vars, extractors, objectives, constraints
  • Dependencies: P3.2
  • Acceptance: Claude can add nodes to canvas

P3.4 - Create spec_remove_node MCP tool

  • File: mcp-server/atomizer-tools/src/tools/spec_tools.ts
  • Action: Tool to remove nodes (with edge cleanup)
  • Dependencies: P3.3
  • Acceptance: Claude can remove nodes

P3.5 - Create spec_validate MCP tool

  • File: mcp-server/atomizer-tools/src/tools/spec_tools.ts
  • Action: Tool to validate spec and return report
  • Dependencies: P1.12
  • Acceptance: Claude can check spec validity

P3.6 - Create spec_add_custom_extractor MCP tool

  • File: mcp-server/atomizer-tools/src/tools/spec_tools.ts
  • Action: Tool to add custom Python functions as extractors
  • Dependencies: P1.7, P3.3
  • Acceptance: Claude can add custom extraction logic

P3.7 - Register spec tools in MCP server

  • File: mcp-server/atomizer-tools/src/index.ts
  • Action: Import and register all spec_* tools
  • Dependencies: P3.1-P3.6
  • Acceptance: Tools appear in MCP tool list

P3.8 - Update ContextBuilder for spec awareness

  • File: atomizer-dashboard/backend/api/services/context_builder.py
  • Action: Include spec summary in Claude context, mention spec tools
  • Dependencies: P3.7
  • Acceptance: Claude knows about spec tools in context

P3.9 - Create custom extractor runtime loader

  • File: optimization_engine/extractors/custom_loader.py
  • Action: Dynamically load and execute custom functions from spec
  • Dependencies: P1.7
  • Acceptance: Custom functions execute during optimization

P3.10 - Integrate custom extractors into optimization runner

  • File: optimization_engine/core/runner.py
  • Action: Check spec for custom extractors, load and use them
  • Dependencies: P3.9
  • Acceptance: Optimization uses custom extractors defined in spec

P3.11 - Add custom extractor node type to canvas

  • File: atomizer-dashboard/frontend/src/components/canvas/nodes/CustomExtractorNode.tsx
  • Action: New node type showing custom function with code preview
  • Dependencies: P2.10
  • Acceptance: Custom extractors display distinctly in canvas

P3.12 - Add code editor for custom extractors

  • File: atomizer-dashboard/frontend/src/components/canvas/panels/CustomExtractorPanel.tsx
  • Action: Monaco editor for viewing/editing custom Python code
  • Dependencies: P3.11
  • Acceptance: Users can view and edit custom function code

P3.13 - Create spec_create_from_description MCP tool

  • File: mcp-server/atomizer-tools/src/tools/spec_tools.ts
  • Action: Tool to create new spec from natural language + model path
  • Dependencies: P3.6, P1.16
  • Acceptance: Claude can create studies from descriptions

P3.14 - Update Claude prompts for spec workflow

  • File: atomizer-dashboard/backend/api/services/context_builder.py
  • Action: Update system prompts to guide Claude on using spec tools
  • Dependencies: P3.13
  • Acceptance: Claude naturally uses spec tools for modifications

PHASE 4: Polish & Testing

P4.1 - Migrate m1_mirror studies

  • Action: Run migration on all m1_mirror_* studies
  • Dependencies: P1.17
  • Acceptance: All studies have valid atomizer_spec.json

P4.2 - Migrate drone_gimbal study

  • Action: Run migration on drone_gimbal study
  • Dependencies: P1.17
  • Acceptance: Study has valid atomizer_spec.json

P4.3 - Migrate all remaining studies

  • Action: Run migration on all studies in studies/
  • Dependencies: P4.1, P4.2
  • Acceptance: All studies migrated, validated

P4.4 - Create spec unit tests

  • File: tests/test_spec_manager.py
  • Action: Unit tests for SpecManager operations
  • Dependencies: P1.7
  • Acceptance: All SpecManager methods tested

P4.5 - Create spec API integration tests

  • File: tests/test_spec_api.py
  • Action: Integration tests for REST endpoints
  • Dependencies: P1.12
  • Acceptance: All endpoints tested

P4.6 - Create migration tests

  • File: tests/test_migrator.py
  • Action: Test migration with various config formats
  • Dependencies: P1.16
  • Acceptance: All config variants migrate correctly

P4.7 - Create frontend component tests

  • File: atomizer-dashboard/frontend/src/__tests__/SpecRenderer.test.tsx
  • Action: Test SpecRenderer with various specs
  • Dependencies: P2.16
  • Acceptance: Canvas renders correctly for all spec types

P4.8 - Create WebSocket sync tests

  • File: tests/test_spec_sync.py
  • Action: Test real-time sync between multiple clients
  • Dependencies: P2.15
  • Acceptance: Changes propagate correctly

P4.9 - Create MCP tools tests

  • File: mcp-server/atomizer-tools/src/__tests__/spec_tools.test.ts
  • Action: Test all spec_* MCP tools
  • Dependencies: P3.7
  • Acceptance: All tools work correctly

P4.10 - End-to-end testing

  • Action: Full workflow test: create study in canvas, modify via Claude, run optimization
  • Dependencies: P4.1-P4.9
  • Acceptance: Complete workflow works

P4.11 - Update documentation

  • Files: docs/04_USER_GUIDES/CANVAS.md, docs/04_USER_GUIDES/DASHBOARD.md
  • Action: Document new spec-based workflow
  • Dependencies: P4.10
  • Acceptance: Documentation reflects new system

P4.12 - Update CLAUDE.md

  • File: CLAUDE.md
  • Action: Add spec tools documentation, update context loading
  • Dependencies: P4.11
  • Acceptance: Claude Code sessions know about spec system

File Summary

New Files to Create

File Phase Purpose
atomizer-dashboard/frontend/src/types/atomizer-spec.ts P1 TypeScript types
optimization_engine/config/spec_models.py P1 Pydantic models
optimization_engine/config/spec_validator.py P1 Validation logic
atomizer-dashboard/backend/api/services/spec_manager.py P1 Core spec service
atomizer-dashboard/backend/api/routes/spec.py P1 REST endpoints
optimization_engine/config/migrator.py P1 Config migration
tools/migrate_to_spec_v2.py P1 Migration CLI
atomizer-dashboard/frontend/src/hooks/useSpecStore.ts P2 Spec state store
atomizer-dashboard/frontend/src/hooks/useSpecSync.ts P2 WebSocket sync
atomizer-dashboard/frontend/src/lib/spec/converter.ts P2 Spec ↔ ReactFlow
atomizer-dashboard/frontend/src/components/canvas/SpecRenderer.tsx P2 New canvas component
mcp-server/atomizer-tools/src/tools/spec_tools.ts P3 MCP tools
optimization_engine/extractors/custom_loader.py P3 Custom function loader
atomizer-dashboard/frontend/src/components/canvas/nodes/CustomExtractorNode.tsx P3 Custom node type
atomizer-dashboard/frontend/src/components/canvas/panels/CustomExtractorPanel.tsx P3 Code editor panel

Files to Modify

File Phase Changes
atomizer-dashboard/backend/api/main.py P1 Mount spec router
mcp-server/atomizer-tools/src/index.ts P3 Register spec tools
atomizer-dashboard/backend/api/services/context_builder.py P3 Update Claude context
optimization_engine/core/runner.py P3 Custom extractor support
atomizer-dashboard/frontend/src/pages/CanvasView.tsx P2 Use SpecRenderer
CLAUDE.md P4 Document spec system

Success Criteria

Phase 1 Complete When:

  • SpecManager can load/save/validate specs
  • All REST endpoints return correct responses
  • Migration tool converts existing configs

Phase 2 Complete When:

  • Canvas renders from AtomizerSpec
  • Edits persist to spec file
  • WebSocket sync works between clients

Phase 3 Complete When:

  • Claude can read and modify specs via MCP
  • Custom extractors work in optimization
  • Claude can create studies from descriptions

Phase 4 Complete When:

  • All existing studies migrated
  • All tests pass
  • Documentation updated

Risk Mitigation

Risk Mitigation
Existing studies break Migration tool with dry-run, keep old configs as backup
WebSocket complexity Start with polling, add WebSocket as enhancement
Custom code security Sandbox execution, syntax validation, no imports
Performance with large specs Lazy loading, incremental updates

Quick Reference

Master Design: docs/plans/UNIFIED_CONFIGURATION_ARCHITECTURE.md Schema: optimization_engine/schemas/atomizer_spec_v2.json This Plan: docs/plans/UNIFIED_CONFIG_EXECUTION_PLAN.md

Execute tasks in order. Each task has clear acceptance criteria. Reference the master design document for detailed specifications.

Reference in New Issue View Git Blame Copy Permalink