Antoine/Atomizer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
89694088a2c72db71cc68ecc6008712e1297949c
Atomizer/DEVELOPMENT.md
Anto01 d228ccec66 refactor: Archive experimental LLM features for MVP stability (Phase 1.1) 
Moved experimental LLM integration code to optimization_engine/future/:
- llm_optimization_runner.py - Runtime LLM API runner
- llm_workflow_analyzer.py - Workflow analysis
- inline_code_generator.py - Auto-generate calculations
- hook_generator.py - Auto-generate hooks
- report_generator.py - LLM report generation
- extractor_orchestrator.py - Extractor orchestration

Added comprehensive optimization_engine/future/README.md explaining:
- MVP LLM strategy (Claude Code skills, not runtime LLM)
- Why files were archived
- When to revisit post-MVP
- Production architecture reference

Production runner confirmed: optimization_engine/runner.py is sole active runner.

This establishes clear separation between:
- Production code (stable, no runtime LLM dependencies)
- Experimental code (archived for post-MVP exploration)

Part of Phase 1: Core Stabilization & Organization for MVP

Generated with Claude Code

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-24 09:12:36 -05:00

620 lines
22 KiB
Markdown
Raw Blame History

# Atomizer Development Guide
**Last Updated**: 2025-11-21
**Current Phase**: Phase 3.2 - Integration Sprint + Documentation
**Status**: 🟢 Core Complete (100%) | ✅ Protocols 10/11/13 Active (100%) | 🎯 Dashboard Live (95%) | 📚 Documentation Reorganized
📘 **Quick Links**:
- [Protocol Specifications](docs/PROTOCOLS.md) - All active protocols consolidated
- [Documentation Index](docs/00_INDEX.md) - Complete documentation navigation
- [README](README.md) - Project overview and quick start
---
## Table of Contents
1. [Current Phase](#current-phase)
2. [Completed Features](#completed-features)
3. [Active Development](#active-development)
4. [Known Issues](#known-issues)
5. [Testing Status](#testing-status)
6. [Phase-by-Phase Progress](#phase-by-phase-progress)
---
## Current Phase
### Phase 3.2: Integration Sprint (🎯 TOP PRIORITY)
**Goal**: Connect LLM intelligence components to production workflow
**Timeline**: 2-4 weeks (Started 2025-11-17)
**Status**: LLM components built and tested individually (85% complete). Need to wire them into production runner.
📋 **Detailed Plan**: [docs/PHASE_3_2_INTEGRATION_PLAN.md](docs/PHASE_3_2_INTEGRATION_PLAN.md)
**Critical Path**:
#### Week 1: Make LLM Mode Accessible (16 hours)
- [ ] **1.1** Create unified entry point `optimization_engine/run_optimization.py` (4h)
- Add `--llm` flag for natural language mode
- Add `--request` parameter for natural language input
- Support both LLM and traditional JSON modes
- Preserve backward compatibility
- [ ] **1.2** Wire LLMOptimizationRunner to production (8h)
- Connect LLMWorkflowAnalyzer to entry point
- Bridge LLMOptimizationRunner → OptimizationRunner
- Pass model updater and simulation runner callables
- Integrate with existing hook system
- [ ] **1.3** Create minimal example (2h)
- Create `examples/llm_mode_demo.py`
- Show natural language → optimization results
- Compare traditional (100 lines) vs LLM (3 lines)
- [ ] **1.4** End-to-end integration test (2h)
- Test with simple_beam_optimization study
- Verify extractors generated correctly
- Validate output matches manual mode
#### Week 2: Robustness & Safety (16 hours)
- [ ] **2.1** Code validation pipeline (6h)
- Create `optimization_engine/code_validator.py`
- Implement syntax validation (ast.parse)
- Implement security scanning (whitelist imports)
- Implement test execution on example OP2
- Add retry with LLM feedback on failure
- [ ] **2.2** Graceful fallback mechanisms (4h)
- Wrap all LLM calls in try/except
- Provide clear error messages
- Offer fallback to manual mode
- Never crash on LLM failure
- [ ] **2.3** LLM audit trail (3h)
- Create `optimization_engine/llm_audit.py`
- Log all LLM requests and responses
- Log generated code with prompts
- Create `llm_audit.json` in study output
- [ ] **2.4** Failure scenario testing (3h)
- Test invalid natural language request
- Test LLM unavailable
- Test generated code syntax errors
- Test validation failures
#### Week 3: Learning System (12 hours)
- [ ] **3.1** Knowledge base implementation (4h)
- Create `optimization_engine/knowledge_base.py`
- Implement `save_session()` - Save successful workflows
- Implement `search_templates()` - Find similar patterns
- Add confidence scoring
- [ ] **3.2** Template extraction (4h)
- Extract reusable patterns from generated code
- Parameterize variable parts
- Save templates with usage examples
- Implement template application to new requests
- [ ] **3.3** ResearchAgent integration (4h)
- Complete ResearchAgent implementation
- Integrate into ExtractorOrchestrator error handling
- Add user example collection workflow
- Save learned knowledge to knowledge base
#### Week 4: Documentation & Discoverability (8 hours)
- [ ] **4.1** Update README (2h)
- Add "🤖 LLM-Powered Mode" section
- Show example command with natural language
- Link to detailed docs
- [ ] **4.2** Create LLM mode documentation (3h)
- Create `docs/LLM_MODE.md`
- Explain how LLM mode works
- Provide usage examples
- Add troubleshooting guide
- [ ] **4.3** Create demo video/GIF (1h)
- Record terminal session
- Show before/after (100 lines → 3 lines)
- Create animated GIF for README
- [ ] **4.4** Update all planning docs (2h)
- Update DEVELOPMENT.md status
- Update DEVELOPMENT_GUIDANCE.md (80-90% → 90-95%)
- Mark Phase 3.2 as ✅ Complete
---
## Completed Features
### ✅ Live Dashboard System (Completed 2025-11-21)
#### Backend (FastAPI + WebSocket)
- [x] **FastAPI Backend** ([atomizer-dashboard/backend/](atomizer-dashboard/backend/))
- REST API endpoints for study management
- WebSocket streaming with file watching (Watchdog)
- Real-time updates (<100ms latency)
- CORS configured for local development
- [x] **REST API Endpoints** ([backend/api/routes/optimization.py](atomizer-dashboard/backend/api/routes/optimization.py))
- `GET /api/optimization/studies` - List all studies
- `GET /api/optimization/studies/{id}/status` - Get study status
- `GET /api/optimization/studies/{id}/history` - Get trial history
- `GET /api/optimization/studies/{id}/pruning` - Get pruning diagnostics
- [x] **WebSocket Streaming** ([backend/api/websocket/optimization_stream.py](atomizer-dashboard/backend/api/websocket/optimization_stream.py))
- File watching on `optimization_history_incremental.json`
- Real-time trial updates via WebSocket
- Pruning alerts and progress updates
- Automatic observer lifecycle management
#### Frontend (HTML + Chart.js)
- [x] **Enhanced Live Dashboard** ([atomizer-dashboard/dashboard-enhanced.html](atomizer-dashboard/dashboard-enhanced.html))
- Real-time WebSocket updates
- Interactive convergence chart (Chart.js)
- Parameter space scatter plot
- Pruning alerts (toast notifications)
- Data export (JSON/CSV)
- Study auto-discovery and selection
- Metric dashboard (trials, best value, pruned count)
#### React Frontend (In Progress)
- [x] **Project Configuration** ([atomizer-dashboard/frontend/](atomizer-dashboard/frontend/))
- React 18 + Vite 5 + TypeScript 5.2
- TailwindCSS 3.3 for styling
- Recharts 2.10 for charts
- Complete build configuration
- [x] **TypeScript Types** ([frontend/src/types/](atomizer-dashboard/frontend/src/types/))
- Complete type definitions for API data
- WebSocket message types
- Chart data structures
- [x] **Custom Hooks** ([frontend/src/hooks/useWebSocket.ts](atomizer-dashboard/frontend/src/hooks/useWebSocket.ts))
- WebSocket connection management
- Auto-reconnection with exponential backoff
- Type-safe message routing
- [x] **Reusable Components** ([frontend/src/components/](atomizer-dashboard/frontend/src/components/))
- Card, MetricCard, Badge, StudyCard components
- TailwindCSS styling with dark theme
- [ ] **Dashboard Page** (Pending manual completion)
- Need to run `npm install`
- Create main.tsx, App.tsx, Dashboard.tsx
- Integrate Recharts for charts
- Test end-to-end
#### Documentation
- [x] **Dashboard Master Plan** ([docs/DASHBOARD_MASTER_PLAN.md](docs/DASHBOARD_MASTER_PLAN.md))
- Complete 3-page architecture (Configurator, Live Dashboard, Results Viewer)
- Tech stack recommendations
- Implementation phases
- [x] **Implementation Status** ([docs/DASHBOARD_IMPLEMENTATION_STATUS.md](docs/DASHBOARD_IMPLEMENTATION_STATUS.md))
- Current progress tracking
- Testing instructions
- Next steps
- [x] **React Implementation Guide** ([docs/DASHBOARD_REACT_IMPLEMENTATION.md](docs/DASHBOARD_REACT_IMPLEMENTATION.md))
- Complete templates for remaining components
- Recharts integration examples
- Troubleshooting guide
- [x] **Session Summary** ([docs/DASHBOARD_SESSION_SUMMARY.md](docs/DASHBOARD_SESSION_SUMMARY.md))
- Features demonstrated
- How to use the dashboard
- Architecture explanation
### ✅ Phase 1: Plugin System & Infrastructure (Completed 2025-01-16)
#### Core Architecture
- [x] **Hook Manager** ([optimization_engine/plugins/hook_manager.py](optimization_engine/plugins/hook_manager.py))
- Hook registration with priority-based execution
- Auto-discovery from plugin directories
- Context passing to all hooks
- Execution history tracking
- [x] **Lifecycle Hooks**
- `pre_solve`: Execute before solver launch
- `post_solve`: Execute after solve, before extraction
- `post_extraction`: Execute after result extraction
#### Logging Infrastructure
- [x] **Detailed Trial Logs** ([detailed_logger.py](optimization_engine/plugins/pre_solve/detailed_logger.py))
- Per-trial log files in `optimization_results/trial_logs/`
- Complete iteration trace with timestamps
- Design variables, configuration, timeline
- Extracted results and constraint evaluations
- [x] **High-Level Optimization Log** ([optimization_logger.py](optimization_engine/plugins/pre_solve/optimization_logger.py))
- `optimization.log` file tracking overall progress
- Configuration summary header
- Compact START/COMPLETE entries per trial
- Easy to scan format for monitoring
- [x] **Result Appenders**
- [log_solve_complete.py](optimization_engine/plugins/post_solve/log_solve_complete.py) - Appends solve completion to trial logs
- [log_results.py](optimization_engine/plugins/post_extraction/log_results.py) - Appends extracted results to trial logs
- [optimization_logger_results.py](optimization_engine/plugins/post_extraction/optimization_logger_results.py) - Appends results to optimization.log
#### Project Organization
- [x] **Studies Structure** ([studies/](studies/))
- Standardized folder layout with `model/`, `optimization_results/`, `analysis/`
- Comprehensive documentation in [studies/README.md](studies/README.md)
- Example study: [bracket_stress_minimization/](studies/bracket_stress_minimization/)
- Template structure for future studies
- [x] **Path Resolution** ([atomizer_paths.py](atomizer_paths.py))
- Intelligent project root detection using marker files
- Helper functions: `root()`, `optimization_engine()`, `studies()`, `tests()`
- `ensure_imports()` for robust module imports
- Works regardless of script location
#### Testing
- [x] **Hook Validation Test** ([test_hooks_with_bracket.py](tests/test_hooks_with_bracket.py))
- Verifies hook loading and execution
- Tests 3 trials with dummy data
- Checks hook execution history
- [x] **Integration Tests**
- [run_5trial_test.py](tests/run_5trial_test.py) - Quick 5-trial optimization
- [test_journal_optimization.py](tests/test_journal_optimization.py) - Full optimization test
#### Runner Enhancements
- [x] **Context Passing** ([runner.py:332,365,412](optimization_engine/runner.py))
- `output_dir` passed to all hook contexts
- Trial number, design variables, extracted results
- Configuration dictionary available to hooks
### ✅ Core Engine (Pre-Phase 1)
- [x] Optuna integration with TPE sampler
- [x] Multi-objective optimization support
- [x] NX journal execution ([nx_solver.py](optimization_engine/nx_solver.py))
- [x] Expression updates ([nx_updater.py](optimization_engine/nx_updater.py))
- [x] OP2 result extraction (stress, displacement)
- [x] Study management with resume capability
- [x] Web dashboard (real-time monitoring)
- [x] Precision control (4-decimal rounding)
---
## Active Development
### In Progress - Dashboard (High Priority)
- [x] Backend API complete (FastAPI + WebSocket)
- [x] HTML dashboard with Chart.js complete
- [x] React project structure and configuration complete
- [ ] **Complete React frontend** (Awaiting manual npm install)
- [ ] Run `npm install` in frontend directory
- [ ] Create main.tsx and App.tsx
- [ ] Create Dashboard.tsx with Recharts
- [ ] Test end-to-end with live optimization
### Up Next - Dashboard (Next Session)
- [ ] Study Configurator page (React)
- [ ] Results Report Viewer page (React)
- [ ] LLM chat interface integration (future)
- [ ] Docker deployment configuration
### In Progress - Phase 3.2 Integration
- [ ] Feature registry creation (Phase 2, Week 1)
- [ ] Claude skill definition (Phase 2, Week 1)
### Up Next (Phase 2, Week 2)
- [ ] Natural language parser
- [ ] Intent classification system
- [ ] Entity extraction for optimization parameters
- [ ] Conversational workflow manager
### Backlog (Phase 3+)
- [ ] Custom function generator (RSS, weighted objectives)
- [ ] Journal script generator
- [ ] Code validation pipeline
- [ ] Result analyzer with statistical analysis
- [ ] Surrogate quality checker
- [ ] HTML/PDF report generator
---
## Known Issues
### Critical
- None currently
### Minor
- [ ] `.claude/settings.local.json` modified during development (contains user-specific settings)
- [ ] Some old bash background processes still running from previous tests
### Documentation
- [ ] Need to add examples of custom hooks to studies/README.md
- [ ] Missing API documentation for hook_manager methods
- [ ] No developer guide for creating new plugins
---
## Testing Status
### Automated Tests
- ✅ **Hook system** - `test_hooks_with_bracket.py` passing
- ✅ **5-trial integration** - `run_5trial_test.py` working
- ✅ **Full optimization** - `test_journal_optimization.py` functional
- ⏳ **Unit tests** - Need to create for individual modules
- ⏳ **CI/CD pipeline** - Not yet set up
### Manual Testing
- ✅ Bracket optimization (50 trials)
- ✅ Log file generation in correct locations
- ✅ Hook execution at all lifecycle points
- ✅ Path resolution across different script locations
- ✅ **Dashboard backend** - REST API and WebSocket tested successfully
- ✅ **HTML dashboard** - Live updates working with Chart.js
- ⏳ **React dashboard** - Pending npm install and completion
- ⏳ Resume functionality with config validation
### Test Coverage
- Hook manager: ~80% (core functionality tested)
- Logging plugins: 100% (tested via integration tests)
- Path resolution: 100% (tested in all scripts)
- Result extractors: ~70% (basic tests exist)
- **Dashboard backend**: ~90% (REST endpoints and WebSocket tested)
- **Dashboard frontend**: ~60% (HTML version tested, React pending)
- Overall: ~65% estimated
---
## Phase-by-Phase Progress
### Phase 1: Plugin System ✅ (100% Complete)
**Completed** (2025-01-16):
- [x] Hook system for optimization lifecycle
- [x] Plugin auto-discovery and registration
- [x] Hook manager with priority-based execution
- [x] Detailed per-trial logs (`trial_logs/`)
- [x] High-level optimization log (`optimization.log`)
- [x] Context passing system for hooks
- [x] Studies folder structure
- [x] Comprehensive studies documentation
- [x] Model file organization (`model/` folder)
- [x] Intelligent path resolution
- [x] Test suite for hook system
**Deferred to Future Phases**:
- Feature registry → Phase 2 (with LLM interface)
- `pre_mesh` and `post_mesh` hooks → Future (not needed for current workflow)
- Custom objective/constraint registration → Phase 3 (Code Generation)
---
### Phase 2: LLM Integration 🟡 (0% Complete)
**Target**: 2 weeks (Started 2025-01-16)
#### Week 1 Todos (Feature Registry & Claude Skill)
- [ ] Create `optimization_engine/feature_registry.json`
- [ ] Extract all current capabilities
- [ ] Draft `.claude/skills/atomizer.md`
- [ ] Test LLM's ability to navigate codebase
#### Week 2 Todos (Natural Language Interface)
- [ ] Implement intent classifier
- [ ] Build entity extractor
- [ ] Create workflow manager
- [ ] Test end-to-end: "Create a stress minimization study"
**Success Criteria**:
- [ ] LLM can create optimization from natural language in <5 turns
- [ ] 90% of user requests understood correctly
- [ ] Zero manual JSON editing required
---
### Phase 3: Code Generation ⏳ (Not Started)
**Target**: 3 weeks
**Key Deliverables**:
- [ ] Custom function generator
- [ ] RSS (Root Sum Square) template
- [ ] Weighted objectives template
- [ ] Custom constraints template
- [ ] Journal script generator
- [ ] Code validation pipeline
- [ ] Safe execution environment
**Success Criteria**:
- [ ] LLM generates 10+ custom functions with zero errors
- [ ] All generated code passes safety validation
- [ ] Users save 50% time vs. manual coding
---
### Phase 4: Analysis & Decision Support ⏳ (Not Started)
**Target**: 3 weeks
**Key Deliverables**:
- [ ] Result analyzer (convergence, sensitivity, outliers)
- [ ] Surrogate model quality checker (R², CV score, confidence intervals)
- [ ] Decision assistant (trade-offs, what-if analysis, recommendations)
**Success Criteria**:
- [ ] Surrogate quality detection 95% accurate
- [ ] Recommendations lead to 30% faster convergence
- [ ] Users report higher confidence in results
---
### Phase 5: Automated Reporting ⏳ (Not Started)
**Target**: 2 weeks
**Key Deliverables**:
- [ ] Report generator with Jinja2 templates
- [ ] Multi-format export (HTML, PDF, Markdown, JSON)
- [ ] LLM-written narrative explanations
**Success Criteria**:
- [ ] Reports generated in <30 seconds
- [ ] Narrative quality rated 4/5 by engineers
- [ ] 80% of reports used without manual editing
---
### Phase 6: NX MCP Enhancement ⏳ (Not Started)
**Target**: 4 weeks
**Key Deliverables**:
- [ ] NX documentation MCP server
- [ ] Advanced NX operations library
- [ ] Feature bank with 50+ pre-built operations
**Success Criteria**:
- [ ] NX MCP answers 95% of API questions correctly
- [ ] Feature bank covers 80% of common workflows
- [ ] Users write 50% less manual journal code
---
### Phase 7: Self-Improving System ⏳ (Not Started)
**Target**: 4 weeks
**Key Deliverables**:
- [ ] Feature learning system
- [ ] Best practices database
- [ ] Continuous documentation generation
**Success Criteria**:
- [ ] 20+ user-contributed features in library
- [ ] Pattern recognition identifies 10+ best practices
- [ ] Documentation auto-updates with zero manual effort
---
## Development Commands
### Running Dashboard
```bash
# Start backend server
cd atomizer-dashboard/backend
python -m uvicorn api.main:app --reload --host 0.0.0.0 --port 8000
# Access HTML dashboard (current)
# Open browser: http://localhost:8000
# Start React frontend (when ready)
cd atomizer-dashboard/frontend
npm install # First time only
npm run dev # Starts on http://localhost:3000
```
### Running Tests
```bash
# Hook validation (3 trials, fast)
python tests/test_hooks_with_bracket.py
# Quick integration test (5 trials)
python tests/run_5trial_test.py
# Full optimization test
python tests/test_journal_optimization.py
```
### Code Quality
```bash
# Run linter (when available)
# pylint optimization_engine/
# Run type checker (when available)
# mypy optimization_engine/
# Run all tests (when test suite is complete)
# pytest tests/
```
### Git Workflow
```bash
# Stage all changes
git add .
# Commit with conventional commits format
git commit -m "feat: description" # New feature
git commit -m "fix: description" # Bug fix
git commit -m "docs: description" # Documentation
git commit -m "test: description" # Tests
git commit -m "refactor: description" # Code refactoring
# Push to GitHub
git push origin main
```
---
## Documentation
### For Developers
- [DEVELOPMENT_ROADMAP.md](DEVELOPMENT_ROADMAP.md) - Strategic vision and phases
- [studies/README.md](studies/README.md) - Studies folder organization
- [CHANGELOG.md](CHANGELOG.md) - Version history
### Dashboard Documentation
- [docs/DASHBOARD_MASTER_PLAN.md](docs/DASHBOARD_MASTER_PLAN.md) - Complete architecture blueprint
- [docs/DASHBOARD_IMPLEMENTATION_STATUS.md](docs/DASHBOARD_IMPLEMENTATION_STATUS.md) - Current progress
- [docs/DASHBOARD_REACT_IMPLEMENTATION.md](docs/DASHBOARD_REACT_IMPLEMENTATION.md) - React implementation guide
- [docs/DASHBOARD_SESSION_SUMMARY.md](docs/DASHBOARD_SESSION_SUMMARY.md) - Session summary
- [atomizer-dashboard/README.md](atomizer-dashboard/README.md) - Dashboard quick start
- [atomizer-dashboard/backend/README.md](atomizer-dashboard/backend/README.md) - Backend API docs
- [atomizer-dashboard/frontend/README.md](atomizer-dashboard/frontend/README.md) - Frontend setup guide
### For Users
- [README.md](README.md) - Project overview and quick start
- [docs/INDEX.md](docs/INDEX.md) - Complete documentation index
- [docs/](docs/) - Additional documentation
---
## Notes
### Architecture Decisions
- **Hook system**: Chose priority-based execution to allow precise control of plugin order
- **Path resolution**: Used marker files instead of environment variables for simplicity
- **Logging**: Two-tier system (detailed trial logs + high-level optimization.log) for different use cases
### Performance Considerations
- Hook execution adds <1s overhead per trial (acceptable for FEA simulations)
- Path resolution caching could improve startup time (future optimization)
- Log file sizes grow linearly with trials (~10KB per trial)
### Future Considerations
- Consider moving to structured logging (JSON) for easier parsing
- May need database for storing hook execution history (currently in-memory)
- Dashboard integration will require WebSocket for real-time log streaming
---
**Last Updated**: 2025-11-21
**Maintained by**: Antoine Polvé (antoine@atomaste.com)
**Repository**: [GitHub - Atomizer](https://github.com/yourusername/Atomizer)
---
## Recent Updates (November 21, 2025)
### Dashboard System Implementation ✅
- **Backend**: FastAPI + WebSocket with real-time file watching complete
- **HTML Dashboard**: Functional dashboard with Chart.js, data export, pruning alerts
- **React Setup**: Complete project configuration, types, hooks, components
- **Documentation**: 5 comprehensive markdown documents covering architecture, implementation, and usage
### Next Immediate Steps
1. Run `npm install` in `atomizer-dashboard/frontend`
2. Create `main.tsx`, `App.tsx`, and `Dashboard.tsx` using provided templates
3. Test React dashboard with live optimization
4. Build Study Configurator page (next major feature)
Reference in New Issue View Git Blame Copy Permalink