Atomizer/docs/PHASE_2_5_INTELLIGENT_GAP_DETECTION.md

Phase 2.5: Intelligent Codebase-Aware Gap Detection

Problem Statement

The current Research Agent uses dumb keyword matching and doesn't understand what already exists in the Atomizer codebase. When a user asks:

"I want to evaluate strain on a part with sol101 and optimize this (minimize) using iterations and optuna to lower it varying all my geometry parameters that contains v_ in its expression"

Current (Wrong) Behavior:

• Detects keyword "geometry"
• Asks user for geometry examples
• Completely misses the actual request

Expected (Correct) Behavior:

Analyzing your optimization request...

Workflow Components Identified:
---------------------------------
1. Run SOL101 analysis                    [KNOWN - nx_solver.py]
2. Extract geometry parameters (v_ prefix) [KNOWN - expression system]
3. Update parameter values                 [KNOWN - parameter updater]
4. Optuna optimization loop               [KNOWN - optimization engine]
5. Extract strain from OP2                [MISSING - not implemented]
6. Minimize strain objective              [SIMPLE - max(strain values)]

Knowledge Gap Analysis:
-----------------------
HAVE:  - OP2 displacement extraction (op2_extractor_example.py)
HAVE:  - OP2 stress extraction (op2_extractor_example.py)
MISSING: - OP2 strain extraction

Research Needed:
----------------
Only need to learn: How to extract strain data from Nastran OP2 files using pyNastran

Would you like me to:
1. Search pyNastran documentation for strain extraction
2. Look for strain extraction examples in op2_extractor_example.py pattern
3. Ask you for an example of strain extraction code

Solution Architecture

1. Codebase Capability Analyzer

Scan Atomizer to build capability index:

class CodebaseCapabilityAnalyzer:
    """Analyzes what Atomizer can already do."""

    def analyze_codebase(self) -> Dict[str, Any]:
        """
        Returns:
        {
            'optimization': {
                'optuna_integration': True,
                'parameter_updating': True,
                'expression_parsing': True
            },
            'simulation': {
                'nx_solver': True,
                'sol101': True,
                'sol103': False
            },
            'result_extraction': {
                'displacement': True,
                'stress': True,
                'strain': False,  # <-- THE GAP!
                'modal': False
            }
        }
        """

2. Workflow Decomposer

Break user request into atomic steps:

class WorkflowDecomposer:
    """Breaks complex requests into atomic workflow steps."""

    def decompose(self, user_request: str) -> List[WorkflowStep]:
        """
        Input: "minimize strain using SOL101 and optuna varying v_ params"

        Output:
        [
            WorkflowStep("identify_parameters", domain="geometry", params={"filter": "v_"}),
            WorkflowStep("update_parameters", domain="geometry", params={"values": "from_optuna"}),
            WorkflowStep("run_analysis", domain="simulation", params={"solver": "SOL101"}),
            WorkflowStep("extract_strain", domain="results", params={"metric": "max_strain"}),
            WorkflowStep("optimize", domain="optimization", params={"objective": "minimize", "algorithm": "optuna"})
        ]
        """

3. Capability Matcher

Match workflow steps to existing capabilities:

class CapabilityMatcher:
    """Matches required workflow steps to existing capabilities."""

    def match(self, workflow_steps, capabilities) -> CapabilityMatch:
        """
        Returns:
        {
            'known_steps': [
                {'step': 'identify_parameters', 'implementation': 'expression_parser.py'},
                {'step': 'update_parameters', 'implementation': 'parameter_updater.py'},
                {'step': 'run_analysis', 'implementation': 'nx_solver.py'},
                {'step': 'optimize', 'implementation': 'optuna_optimizer.py'}
            ],
            'unknown_steps': [
                {'step': 'extract_strain', 'similar_to': 'extract_stress', 'gap': 'strain_from_op2'}
            ],
            'confidence': 0.80  # 4/5 steps known
        }
        """

4. Targeted Research Planner

Create research plan ONLY for missing pieces:

class TargetedResearchPlanner:
    """Creates research plan focused on actual gaps."""

    def plan(self, unknown_steps) -> ResearchPlan:
        """
        For gap='strain_from_op2', similar_to='stress_from_op2':

        Research Plan:
        1. Read existing op2_extractor_example.py to understand pattern
        2. Search pyNastran docs for strain extraction API
        3. If not found, ask user for strain extraction example
        4. Generate extract_strain() function following same pattern as extract_stress()
        """

Implementation Plan

Week 1: Capability Analysis

• Map existing Atomizer capabilities
• Build capability index from code
• Create capability query system

Week 2: Workflow Decomposition

• Build workflow step extractor
• Create domain classifier
• Implement step-to-capability matcher

Week 3: Intelligent Gap Detection

• Integrate all components
• Test with strain optimization request
• Verify correct gap identification

Success Criteria

Test Input: "minimize strain using SOL101 and optuna varying v_ parameters"

Expected Output:

Request Analysis Complete
-------------------------

Known Capabilities (80%):
- Parameter identification (v_ prefix filter)
- Parameter updating
- SOL101 simulation execution
- Optuna optimization loop

Missing Capability (20%):
- Strain extraction from OP2 files

Recommendation:
The only missing piece is extracting strain data from Nastran OP2 output files.
I found a similar implementation for stress extraction in op2_extractor_example.py.

Would you like me to:
1. Research pyNastran strain extraction API
2. Generate extract_max_strain() function following the stress extraction pattern
3. Integrate into your optimization workflow

Research needed: Minimal (1 function, ~50 lines of code)

Benefits

1. Accurate Gap Detection: Only identifies actual missing capabilities
2. Minimal Research: Focuses effort on real unknowns
3. Leverages Existing Code: Understands what you already have
4. Better UX: Clear explanation of what's known vs unknown
5. Faster Iterations: Doesn't waste time on known capabilities

Current Status

• Problem identified
• Solution architecture designed
• Implementation completed
• All tests passing

Implementation Summary

Phase 2.5 has been successfully implemented with 4 core components:

1. CodebaseCapabilityAnalyzer (codebase_analyzer.py)

   • Scans Atomizer codebase for existing capabilities
   • Identifies what's implemented vs missing
   • Finds similar capabilities for pattern reuse

2. WorkflowDecomposer (workflow_decomposer.py)

   • Breaks user requests into atomic workflow steps
   • Extracts parameters from natural language
   • Classifies steps by domain

3. CapabilityMatcher (capability_matcher.py)

   • Matches workflow steps to existing code
   • Identifies actual knowledge gaps
   • Calculates confidence based on pattern similarity

4. TargetedResearchPlanner (targeted_research_planner.py)

   • Creates focused research plans
   • Leverages similar capabilities when available
   • Prioritizes research sources

Test Results

Run the comprehensive test:

python tests/test_phase_2_5_intelligent_gap_detection.py

Test Output (strain optimization request):

• Workflow: 5 steps identified
• Known: 4/5 steps (80% coverage)
• Missing: Only strain extraction
• Similar: Can adapt from displacement/stress
• Overall confidence: 90%
• Research plan: 4 focused steps

Next Steps

1. Integrate Phase 2.5 with existing Research Agent
2. Update interactive session to use new gap detection
3. Test with diverse optimization requests
4. Build MCP integration for documentation search