Atomizer/knowledge_base/README.md

# Knowledge Base

> Persistent storage of learned patterns, schemas, and research findings for autonomous feature generation

**Purpose**: Enable Atomizer to learn from user examples, documentation, and research sessions, building a growing repository of knowledge that makes future feature generation faster and more accurate.

---

## Folder Structure

```
knowledge_base/
├── nx_research/           # NX-specific learned patterns and schemas
│   ├── material_xml_schema.md
│   ├── journal_script_patterns.md
│   ├── load_bc_patterns.md
│   └── best_practices.md
├── research_sessions/     # Detailed logs of each research session
│   └── [YYYY-MM-DD]_[topic]/
│       ├── user_question.txt        # Original user request
│       ├── sources_consulted.txt    # Where information came from
│       ├── findings.md              # What was learned
│       └── decision_rationale.md    # Why this approach was chosen
└── templates/             # Reusable code patterns learned from research
    ├── xml_generation_template.py
    ├── journal_script_template.py
    └── custom_extractor_template.py
```

---

## Research Workflow

### 1. Knowledge Gap Detection
When an LLM encounters a request it cannot fulfill:
```python
# Search feature registry
gap = research_agent.identify_knowledge_gap("Create NX material XML")
# Returns: {'missing_features': ['material_generator'], 'confidence': 0.2}
```

### 2. Research Plan Creation
Prioritize sources: **User Examples** > **NX MCP** > **Web Documentation**
```python
plan = research_agent.create_research_plan(gap)
# Returns: [
#   {'step': 1, 'action': 'ask_user_for_example', 'priority': 'high'},
#   {'step': 2, 'action': 'query_nx_mcp', 'priority': 'medium'},
#   {'step': 3, 'action': 'web_search', 'query': 'NX material XML', 'priority': 'low'}
# ]
```

### 3. Interactive Research
Ask user first for concrete examples:
```
LLM: "I don't have a feature for NX material XMLs yet.
     Do you have an example .xml file I can learn from?"

User: [uploads steel_material.xml]

LLM: [Analyzes structure, extracts schema, identifies patterns]
```

### 4. Knowledge Synthesis
Combine findings from multiple sources:
```python
findings = {
    'user_example': 'steel_material.xml',
    'nx_mcp_docs': 'PhysicalMaterial schema',
    'web_docs': 'NXOpen material properties API'
}

knowledge = research_agent.synthesize_knowledge(findings)
# Returns: {
#   'schema': {...},
#   'patterns': [...],
#   'confidence': 0.85
# }
```

### 5. Feature Generation
Create new feature following learned patterns:
```python
feature_spec = research_agent.design_feature(knowledge)
# Generates:
# - optimization_engine/custom_functions/nx_material_generator.py
# - knowledge_base/nx_research/material_xml_schema.md
# - knowledge_base/templates/xml_generation_template.py
```

### 6. Documentation & Integration
Save research session and update registries:
```python
research_agent.document_session(
    topic='nx_materials',
    findings=findings,
    generated_files=['nx_material_generator.py'],
    confidence=0.85
)
# Creates: knowledge_base/research_sessions/2025-01-16_nx_materials/
```

---

## Confidence Tracking

Knowledge is tagged with confidence scores based on source:

| Source | Confidence | Reliability |
|--------|-----------|-------------|
| User-validated example | 0.95 | Highest - user confirmed it works |
| NX MCP (official docs) | 0.85 | High - authoritative source |
| NXOpenTSE (community) | 0.70 | Medium - community-verified |
| Web search (generic) | 0.50 | Low - needs validation |

**Rule**: Only generate code if combined confidence > 0.70

---

## Knowledge Retrieval

Before starting new research, search existing knowledge base:

```python
# Check if we already know about this topic
existing = research_agent.search_knowledge_base("material XML")
if existing and existing['confidence'] > 0.8:
    # Use existing template
    template = load_template(existing['template_path'])
else:
    # Start new research session
    research_agent.execute_research(topic="material XML")
```

---

## Best Practices

### For NX Research
- Always save journal script patterns with comments explaining NXOpen API calls
- Document version compatibility (e.g., "Tested on NX 2412")
- Include error handling patterns (common NX exceptions)
- Store unit conversion patterns (mm/m, MPa/Pa, etc.)

### For Research Sessions
- Save user's original question verbatim
- Document ALL sources consulted (with URLs or file paths)
- Explain decision rationale (why this approach over alternatives)
- Include confidence assessment with justification

### For Templates
- Make templates parameterizable (use Jinja2 or similar)
- Include type hints and docstrings
- Add validation logic (check inputs before execution)
- Document expected inputs/outputs

---

## Example Research Session

### Session: `2025-01-16_nx_materials`

**User Question**:
```
"Please create a new material XML for NX with titanium Ti-6Al-4V properties"
```

**Sources Consulted**:
1. User provided: `steel_material.xml` (existing NX material)
2. NX MCP query: "PhysicalMaterial XML schema"
3. Web search: "Titanium Ti-6Al-4V material properties"

**Findings**:
- XML schema learned from user example
- Material properties from web search
- Validation: User confirmed generated XML loads in NX

**Generated Files**:
1. `optimization_engine/custom_functions/nx_material_generator.py`
2. `knowledge_base/nx_research/material_xml_schema.md`
3. `knowledge_base/templates/xml_generation_template.py`

**Confidence**: 0.90 (user-validated)

**Decision Rationale**:
Chose XML generation over direct NXOpen API because:
- XML is version-agnostic (works across NX versions)
- User already had XML workflow established
- Easier for user to inspect/validate generated files

---

## Future Enhancements

### Phase 2 (Current)
- Interactive research workflow
- Knowledge base structure
- Basic pattern learning

### Phase 3-4
- Multi-source synthesis (combine user + MCP + web)
- Automatic template extraction from code
- Pattern recognition across sessions

### Phase 7-8
- Community knowledge sharing
- Pattern evolution (refine templates based on usage)
- Predictive research (anticipate knowledge gaps)

---

**Last Updated**: 2025-01-16
**Related Docs**: [DEVELOPMENT_ROADMAP.md](../DEVELOPMENT_ROADMAP.md), [FEATURE_REGISTRY_ARCHITECTURE.md](../docs/FEATURE_REGISTRY_ARCHITECTURE.md)