# 🔧 08 — System Implementation Status

> How the multi-agent system actually works right now, as built.
> Last updated: 2026-02-15

---

## 1. Architecture Overview

**Multi-Instance Cluster:** 8 independent OpenClaw gateway processes, one per agent. Each has its own systemd service, Discord bot token, port, and state directory.

```
┌──────────────────────────────────────────────────────────────────┐
│                        T420 (clawdbot)                           │
│                                                                  │
│  ┌────────────────────────────────────────────────────────────┐  │
│  │  OpenClaw Gateway — Mario (main instance)                  │  │
│  │  Port 18789 │ Slack: Antoine's personal workspace          │  │
│  │  State: ~/.openclaw/                                       │  │
│  └────────────────────────────────────────────────────────────┘  │
│                                                                  │
│  ┌──────────────── Atomizer Cluster ────────────────────────┐   │
│  │                                                           │   │
│  │  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐      │   │
│  │  │  Manager     │  │  Tech Lead  │  │  Secretary   │      │   │
│  │  │  :18800      │  │  :18804     │  │  :18808      │      │   │
│  │  │  Opus 4.6    │  │  Opus 4.6   │  │  Gemini 2.5  │      │   │
│  │  └──────┬───────┘  └──────┬──────┘  └──────┬───────┘      │   │
│  │         │                 │                 │              │   │
│  │  ┌──────┴───────┐  ┌─────┴──────┐  ┌──────┴───────┐      │   │
│  │  │  Auditor     │  │  Optimizer  │  │ Study Builder│      │   │
│  │  │  :18812      │  │  :18816     │  │  :18820      │      │   │
│  │  │  Opus 4.6    │  │  Sonnet 4.5 │  │  Sonnet 4.5  │      │   │
│  │  └──────────────┘  └────────────┘  └──────────────┘      │   │
│  │                                                           │   │
│  │  ┌─────────────┐  ┌─────────────┐                        │   │
│  │  │  NX Expert   │  │  Webster    │                        │   │
│  │  │  :18824      │  │  :18828     │                        │   │
│  │  │  Sonnet 4.5  │  │  Gemini 2.5 │                        │   │
│  │  └─────────────┘  └─────────────┘                        │   │
│  │                                                           │   │
│  │  Inter-agent: hooks API (curl between ports)              │   │
│  │  Shared token: 31422bb39bc9e7a4d34f789d8a7cbc582dece8dd…  │   │
│  └───────────────────────────────────────────────────────────┘   │
│                                                                  │
└──────────────────────────────────────────────────────────────────┘
                          │
                          ▼
┌──────────────────────────────────────────────────────────────────┐
│                 Discord: Atomizer-HQ Server                      │
│                 Guild: 1471858733452890132                        │
│                                                                  │
│  📋 COMMAND: #ceo-office, #announcements, #daily-standup         │
│  🔧 ENGINEERING: #technical, #code-review, #fea-analysis, #nx   │
│  📊 OPERATIONS: #task-board, #meeting-notes, #reports            │
│  🔬 RESEARCH: #literature, #materials-data                       │
│  🏗️ PROJECTS: #active-projects                                  │
│  📚 KNOWLEDGE: #knowledge-base, #lessons-learned                 │
│  🤖 SYSTEM: #agent-logs, #inter-agent, #it-ops                  │
│                                                                  │
│  Each agent = its own Discord bot with unique name & avatar      │
└──────────────────────────────────────────────────────────────────┘
```

---

## 2. Why Multi-Instance (Not Single Gateway)

OpenClaw's native Discord provider (`@buape/carbon`) has a race condition bug when multiple bot tokens connect from one process. Since we need 8 separate bot accounts, we run 8 separate processes — each handles exactly one token, bypassing the bug entirely.

**Advantages over previous bridge approach:**
- Native Discord streaming, threads, reactions, attachments
- Fault isolation — one agent crashing doesn't take down the others
- No middleware polling session files on disk
- Each agent appears as its own Discord user with independent presence

---

## 3. Port Map

| Agent | Port | Model | Notes |
|-------|------|-------|-------|
| Manager | 18800 | Opus 4.6 | Orchestrates, delegates. Heartbeat disabled (Discord delivery bug) |
| Tech Lead | 18804 | Opus 4.6 | Technical authority |
| Secretary | 18808 | Gemini 2.5 Pro | Task tracking, notes. Changed from Codex 2026-02-15 (OAuth expired) |
| Auditor | 18812 | Gemini 2.5 Pro | Quality review. Changed from Codex 2026-02-15 (OAuth expired) |
| Optimizer | 18816 | Sonnet 4.5 | Optimization work |
| Study Builder | 18820 | Gemini 2.5 Pro | Study setup. Changed from Codex 2026-02-15 (OAuth expired) |
| NX Expert | 18824 | Sonnet 4.5 | CAD/NX work |
| Webster | 18828 | Gemini 2.5 Pro | Research. Heartbeat disabled (Discord delivery bug) |

> **⚠️ Port spacing = 4.** OpenClaw uses port N AND N+3 (browser service). Never assign adjacent ports.

---

## 4. Systemd Setup

### Template Service
File: `~/.config/systemd/user/openclaw-atomizer@.service`

```ini
[Unit]
Description=OpenClaw Atomizer - %i
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/node /home/papa/.local/lib/node_modules/openclaw/dist/index.js gateway
Environment=PATH=/home/papa/.local/bin:/usr/local/bin:/usr/bin:/bin
Environment=HOME=/home/papa
Environment=OPENCLAW_STATE_DIR=/home/papa/atomizer/instances/%i
Environment=OPENCLAW_CONFIG_PATH=/home/papa/atomizer/instances/%i/openclaw.json
Environment=OPENCLAW_GATEWAY_TOKEN=31422bb39bc9e7a4d34f789d8a7cbc582dece8dd170dadd1
EnvironmentFile=/home/papa/atomizer/instances/%i/env
EnvironmentFile=/home/papa/atomizer/config/.discord-tokens.env
Restart=always
RestartSec=5
StartLimitIntervalSec=60
StartLimitBurst=5

[Install]
WantedBy=default.target
```

### Cluster Management Script
File: `~/atomizer/cluster.sh`

```bash
# Start all:   bash cluster.sh start
# Stop all:    bash cluster.sh stop
# Restart all: bash cluster.sh restart
# Status:      bash cluster.sh status
# Logs:        bash cluster.sh logs [agent-name]
```

---

## 5. File System Layout

```
~/atomizer/
├── cluster.sh                     ← Cluster management script
├── config/
│   ├── .discord-tokens.env        ← All 8 bot tokens (env vars)
│   └── atomizer-discord.env       ← Legacy (can remove)
├── instances/                     ← Per-agent OpenClaw state
│   ├── manager/
│   │   ├── openclaw.json          ← Agent config (1 agent per instance)
│   │   ├── env                    ← Instance-specific env vars
│   │   └── agents/main/sessions/  ← Session data (auto-created)
│   ├── tech-lead/
│   ├── secretary/
│   ├── auditor/
│   ├── optimizer/
│   ├── study-builder/
│   ├── nx-expert/
│   └── webster/
├── workspaces/                    ← Agent workspaces (SOUL, AGENTS, memory)
│   ├── manager/
│   │   ├── SOUL.md
│   │   ├── AGENTS.md
│   │   ├── MEMORY.md
│   │   └── memory/
│   ├── secretary/
│   ├── technical-lead/
│   ├── auditor/
│   ├── optimizer/
│   ├── study-builder/
│   ├── nx-expert/
│   ├── webster/
│   └── shared/                    ← Shared context (CLUSTER.md, protocols)
└── tools/
    └── nxopen-mcp/                ← NX Open MCP server (for CAD)
```

**Key distinction:** `instances/` = OpenClaw runtime state (configs, sessions, SQLite). `workspaces/` = agent personality and memory (SOUL.md, AGENTS.md, etc.).

---

## 6. Inter-Agent Communication

### Delegation Skill (Primary Method)
Manager and Tech Lead use the `delegate` skill to assign tasks to other agents. The skill wraps the OpenClaw Hooks API with port mapping, auth, error handling, and logging.

**Location:** `/home/papa/atomizer/workspaces/shared/skills/delegate/`
**Installed on:** Manager, Tech Lead (symlinked from shared)

```bash
# Usage
bash /home/papa/atomizer/workspaces/shared/skills/delegate/delegate.sh <agent> "<instruction>" [options]

# Examples
delegate.sh webster "Find CTE of Zerodur Class 0 between 20-40°C"
delegate.sh nx-expert "Mesh the M2 mirror" --channel C0AEJV13TEU --deliver
delegate.sh auditor "Review thermal analysis" --no-deliver
```

**How it works:**
1. Looks up the target agent's port from hardcoded port map
2. Checks if the target is running
3. POSTs to `http://127.0.0.1:PORT/hooks/agent` with auth token
4. Target agent processes the task asynchronously in an isolated session
5. Response delivered to Discord if `--deliver` is set

**Options:** `--channel <id>`, `--deliver` (default), `--no-deliver`

### Delegation Authority
| Agent | Can Delegate To |
|-------|----------------|
| Manager | All agents |
| Tech Lead | All agents except Manager |
| All others | Cannot delegate — request via Manager or Tech Lead |

### Hooks Protocol
All agents follow `/home/papa/atomizer/workspaces/shared/HOOKS-PROTOCOL.md`:
- Hook messages = **high-priority assignments**, processed before other work
- After completing tasks, agents **append** status to `shared/project_log.md`
- Only the Manager updates `shared/PROJECT_STATUS.md` (gatekeeper pattern)

### Raw Hooks API (Reference)
The delegate skill wraps this, but for reference:
```bash
curl -s -X POST http://127.0.0.1:PORT/hooks/agent \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer 31422bb39bc9e7a4d34f789d8a7cbc582dece8dd170dadd1" \
  -d '{"message": "your request here", "deliver": true, "channel": "discord"}'
```

### sessions_send / sessions_spawn
Agents configured with `agentToAgent.enabled: true` can use OpenClaw's built-in `sessions_send` and `sessions_spawn` tools to communicate within the same instance. Cross-instance communication requires the hooks API / delegate skill.

---

## 7. Current Status

### ✅ Working
- All 8 instances running as systemd services (auto-start on boot)
- Each agent has its own Discord bot identity (name, avatar, presence)
- Native Discord features: streaming, typing indicators, message chunking
- Agent workspaces with SOUL.md, AGENTS.md, MEMORY.md
- Hooks API enabled on all instances (Google Gemini + Anthropic auth configured)
- **Delegation skill deployed** — Manager and Tech Lead can delegate tasks to any agent via `delegate.sh`
- **Hooks protocol** — all agents know how to receive and prioritize delegated tasks
- **Gatekeeper pattern** — Manager owns PROJECT_STATUS.md; others append to project_log.md
- Cluster management via `cluster.sh`
- Estimated total RAM: ~4.2GB for 8 instances

### ❌ Known Issues
- ~~**DELEGATE syntax is fake**~~ → ✅ RESOLVED (2026-02-14): Replaced with `delegate.sh` skill using hooks API
- **Discord "Ambiguous recipient" bug** (2026-02-15): OpenClaw Discord plugin requires `user:` or `channel:` prefix for message targets. When heartbeat tries to reply to a session that originated from a Discord DM, it uses the bare user ID → delivery fails. **Workaround:** Heartbeat disabled on Manager + Webster. Other agents unaffected (their sessions don't originate from Discord DMs). Proper fix requires OpenClaw patch to auto-infer `user:` for known user IDs.
- **Codex OAuth expired** (2026-02-15): `refresh_token_reused` error — multiple instances racing to refresh the same shared Codex token. Secretary, Auditor, Study-Builder switched to Gemini 2.5 Pro. To restore Codex: Antoine must re-run `codex login` via SSH tunnel, then run `~/atomizer/scripts/sync-codex-tokens.sh`.
- **No automated orchestration layer:** Manager delegates manually (but now has proper tooling to do so — orchestrate.sh, workflow engine)
- **5 agents not yet created:** Post-Processor, Reporter, Developer, Knowledge Base, IT (from the original 13-agent plan)
- **Windows execution bridge** (`atomizer_job_watcher.py`): exists but not connected end-to-end

---

## 8. Evolution History

| Date | Phase | What Changed |
|------|-------|-------------|
| 2026-02-07 | Phase 0 | Vision doc created, 13-agent plan designed |
| 2026-02-08 | Phase 0 | Single gateway (port 18790) running on Slack |
| 2026-02-13 | Discord Migration | Discord server created, 8 bot tokens obtained |
| 2026-02-14 (AM) | Bridge Attempt | discord-bridge.js built — worked but fragile (no streaming, polled session files) |
| 2026-02-14 (PM) | **Multi-Instance Cluster** | Pivoted to 8 independent OpenClaw instances. Bridge killed. Native Discord restored. |
| 2026-02-14 (PM) | **Delegation System** | Built `delegate.sh` skill, hooks protocol, gatekeeper pattern. Fake DELEGATE syntax replaced with real hooks API calls. Google Gemini auth added to all instances. |
| 2026-02-15 | **Orchestration Engine** | Phases 1-3 complete: synchronous delegation (`orchestrate.py`), smart routing (capability registry), hierarchical delegation (Tech-Lead + Optimizer can sub-delegate), YAML workflow engine with parallel execution + approval gates. See `10-ORCHESTRATION-ENGINE-PLAN.md`. |
| 2026-02-15 | **Stability Fixes** | Discord heartbeat delivery bug identified (ambiguous recipient). Codex OAuth token expired (refresh_token_reused). Heartbeat disabled on Manager + Webster. Secretary/Auditor/Study-Builder switched from Codex to Gemini 2.5 Pro. HEARTBEAT.md created for all agents. |

---

*Created: 2026-02-14 by Mario*
*This is the "as-built" document — updated as implementation evolves.*