homeai/homeai-voice/VOICE_PIPELINE_SETUP.md

# Voice Pipeline Setup Guide

> Complete guide to setting up the end-to-end voice pipeline with OpenClaw integration

---

## Network Configuration

**Important**: Services are split across two machines:

| Service | Port | Location |
|---------|------|----------|
| Wyoming STT (Whisper large-v3) | 10300 | Mac Mini (10.0.0.101) |
| Wyoming TTS (Kokoro ONNX) | 10301 | Mac Mini (10.0.0.101) |
| Wyoming Satellite | 10700 | Mac Mini (10.0.0.101) |
| openWakeWord | - | Mac Mini (10.0.0.101) |
| OpenClaw Gateway | 8080 | Mac Mini (10.0.0.101) |
| Ollama | 11434 | Mac Mini (10.0.0.101) |
| Home Assistant (Docker) | 8123 | Server (10.0.0.199) |

**All integrations must point to 10.0.0.101 (Mac Mini), not 10.0.0.199 (HA server).**

---

## Current Status

### ✅ Services Running on Mac Mini (10.0.0.101)

| Service | Port | Status |
|---------|------|--------|
| Wyoming STT | 10300 | ✅ Running |
| Wyoming TTS | 10301 | ✅ Running |
| Wyoming Satellite | 10700 | ✅ Running |
| openWakeWord | - | ✅ Running |
| OpenClaw Gateway | 8080 | ✅ Running |
| Ollama | 11434 | ✅ Running |

### ✅ Completed
- Wyoming STT/TTS services installed and running
- Wyoming Satellite installed and running
- OpenClaw agent configured with home-assistant skill
- Custom OpenClaw conversation component created

### 🔄 Next Steps
1. Install OpenClaw conversation component in Home Assistant
2. Configure Wyoming integrations in HA
3. Create voice assistant pipeline with OpenClaw
4. Test the full voice loop

---

## Step 1: Install OpenClaw Conversation Component

Home Assistant is running in Docker on server 10.0.0.199. Use the automated installation script.

### Option A: Automated Installation (Recommended)

```bash
# From Mac Mini, run the installation script
cd ~/gitea/homeai/homeai-agent/custom_components
./install-to-docker-ha.sh

# The script will:
# 1. Create a tarball of the component
# 2. Copy it to the HA server via SCP
# 3. Extract it into the HA Docker container
# 4. Provide next steps
```

**Requirements:**
- SSH access to 10.0.0.199
- SSH keys configured (or password access)

### Option B: Manual Installation via SSH

```bash
# 1. Create tarball
cd ~/gitea/homeai/homeai-agent/custom_components
tar -czf openclaw_conversation.tar.gz openclaw_conversation/

# 2. Copy to HA server
scp openclaw_conversation.tar.gz 10.0.0.199:/tmp/

# 3. SSH to HA server and install
ssh 10.0.0.199
CONTAINER=$(docker ps --filter "name=homeassistant" --format "{{.Names}}" | head -n 1)
docker cp /tmp/openclaw_conversation.tar.gz $CONTAINER:/tmp/
docker exec $CONTAINER sh -c 'cd /config/custom_components && tar -xzf /tmp/openclaw_conversation.tar.gz'
docker restart $CONTAINER
```

### Option D: Using Home Assistant File Editor (Manual)

1. Open Home Assistant UI at http://10.0.0.199:8123
2. Install the **File Editor** add-on if not already installed
3. Create directory: `/config/custom_components/openclaw_conversation/`
4. Copy each file from `homeai-agent/custom_components/openclaw_conversation/`:
   - `__init__.py`
   - `config_flow.py`
   - `const.py`
   - `conversation.py`
   - `manifest.json`
   - `strings.json`

### Verify Installation

After installation, restart Home Assistant:

```bash
# Via SSH
ssh 10.0.0.199 'docker restart homeassistant'

# Or via HA UI
# Settings → System → Restart
```

Check logs for any errors:
- **Settings → System → Logs**
- Look for "OpenClaw Conversation" in the logs

---

## Step 2: Configure Wyoming Integrations

### Add Wyoming STT (Speech-to-Text)

1. Go to **Settings → Devices & Services → Add Integration**
2. Search for **"Wyoming Protocol"**
3. Configure:
   - **Host**: `10.0.0.101` ⚠️ **Mac Mini IP, not HA server IP (10.0.0.199)**
   - **Port**: `10300`
   - **Name**: `Mac Mini STT`
4. Click **Submit**

### Add Wyoming TTS (Text-to-Speech)

1. Click **Add Integration** again
2. Search for **"Wyoming Protocol"**
3. Configure:
   - **Host**: `10.0.0.101` ⚠️ **Mac Mini IP**
   - **Port**: `10301`
   - **Name**: `Mac Mini TTS`
4. Click **Submit**

### Add Wyoming Satellite

1. Click **Add Integration** again
2. Search for **"Wyoming Protocol"**
3. Configure:
   - **Host**: `10.0.0.101` ⚠️ **Mac Mini IP**
   - **Port**: `10700`
   - **Name**: `Mac Mini Living Room`
4. Click **Submit**

### Verify Integrations

All three Wyoming integrations should appear in **Settings → Devices & Services**.

---

## Step 3: Add OpenClaw Conversation Agent

### Via UI (Recommended)

1. Go to **Settings → Devices & Services → Add Integration**
2. Search for **"OpenClaw Conversation"**
3. Configure:
   - **OpenClaw Host**: `10.0.0.101` ⚠️ **Mac Mini IP, not HA server IP (10.0.0.199)**
   - **OpenClaw Port**: `8080`
   - **Agent Name**: `main`
   - **Timeout**: `30` seconds
4. Click **Submit**

### Via YAML (Alternative)

Add to `/config/configuration.yaml`:

```yaml
openclaw_conversation:
  openclaw_host: 10.0.0.101  # Mac Mini IP
  openclaw_port: 8080
  agent_name: main
  timeout: 30
```

Then restart Home Assistant.

---

## Step 4: Create Voice Assistant Pipeline

1. Go to **Settings → Voice Assistants**
2. Click **Add Assistant**
3. Configure:
   - **Name**: `HomeAI with OpenClaw`
   - **Language**: `English`
   - **Speech-to-Text**: Select `Mac Mini STT` (Wyoming)
   - **Conversation Agent**: Select `OpenClaw Conversation`
   - **Text-to-Speech**: Select `Mac Mini TTS` (Wyoming)
4. Click **Create**

### Set as Default

1. In **Settings → Voice Assistants**
2. Click the three dots next to "HomeAI with OpenClaw"
3. Select **Set as preferred**

---

## Step 5: Test the Pipeline

### Test 1: Text Input → TTS Output

1. Open Home Assistant UI
2. Click the **Assist** icon (microphone) in the top-right corner
3. Type: `"What time is it?"`
4. Press Enter

**Expected Result**: You should hear a spoken response via Kokoro TTS

### Test 2: Voice Input → OpenClaw → TTS Output

1. Ensure Wyoming Satellite is running on Mac Mini:
   ```bash
   launchctl list | grep wyoming-satellite
   ```

2. Say the wake word: **"Hey Jarvis"**
3. Wait for the beep/acknowledgment
4. Speak: **"What time is it?"**

**Expected Result**: You should hear a spoken response

### Test 3: Home Assistant Control via Voice

1. Say: **"Hey Jarvis"**
2. Speak: **"Turn on the reading lamp"**

**Expected Result**: 
- OpenClaw processes the request
- Home Assistant skill executes the action
- Light turns on
- You hear a confirmation via TTS

---

## Troubleshooting

### Issue: OpenClaw Conversation not appearing in integrations

**Solution**:
1. Verify files are in `/config/custom_components/openclaw_conversation/`
2. Check Home Assistant logs for errors
3. Ensure `manifest.json` is valid JSON
4. Restart Home Assistant

### Issue: Wyoming services not connecting

**Solution**:
1. Verify services are running on Mac Mini:
   ```bash
   launchctl list | grep wyoming
   nc -z 10.0.0.199 10300  # Test STT
   nc -z 10.0.0.199 10301  # Test TTS
   nc -z 10.0.0.199 10700  # Test Satellite
   ```

2. Check firewall rules on Mac Mini
3. Verify Home Assistant can reach Mac Mini network

### Issue: OpenClaw not responding

**Solution**:
1. Verify OpenClaw is running:
   ```bash
   launchctl list | grep openclaw
   pgrep -f openclaw
   ```

2. Test OpenClaw CLI directly:
   ```bash
   openclaw agent --message "Hello" --agent main
   ```

3. Check OpenClaw logs:
   ```bash
   tail -f /tmp/homeai-openclaw.log
   ```

4. Verify OpenClaw can reach Home Assistant:
   ```bash
   curl http://10.0.0.199:8123/api/
   ```

### Issue: No audio output from satellite

**Solution**:
1. Check satellite logs:
   ```bash
   tail -f /tmp/homeai-wyoming-satellite.log
   ```

2. Test audio output:
   ```bash
   afplay /System/Library/Sounds/Glass.aiff
   ```

3. Verify SoX is installed:
   ```bash
   which play
   brew install sox
   ```

### Issue: Wake word not detected

**Solution**:
1. Check wakeword service:
   ```bash
   launchctl list | grep wakeword
   ```

2. Test microphone input:
   ```bash
   # Record a test
   rec -r 16000 -c 1 test.wav trim 0 5
   ```

3. Adjust wake word threshold in satellite config

---

## Voice Pipeline Flow

```
┌─────────────────┐
│   USB Mic       │
│  (Mac Mini)     │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  Wake Word      │
│  Detection      │
│  (hey_jarvis)   │
└────────┬────────┘
         │ wake detected
         ▼
┌─────────────────┐
│ Wyoming         │
│ Satellite       │
│ :10700          │
└────────┬────────┘
         │ audio stream
         ▼
┌─────────────────┐
│ Wyoming STT     │
│ (Whisper)       │
│ :10300          │
└────────┬────────┘
         │ transcript
         ▼
┌─────────────────┐
│ Home Assistant  │
│ Voice Pipeline  │
└────────┬────────┘
         │ text
         ▼
┌─────────────────┐
│ OpenClaw        │
│ Conversation    │
│ Agent           │
└────────┬────────┘
         │ message
         ▼
┌─────────────────┐
│ OpenClaw        │
│ Gateway         │
│ :8080           │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│ Ollama LLM      │
│ + Skills        │
│ :11434          │
└────────┬────────┘
         │ response
         ▼
┌─────────────────┐
│ Wyoming TTS     │
│ (Kokoro)        │
│ :10301          │
└────────┬────────┘
         │ audio
         ▼
┌─────────────────┐
│   Speaker       │
│  (Mac Mini)     │
└─────────────────┘
```

---

## Next Steps After Setup

1. **Install Chatterbox TTS** for voice cloning
2. **Set up mem0** for long-term memory
3. **Configure n8n workflows** for automation
4. **Add Uptime Kuma monitors** for all services
5. **Begin ESP32 satellite setup** (Phase 4)

---

## Files Reference

| File | Purpose |
|------|---------|
| [`homeai-agent/custom_components/openclaw_conversation/`](../homeai-agent/custom_components/openclaw_conversation/) | Custom HA component |
| [`homeai-agent/skills/home-assistant/openclaw_bridge.py`](../homeai-agent/skills/home-assistant/openclaw_bridge.py) | Bridge script |
| [`homeai-voice/scripts/launchd/`](scripts/launchd/) | Service plists |
| [`plans/ha-voice-pipeline-implementation.md`](../plans/ha-voice-pipeline-implementation.md) | Detailed implementation plan |
| [`plans/voice-loop-integration.md`](../plans/voice-loop-integration.md) | Architecture options |

---

## Success Criteria

- [ ] Wyoming STT/TTS/Satellite appear in HA integrations
- [ ] OpenClaw Conversation agent appears in HA integrations
- [ ] Voice assistant pipeline created with OpenClaw
- [ ] Typed query in Assist returns spoken response
- [ ] Voice query via satellite returns spoken response
- [ ] "Turn on the reading lamp" command works end-to-end
- [ ] Latency under 5 seconds from wake to response
- [ ] All services survive Mac Mini reboot