# Home Assistant Voice Pipeline Implementation Plan

> Created: 2026-03-07 | Phase: 2.1 - HA Voice Pipeline Setup

---

## Current State Summary

### ✅ Running Services
| Service | Port | Status | Location |
|---------|------|--------|----------|
| Wyoming STT (Whisper large-v3) | 10300 | ✅ Running | Mac Mini |
| Wyoming TTS (Kokoro ONNX) | 10301 | ✅ Running | Mac Mini |
| Wyoming Satellite | 10700 | ✅ Running | Mac Mini |
| OpenClaw Gateway | 8080 | ✅ Running | Mac Mini |
| Home Assistant | 8123 | ✅ Running | Docker (10.0.0.199) |
| Ollama | 11434 | ✅ Running | Mac Mini |

### ✅ Completed Capabilities
- OpenClaw tool calling works with `qwen2.5:7b`
- Home Assistant skill tested: "Turn on the study shelves light" ✓
- Wyoming test pipeline passes (3/3 tests)

---

## Implementation Steps

### Phase 1: Home Assistant Wyoming Integration (Manual UI Steps)

#### Step 1.1: Add Wyoming Protocol Integration

1. Open Home Assistant UI → **Settings → Devices & Services → Add Integration**
2. Search for **"Wyoming Protocol"**
3. Add the following three services:

**Speech-to-Text (STT):**
- Host: `10.0.0.199`
- Port: `10300`
- Name: `Mac Mini STT`

**Text-to-Speech (TTS):**
- Host: `10.0.0.199`
- Port: `10301`
- Name: `Mac Mini TTS`

**Satellite:**
- Host: `10.0.0.199`
- Port: `10700`
- Name: `Mac Mini Living Room`

#### Step 1.2: Create Voice Assistant Pipeline

1. Go to **Settings → Voice Assistants**
2. Click **Add Pipeline**
3. Configure:
   - **Name**: "HomeAI with OpenClaw"
   - **Speech-to-Text**: Select "Mac Mini STT" (Wyoming)
   - **Conversation Agent**: Select "Home Assistant" (for initial testing)
   - **Text-to-Speech**: Select "Mac Mini TTS" (Wyoming)
   - **Language**: English

4. Save the pipeline

#### Step 1.3: Assign Pipeline to Assist

1. Go to **Settings → Voice Assistants**
2. Click **Assist** tab
3. Set the default pipeline to "HomeAI with OpenClaw"

---

### Phase 2: Test Basic Voice Pipeline

#### Step 2.1: Test via Browser (Typed Query)

1. Open Home Assistant UI
2. Click the **Assist** icon (microphone) in the top-right corner
3. Type: "What time is it?"
4. Expected: You should hear a spoken response via Kokoro TTS

#### Step 2.2: Test via Satellite (Voice)

1. Ensure Wyoming Satellite is running: `launchctl list com.homeai.wyoming-satellite`
2. Say the wake word: "Hey Jarvis" (or configured wake word)
3. Speak: "What time is it?"
4. Expected: You should hear a spoken response

#### Step 2.3: Verify Pipeline Components

Check logs for each component:
```bash
# STT logs
tail -f /tmp/homeai-wyoming-stt.log

# TTS logs
tail -f /tmp/homeai-wyoming-tts.log

# Satellite logs
tail -f /tmp/homeai-wyoming-satellite.log
```

---

### Phase 3: OpenClaw Integration

#### Step 3.1: Add Shell Command to HA

Edit Home Assistant `configuration.yaml` (via File Editor add-on or VS Code Server):

```yaml
shell_command:
  openclaw_chat: 'python3 /Users/aodhan/gitea/homeai/homeai-agent/skills/home-assistant/openclaw_bridge.py "{{ message }}" --raw'
```

Restart Home Assistant to load the new configuration.

#### Step 3.2: Create OpenClaw Conversation Automation

Create a new automation in HA UI:

**Automation: "Voice Command via OpenClaw"**

```yaml
alias: "Voice Command via OpenClaw"
description: "Routes voice commands to OpenClaw for processing"
trigger:
  - platform: conversation
    command:
      - "ask jarvis *"
      - "ask assistant *"
      - "hey jarvis *"
condition: []
action:
  - service: shell_command.openclaw_chat
    data:
      message: "{{ trigger.slots.command }}"
    response_variable: openclaw_response
  
  - service: tts.speak
    data:
      media_player_entity_id: media_player.mac_mini_speaker
      message: "{{ openclaw_response }}"
      tts: wyoming_tts
mode: single
```

**Alternative: Direct Conversation Agent**

For tighter integration, create a custom conversation agent that calls OpenClaw directly. This requires a custom component.

#### Step 3.3: Test OpenClaw Integration

1. Open HA Assist
2. Type: "ask jarvis turn on the reading lamp"
3. Expected:
   - Command sent to OpenClaw
   - OpenClaw processes via home-assistant skill
   - Light turns on
   - TTS confirmation spoken

---

### Phase 4: Advanced Configuration

#### Step 4.1: Create Custom Conversation Agent (Optional)

For full OpenClaw integration as the primary conversation agent:

1. Create custom component: `custom_components/openclaw_conversation/`
2. Implement `async_process` method that calls OpenClaw
3. Configure in `configuration.yaml`:

```yaml
conversation:
  - name: OpenClaw
    agent_id: openclaw_agent
```

#### Step 4.2: Add Intent Scripts

For specific intents that bypass OpenClaw:

```yaml
intent_script:
  HassTurnOn:
    action:
      - service: homeassistant.turn_on
        target:
          entity_id: "{{ name }}"
    speech:
      text: "Turned on {{ name }}"
```

---

## Verification Checklist

### Basic Pipeline
- [ ] Wyoming STT appears in HA Integrations
- [ ] Wyoming TTS appears in HA Integrations
- [ ] Wyoming Satellite appears in HA Integrations
- [ ] Voice Assistant pipeline created and assigned
- [ ] Typed query in Assist returns spoken response
- [ ] Voice query via satellite returns spoken response

### OpenClaw Integration
- [ ] Shell command `openclaw_chat` configured
- [ ] Automation triggers on conversation intent
- [ ] OpenClaw receives and processes command
- [ ] HA action executed (e.g., light turns on)
- [ ] TTS confirmation spoken

### Performance
- [ ] Latency under 5 seconds from wake to response
- [ ] STT transcription accurate
- [ ] TTS audio clear and natural
- [ ] No errors in service logs

---

## Troubleshooting

### Issue: Wyoming services not appearing in HA
**Solution:**
1. Verify services are running: `nc -z 10.0.0.199 10300`
2. Check HA can reach Mac Mini: Test from HA container
3. Verify Wyoming protocol version compatibility

### Issue: No audio output
**Solution:**
1. Check SoX installation: `which play`
2. Test audio directly: `afplay /System/Library/Sounds/Glass.aiff`
3. Check satellite logs: `tail -f /tmp/homeai-wyoming-satellite-error.log`

### Issue: OpenClaw not responding
**Solution:**
1. Verify OpenClaw running: `pgrep -f openclaw`
2. Test CLI directly: `openclaw agent --message "Hello" --agent main`
3. Check bridge script permissions: `chmod +x openclaw_bridge.py`

### Issue: STT not transcribing
**Solution:**
1. Check STT logs: `tail -f /tmp/homeai-wyoming-stt.log`
2. Verify Whisper model loaded
3. Test with sample audio file

---

## Next Steps After Completion

1. **Install Chatterbox TTS** for voice cloning
2. **Set up mem0** for long-term memory
3. **Configure n8n workflows** for automation
4. **Add Uptime Kuma monitors** for all services
5. **Begin ESP32 satellite setup** (Phase 4)

---

## File References

| File | Purpose |
|------|---------|
| `homeai-agent/skills/home-assistant/openclaw_bridge.py` | Bridge script for HA → OpenClaw |
| `homeai-agent/skills/home-assistant/ha-configuration.yaml` | Example HA configuration |
| `homeai-voice/wyoming/test-pipeline.sh` | Pipeline smoke test |
| `homeai-voice/scripts/launchd/com.homeai.wyoming-satellite.plist` | Satellite service config |