homeai/plans/ha-voice-pipeline-implementation.md

Home Assistant Voice Pipeline Implementation Plan

Created: 2026-03-07 | Phase: 2.1 - HA Voice Pipeline Setup

---

Current State Summary

✅ Running Services

Service	Port	Status	Location
Wyoming STT (Whisper large-v3)	10300	✅ Running	Mac Mini
Wyoming TTS (Kokoro ONNX)	10301	✅ Running	Mac Mini
Wyoming Satellite	10700	✅ Running	Mac Mini
OpenClaw Gateway	8080	✅ Running	Mac Mini
Home Assistant	8123	✅ Running	Docker (10.0.0.199)
Ollama	11434	✅ Running	Mac Mini

✅ Completed Capabilities

• OpenClaw tool calling works with qwen2.5:7b
• Home Assistant skill tested: "Turn on the study shelves light" ✓
• Wyoming test pipeline passes (3/3 tests)

---

Implementation Steps

Phase 1: Home Assistant Wyoming Integration (Manual UI Steps)

Step 1.1: Add Wyoming Protocol Integration

1. Open Home Assistant UI → Settings → Devices & Services → Add Integration
2. Search for "Wyoming Protocol"
3. Add the following three services:

Speech-to-Text (STT):

• Host: 10.0.0.199
• Port: 10300
• Name: Mac Mini STT

Text-to-Speech (TTS):

• Host: 10.0.0.199
• Port: 10301
• Name: Mac Mini TTS

Satellite:

• Host: 10.0.0.199
• Port: 10700
• Name: Mac Mini Living Room

Step 1.2: Create Voice Assistant Pipeline

1. Go to Settings → Voice Assistants

2. Click Add Pipeline

3. Configure:

   • Name: "HomeAI with OpenClaw"
   • Speech-to-Text: Select "Mac Mini STT" (Wyoming)
   • Conversation Agent: Select "Home Assistant" (for initial testing)
   • Text-to-Speech: Select "Mac Mini TTS" (Wyoming)
   • Language: English

4. Save the pipeline

Step 1.3: Assign Pipeline to Assist

1. Go to Settings → Voice Assistants
2. Click Assist tab
3. Set the default pipeline to "HomeAI with OpenClaw"

---

Phase 2: Test Basic Voice Pipeline

Step 2.1: Test via Browser (Typed Query)

1. Open Home Assistant UI
2. Click the Assist icon (microphone) in the top-right corner
3. Type: "What time is it?"
4. Expected: You should hear a spoken response via Kokoro TTS

Step 2.2: Test via Satellite (Voice)

1. Ensure Wyoming Satellite is running: launchctl list com.homeai.wyoming-satellite
2. Say the wake word: "Hey Jarvis" (or configured wake word)
3. Speak: "What time is it?"
4. Expected: You should hear a spoken response

Step 2.3: Verify Pipeline Components

Check logs for each component:

# STT logs
tail -f /tmp/homeai-wyoming-stt.log

# TTS logs
tail -f /tmp/homeai-wyoming-tts.log

# Satellite logs
tail -f /tmp/homeai-wyoming-satellite.log

---

Phase 3: OpenClaw Integration

Step 3.1: Add Shell Command to HA

Edit Home Assistant configuration.yaml (via File Editor add-on or VS Code Server):

shell_command:
  openclaw_chat: 'python3 /Users/aodhan/gitea/homeai/homeai-agent/skills/home-assistant/openclaw_bridge.py "{{ message }}" --raw'

Restart Home Assistant to load the new configuration.

Step 3.2: Create OpenClaw Conversation Automation

Create a new automation in HA UI:

Automation: "Voice Command via OpenClaw"

alias: "Voice Command via OpenClaw"
description: "Routes voice commands to OpenClaw for processing"
trigger:
  - platform: conversation
    command:
      - "ask jarvis *"
      - "ask assistant *"
      - "hey jarvis *"
condition: []
action:
  - service: shell_command.openclaw_chat
    data:
      message: "{{ trigger.slots.command }}"
    response_variable: openclaw_response
  
  - service: tts.speak
    data:
      media_player_entity_id: media_player.mac_mini_speaker
      message: "{{ openclaw_response }}"
      tts: wyoming_tts
mode: single

Alternative: Direct Conversation Agent

For tighter integration, create a custom conversation agent that calls OpenClaw directly. This requires a custom component.

Step 3.3: Test OpenClaw Integration

1. Open HA Assist
2. Type: "ask jarvis turn on the reading lamp"
3. Expected:
   • Command sent to OpenClaw
   • OpenClaw processes via home-assistant skill
   • Light turns on
   • TTS confirmation spoken

---

Phase 4: Advanced Configuration

Step 4.1: Create Custom Conversation Agent (Optional)

For full OpenClaw integration as the primary conversation agent:

1. Create custom component: custom_components/openclaw_conversation/
2. Implement async_process method that calls OpenClaw
3. Configure in configuration.yaml:

conversation:
  - name: OpenClaw
    agent_id: openclaw_agent

Step 4.2: Add Intent Scripts

For specific intents that bypass OpenClaw:

intent_script:
  HassTurnOn:
    action:
      - service: homeassistant.turn_on
        target:
          entity_id: "{{ name }}"
    speech:
      text: "Turned on {{ name }}"

---

Verification Checklist

Basic Pipeline

• Wyoming STT appears in HA Integrations
• Wyoming TTS appears in HA Integrations
• Wyoming Satellite appears in HA Integrations
• Voice Assistant pipeline created and assigned
• Typed query in Assist returns spoken response
• Voice query via satellite returns spoken response

OpenClaw Integration

• Shell command openclaw_chat configured
• Automation triggers on conversation intent
• OpenClaw receives and processes command
• HA action executed (e.g., light turns on)
• TTS confirmation spoken

Performance

• Latency under 5 seconds from wake to response
• STT transcription accurate
• TTS audio clear and natural
• No errors in service logs

---

Troubleshooting

Issue: Wyoming services not appearing in HA

Solution:

1. Verify services are running: nc -z 10.0.0.199 10300
2. Check HA can reach Mac Mini: Test from HA container
3. Verify Wyoming protocol version compatibility

Issue: No audio output

Solution:

1. Check SoX installation: which play
2. Test audio directly: afplay /System/Library/Sounds/Glass.aiff
3. Check satellite logs: tail -f /tmp/homeai-wyoming-satellite-error.log

Issue: OpenClaw not responding

Solution:

1. Verify OpenClaw running: pgrep -f openclaw
2. Test CLI directly: openclaw agent --message "Hello" --agent main
3. Check bridge script permissions: chmod +x openclaw_bridge.py

Issue: STT not transcribing

Solution:

1. Check STT logs: tail -f /tmp/homeai-wyoming-stt.log
2. Verify Whisper model loaded
3. Test with sample audio file

---

Next Steps After Completion

1. Install Chatterbox TTS for voice cloning
2. Set up mem0 for long-term memory
3. Configure n8n workflows for automation
4. Add Uptime Kuma monitors for all services
5. Begin ESP32 satellite setup (Phase 4)

---

File References

File	Purpose
homeai-agent/skills/home-assistant/openclaw_bridge.py	Bridge script for HA → OpenClaw
homeai-agent/skills/home-assistant/ha-configuration.yaml	Example HA configuration
homeai-voice/wyoming/test-pipeline.sh	Pipeline smoke test
homeai-voice/scripts/launchd/com.homeai.wyoming-satellite.plist	Satellite service config