feat(phase-04): Wyoming Satellite integration + OpenClaw HA components
## Voice Pipeline (P3) - Replace openWakeWord daemon with Wyoming Satellite approach - Add Wyoming Satellite service on port 10700 for HA voice pipeline - Update setup.sh with cross-platform sed compatibility (macOS/Linux) - Add version field to Kokoro TTS voice info - Update launchd service loader to use Wyoming Satellite ## Home Assistant Integration (P4) - Add custom conversation agent component (openclaw_conversation) - Fix: Use IntentResponse instead of plain strings (HA API requirement) - Support both HTTP API and CLI fallback modes - Config flow for easy HA UI setup - Add OpenClaw bridge scripts (Python + Bash) - Add ha-ctl utility for HA entity control - Fix: Use context manager for token file reading - Add HA configuration examples and documentation ## Infrastructure - Add mem0 backup automation (launchd + script) - Add n8n workflow templates (morning briefing, notification router) - Add VS Code workspace configuration - Reorganize model files into categorized folders: - lmstudio-community/ - mlx-community/ - bartowski/ - mradermacher/ ## Documentation - Update PROJECT_PLAN.md with Wyoming Satellite architecture - Update TODO.md with completed Wyoming integration tasks - Add OPENCLAW_INTEGRATION.md for HA setup guide ## Testing - Verified Wyoming services running (STT:10300, TTS:10301, Satellite:10700) - Verified OpenClaw CLI accessibility - Confirmed cross-platform compatibility fixes
This commit is contained in:
Aodhan Collins
266
plans/ha-voice-pipeline-implementation.md
Normal file
266
plans/ha-voice-pipeline-implementation.md
Normal file
@@ -0,0 +1,266 @@
|
||||
# Home Assistant Voice Pipeline Implementation Plan
|
||||
|
||||
> Created: 2026-03-07 | Phase: 2.1 - HA Voice Pipeline Setup
|
||||
|
||||
---
|
||||
|
||||
## Current State Summary
|
||||
|
||||
### ✅ Running Services
|
||||
| Service | Port | Status | Location |
|
||||
|---------|------|--------|----------|
|
||||
| Wyoming STT (Whisper large-v3) | 10300 | ✅ Running | Mac Mini |
|
||||
| Wyoming TTS (Kokoro ONNX) | 10301 | ✅ Running | Mac Mini |
|
||||
| Wyoming Satellite | 10700 | ✅ Running | Mac Mini |
|
||||
| OpenClaw Gateway | 8080 | ✅ Running | Mac Mini |
|
||||
| Home Assistant | 8123 | ✅ Running | Docker (10.0.0.199) |
|
||||
| Ollama | 11434 | ✅ Running | Mac Mini |
|
||||
|
||||
### ✅ Completed Capabilities
|
||||
- OpenClaw tool calling works with `qwen2.5:7b`
|
||||
- Home Assistant skill tested: "Turn on the study shelves light" ✓
|
||||
- Wyoming test pipeline passes (3/3 tests)
|
||||
|
||||
---
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Home Assistant Wyoming Integration (Manual UI Steps)
|
||||
|
||||
#### Step 1.1: Add Wyoming Protocol Integration
|
||||
|
||||
1. Open Home Assistant UI → **Settings → Devices & Services → Add Integration**
|
||||
2. Search for **"Wyoming Protocol"**
|
||||
3. Add the following three services:
|
||||
|
||||
**Speech-to-Text (STT):**
|
||||
- Host: `10.0.0.199`
|
||||
- Port: `10300`
|
||||
- Name: `Mac Mini STT`
|
||||
|
||||
**Text-to-Speech (TTS):**
|
||||
- Host: `10.0.0.199`
|
||||
- Port: `10301`
|
||||
- Name: `Mac Mini TTS`
|
||||
|
||||
**Satellite:**
|
||||
- Host: `10.0.0.199`
|
||||
- Port: `10700`
|
||||
- Name: `Mac Mini Living Room`
|
||||
|
||||
#### Step 1.2: Create Voice Assistant Pipeline
|
||||
|
||||
1. Go to **Settings → Voice Assistants**
|
||||
2. Click **Add Pipeline**
|
||||
3. Configure:
|
||||
- **Name**: "HomeAI with OpenClaw"
|
||||
- **Speech-to-Text**: Select "Mac Mini STT" (Wyoming)
|
||||
- **Conversation Agent**: Select "Home Assistant" (for initial testing)
|
||||
- **Text-to-Speech**: Select "Mac Mini TTS" (Wyoming)
|
||||
- **Language**: English
|
||||
|
||||
4. Save the pipeline
|
||||
|
||||
#### Step 1.3: Assign Pipeline to Assist
|
||||
|
||||
1. Go to **Settings → Voice Assistants**
|
||||
2. Click **Assist** tab
|
||||
3. Set the default pipeline to "HomeAI with OpenClaw"
|
||||
|
||||
---
|
||||
|
||||
### Phase 2: Test Basic Voice Pipeline
|
||||
|
||||
#### Step 2.1: Test via Browser (Typed Query)
|
||||
|
||||
1. Open Home Assistant UI
|
||||
2. Click the **Assist** icon (microphone) in the top-right corner
|
||||
3. Type: "What time is it?"
|
||||
4. Expected: You should hear a spoken response via Kokoro TTS
|
||||
|
||||
#### Step 2.2: Test via Satellite (Voice)
|
||||
|
||||
1. Ensure Wyoming Satellite is running: `launchctl list com.homeai.wyoming-satellite`
|
||||
2. Say the wake word: "Hey Jarvis" (or configured wake word)
|
||||
3. Speak: "What time is it?"
|
||||
4. Expected: You should hear a spoken response
|
||||
|
||||
#### Step 2.3: Verify Pipeline Components
|
||||
|
||||
Check logs for each component:
|
||||
```bash
|
||||
# STT logs
|
||||
tail -f /tmp/homeai-wyoming-stt.log
|
||||
|
||||
# TTS logs
|
||||
tail -f /tmp/homeai-wyoming-tts.log
|
||||
|
||||
# Satellite logs
|
||||
tail -f /tmp/homeai-wyoming-satellite.log
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Phase 3: OpenClaw Integration
|
||||
|
||||
#### Step 3.1: Add Shell Command to HA
|
||||
|
||||
Edit Home Assistant `configuration.yaml` (via File Editor add-on or VS Code Server):
|
||||
|
||||
```yaml
|
||||
shell_command:
|
||||
openclaw_chat: 'python3 /Users/aodhan/gitea/homeai/homeai-agent/skills/home-assistant/openclaw_bridge.py "{{ message }}" --raw'
|
||||
```
|
||||
|
||||
Restart Home Assistant to load the new configuration.
|
||||
|
||||
#### Step 3.2: Create OpenClaw Conversation Automation
|
||||
|
||||
Create a new automation in HA UI:
|
||||
|
||||
**Automation: "Voice Command via OpenClaw"**
|
||||
|
||||
```yaml
|
||||
alias: "Voice Command via OpenClaw"
|
||||
description: "Routes voice commands to OpenClaw for processing"
|
||||
trigger:
|
||||
- platform: conversation
|
||||
command:
|
||||
- "ask jarvis *"
|
||||
- "ask assistant *"
|
||||
- "hey jarvis *"
|
||||
condition: []
|
||||
action:
|
||||
- service: shell_command.openclaw_chat
|
||||
data:
|
||||
message: "{{ trigger.slots.command }}"
|
||||
response_variable: openclaw_response
|
||||
|
||||
- service: tts.speak
|
||||
data:
|
||||
media_player_entity_id: media_player.mac_mini_speaker
|
||||
message: "{{ openclaw_response }}"
|
||||
tts: wyoming_tts
|
||||
mode: single
|
||||
```
|
||||
|
||||
**Alternative: Direct Conversation Agent**
|
||||
|
||||
For tighter integration, create a custom conversation agent that calls OpenClaw directly. This requires a custom component.
|
||||
|
||||
#### Step 3.3: Test OpenClaw Integration
|
||||
|
||||
1. Open HA Assist
|
||||
2. Type: "ask jarvis turn on the reading lamp"
|
||||
3. Expected:
|
||||
- Command sent to OpenClaw
|
||||
- OpenClaw processes via home-assistant skill
|
||||
- Light turns on
|
||||
- TTS confirmation spoken
|
||||
|
||||
---
|
||||
|
||||
### Phase 4: Advanced Configuration
|
||||
|
||||
#### Step 4.1: Create Custom Conversation Agent (Optional)
|
||||
|
||||
For full OpenClaw integration as the primary conversation agent:
|
||||
|
||||
1. Create custom component: `custom_components/openclaw_conversation/`
|
||||
2. Implement `async_process` method that calls OpenClaw
|
||||
3. Configure in `configuration.yaml`:
|
||||
|
||||
```yaml
|
||||
conversation:
|
||||
- name: OpenClaw
|
||||
agent_id: openclaw_agent
|
||||
```
|
||||
|
||||
#### Step 4.2: Add Intent Scripts
|
||||
|
||||
For specific intents that bypass OpenClaw:
|
||||
|
||||
```yaml
|
||||
intent_script:
|
||||
HassTurnOn:
|
||||
action:
|
||||
- service: homeassistant.turn_on
|
||||
target:
|
||||
entity_id: "{{ name }}"
|
||||
speech:
|
||||
text: "Turned on {{ name }}"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Verification Checklist
|
||||
|
||||
### Basic Pipeline
|
||||
- [ ] Wyoming STT appears in HA Integrations
|
||||
- [ ] Wyoming TTS appears in HA Integrations
|
||||
- [ ] Wyoming Satellite appears in HA Integrations
|
||||
- [ ] Voice Assistant pipeline created and assigned
|
||||
- [ ] Typed query in Assist returns spoken response
|
||||
- [ ] Voice query via satellite returns spoken response
|
||||
|
||||
### OpenClaw Integration
|
||||
- [ ] Shell command `openclaw_chat` configured
|
||||
- [ ] Automation triggers on conversation intent
|
||||
- [ ] OpenClaw receives and processes command
|
||||
- [ ] HA action executed (e.g., light turns on)
|
||||
- [ ] TTS confirmation spoken
|
||||
|
||||
### Performance
|
||||
- [ ] Latency under 5 seconds from wake to response
|
||||
- [ ] STT transcription accurate
|
||||
- [ ] TTS audio clear and natural
|
||||
- [ ] No errors in service logs
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Issue: Wyoming services not appearing in HA
|
||||
**Solution:**
|
||||
1. Verify services are running: `nc -z 10.0.0.199 10300`
|
||||
2. Check HA can reach Mac Mini: Test from HA container
|
||||
3. Verify Wyoming protocol version compatibility
|
||||
|
||||
### Issue: No audio output
|
||||
**Solution:**
|
||||
1. Check SoX installation: `which play`
|
||||
2. Test audio directly: `afplay /System/Library/Sounds/Glass.aiff`
|
||||
3. Check satellite logs: `tail -f /tmp/homeai-wyoming-satellite-error.log`
|
||||
|
||||
### Issue: OpenClaw not responding
|
||||
**Solution:**
|
||||
1. Verify OpenClaw running: `pgrep -f openclaw`
|
||||
2. Test CLI directly: `openclaw agent --message "Hello" --agent main`
|
||||
3. Check bridge script permissions: `chmod +x openclaw_bridge.py`
|
||||
|
||||
### Issue: STT not transcribing
|
||||
**Solution:**
|
||||
1. Check STT logs: `tail -f /tmp/homeai-wyoming-stt.log`
|
||||
2. Verify Whisper model loaded
|
||||
3. Test with sample audio file
|
||||
|
||||
---
|
||||
|
||||
## Next Steps After Completion
|
||||
|
||||
1. **Install Chatterbox TTS** for voice cloning
|
||||
2. **Set up mem0** for long-term memory
|
||||
3. **Configure n8n workflows** for automation
|
||||
4. **Add Uptime Kuma monitors** for all services
|
||||
5. **Begin ESP32 satellite setup** (Phase 4)
|
||||
|
||||
---
|
||||
|
||||
## File References
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `homeai-agent/skills/home-assistant/openclaw_bridge.py` | Bridge script for HA → OpenClaw |
|
||||
| `homeai-agent/skills/home-assistant/ha-configuration.yaml` | Example HA configuration |
|
||||
| `homeai-voice/wyoming/test-pipeline.sh` | Pipeline smoke test |
|
||||
| `homeai-voice/scripts/launchd/com.homeai.wyoming-satellite.plist` | Satellite service config |
|
||||
Reference in New Issue
Block a user