diff --git a/TODO.md b/TODO.md index b509854..be8eaec 100644 --- a/TODO.md +++ b/TODO.md @@ -44,10 +44,12 @@ - [x] Write + load openWakeWord launchd plist (`com.homeai.wakeword`) — DISABLED, replaced by Wyoming satellite - [x] Write `wyoming/test-pipeline.sh` — smoke test (3/3 passing) - [x] Install Wyoming satellite — handles wake word via HA voice pipeline -- [x] Connect Home Assistant Wyoming integration (STT + TTS + Satellite) - [x] Install Wyoming satellite for Mac Mini (port 10700) -- [ ] Create HA Voice Assistant pipeline with OpenClaw conversation agent +- [x] Write OpenClaw conversation custom component for Home Assistant +- [~] Connect Home Assistant Wyoming integration (STT + TTS + Satellite) — ready to configure in HA UI +- [~] Create HA Voice Assistant pipeline with OpenClaw conversation agent — component ready, needs HA UI setup - [ ] Test HA Assist via browser: type query → hear spoken response +- [ ] Test full voice loop: wake word → STT → OpenClaw → TTS → audio playback - [ ] Install Chatterbox TTS (MPS build), test with sample `.wav` - [ ] Install Qwen3-TTS via MLX (fallback) - [ ] Train custom wake word using character name diff --git a/VOICE_PIPELINE_STATUS.md b/VOICE_PIPELINE_STATUS.md new file mode 100644 index 0000000..33f61ca --- /dev/null +++ b/VOICE_PIPELINE_STATUS.md @@ -0,0 +1,349 @@ +# Voice Pipeline Status Report + +> Last Updated: 2026-03-08 + +--- + +## Executive Summary + +The voice pipeline backend is **fully operational** on the Mac Mini. All services are running and tested: + +- ✅ Wyoming STT (Whisper large-v3) - Port 10300 +- ✅ Wyoming TTS (Kokoro ONNX) - Port 10301 +- ✅ Wyoming Satellite (wake word + audio) - Port 10700 +- ✅ OpenClaw Agent (LLM + skills) - Port 8080 +- ✅ Ollama (local LLM runtime) - Port 11434 + +**Next Step**: Manual Home Assistant UI configuration to connect the pipeline. + +--- + +## What's Working ✅ + +### 1. Speech-to-Text (STT) +- **Service**: Wyoming Faster Whisper +- **Model**: large-v3 (multilingual, high accuracy) +- **Port**: 10300 +- **Status**: Running via launchd (`com.homeai.wyoming-stt`) +- **Test**: `nc -z localhost 10300` ✓ + +### 2. Text-to-Speech (TTS) +- **Service**: Wyoming Kokoro ONNX +- **Voice**: af_heart (default, configurable) +- **Port**: 10301 +- **Status**: Running via launchd (`com.homeai.wyoming-tts`) +- **Test**: `nc -z localhost 10301` ✓ + +### 3. Wyoming Satellite +- **Function**: Wake word detection + audio capture/playback +- **Wake Word**: "hey_jarvis" (openWakeWord model) +- **Port**: 10700 +- **Status**: Running via launchd (`com.homeai.wyoming-satellite`) +- **Test**: `nc -z localhost 10700` ✓ + +### 4. OpenClaw Agent +- **Function**: AI agent with tool calling (home automation, etc.) +- **Gateway**: WebSocket + CLI +- **Port**: 8080 +- **Status**: Running via launchd (`com.homeai.openclaw`) +- **Skills**: home-assistant, voice-assistant +- **Test**: `openclaw agent --message "Hello" --agent main` ✓ + +### 5. Ollama LLM +- **Models**: llama3.3:70b, qwen2.5:7b, and others +- **Port**: 11434 +- **Status**: Running natively +- **Test**: `ollama list` ✓ + +### 6. Home Assistant Integration +- **Custom Component**: OpenClaw Conversation agent created +- **Location**: `homeai-agent/custom_components/openclaw_conversation/` +- **Features**: + - Full conversation agent implementation + - Config flow for UI setup + - CLI fallback if HTTP unavailable + - Error handling and logging +- **Status**: Ready for installation + +--- + +## What's Pending 🔄 + +### Manual Steps Required (Home Assistant UI) + +These steps require access to the Home Assistant web interface at http://10.0.0.199:8123: + +1. **Install OpenClaw Conversation Component** + - Copy component to HA server's `/config/custom_components/` + - Restart Home Assistant + - See: [`homeai-voice/VOICE_PIPELINE_SETUP.md`](homeai-voice/VOICE_PIPELINE_SETUP.md) + +2. **Add Wyoming Integrations** + - Settings → Devices & Services → Add Integration → Wyoming Protocol + - Add STT (10.0.0.199:10300) + - Add TTS (10.0.0.199:10301) + - Add Satellite (10.0.0.199:10700) + +3. **Add OpenClaw Conversation** + - Settings → Devices & Services → Add Integration → OpenClaw Conversation + - Configure: host=10.0.0.199, port=8080, agent=main + +4. **Create Voice Assistant Pipeline** + - Settings → Voice Assistants → Add Assistant + - Name: "HomeAI with OpenClaw" + - STT: Mac Mini STT + - Conversation: OpenClaw Conversation + - TTS: Mac Mini TTS + - Set as preferred + +5. **Test the Pipeline** + - Type test: "What time is it?" in HA Assist + - Voice test: "Hey Jarvis, turn on the reading lamp" + +### Future Enhancements + +6. **Chatterbox TTS** - Voice cloning for character personality +7. **Qwen3-TTS** - Alternative voice synthesis via MLX +8. **Custom Wake Word** - Train with character's name +9. **Uptime Kuma** - Add monitoring for all services + +--- + +## Architecture + +``` +┌──────────────────────────────────────────────────────────────┐ +│ Mac Mini M4 Pro │ +│ (10.0.0.199) │ +├──────────────────────────────────────────────────────────────┤ +│ │ +│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ +│ │ Wyoming │ │ Wyoming │ │ Wyoming │ │ +│ │ STT │ │ TTS │ │ Satellite │ │ +│ │ :10300 │ │ :10301 │ │ :10700 │ │ +│ └─────────────┘ └─────────────┘ └─────────────┘ │ +│ │ +│ ┌─────────────┐ ┌─────────────┐ │ +│ │ OpenClaw │ │ Ollama │ │ +│ │ Gateway │ │ LLM │ │ +│ │ :8080 │ │ :11434 │ │ +│ └─────────────┘ └─────────────┘ │ +│ │ +└──────────────────────────────────────────────────────────────┘ + ▲ + │ Wyoming Protocol + HTTP API + │ +┌──────────────────────────────────────────────────────────────┐ +│ Home Assistant Server │ +│ (10.0.0.199) │ +├──────────────────────────────────────────────────────────────┤ +│ │ +│ ┌─────────────────────────────────────────────────────┐ │ +│ │ Voice Assistant Pipeline │ │ +│ │ │ │ +│ │ Wyoming STT → OpenClaw Conversation → Wyoming TTS │ │ +│ └─────────────────────────────────────────────────────┘ │ +│ │ +│ ┌─────────────────────────────────────────────────────┐ │ +│ │ OpenClaw Conversation Custom Component │ │ +│ │ (Routes to OpenClaw Gateway on Mac Mini) │ │ +│ └─────────────────────────────────────────────────────┘ │ +│ │ +└──────────────────────────────────────────────────────────────┘ +``` + +--- + +## Voice Flow Example + +**User**: "Hey Jarvis, turn on the reading lamp" + +1. **Wake Word Detection** (Wyoming Satellite) + - Detects "Hey Jarvis" + - Starts recording audio + +2. **Speech-to-Text** (Wyoming STT) + - Transcribes: "turn on the reading lamp" + - Sends text to Home Assistant + +3. **Conversation Processing** (HA → OpenClaw) + - HA Voice Pipeline receives text + - Routes to OpenClaw Conversation agent + - OpenClaw Gateway processes request + +4. **LLM Processing** (Ollama) + - llama3.3:70b generates response + - Identifies intent: control light + - Calls home-assistant skill + +5. **Action Execution** (Home Assistant API) + - OpenClaw calls HA REST API + - Turns on "reading lamp" entity + - Returns confirmation + +6. **Text-to-Speech** (Wyoming TTS) + - Generates audio: "I've turned on the reading lamp" + - Sends to Wyoming Satellite + +7. **Audio Playback** (Mac Mini Speaker) + - Plays confirmation audio + - User hears response + +**Total Latency**: Target < 5 seconds + +--- + +## Service Management + +### Check All Services + +```bash +# Quick health check +./homeai-voice/scripts/test-services.sh + +# Individual service status +launchctl list | grep homeai +``` + +### Restart a Service + +```bash +# Example: Restart STT +launchctl unload ~/Library/LaunchAgents/com.homeai.wyoming-stt.plist +launchctl load ~/Library/LaunchAgents/com.homeai.wyoming-stt.plist +``` + +### View Logs + +```bash +# STT logs +tail -f /tmp/homeai-wyoming-stt.log + +# TTS logs +tail -f /tmp/homeai-wyoming-tts.log + +# Satellite logs +tail -f /tmp/homeai-wyoming-satellite.log + +# OpenClaw logs +tail -f /tmp/homeai-openclaw.log +``` + +--- + +## Key Documentation + +| Document | Purpose | +|----------|---------| +| [`homeai-voice/VOICE_PIPELINE_SETUP.md`](homeai-voice/VOICE_PIPELINE_SETUP.md) | Complete setup guide with step-by-step HA configuration | +| [`homeai-voice/RESUME_WORK.md`](homeai-voice/RESUME_WORK.md) | Quick reference for resuming work | +| [`homeai-agent/custom_components/openclaw_conversation/README.md`](homeai-agent/custom_components/openclaw_conversation/README.md) | Custom component documentation | +| [`plans/ha-voice-pipeline-implementation.md`](plans/ha-voice-pipeline-implementation.md) | Detailed implementation plan | +| [`plans/voice-loop-integration.md`](plans/voice-loop-integration.md) | Architecture options and decisions | + +--- + +## Testing + +### Automated Tests + +```bash +# Service health check +./homeai-voice/scripts/test-services.sh + +# OpenClaw test +openclaw agent --message "What time is it?" --agent main + +# Home Assistant skill test +openclaw agent --message "Turn on the reading lamp" --agent main +``` + +### Manual Tests + +1. **Type Test** (HA Assist) + - Open HA UI → Click Assist icon + - Type: "What time is it?" + - Expected: Hear spoken response + +2. **Voice Test** (Wyoming Satellite) + - Say: "Hey Jarvis" + - Wait for beep + - Say: "What time is it?" + - Expected: Hear spoken response + +3. **Home Control Test** + - Say: "Hey Jarvis" + - Say: "Turn on the reading lamp" + - Expected: Light turns on + confirmation + +--- + +## Troubleshooting + +### Services Not Running + +```bash +# Check launchd +launchctl list | grep homeai + +# Reload all services +./homeai-voice/scripts/load-all-launchd.sh +``` + +### Network Issues + +```bash +# Test from Mac Mini to HA +curl http://10.0.0.199:8123/api/ + +# Test ports +nc -z localhost 10300 # STT +nc -z localhost 10301 # TTS +nc -z localhost 10700 # Satellite +nc -z localhost 8080 # OpenClaw +``` + +### Audio Issues + +```bash +# Test microphone +rec -r 16000 -c 1 test.wav trim 0 5 + +# Test speaker +afplay /System/Library/Sounds/Glass.aiff +``` + +--- + +## Next Actions + +1. **Access Home Assistant UI** at http://10.0.0.199:8123 +2. **Follow setup guide**: [`homeai-voice/VOICE_PIPELINE_SETUP.md`](homeai-voice/VOICE_PIPELINE_SETUP.md) +3. **Install OpenClaw component** (see Step 1 in setup guide) +4. **Configure Wyoming integrations** (see Step 2 in setup guide) +5. **Create voice pipeline** (see Step 4 in setup guide) +6. **Test end-to-end** (see Step 5 in setup guide) + +--- + +## Success Metrics + +- [ ] All services show green in health check +- [ ] Wyoming integrations appear in HA +- [ ] OpenClaw Conversation agent registered +- [ ] Voice pipeline created and set as default +- [ ] Typed query returns spoken response +- [ ] Voice query via satellite works +- [ ] Home control via voice works +- [ ] End-to-end latency < 5 seconds +- [ ] Services survive Mac Mini reboot + +--- + +## Project Context + +This is **Phase 2** of the HomeAI project. See [`TODO.md`](TODO.md) for the complete project roadmap. + +**Previous Phase**: Phase 1 - Foundation (Infrastructure + LLM) ✅ Complete +**Current Phase**: Phase 2 - Voice Pipeline 🔄 Backend Complete, HA Integration Pending +**Next Phase**: Phase 3 - Agent & Character (mem0, character system, workflows) diff --git a/homeai-agent/custom_components/install-to-docker-ha.sh b/homeai-agent/custom_components/install-to-docker-ha.sh new file mode 100755 index 0000000..0e8dcc6 --- /dev/null +++ b/homeai-agent/custom_components/install-to-docker-ha.sh @@ -0,0 +1,115 @@ +#!/usr/bin/env bash +# Install OpenClaw Conversation component to Docker Home Assistant on 10.0.0.199 + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +COMPONENT_NAME="openclaw_conversation" +HA_HOST="${HA_HOST:-10.0.0.199}" +HA_CONTAINER="${HA_CONTAINER:-homeassistant}" + +echo "Installing OpenClaw Conversation to Docker Home Assistant" +echo "==========================================================" +echo "Host: $HA_HOST" +echo "Container: $HA_CONTAINER" +echo "" + +# Check if we can reach the host +if ! ping -c 1 -W 2 "$HA_HOST" &>/dev/null; then + echo "Error: Cannot reach $HA_HOST" + echo "Please ensure the server is accessible" + exit 1 +fi + +# Create temporary tarball +TEMP_DIR=$(mktemp -d) +TARBALL="$TEMP_DIR/openclaw_conversation.tar.gz" + +echo "Creating component archive..." +cd "$SCRIPT_DIR" +tar -czf "$TARBALL" \ + --exclude='*.pyc' \ + --exclude='__pycache__' \ + --exclude='.DS_Store' \ + "$COMPONENT_NAME" + +echo "✓ Archive created: $(du -h "$TARBALL" | cut -f1)" +echo "" + +# Copy to remote host +echo "Copying to $HA_HOST:/tmp/..." +if scp -q "$TARBALL" "$HA_HOST:/tmp/openclaw_conversation.tar.gz"; then + echo "✓ File copied successfully" +else + echo "✗ Failed to copy file" + echo "" + echo "Troubleshooting:" + echo " 1. Ensure SSH access is configured: ssh $HA_HOST" + echo " 2. Check SSH keys are set up" + echo " 3. Try manual copy: scp $TARBALL $HA_HOST:/tmp/" + rm -rf "$TEMP_DIR" + exit 1 +fi + +# Extract into container +echo "" +echo "Installing into Home Assistant container..." +ssh "$HA_HOST" << 'EOF' +# Find the Home Assistant container +CONTAINER=$(docker ps --filter "name=homeassistant" --format "{{.Names}}" | head -n 1) + +if [ -z "$CONTAINER" ]; then + echo "Error: Home Assistant container not found" + echo "Available containers:" + docker ps --format "{{.Names}}" + exit 1 +fi + +echo "Found container: $CONTAINER" + +# Copy tarball into container +docker cp /tmp/openclaw_conversation.tar.gz "$CONTAINER:/tmp/" + +# Extract into custom_components +docker exec "$CONTAINER" sh -c ' + mkdir -p /config/custom_components + cd /config/custom_components + tar -xzf /tmp/openclaw_conversation.tar.gz + rm /tmp/openclaw_conversation.tar.gz + ls -la openclaw_conversation/ +' + +# Cleanup +rm /tmp/openclaw_conversation.tar.gz + +echo "" +echo "✓ Component installed successfully!" +EOF + +# Cleanup local temp +rm -rf "$TEMP_DIR" + +echo "" +echo "==========================================================" +echo "Installation complete!" +echo "" +echo "Next steps:" +echo " 1. Restart Home Assistant:" +echo " ssh $HA_HOST 'docker restart $HA_CONTAINER'" +echo "" +echo " 2. Open Home Assistant UI: http://$HA_HOST:8123" +echo "" +echo " 3. Go to Settings → Devices & Services → Add Integration" +echo "" +echo " 4. Search for 'OpenClaw Conversation'" +echo "" +echo " 5. Configure:" +echo " - OpenClaw Host: 10.0.0.101 ⚠️ (Mac Mini IP, NOT $HA_HOST)" +echo " - OpenClaw Port: 8081 (HTTP Bridge port)" +echo " - Agent Name: main" +echo " - Timeout: 30" +echo "" +echo " IMPORTANT: All services (OpenClaw, Wyoming STT/TTS/Satellite) run on" +echo " 10.0.0.101 (Mac Mini), not $HA_HOST (HA server)" +echo "" +echo "See VOICE_PIPELINE_SETUP.md for complete configuration guide" diff --git a/homeai-agent/custom_components/install.sh b/homeai-agent/custom_components/install.sh index a989bc1..74ace38 100755 --- a/homeai-agent/custom_components/install.sh +++ b/homeai-agent/custom_components/install.sh @@ -52,12 +52,12 @@ if [[ -d "$TARGET_DIR" && -f "$TARGET_DIR/manifest.json" ]]; then echo " 1. Restart Home Assistant" echo " 2. Go to Settings → Devices & Services → Add Integration" echo " 3. Search for 'OpenClaw Conversation'" - echo " 4. Configure the settings (host: localhost, port: 8080)" + echo " 4. Configure the settings (host: localhost, port: 8081)" echo "" echo " Or add to configuration.yaml:" echo " openclaw_conversation:" echo " openclaw_host: localhost" - echo " openclaw_port: 8080" + echo " openclaw_port: 8081" echo " agent_name: main" echo " timeout: 30" else diff --git a/homeai-agent/custom_components/openclaw_conversation/README.md b/homeai-agent/custom_components/openclaw_conversation/README.md index cb3b49e..2a7a589 100644 --- a/homeai-agent/custom_components/openclaw_conversation/README.md +++ b/homeai-agent/custom_components/openclaw_conversation/README.md @@ -26,7 +26,7 @@ A custom conversation agent for Home Assistant that routes all voice/text querie 4. Search for "OpenClaw Conversation" 5. Configure the settings: - **OpenClaw Host**: `localhost` (or IP of Mac Mini) - - **OpenClaw Port**: `8080` + - **OpenClaw Port**: `8081` (HTTP Bridge) - **Agent Name**: `main` (or your configured agent) - **Timeout**: `30` seconds @@ -49,7 +49,7 @@ Add to your `configuration.yaml`: ```yaml openclaw_conversation: openclaw_host: localhost - openclaw_port: 8080 + openclaw_port: 8081 agent_name: main timeout: 30 ``` @@ -95,7 +95,7 @@ Once configured, the OpenClaw agent will be available as a conversation agent in 1. Verify OpenClaw host/port settings 2. Ensure OpenClaw is accessible from HA container/host -3. Check network connectivity: `curl http://localhost:8080/status` +3. Check network connectivity: `curl http://localhost:8081/status` ## Files diff --git a/homeai-agent/custom_components/openclaw_conversation/__init__.py b/homeai-agent/custom_components/openclaw_conversation/__init__.py index 69d43c8..7a183af 100644 --- a/homeai-agent/custom_components/openclaw_conversation/__init__.py +++ b/homeai-agent/custom_components/openclaw_conversation/__init__.py @@ -22,7 +22,7 @@ from .const import ( DEFAULT_TIMEOUT, DOMAIN, ) -from .conversation import OpenClawCLIAgent +from .conversation import OpenClawAgent, OpenClawCLIAgent _LOGGER = logging.getLogger(__name__) @@ -76,11 +76,11 @@ async def async_setup_entry(hass: HomeAssistant, entry: ConfigEntry) -> bool: # Store entry data hass.data[DOMAIN][entry.entry_id] = entry.data - # Register the conversation agent - agent = OpenClawCLIAgent(hass, entry.data) + # Register the conversation agent (HTTP-based for cross-network access) + agent = OpenClawAgent(hass, entry.data) from homeassistant.components import conversation - conversation.async_set_agent(hass, DOMAIN, agent) + conversation.async_set_agent(hass, entry, agent) _LOGGER.info("OpenClaw Conversation agent registered from config entry") @@ -91,7 +91,7 @@ async def async_unload_entry(hass: HomeAssistant, entry: ConfigEntry) -> bool: """Unload a config entry.""" # Unregister the conversation agent from homeassistant.components import conversation - conversation.async_unset_agent(hass, DOMAIN) + conversation.async_unset_agent(hass, entry) hass.data[DOMAIN].pop(entry.entry_id, None) diff --git a/homeai-agent/custom_components/openclaw_conversation/const.py b/homeai-agent/custom_components/openclaw_conversation/const.py index a8bade6..098635b 100644 --- a/homeai-agent/custom_components/openclaw_conversation/const.py +++ b/homeai-agent/custom_components/openclaw_conversation/const.py @@ -10,7 +10,7 @@ CONF_TIMEOUT = "timeout" # Defaults DEFAULT_HOST = "localhost" -DEFAULT_PORT = 8080 +DEFAULT_PORT = 8081 # OpenClaw HTTP Bridge (not 8080 gateway) DEFAULT_AGENT = "main" DEFAULT_TIMEOUT = 30 diff --git a/homeai-agent/custom_components/openclaw_conversation/conversation.py b/homeai-agent/custom_components/openclaw_conversation/conversation.py index f378dd4..a09d379 100644 --- a/homeai-agent/custom_components/openclaw_conversation/conversation.py +++ b/homeai-agent/custom_components/openclaw_conversation/conversation.py @@ -187,8 +187,6 @@ class OpenClawCLIAgent(AbstractConversationAgent): async def _call_openclaw_cli(self, message: str) -> str: """Call OpenClaw CLI and return the response.""" - import subprocess - cmd = [ "openclaw", "agent", @@ -196,6 +194,7 @@ class OpenClawCLIAgent(AbstractConversationAgent): "--agent", self.agent_name, ] + proc = None try: proc = await asyncio.create_subprocess_exec( *cmd, @@ -215,6 +214,9 @@ class OpenClawCLIAgent(AbstractConversationAgent): return stdout.decode().strip() except asyncio.TimeoutError: + if proc is not None: + proc.kill() + await proc.wait() _LOGGER.error("Timeout calling OpenClaw CLI") return "I'm sorry, the request timed out." except FileNotFoundError: diff --git a/homeai-agent/custom_components/package-for-ha.sh b/homeai-agent/custom_components/package-for-ha.sh new file mode 100755 index 0000000..ebf6636 --- /dev/null +++ b/homeai-agent/custom_components/package-for-ha.sh @@ -0,0 +1,46 @@ +#!/usr/bin/env bash +# Package OpenClaw Conversation component for Home Assistant installation + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +COMPONENT_NAME="openclaw_conversation" +OUTPUT_DIR="$SCRIPT_DIR/dist" +TIMESTAMP=$(date +%Y%m%d_%H%M%S) +ARCHIVE_NAME="openclaw_conversation_${TIMESTAMP}.tar.gz" + +echo "Packaging OpenClaw Conversation component..." +echo "" + +# Create dist directory +mkdir -p "$OUTPUT_DIR" + +# Create tarball +cd "$SCRIPT_DIR" +tar -czf "$OUTPUT_DIR/$ARCHIVE_NAME" \ + --exclude='*.pyc' \ + --exclude='__pycache__' \ + --exclude='.DS_Store' \ + "$COMPONENT_NAME" + +# Create latest symlink +cd "$OUTPUT_DIR" +ln -sf "$ARCHIVE_NAME" openclaw_conversation_latest.tar.gz + +echo "✓ Package created: $OUTPUT_DIR/$ARCHIVE_NAME" +echo "" +echo "Installation instructions:" +echo "" +echo "1. Copy to Home Assistant server:" +echo " scp $OUTPUT_DIR/$ARCHIVE_NAME user@10.0.0.199:/tmp/" +echo "" +echo "2. SSH into Home Assistant server:" +echo " ssh user@10.0.0.199" +echo "" +echo "3. Extract to custom_components:" +echo " cd /config/custom_components" +echo " tar -xzf /tmp/$ARCHIVE_NAME" +echo "" +echo "4. Restart Home Assistant" +echo "" +echo "Or use the install.sh script for automated installation." diff --git a/homeai-agent/launchd/com.homeai.openclaw-bridge.plist b/homeai-agent/launchd/com.homeai.openclaw-bridge.plist new file mode 100644 index 0000000..df3b19f --- /dev/null +++ b/homeai-agent/launchd/com.homeai.openclaw-bridge.plist @@ -0,0 +1,40 @@ + + + + + Label + com.homeai.openclaw-bridge + + ProgramArguments + + /opt/homebrew/bin/python3 + /Users/aodhan/gitea/homeai/homeai-agent/openclaw-http-bridge.py + --port + 8081 + --host + 0.0.0.0 + + + RunAtLoad + + + KeepAlive + + + StandardOutPath + /tmp/homeai-openclaw-bridge.log + + StandardErrorPath + /tmp/homeai-openclaw-bridge-error.log + + ThrottleInterval + 10 + + EnvironmentVariables + + PATH + /opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin + + + diff --git a/homeai-agent/openclaw-http-bridge.py b/homeai-agent/openclaw-http-bridge.py new file mode 100644 index 0000000..5d37de5 --- /dev/null +++ b/homeai-agent/openclaw-http-bridge.py @@ -0,0 +1,141 @@ +#!/usr/bin/env python3 +""" +OpenClaw HTTP Bridge + +A simple HTTP server that translates HTTP POST requests to OpenClaw CLI calls. +This allows Home Assistant (running in Docker on a different machine) to +communicate with OpenClaw via HTTP. + +Usage: + python3 openclaw-http-bridge.py [--port 8081] + +Endpoints: + POST /api/agent/message + { + "message": "Your message here", + "agent": "main" + } + + Returns: + { + "response": "OpenClaw response text" + } +""" + +import argparse +import json +import subprocess +import sys +from http.server import HTTPServer, BaseHTTPRequestHandler +from urllib.parse import urlparse + + +class OpenClawBridgeHandler(BaseHTTPRequestHandler): + """HTTP request handler for OpenClaw bridge.""" + + def log_message(self, format, *args): + """Log requests to stderr.""" + print(f"[OpenClaw Bridge] {self.address_string()} - {format % args}") + + def _send_json_response(self, status_code: int, data: dict): + """Send a JSON response.""" + self.send_response(status_code) + self.send_header("Content-Type", "application/json") + self.end_headers() + self.wfile.write(json.dumps(data).encode()) + + def do_POST(self): + """Handle POST requests.""" + parsed_path = urlparse(self.path) + + # Only handle the agent message endpoint + if parsed_path.path != "/api/agent/message": + self._send_json_response(404, {"error": "Not found"}) + return + + # Read request body + content_length = int(self.headers.get("Content-Length", 0)) + if content_length == 0: + self._send_json_response(400, {"error": "Empty request body"}) + return + + try: + body = self.rfile.read(content_length).decode() + data = json.loads(body) + except json.JSONDecodeError: + self._send_json_response(400, {"error": "Invalid JSON"}) + return + + # Extract parameters + message = data.get("message", "").strip() + agent = data.get("agent", "main") + + if not message: + self._send_json_response(400, {"error": "Message is required"}) + return + + # Call OpenClaw CLI (use full path for launchd compatibility) + try: + result = subprocess.run( + ["/opt/homebrew/bin/openclaw", "agent", "--message", message, "--agent", agent], + capture_output=True, + text=True, + timeout=30, + check=True + ) + response_text = result.stdout.strip() + self._send_json_response(200, {"response": response_text}) + except subprocess.TimeoutExpired: + self._send_json_response(504, {"error": "OpenClaw command timed out"}) + except subprocess.CalledProcessError as e: + error_msg = e.stderr.strip() if e.stderr else "OpenClaw command failed" + self._send_json_response(500, {"error": error_msg}) + except FileNotFoundError: + self._send_json_response(500, {"error": "OpenClaw CLI not found"}) + except Exception as e: + self._send_json_response(500, {"error": str(e)}) + + def do_GET(self): + """Handle GET requests (health check).""" + parsed_path = urlparse(self.path) + + if parsed_path.path == "/status" or parsed_path.path == "/": + self._send_json_response(200, { + "status": "ok", + "service": "OpenClaw HTTP Bridge", + "version": "1.0.0" + }) + else: + self._send_json_response(404, {"error": "Not found"}) + + +def main(): + """Run the HTTP bridge server.""" + parser = argparse.ArgumentParser(description="OpenClaw HTTP Bridge") + parser.add_argument( + "--port", + type=int, + default=8081, + help="Port to listen on (default: 8081)" + ) + parser.add_argument( + "--host", + default="0.0.0.0", + help="Host to bind to (default: 0.0.0.0)" + ) + args = parser.parse_args() + + server = HTTPServer((args.host, args.port), OpenClawBridgeHandler) + print(f"OpenClaw HTTP Bridge running on http://{args.host}:{args.port}") + print(f"Endpoint: POST http://{args.host}:{args.port}/api/agent/message") + print("Press Ctrl+C to stop") + + try: + server.serve_forever() + except KeyboardInterrupt: + print("\nShutting down...") + server.shutdown() + + +if __name__ == "__main__": + main() diff --git a/homeai-voice/TROUBLESHOOTING.md b/homeai-voice/TROUBLESHOOTING.md new file mode 100644 index 0000000..f2efc92 --- /dev/null +++ b/homeai-voice/TROUBLESHOOTING.md @@ -0,0 +1,420 @@ +# Voice Pipeline Troubleshooting Guide + +> Common issues and solutions for the voice pipeline setup + +--- + +## Network Configuration + +**Important**: The services are split across two machines: + +| Service | Machine | IP Address | +|---------|---------|------------| +| OpenClaw Gateway | Mac Mini | 10.0.0.101 | +| Wyoming STT | Mac Mini | 10.0.0.101 | +| Wyoming TTS | Mac Mini | 10.0.0.101 | +| Wyoming Satellite | Mac Mini | 10.0.0.101 | +| Ollama | Mac Mini | 10.0.0.101 | +| Home Assistant | Server (Docker) | 10.0.0.199 | + +--- + +## Issue: OpenClaw Conversation Cannot Connect + +### Symptoms +- Integration installed but shows connection error +- HA logs show timeout or connection refused +- Error: "Cannot connect to OpenClaw service" + +### Root Cause +The OpenClaw Conversation integration is configured with the wrong host IP. It needs to point to the Mac Mini (10.0.0.101), not the HA server (10.0.0.199). + +### Solution + +1. **Open Home Assistant UI** at http://10.0.0.199:8123 + +2. **Go to Settings → Devices & Services** + +3. **Find "OpenClaw Conversation"** integration + +4. **Click "Configure"** (or delete and re-add) + +5. **Set the correct configuration:** + - **OpenClaw Host**: `10.0.0.101` (Mac Mini IP, NOT 10.0.0.199) + - **OpenClaw Port**: `8080` + - **Agent Name**: `main` + - **Timeout**: `30` + +6. **Save** and verify connection + +### Verify Network Connectivity + +From the HA server, test if it can reach OpenClaw: + +```bash +# SSH to HA server +ssh 10.0.0.199 + +# Test OpenClaw connectivity +curl http://10.0.0.101:8080/status + +# Or use nc +nc -z 10.0.0.101 8080 && echo "OpenClaw reachable" || echo "Cannot reach OpenClaw" +``` + +From the Mac Mini, verify OpenClaw is listening: + +```bash +# Check OpenClaw is running +launchctl list | grep openclaw + +# Check it's listening on all interfaces +lsof -i :8080 + +# Test locally +curl http://localhost:8080/status +``` + +--- + +## Issue: Wyoming Services Cannot Connect + +### Symptoms +- Wyoming integrations show as unavailable +- HA cannot reach STT/TTS services +- Timeout errors in HA logs + +### Solution + +Wyoming services are also on the Mac Mini (10.0.0.101): + +1. **Go to Settings → Devices & Services** + +2. **For each Wyoming integration**, verify the host is set to **10.0.0.101**: + - Wyoming STT: `10.0.0.101:10300` + - Wyoming TTS: `10.0.0.101:10301` + - Wyoming Satellite: `10.0.0.101:10700` + +3. **Test connectivity from HA server:** + +```bash +ssh 10.0.0.199 +nc -z 10.0.0.101 10300 # STT +nc -z 10.0.0.101 10301 # TTS +nc -z 10.0.0.101 10700 # Satellite +``` + +--- + +## Issue: Firewall Blocking Connections + +### Symptoms +- Services work locally on Mac Mini +- Cannot connect from HA server +- Connection timeout errors + +### Solution + +Check Mac Mini firewall settings: + +```bash +# Check firewall status +sudo /usr/libexec/ApplicationFirewall/socketfilterfw --getglobalstate + +# If enabled, add exceptions for the services +sudo /usr/libexec/ApplicationFirewall/socketfilterfw --add /opt/homebrew/bin/ollama +sudo /usr/libexec/ApplicationFirewall/socketfilterfw --add /usr/local/bin/openclaw + +# Or temporarily disable for testing (not recommended for production) +sudo /usr/libexec/ApplicationFirewall/socketfilterfw --setglobalstate off +``` + +--- + +## Issue: OpenClaw CLI Not Found in HA Container + +### Symptoms +- Integration uses CLI fallback +- Error: "OpenClaw CLI not found" +- Component works but responses fail + +### Root Cause +The `openclaw` command is not available inside the HA Docker container. The integration should use the HTTP API, not CLI. + +### Solution + +The OpenClawCLIAgent is a fallback. Ensure the integration is using the HTTP API: + +1. Check the integration configuration uses the correct host/port +2. Verify OpenClaw Gateway is accessible via HTTP +3. The component will automatically use HTTP if available + +--- + +## Issue: Voice Pipeline Not Responding + +### Symptoms +- Wake word detected but no response +- Audio captured but not transcribed +- Transcription works but no TTS output + +### Debugging Steps + +1. **Check all services are running:** + +```bash +# On Mac Mini +./homeai-voice/scripts/test-services.sh +``` + +2. **Test each component individually:** + +```bash +# Test wake word detection +# Say "Hey Jarvis" and check satellite logs +tail -f /tmp/homeai-wyoming-satellite.log + +# Test STT +# Check if audio is being transcribed +tail -f /tmp/homeai-wyoming-stt.log + +# Test OpenClaw +openclaw agent --message "Hello" --agent main + +# Test TTS +tail -f /tmp/homeai-wyoming-tts.log +``` + +3. **Check HA Voice Pipeline configuration:** + - Settings → Voice Assistants + - Verify pipeline uses correct STT, Conversation, and TTS + - Ensure OpenClaw Conversation is selected + +4. **Test from HA Assist:** + - Type a query in HA Assist panel + - Check if you get a response + - This bypasses wake word and audio capture + +--- + +## Monitoring Wake Word Detection + +To see when the wake word ("Hey Jarvis") is being detected in real-time: + +### Option 1: Watch Satellite Logs (Recommended) + +The Wyoming Satellite handles wake word detection and audio streaming: + +```bash +# Terminal 1: Watch satellite logs +tail -f /tmp/homeai-wyoming-satellite.log +``` + +**What to look for:** +- `Wake word detected` - Wake word was heard +- `Streaming audio` - Audio being sent to STT +- `Connected to server` - Connection status + +### Option 2: Watch Wake Word Service Logs + +```bash +# Terminal 1: Watch wake word detection logs +tail -f /tmp/homeai-wakeword.log +``` + +### Option 3: Watch All Voice Pipeline Logs + +```bash +# Terminal 1: Watch all voice-related logs +tail -f /tmp/homeai-*.log | grep -E "(wake|satellite|stt|tts|openclaw)" +``` + +### Test Wake Word Detection + +While watching the logs, try this: + +1. **Say clearly**: "Hey Jarvis" (or your configured wake word) +2. **Wait** for the acknowledgment beep +3. **Speak your command**: "What time is it?" +4. **Check logs** for activity + +### Expected Log Output + +When wake word is detected, you should see: + +``` +[wyoming_satellite] Wake word detected +[wyoming_satellite] Streaming audio to stt +[wyoming_satellite] Connected to 10.0.0.101:10300 +``` + +--- + +## Issue: Audio Playback Not Working + +### Symptoms +- Pipeline works but no audio output +- TTS generates audio but satellite doesn't play it +- Silent responses + +### Solution + +1. **Check audio output device:** + +```bash +# On Mac Mini +afplay /System/Library/Sounds/Glass.aiff +``` + +2. **Check satellite configuration:** + +```bash +# View satellite config +cat ~/Library/LaunchAgents/com.homeai.wyoming-satellite.plist + +# Check logs for audio errors +tail -f /tmp/homeai-wyoming-satellite.log +``` + +3. **Verify SoX is installed:** + +```bash +which play +brew install sox +``` + +--- + +## Issue: High Latency (>5 seconds) + +### Symptoms +- Long delay between wake word and response +- Slow transcription or TTS generation + +### Solutions + +1. **Check network latency:** + +```bash +# From HA server to Mac Mini +ping 10.0.0.101 +``` + +2. **Check Ollama model size:** + +```bash +# Smaller models are faster +ollama list + +# Switch to faster model in OpenClaw config +# qwen2.5:7b is faster than llama3.3:70b +``` + +3. **Check system resources:** + +```bash +# On Mac Mini +top -l 1 | grep -E "CPU|PhysMem" +``` + +--- + +## Correct Configuration Summary + +### OpenClaw Conversation Integration +- Host: `10.0.0.101` (Mac Mini) +- Port: `8080` +- Agent: `main` +- Timeout: `30` + +### Wyoming STT Integration +- Host: `10.0.0.101` (Mac Mini) +- Port: `10300` + +### Wyoming TTS Integration +- Host: `10.0.0.101` (Mac Mini) +- Port: `10301` + +### Wyoming Satellite Integration +- Host: `10.0.0.101` (Mac Mini) +- Port: `10700` + +--- + +## Testing Checklist + +- [ ] All services running on Mac Mini (10.0.0.101) +- [ ] HA can ping Mac Mini: `ping 10.0.0.101` +- [ ] HA can reach OpenClaw: `curl http://10.0.0.101:8080/status` +- [ ] HA can reach Wyoming STT: `nc -z 10.0.0.101 10300` +- [ ] HA can reach Wyoming TTS: `nc -z 10.0.0.101 10301` +- [ ] HA can reach Wyoming Satellite: `nc -z 10.0.0.101 10700` +- [ ] OpenClaw Conversation integration configured with 10.0.0.101 +- [ ] Wyoming integrations configured with 10.0.0.101 +- [ ] Voice pipeline created and set as default +- [ ] Test query in HA Assist returns response + +--- + +## Bugs Fixed During Setup + +The following bugs were discovered and fixed during initial setup (2026-03-08): + +### 1. OpenClaw Network Binding + +**Problem**: OpenClaw gateway was only listening on localhost (127.0.0.1), not accessible from HA server. + +**Fix**: Added `"bind": "lan"` to `~/.openclaw/openclaw.json`: + +```json +{ + "gateway": { + "port": 8080, + "mode": "local", + "bind": "lan", + "auth": { "token": "..." } + } +} +``` + +### 2. Custom Component API Error + +**Problem**: `async_set_agent()` was being called with `DOMAIN` (string) instead of `entry` (ConfigEntry object). + +**Fix**: Changed parameter in `homeai-agent/custom_components/openclaw_conversation/__init__.py`: + +```python +# Line 83 +conversation.async_set_agent(hass, entry, agent) # Was: DOMAIN + +# Line 94 +conversation.async_unset_agent(hass, entry) # Was: DOMAIN +``` + +### 3. TTS Server Missing Version + +**Problem**: `TtsProgram` initialization was missing required `version` parameter. + +**Fix**: Added `version="1.0.0"` in `homeai-voice/tts/wyoming_kokoro_server.py` line 58. + +### 4. Voice Commands Not Working (CLI Not in Docker) + +**Problem**: HA Docker container couldn't access `openclaw` CLI. + +**Fix**: Created OpenClaw HTTP Bridge (`homeai-agent/openclaw-http-bridge.py`) on port 8081 that translates HTTP POST requests to OpenClaw CLI calls. The custom component now uses port 8081 (HTTP bridge) instead of 8080 (gateway). + +--- + +## Getting Help + +If issues persist: + +1. **Check service logs:** + - Mac Mini: `/tmp/homeai-*.log` + - HA: Settings → System → Logs + +2. **Verify network connectivity** between machines + +3. **Test each component** individually before testing the full pipeline + +4. **Review configuration** in [`VOICE_PIPELINE_SETUP.md`](VOICE_PIPELINE_SETUP.md) diff --git a/homeai-voice/VOICE_PIPELINE_SETUP.md b/homeai-voice/VOICE_PIPELINE_SETUP.md new file mode 100644 index 0000000..759e399 --- /dev/null +++ b/homeai-voice/VOICE_PIPELINE_SETUP.md @@ -0,0 +1,435 @@ +# Voice Pipeline Setup Guide + +> Complete guide to setting up the end-to-end voice pipeline with OpenClaw integration + +--- + +## Network Configuration + +**Important**: Services are split across two machines: + +| Service | Port | Location | +|---------|------|----------| +| Wyoming STT (Whisper large-v3) | 10300 | Mac Mini (10.0.0.101) | +| Wyoming TTS (Kokoro ONNX) | 10301 | Mac Mini (10.0.0.101) | +| Wyoming Satellite | 10700 | Mac Mini (10.0.0.101) | +| openWakeWord | - | Mac Mini (10.0.0.101) | +| OpenClaw Gateway | 8080 | Mac Mini (10.0.0.101) | +| Ollama | 11434 | Mac Mini (10.0.0.101) | +| Home Assistant (Docker) | 8123 | Server (10.0.0.199) | + +**All integrations must point to 10.0.0.101 (Mac Mini), not 10.0.0.199 (HA server).** + +--- + +## Current Status + +### ✅ Services Running on Mac Mini (10.0.0.101) + +| Service | Port | Status | +|---------|------|--------| +| Wyoming STT | 10300 | ✅ Running | +| Wyoming TTS | 10301 | ✅ Running | +| Wyoming Satellite | 10700 | ✅ Running | +| openWakeWord | - | ✅ Running | +| OpenClaw Gateway | 8080 | ✅ Running | +| Ollama | 11434 | ✅ Running | + +### ✅ Completed +- Wyoming STT/TTS services installed and running +- Wyoming Satellite installed and running +- OpenClaw agent configured with home-assistant skill +- Custom OpenClaw conversation component created + +### 🔄 Next Steps +1. Install OpenClaw conversation component in Home Assistant +2. Configure Wyoming integrations in HA +3. Create voice assistant pipeline with OpenClaw +4. Test the full voice loop + +--- + +## Step 1: Install OpenClaw Conversation Component + +Home Assistant is running in Docker on server 10.0.0.199. Use the automated installation script. + +### Option A: Automated Installation (Recommended) + +```bash +# From Mac Mini, run the installation script +cd ~/gitea/homeai/homeai-agent/custom_components +./install-to-docker-ha.sh + +# The script will: +# 1. Create a tarball of the component +# 2. Copy it to the HA server via SCP +# 3. Extract it into the HA Docker container +# 4. Provide next steps +``` + +**Requirements:** +- SSH access to 10.0.0.199 +- SSH keys configured (or password access) + +### Option B: Manual Installation via SSH + +```bash +# 1. Create tarball +cd ~/gitea/homeai/homeai-agent/custom_components +tar -czf openclaw_conversation.tar.gz openclaw_conversation/ + +# 2. Copy to HA server +scp openclaw_conversation.tar.gz 10.0.0.199:/tmp/ + +# 3. SSH to HA server and install +ssh 10.0.0.199 +CONTAINER=$(docker ps --filter "name=homeassistant" --format "{{.Names}}" | head -n 1) +docker cp /tmp/openclaw_conversation.tar.gz $CONTAINER:/tmp/ +docker exec $CONTAINER sh -c 'cd /config/custom_components && tar -xzf /tmp/openclaw_conversation.tar.gz' +docker restart $CONTAINER +``` + +### Option D: Using Home Assistant File Editor (Manual) + +1. Open Home Assistant UI at http://10.0.0.199:8123 +2. Install the **File Editor** add-on if not already installed +3. Create directory: `/config/custom_components/openclaw_conversation/` +4. Copy each file from `homeai-agent/custom_components/openclaw_conversation/`: + - `__init__.py` + - `config_flow.py` + - `const.py` + - `conversation.py` + - `manifest.json` + - `strings.json` + +### Verify Installation + +After installation, restart Home Assistant: + +```bash +# Via SSH +ssh 10.0.0.199 'docker restart homeassistant' + +# Or via HA UI +# Settings → System → Restart +``` + +Check logs for any errors: +- **Settings → System → Logs** +- Look for "OpenClaw Conversation" in the logs + +--- + +## Step 2: Configure Wyoming Integrations + +### Add Wyoming STT (Speech-to-Text) + +1. Go to **Settings → Devices & Services → Add Integration** +2. Search for **"Wyoming Protocol"** +3. Configure: + - **Host**: `10.0.0.101` ⚠️ **Mac Mini IP, not HA server IP (10.0.0.199)** + - **Port**: `10300` + - **Name**: `Mac Mini STT` +4. Click **Submit** + +### Add Wyoming TTS (Text-to-Speech) + +1. Click **Add Integration** again +2. Search for **"Wyoming Protocol"** +3. Configure: + - **Host**: `10.0.0.101` ⚠️ **Mac Mini IP** + - **Port**: `10301` + - **Name**: `Mac Mini TTS` +4. Click **Submit** + +### Add Wyoming Satellite + +1. Click **Add Integration** again +2. Search for **"Wyoming Protocol"** +3. Configure: + - **Host**: `10.0.0.101` ⚠️ **Mac Mini IP** + - **Port**: `10700` + - **Name**: `Mac Mini Living Room` +4. Click **Submit** + +### Verify Integrations + +All three Wyoming integrations should appear in **Settings → Devices & Services**. + +--- + +## Step 3: Add OpenClaw Conversation Agent + +### Via UI (Recommended) + +1. Go to **Settings → Devices & Services → Add Integration** +2. Search for **"OpenClaw Conversation"** +3. Configure: + - **OpenClaw Host**: `10.0.0.101` ⚠️ **Mac Mini IP, not HA server IP (10.0.0.199)** + - **OpenClaw Port**: `8080` + - **Agent Name**: `main` + - **Timeout**: `30` seconds +4. Click **Submit** + +### Via YAML (Alternative) + +Add to `/config/configuration.yaml`: + +```yaml +openclaw_conversation: + openclaw_host: 10.0.0.101 # Mac Mini IP + openclaw_port: 8080 + agent_name: main + timeout: 30 +``` + +Then restart Home Assistant. + +--- + +## Step 4: Create Voice Assistant Pipeline + +1. Go to **Settings → Voice Assistants** +2. Click **Add Assistant** +3. Configure: + - **Name**: `HomeAI with OpenClaw` + - **Language**: `English` + - **Speech-to-Text**: Select `Mac Mini STT` (Wyoming) + - **Conversation Agent**: Select `OpenClaw Conversation` + - **Text-to-Speech**: Select `Mac Mini TTS` (Wyoming) +4. Click **Create** + +### Set as Default + +1. In **Settings → Voice Assistants** +2. Click the three dots next to "HomeAI with OpenClaw" +3. Select **Set as preferred** + +--- + +## Step 5: Test the Pipeline + +### Test 1: Text Input → TTS Output + +1. Open Home Assistant UI +2. Click the **Assist** icon (microphone) in the top-right corner +3. Type: `"What time is it?"` +4. Press Enter + +**Expected Result**: You should hear a spoken response via Kokoro TTS + +### Test 2: Voice Input → OpenClaw → TTS Output + +1. Ensure Wyoming Satellite is running on Mac Mini: + ```bash + launchctl list | grep wyoming-satellite + ``` + +2. Say the wake word: **"Hey Jarvis"** +3. Wait for the beep/acknowledgment +4. Speak: **"What time is it?"** + +**Expected Result**: You should hear a spoken response + +### Test 3: Home Assistant Control via Voice + +1. Say: **"Hey Jarvis"** +2. Speak: **"Turn on the reading lamp"** + +**Expected Result**: +- OpenClaw processes the request +- Home Assistant skill executes the action +- Light turns on +- You hear a confirmation via TTS + +--- + +## Troubleshooting + +### Issue: OpenClaw Conversation not appearing in integrations + +**Solution**: +1. Verify files are in `/config/custom_components/openclaw_conversation/` +2. Check Home Assistant logs for errors +3. Ensure `manifest.json` is valid JSON +4. Restart Home Assistant + +### Issue: Wyoming services not connecting + +**Solution**: +1. Verify services are running on Mac Mini: + ```bash + launchctl list | grep wyoming + nc -z 10.0.0.199 10300 # Test STT + nc -z 10.0.0.199 10301 # Test TTS + nc -z 10.0.0.199 10700 # Test Satellite + ``` + +2. Check firewall rules on Mac Mini +3. Verify Home Assistant can reach Mac Mini network + +### Issue: OpenClaw not responding + +**Solution**: +1. Verify OpenClaw is running: + ```bash + launchctl list | grep openclaw + pgrep -f openclaw + ``` + +2. Test OpenClaw CLI directly: + ```bash + openclaw agent --message "Hello" --agent main + ``` + +3. Check OpenClaw logs: + ```bash + tail -f /tmp/homeai-openclaw.log + ``` + +4. Verify OpenClaw can reach Home Assistant: + ```bash + curl http://10.0.0.199:8123/api/ + ``` + +### Issue: No audio output from satellite + +**Solution**: +1. Check satellite logs: + ```bash + tail -f /tmp/homeai-wyoming-satellite.log + ``` + +2. Test audio output: + ```bash + afplay /System/Library/Sounds/Glass.aiff + ``` + +3. Verify SoX is installed: + ```bash + which play + brew install sox + ``` + +### Issue: Wake word not detected + +**Solution**: +1. Check wakeword service: + ```bash + launchctl list | grep wakeword + ``` + +2. Test microphone input: + ```bash + # Record a test + rec -r 16000 -c 1 test.wav trim 0 5 + ``` + +3. Adjust wake word threshold in satellite config + +--- + +## Voice Pipeline Flow + +``` +┌─────────────────┐ +│ USB Mic │ +│ (Mac Mini) │ +└────────┬────────┘ + │ + ▼ +┌─────────────────┐ +│ Wake Word │ +│ Detection │ +│ (hey_jarvis) │ +└────────┬────────┘ + │ wake detected + ▼ +┌─────────────────┐ +│ Wyoming │ +│ Satellite │ +│ :10700 │ +└────────┬────────┘ + │ audio stream + ▼ +┌─────────────────┐ +│ Wyoming STT │ +│ (Whisper) │ +│ :10300 │ +└────────┬────────┘ + │ transcript + ▼ +┌─────────────────┐ +│ Home Assistant │ +│ Voice Pipeline │ +└────────┬────────┘ + │ text + ▼ +┌─────────────────┐ +│ OpenClaw │ +│ Conversation │ +│ Agent │ +└────────┬────────┘ + │ message + ▼ +┌─────────────────┐ +│ OpenClaw │ +│ Gateway │ +│ :8080 │ +└────────┬────────┘ + │ + ▼ +┌─────────────────┐ +│ Ollama LLM │ +│ + Skills │ +│ :11434 │ +└────────┬────────┘ + │ response + ▼ +┌─────────────────┐ +│ Wyoming TTS │ +│ (Kokoro) │ +│ :10301 │ +└────────┬────────┘ + │ audio + ▼ +┌─────────────────┐ +│ Speaker │ +│ (Mac Mini) │ +└─────────────────┘ +``` + +--- + +## Next Steps After Setup + +1. **Install Chatterbox TTS** for voice cloning +2. **Set up mem0** for long-term memory +3. **Configure n8n workflows** for automation +4. **Add Uptime Kuma monitors** for all services +5. **Begin ESP32 satellite setup** (Phase 4) + +--- + +## Files Reference + +| File | Purpose | +|------|---------| +| [`homeai-agent/custom_components/openclaw_conversation/`](../homeai-agent/custom_components/openclaw_conversation/) | Custom HA component | +| [`homeai-agent/skills/home-assistant/openclaw_bridge.py`](../homeai-agent/skills/home-assistant/openclaw_bridge.py) | Bridge script | +| [`homeai-voice/scripts/launchd/`](scripts/launchd/) | Service plists | +| [`plans/ha-voice-pipeline-implementation.md`](../plans/ha-voice-pipeline-implementation.md) | Detailed implementation plan | +| [`plans/voice-loop-integration.md`](../plans/voice-loop-integration.md) | Architecture options | + +--- + +## Success Criteria + +- [ ] Wyoming STT/TTS/Satellite appear in HA integrations +- [ ] OpenClaw Conversation agent appears in HA integrations +- [ ] Voice assistant pipeline created with OpenClaw +- [ ] Typed query in Assist returns spoken response +- [ ] Voice query via satellite returns spoken response +- [ ] "Turn on the reading lamp" command works end-to-end +- [ ] Latency under 5 seconds from wake to response +- [ ] All services survive Mac Mini reboot diff --git a/homeai-voice/WYOMING_SATELLITE_SETUP.md b/homeai-voice/WYOMING_SATELLITE_SETUP.md new file mode 100644 index 0000000..ed88ee4 --- /dev/null +++ b/homeai-voice/WYOMING_SATELLITE_SETUP.md @@ -0,0 +1,195 @@ +# Wyoming Satellite Setup Guide + +> How to configure the Wyoming Satellite wizard in Home Assistant + +--- + +## When Adding Wyoming Satellite Integration + +When you add the Wyoming Satellite integration, Home Assistant will open a wizard to configure a voice assistant. Here's what to do: + +--- + +## Option 1: Skip Wizard and Configure Later (Recommended) + +**Best approach if you haven't created the OpenClaw pipeline yet:** + +1. **Skip/Cancel the wizard** - just add the satellite integration without configuring the pipeline +2. The satellite will be added but not assigned to a pipeline yet +3. Continue with creating the voice assistant pipeline (see below) +4. Come back and assign the satellite to the pipeline later + +--- + +## Option 2: Use Default Pipeline Temporarily + +**If you want to test the satellite immediately:** + +1. In the wizard, select **"Home Assistant"** as the pipeline (default) +2. This will use HA's built-in conversation agent (not OpenClaw) +3. You can test basic commands like "What time is it?" +4. Later, switch to the OpenClaw pipeline once it's created + +--- + +## Creating the Voice Assistant Pipeline + +**Do this BEFORE configuring the satellite (or after if you used Option 2):** + +### Step 1: Create the Pipeline + +1. Go to **Settings → Voice Assistants** +2. Click **Add Assistant** +3. Configure: + - **Name**: `HomeAI with OpenClaw` + - **Language**: `English` + - **Speech-to-Text**: Select `Mac Mini STT` (Wyoming) + - **Conversation Agent**: Select `OpenClaw Conversation` + - **Text-to-Speech**: Select `Mac Mini TTS` (Wyoming) +4. Click **Create** + +### Step 2: Set as Preferred (Optional) + +1. In the Voice Assistants list, find "HomeAI with OpenClaw" +2. Click the three dots (⋮) +3. Select **Set as preferred** + +This makes it the default pipeline for all new satellites. + +--- + +## Assigning Satellite to Pipeline + +### If You Skipped the Wizard + +1. Go to **Settings → Devices & Services** +2. Find **Wyoming Protocol** (the satellite entry) +3. Click **Configure** +4. Select **Pipeline**: `HomeAI with OpenClaw` +5. Click **Submit** + +### If You Used the Default Pipeline + +1. Go to **Settings → Devices & Services** +2. Find **Wyoming Protocol** (the satellite entry) +3. Click **Configure** +4. Change **Pipeline** from "Home Assistant" to `HomeAI with OpenClaw` +5. Click **Submit** + +--- + +## Satellite Configuration Details + +The wizard may ask for these details: + +| Field | Value | Notes | +|-------|-------|-------| +| **Name** | `Mac Mini Living Room` | Or any name you prefer | +| **Pipeline** | `HomeAI with OpenClaw` | Select after creating it | +| **Wake Word** | `hey_jarvis` | Should be auto-detected | +| **Audio Input** | Default | Detected from satellite | +| **Audio Output** | Default | Detected from satellite | + +--- + +## Complete Voice Pipeline Flow + +Once configured, the flow will be: + +``` +1. Say "Hey Jarvis" → Wake word detected by satellite +2. Satellite captures audio → Sends to Wyoming STT (10.0.0.101:10300) +3. STT transcribes → Sends text to HA Voice Pipeline +4. HA routes to OpenClaw Conversation agent +5. OpenClaw processes → Calls Ollama LLM + skills +6. Response generated → Sent to Wyoming TTS (10.0.0.101:10301) +7. TTS generates audio → Sent back to satellite +8. Satellite plays audio → You hear the response +``` + +--- + +## Testing the Pipeline + +### Test 1: Via HA Assist (No Wake Word) + +1. Open Home Assistant UI +2. Click the **Assist** icon (microphone) in top-right +3. Type: `"What time is it?"` +4. Press Enter +5. **Expected**: You should hear a spoken response via TTS + +### Test 2: Via Satellite (With Wake Word) + +1. Say: **"Hey Jarvis"** +2. Wait for acknowledgment beep +3. Say: **"What time is it?"** +4. **Expected**: You should hear a spoken response + +### Test 3: Home Control + +1. Say: **"Hey Jarvis"** +2. Say: **"Turn on the reading lamp"** +3. **Expected**: + - Light turns on + - You hear confirmation: "I've turned on the reading lamp" + +--- + +## Troubleshooting + +### Satellite Not Responding + +1. **Check satellite is online**: + - Settings → Devices & Services → Wyoming Protocol + - Should show "Connected" + +2. **Check pipeline is assigned**: + - Configure satellite → Verify pipeline is set + +3. **Check satellite logs** on Mac Mini: + ```bash + tail -f /tmp/homeai-wyoming-satellite.log + ``` + +### Wake Word Not Detected + +1. **Check microphone**: + - Satellite logs should show audio input + - Try speaking louder or closer to mic + +2. **Adjust wake word sensitivity**: + - May need to configure threshold in satellite settings + +### No Audio Output + +1. **Check speaker**: + ```bash + afplay /System/Library/Sounds/Glass.aiff + ``` + +2. **Check TTS is working**: + - Test via HA Assist (type query) + - Should hear response + +--- + +## Summary + +**Recommended Setup Order:** + +1. ✅ Add Wyoming STT integration (10.0.0.101:10300) +2. ✅ Add Wyoming TTS integration (10.0.0.101:10301) +3. ✅ Add OpenClaw Conversation integration (10.0.0.101:8080) +4. ✅ Create voice assistant pipeline "HomeAI with OpenClaw" +5. ✅ Add Wyoming Satellite integration (10.0.0.101:10700) +6. ✅ Assign satellite to "HomeAI with OpenClaw" pipeline +7. ✅ Test the complete voice loop + +--- + +## Related Documentation + +- [`VOICE_PIPELINE_SETUP.md`](VOICE_PIPELINE_SETUP.md) - Complete setup guide +- [`TROUBLESHOOTING.md`](TROUBLESHOOTING.md) - Troubleshooting guide +- [`OPENCLAW_NETWORK_FIX.md`](OPENCLAW_NETWORK_FIX.md) - Network access fix diff --git a/homeai-voice/scripts/test-services.sh b/homeai-voice/scripts/test-services.sh new file mode 100755 index 0000000..238b499 --- /dev/null +++ b/homeai-voice/scripts/test-services.sh @@ -0,0 +1,140 @@ +#!/usr/bin/env bash +# Test all voice pipeline services are running and accessible + +set -euo pipefail + +echo "Testing Voice Pipeline Services..." +echo "==================================" +echo "" + +# Colors for output +GREEN='\033[0;32m' +RED='\033[0;31m' +YELLOW='\033[1;33m' +NC='\033[0m' # No Color + +# Test function +test_service() { + local name=$1 + local host=$2 + local port=$3 + + if nc -z -w 2 "$host" "$port" 2>/dev/null; then + echo -e "${GREEN}✓${NC} $name ($host:$port)" + return 0 + else + echo -e "${RED}✗${NC} $name ($host:$port) - NOT ACCESSIBLE" + return 1 + fi +} + +# Test launchd service +test_launchd() { + local name=$1 + local service=$2 + + if launchctl list | grep -q "$service"; then + echo -e "${GREEN}✓${NC} $name (launchd: $service)" + return 0 + else + echo -e "${RED}✗${NC} $name (launchd: $service) - NOT RUNNING" + return 1 + fi +} + +# Test command availability +test_command() { + local name=$1 + local cmd=$2 + + if command -v "$cmd" &> /dev/null; then + echo -e "${GREEN}✓${NC} $name command available" + return 0 + else + echo -e "${RED}✗${NC} $name command NOT FOUND" + return 1 + fi +} + +echo "1. Network Services" +echo "-------------------" +test_service "Wyoming STT" "localhost" "10300" +test_service "Wyoming TTS" "localhost" "10301" +test_service "Wyoming Satellite" "localhost" "10700" +test_service "OpenClaw Gateway" "localhost" "8080" +test_service "Ollama" "localhost" "11434" +test_service "Home Assistant" "10.0.0.199" "8123" +echo "" + +echo "2. Launchd Services" +echo "-------------------" +test_launchd "Wyoming STT" "com.homeai.wyoming-stt" +test_launchd "Wyoming TTS" "com.homeai.wyoming-tts" +test_launchd "Wyoming Satellite" "com.homeai.wyoming-satellite" +test_launchd "Wake Word" "com.homeai.wakeword" +test_launchd "OpenClaw" "com.homeai.openclaw" +test_launchd "Ollama" "com.homeai.ollama" +echo "" + +echo "3. Commands" +echo "-----------" +test_command "OpenClaw" "openclaw" +test_command "Ollama" "ollama" +test_command "SoX (play)" "play" +test_command "SoX (rec)" "rec" +echo "" + +echo "4. Wyoming Protocol Test" +echo "------------------------" +if command -v wyoming-client &> /dev/null; then + echo -e "${YELLOW}Testing STT...${NC}" + # Would need a test audio file + echo " (Manual test required with audio file)" + + echo -e "${YELLOW}Testing TTS...${NC}" + # Would need Wyoming client + echo " (Manual test required with Wyoming client)" +else + echo -e "${YELLOW}⚠${NC} wyoming-client not installed (optional)" +fi +echo "" + +echo "5. OpenClaw Test" +echo "----------------" +if command -v openclaw &> /dev/null; then + echo -e "${YELLOW}Testing OpenClaw agent...${NC}" + if timeout 10 openclaw agent --message "Hello" --agent main &>/dev/null; then + echo -e "${GREEN}✓${NC} OpenClaw agent responding" + else + echo -e "${RED}✗${NC} OpenClaw agent not responding" + fi +else + echo -e "${RED}✗${NC} OpenClaw command not found" +fi +echo "" + +echo "6. Audio Devices" +echo "----------------" +if command -v rec &> /dev/null; then + echo "Input devices:" + rec -n stat trim 0 0.1 2>&1 | grep -i "input" || echo " (Unable to detect)" + + echo "Output devices:" + if command -v afplay &> /dev/null; then + echo -e "${GREEN}✓${NC} afplay available for audio output" + else + echo -e "${RED}✗${NC} afplay not available" + fi +else + echo -e "${YELLOW}⚠${NC} SoX not installed - audio recording unavailable" +fi +echo "" + +echo "==================================" +echo "Test complete!" +echo "" +echo "Next steps:" +echo "1. Install OpenClaw conversation component in Home Assistant" +echo "2. Configure Wyoming integrations in HA UI" +echo "3. Create voice assistant pipeline" +echo "4. Test with: 'Hey Jarvis, what time is it?'" diff --git a/homeai-voice/tts/wyoming_kokoro_server.py b/homeai-voice/tts/wyoming_kokoro_server.py index c0d0a9c..730af20 100644 --- a/homeai-voice/tts/wyoming_kokoro_server.py +++ b/homeai-voice/tts/wyoming_kokoro_server.py @@ -56,6 +56,7 @@ class KokoroEventHandler(AsyncEventHandler): url="https://github.com/thewh1teagle/kokoro-onnx", ), installed=True, + version="1.0.0", voices=[ TtsVoice( name=self._default_voice,