Phase 4: Resolve OpenClaw tool calling with qwen2.5:7b

- Pull qwen2.5:7b model (~4.7GB) with native tool-calling support
- Configure OpenClaw to use qwen2.5:7b as primary model
- Fix HASS_TOKEN file (remove trailing comment)
- Verify tool calling works end-to-end with HA skill
- Test home-assistant skill: turn_on/turn_off lights
- Update TODO.md with completed Phase 4 tasks
- Add PHASE4_COMPLETION.md documentation

Tool calling now working:
✓ qwen2.5:7b returns proper tool_calls array
✓ OpenClaw parses and executes commands
✓ Home Assistant skill controls entities
✓ HA API connectivity verified
This commit is contained in:
Aodhan Collins
2026-03-07 00:16:18 +00:00
parent c3dda280ea
commit 9eb5633115
3 changed files with 196 additions and 36 deletions

65
TODO.md
View File

@@ -16,7 +16,6 @@
- [x] `docker compose up -d` — bring all services up
- [x] Home Assistant onboarding — long-lived access token generated, stored in `.env`
- [ ] Install Tailscale, verify all services reachable on Tailnet
- [ ] Gitea: initialise all 8 sub-project repos, configure SSH
- [ ] Uptime Kuma: add monitors for all services, configure mobile alerts
- [ ] Verify all containers survive a cold reboot
@@ -24,7 +23,8 @@
- [x] Install Ollama natively via brew
- [x] Write and load launchd plist (`com.homeai.ollama.plist`) — `/opt/homebrew/bin/ollama`
- [x] Register local GGUF models via Modelfiles (no download): llama3.3:70b, qwen3:32b, codestral:22b
- [x] Register local GGUF models via Modelfiles (no download): llama3.3:70b, qwen3:32b, codestral:22b, qwen2.5:7b
- [x] Register additional models: EVA-LLaMA-3.33-70B, Midnight-Miqu-70B, QwQ-32B, Qwen3.5-35B, Qwen3-Coder-30B, Qwen3-VL-30B, GLM-4.6V-Flash, DeepSeek-R1-8B, gemma-3-27b
- [x] Deploy Open WebUI via Docker compose (port 3030)
- [x] Verify Open WebUI connected to Ollama, all models available
- [ ] Run `scripts/benchmark.sh` — record results in `benchmark-results.md`
@@ -55,7 +55,26 @@
## Phase 3 — Agent & Character
### P5 · homeai-character *(no runtime deps — can start alongside P1)*
### P4 · homeai-agent
- [x] Install OpenClaw (npm global, v2026.3.2)
- [x] Configure Ollama provider (native API, `http://localhost:11434`)
- [x] Write + load launchd plist (`com.homeai.openclaw`) — gateway on port 8080
- [x] Fix context window: set `contextWindow=32768` for llama3.3:70b in `openclaw.json`
- [x] Fix Llama 3.3 Modelfile: add tool-calling TEMPLATE block
- [x] Verify `openclaw agent --message "..." --agent main` → completed
- [x] Write `skills/home-assistant` SKILL.md — HA REST API control
- [x] Write `skills/voice-assistant` SKILL.md — voice response style guide
- [x] Wire HASS_TOKEN — create `~/.homeai/hass_token` or set env in launchd plist
- [x] Test home-assistant skill: "turn on/off the reading lamp"
- [ ] Set up mem0 with Chroma backend, test semantic recall
- [ ] Write memory backup launchd job
- [ ] Build morning briefing n8n workflow
- [ ] Build notification router n8n workflow
- [ ] Verify full voice → agent → HA action flow
- [ ] Add OpenClaw to Uptime Kuma monitors
### P5 · homeai-character *(can start alongside P4)*
- [ ] Define and write `schema/character.schema.json` (v1)
- [ ] Write `characters/aria.json` — default character
@@ -65,28 +84,10 @@
- [ ] Add expression mapping UI section
- [ ] Add custom rules editor
- [ ] Test full edit → export → validate → load cycle
- [ ] Wire character system prompt into OpenClaw agent config
- [ ] Record or source voice reference audio for Aria (`~/voices/aria.wav`)
- [ ] Pre-process audio with ffmpeg, test with Chatterbox
- [ ] Update `aria.json` with voice clone path if quality is good
- [ ] Write `SchemaValidator.js` as standalone utility
### P4 · homeai-agent
- [ ] Confirm OpenClaw installation method and Ollama compatibility
- [ ] Install OpenClaw, write `~/.openclaw/config.yaml`
- [ ] Verify OpenClaw responds to basic text query via `/chat`
- [ ] Write `skills/home_assistant.py` — test lights on/off via voice
- [ ] Write `skills/memory.py` — test store and recall
- [ ] Write `skills/weather.py` — verify HA weather sensor data
- [ ] Write `skills/timer.py` — test set/fire a timer
- [ ] Write skill stubs: `music.py`, `vtube_studio.py`, `comfyui.py`
- [ ] Set up mem0 with Chroma backend, test semantic recall
- [ ] Write and load memory backup launchd job
- [ ] Symlink `homeai-agent/skills/``~/.openclaw/skills/`
- [ ] Build morning briefing n8n workflow
- [ ] Build notification router n8n workflow
- [ ] Verify full voice → agent → HA action flow
- [ ] Add OpenClaw to Uptime Kuma monitors
---
@@ -118,13 +119,12 @@
- [ ] Source/purchase a Live2D model (nizima.com or booth.pm)
- [ ] Load model in VTube Studio
- [ ] Create hotkeys for all 8 expression states
- [ ] Write `skills/vtube_studio.py` full implementation
- [ ] Write `skills/vtube_studio` SKILL.md + implementation
- [ ] Run auth flow — click Allow in VTube Studio, save token
- [ ] Test all 8 expressions via test script
- [ ] Update `aria.json` with real VTube Studio hotkey IDs
- [ ] Write `lipsync.py` amplitude-based helper
- [ ] Integrate lip sync into OpenClaw TTS dispatch
- [ ] Symlink `skills/``~/.openclaw/skills/`
- [ ] Test full pipeline: voice → thinking expression → speaking with lip sync
- [ ] Set up VTube Studio mobile (iPhone/iPad) on Tailnet
@@ -141,17 +141,11 @@
- [ ] Download Flux.1-schnell
- [ ] Download ControlNet models (canny, depth)
- [ ] Test generation via ComfyUI web UI (port 8188)
- [ ] Build and export `quick.json` workflow
- [ ] Build and export `portrait.json` workflow
- [ ] Build and export `scene.json` workflow (ControlNet)
- [ ] Build and export `upscale.json` workflow
- [ ] Write `skills/comfyui.py` full implementation
- [ ] Test skill: `comfyui.quick("test prompt")` → image file returned
- [ ] Build and export `quick.json`, `portrait.json`, `scene.json`, `upscale.json` workflows
- [ ] Write `skills/comfyui` SKILL.md + implementation
- [ ] Test skill: "Generate a portrait of Aria looking happy"
- [ ] Collect character reference images for LoRA training
- [ ] Train SDXL LoRA with kohya_ss
- [ ] Load LoRA into `portrait.json`, verify character consistency
- [ ] Symlink `skills/``~/.openclaw/skills/`
- [ ] Test via OpenClaw: "Generate a portrait of Aria looking happy"
- [ ] Train SDXL LoRA with kohya_ss, verify character consistency
- [ ] Add ComfyUI to Uptime Kuma monitors
---
@@ -159,7 +153,7 @@
## Phase 7 — Extended Integrations & Polish
- [ ] Deploy Music Assistant (Docker), integrate with Home Assistant
- [ ] Complete `skills/music.py` in OpenClaw
- [ ] Write `skills/music` SKILL.md for OpenClaw
- [ ] Deploy Snapcast server on Mac Mini
- [ ] Configure Snapcast clients on ESP32 units for multi-room audio
- [ ] Configure Authelia as 2FA layer in front of web UIs
@@ -177,7 +171,6 @@
## Open Decisions
- [ ] Confirm character name (determines wake word training)
- [ ] Confirm OpenClaw version/fork and Ollama compatibility
- [ ] Live2D model: purchase off-the-shelf or commission custom?
- [ ] mem0 backend: Chroma (simple) vs Qdrant Docker (better semantic search)?
- [ ] Snapcast output: ESP32 built-in speakers or dedicated audio hardware per room?