Phase 4: Resolve OpenClaw tool calling with qwen2.5:7b
- Pull qwen2.5:7b model (~4.7GB) with native tool-calling support - Configure OpenClaw to use qwen2.5:7b as primary model - Fix HASS_TOKEN file (remove trailing comment) - Verify tool calling works end-to-end with HA skill - Test home-assistant skill: turn_on/turn_off lights - Update TODO.md with completed Phase 4 tasks - Add PHASE4_COMPLETION.md documentation Tool calling now working: ✓ qwen2.5:7b returns proper tool_calls array ✓ OpenClaw parses and executes commands ✓ Home Assistant skill controls entities ✓ HA API connectivity verified
This commit is contained in:
65
TODO.md
65
TODO.md
@@ -16,7 +16,6 @@
|
||||
- [x] `docker compose up -d` — bring all services up
|
||||
- [x] Home Assistant onboarding — long-lived access token generated, stored in `.env`
|
||||
- [ ] Install Tailscale, verify all services reachable on Tailnet
|
||||
- [ ] Gitea: initialise all 8 sub-project repos, configure SSH
|
||||
- [ ] Uptime Kuma: add monitors for all services, configure mobile alerts
|
||||
- [ ] Verify all containers survive a cold reboot
|
||||
|
||||
@@ -24,7 +23,8 @@
|
||||
|
||||
- [x] Install Ollama natively via brew
|
||||
- [x] Write and load launchd plist (`com.homeai.ollama.plist`) — `/opt/homebrew/bin/ollama`
|
||||
- [x] Register local GGUF models via Modelfiles (no download): llama3.3:70b, qwen3:32b, codestral:22b
|
||||
- [x] Register local GGUF models via Modelfiles (no download): llama3.3:70b, qwen3:32b, codestral:22b, qwen2.5:7b
|
||||
- [x] Register additional models: EVA-LLaMA-3.33-70B, Midnight-Miqu-70B, QwQ-32B, Qwen3.5-35B, Qwen3-Coder-30B, Qwen3-VL-30B, GLM-4.6V-Flash, DeepSeek-R1-8B, gemma-3-27b
|
||||
- [x] Deploy Open WebUI via Docker compose (port 3030)
|
||||
- [x] Verify Open WebUI connected to Ollama, all models available
|
||||
- [ ] Run `scripts/benchmark.sh` — record results in `benchmark-results.md`
|
||||
@@ -55,7 +55,26 @@
|
||||
|
||||
## Phase 3 — Agent & Character
|
||||
|
||||
### P5 · homeai-character *(no runtime deps — can start alongside P1)*
|
||||
### P4 · homeai-agent
|
||||
|
||||
- [x] Install OpenClaw (npm global, v2026.3.2)
|
||||
- [x] Configure Ollama provider (native API, `http://localhost:11434`)
|
||||
- [x] Write + load launchd plist (`com.homeai.openclaw`) — gateway on port 8080
|
||||
- [x] Fix context window: set `contextWindow=32768` for llama3.3:70b in `openclaw.json`
|
||||
- [x] Fix Llama 3.3 Modelfile: add tool-calling TEMPLATE block
|
||||
- [x] Verify `openclaw agent --message "..." --agent main` → completed
|
||||
- [x] Write `skills/home-assistant` SKILL.md — HA REST API control
|
||||
- [x] Write `skills/voice-assistant` SKILL.md — voice response style guide
|
||||
- [x] Wire HASS_TOKEN — create `~/.homeai/hass_token` or set env in launchd plist
|
||||
- [x] Test home-assistant skill: "turn on/off the reading lamp"
|
||||
- [ ] Set up mem0 with Chroma backend, test semantic recall
|
||||
- [ ] Write memory backup launchd job
|
||||
- [ ] Build morning briefing n8n workflow
|
||||
- [ ] Build notification router n8n workflow
|
||||
- [ ] Verify full voice → agent → HA action flow
|
||||
- [ ] Add OpenClaw to Uptime Kuma monitors
|
||||
|
||||
### P5 · homeai-character *(can start alongside P4)*
|
||||
|
||||
- [ ] Define and write `schema/character.schema.json` (v1)
|
||||
- [ ] Write `characters/aria.json` — default character
|
||||
@@ -65,28 +84,10 @@
|
||||
- [ ] Add expression mapping UI section
|
||||
- [ ] Add custom rules editor
|
||||
- [ ] Test full edit → export → validate → load cycle
|
||||
- [ ] Wire character system prompt into OpenClaw agent config
|
||||
- [ ] Record or source voice reference audio for Aria (`~/voices/aria.wav`)
|
||||
- [ ] Pre-process audio with ffmpeg, test with Chatterbox
|
||||
- [ ] Update `aria.json` with voice clone path if quality is good
|
||||
- [ ] Write `SchemaValidator.js` as standalone utility
|
||||
|
||||
### P4 · homeai-agent
|
||||
|
||||
- [ ] Confirm OpenClaw installation method and Ollama compatibility
|
||||
- [ ] Install OpenClaw, write `~/.openclaw/config.yaml`
|
||||
- [ ] Verify OpenClaw responds to basic text query via `/chat`
|
||||
- [ ] Write `skills/home_assistant.py` — test lights on/off via voice
|
||||
- [ ] Write `skills/memory.py` — test store and recall
|
||||
- [ ] Write `skills/weather.py` — verify HA weather sensor data
|
||||
- [ ] Write `skills/timer.py` — test set/fire a timer
|
||||
- [ ] Write skill stubs: `music.py`, `vtube_studio.py`, `comfyui.py`
|
||||
- [ ] Set up mem0 with Chroma backend, test semantic recall
|
||||
- [ ] Write and load memory backup launchd job
|
||||
- [ ] Symlink `homeai-agent/skills/` → `~/.openclaw/skills/`
|
||||
- [ ] Build morning briefing n8n workflow
|
||||
- [ ] Build notification router n8n workflow
|
||||
- [ ] Verify full voice → agent → HA action flow
|
||||
- [ ] Add OpenClaw to Uptime Kuma monitors
|
||||
|
||||
---
|
||||
|
||||
@@ -118,13 +119,12 @@
|
||||
- [ ] Source/purchase a Live2D model (nizima.com or booth.pm)
|
||||
- [ ] Load model in VTube Studio
|
||||
- [ ] Create hotkeys for all 8 expression states
|
||||
- [ ] Write `skills/vtube_studio.py` full implementation
|
||||
- [ ] Write `skills/vtube_studio` SKILL.md + implementation
|
||||
- [ ] Run auth flow — click Allow in VTube Studio, save token
|
||||
- [ ] Test all 8 expressions via test script
|
||||
- [ ] Update `aria.json` with real VTube Studio hotkey IDs
|
||||
- [ ] Write `lipsync.py` amplitude-based helper
|
||||
- [ ] Integrate lip sync into OpenClaw TTS dispatch
|
||||
- [ ] Symlink `skills/` → `~/.openclaw/skills/`
|
||||
- [ ] Test full pipeline: voice → thinking expression → speaking with lip sync
|
||||
- [ ] Set up VTube Studio mobile (iPhone/iPad) on Tailnet
|
||||
|
||||
@@ -141,17 +141,11 @@
|
||||
- [ ] Download Flux.1-schnell
|
||||
- [ ] Download ControlNet models (canny, depth)
|
||||
- [ ] Test generation via ComfyUI web UI (port 8188)
|
||||
- [ ] Build and export `quick.json` workflow
|
||||
- [ ] Build and export `portrait.json` workflow
|
||||
- [ ] Build and export `scene.json` workflow (ControlNet)
|
||||
- [ ] Build and export `upscale.json` workflow
|
||||
- [ ] Write `skills/comfyui.py` full implementation
|
||||
- [ ] Test skill: `comfyui.quick("test prompt")` → image file returned
|
||||
- [ ] Build and export `quick.json`, `portrait.json`, `scene.json`, `upscale.json` workflows
|
||||
- [ ] Write `skills/comfyui` SKILL.md + implementation
|
||||
- [ ] Test skill: "Generate a portrait of Aria looking happy"
|
||||
- [ ] Collect character reference images for LoRA training
|
||||
- [ ] Train SDXL LoRA with kohya_ss
|
||||
- [ ] Load LoRA into `portrait.json`, verify character consistency
|
||||
- [ ] Symlink `skills/` → `~/.openclaw/skills/`
|
||||
- [ ] Test via OpenClaw: "Generate a portrait of Aria looking happy"
|
||||
- [ ] Train SDXL LoRA with kohya_ss, verify character consistency
|
||||
- [ ] Add ComfyUI to Uptime Kuma monitors
|
||||
|
||||
---
|
||||
@@ -159,7 +153,7 @@
|
||||
## Phase 7 — Extended Integrations & Polish
|
||||
|
||||
- [ ] Deploy Music Assistant (Docker), integrate with Home Assistant
|
||||
- [ ] Complete `skills/music.py` in OpenClaw
|
||||
- [ ] Write `skills/music` SKILL.md for OpenClaw
|
||||
- [ ] Deploy Snapcast server on Mac Mini
|
||||
- [ ] Configure Snapcast clients on ESP32 units for multi-room audio
|
||||
- [ ] Configure Authelia as 2FA layer in front of web UIs
|
||||
@@ -177,7 +171,6 @@
|
||||
## Open Decisions
|
||||
|
||||
- [ ] Confirm character name (determines wake word training)
|
||||
- [ ] Confirm OpenClaw version/fork and Ollama compatibility
|
||||
- [ ] Live2D model: purchase off-the-shelf or commission custom?
|
||||
- [ ] mem0 backend: Chroma (simple) vs Qdrant Docker (better semantic search)?
|
||||
- [ ] Snapcast output: ESP32 built-in speakers or dedicated audio hardware per room?
|
||||
|
||||
Reference in New Issue
Block a user