feat: Music Assistant, Claude primary LLM, model tag in chat, setup.sh rewrite
- Deploy Music Assistant on Pi (10.0.0.199:8095) with host networking for Chromecast mDNS discovery, Spotify + SMB library support - Switch primary LLM from Ollama to Claude Sonnet 4 (Anthropic API), local models remain as fallback - Add model info tag under each assistant message in dashboard chat, persisted in conversation JSON - Rewrite homeai-agent/setup.sh: loads .env, injects API keys into plists, symlinks plists to ~/Library/LaunchAgents/, smoke tests services - Update install_service() in common.sh to use symlinks instead of copies - Open UFW ports on Pi for Music Assistant (8095, 8097, 8927) - Add ANTHROPIC_API_KEY to openclaw + bridge launchd plists Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
39
CLAUDE.md
39
CLAUDE.md
@@ -18,14 +18,15 @@ A self-hosted, always-on personal AI assistant running on a **Mac Mini M4 Pro (6
|
||||
| Storage | 1TB SSD |
|
||||
| Network | Gigabit Ethernet |
|
||||
|
||||
All AI inference runs locally on this machine. No cloud dependency required (cloud APIs optional).
|
||||
Primary LLM is Claude Sonnet 4 via Anthropic API. Local Ollama models available as fallback. All other inference (STT, TTS, image gen) runs locally.
|
||||
|
||||
---
|
||||
|
||||
## Core Stack
|
||||
|
||||
### AI & LLM
|
||||
- **Ollama** — local LLM runtime (target models: Llama 3.3 70B, Qwen 2.5 72B)
|
||||
- **Claude Sonnet 4** — primary LLM via Anthropic API (`anthropic/claude-sonnet-4-20250514`), used for all agent interactions
|
||||
- **Ollama** — local LLM runtime (fallback models: Llama 3.3 70B, Qwen 3.5 35B-A3B, Qwen 2.5 7B)
|
||||
- **Model keep-warm daemon** — `preload-models.sh` runs as a loop, checks every 5 min, re-pins evicted models with `keep_alive=-1`. Keeps `qwen2.5:7b` (small/fast) and `$HOMEAI_MEDIUM_MODEL` (default: `qwen3.5:35b-a3b`) always loaded in VRAM. Medium model is configurable via env var for per-persona model assignment.
|
||||
- **Open WebUI** — browser-based chat interface, runs as Docker container
|
||||
|
||||
@@ -45,13 +46,15 @@ All AI inference runs locally on this machine. No cloud dependency required (clo
|
||||
### Smart Home
|
||||
- **Home Assistant** — smart home control platform (Docker)
|
||||
- **Wyoming Protocol** — bridges Whisper STT + Kokoro/Piper TTS into Home Assistant
|
||||
- **Music Assistant** — self-hosted music control, integrates with Home Assistant
|
||||
- **Music Assistant** — self-hosted music control (Docker on Pi at 10.0.0.199:8095), Spotify + SMB library + Chromecast players
|
||||
- **Snapcast** — multi-room synchronised audio output
|
||||
|
||||
### AI Agent / Orchestration
|
||||
- **OpenClaw** — primary AI agent layer; receives voice commands, calls tools, manages personality
|
||||
- **OpenClaw Skills** — 13 skills total: home-assistant, image-generation, voice-assistant, vtube-studio, memory, service-monitor, character, routine, music, workflow, gitea, calendar, mode
|
||||
- **n8n** — visual workflow automation (Docker), chains AI actions
|
||||
- **Character Memory System** — two-tier JSON-based memories (personal per-character + general shared), injected into LLM system prompt with budget truncation
|
||||
- **Public/Private Mode** — routes requests to local Ollama (private) or cloud LLMs (public) with per-category overrides via `active-mode.json`. Default primary model is Claude Sonnet 4.
|
||||
|
||||
### Character & Personality
|
||||
- **Character Schema v2** — JSON spec with background, dialogue_style, appearance, skills, gaze_presets (v1 auto-migrated)
|
||||
@@ -93,7 +96,8 @@ ESP32-S3-BOX-3 (room)
|
||||
→ HA conversation agent → OpenClaw HTTP Bridge
|
||||
→ Bridge resolves character (satellite_id → character mapping)
|
||||
→ Bridge builds system prompt (profile + memories) and writes TTS config to state file
|
||||
→ OpenClaw CLI → Ollama LLM generates response
|
||||
→ Bridge checks active-mode.json for model routing (private=local, public=cloud)
|
||||
→ OpenClaw CLI → LLM generates response (Claude Sonnet 4 default, Ollama fallback)
|
||||
→ Response dispatched:
|
||||
→ Wyoming TTS reads state file → routes to Kokoro (local) or ElevenLabs (cloud)
|
||||
→ Audio sent back to ESP32-S3-BOX-3 (spoken response)
|
||||
@@ -154,11 +158,13 @@ This works for both ESP32/HA pipeline and dashboard chat.
|
||||
4. **OpenClaw** — installed, onboarded, connected to Ollama and Home Assistant ✅
|
||||
5. **ESP32-S3-BOX-3** — ESPHome flash, Wyoming Satellite, display faces ✅
|
||||
6. **Character system** — schema v2, dashboard editor, memory system, per-character TTS routing ✅
|
||||
7. **Animated visual** — PNG/GIF character visual for the web assistant (initial visual layer)
|
||||
8. **Android app** — companion app for mobile access to the assistant
|
||||
9. **ComfyUI** — image generation online, character-consistent model workflows
|
||||
10. **Extended integrations** — n8n workflows, Music Assistant, Snapcast, Gitea, code-server
|
||||
11. **Polish** — Authelia, Tailscale hardening, iOS widgets
|
||||
7. **OpenClaw skills expansion** — 9 new skills (memory, monitor, character, routine, music, workflow, gitea, calendar, mode) + public/private mode routing ✅
|
||||
8. **Music Assistant** — deployed on Pi (10.0.0.199:8095), Spotify + SMB + Chromecast players ✅
|
||||
9. **Animated visual** — PNG/GIF character visual for the web assistant (initial visual layer)
|
||||
10. **Android app** — companion app for mobile access to the assistant
|
||||
11. **ComfyUI** — image generation online, character-consistent model workflows
|
||||
12. **Extended integrations** — Snapcast, code-server
|
||||
13. **Polish** — Authelia, Tailscale hardening, iOS widgets
|
||||
|
||||
### Stretch Goals
|
||||
- **Live2D / VTube Studio** — full Live2D model with WebSocket API bridge (requires learning Live2D tooling)
|
||||
@@ -167,18 +173,27 @@ This works for both ESP32/HA pipeline and dashboard chat.
|
||||
|
||||
## Key Paths & Conventions
|
||||
|
||||
- All Docker compose files: `~/server/docker/`
|
||||
- Launchd plists (source): `homeai-*/launchd/` (symlinked to `~/Library/LaunchAgents/`)
|
||||
- Docker compose (Mac Mini): `homeai-infra/docker/docker-compose.yml`
|
||||
- Docker compose (Pi/SELBINA): `~/docker/selbina/` on 10.0.0.199
|
||||
- OpenClaw skills: `~/.openclaw/skills/`
|
||||
- OpenClaw workspace tools: `~/.openclaw/workspace/TOOLS.md`
|
||||
- OpenClaw config: `~/.openclaw/openclaw.json`
|
||||
- Character configs: `~/homeai-data/characters/`
|
||||
- Character memories: `~/homeai-data/memories/`
|
||||
- Conversation history: `~/homeai-data/conversations/`
|
||||
- Active TTS state: `~/homeai-data/active-tts-voice.json`
|
||||
- Active mode state: `~/homeai-data/active-mode.json`
|
||||
- Satellite → character map: `~/homeai-data/satellite-map.json`
|
||||
- Local routines: `~/homeai-data/routines/`
|
||||
- Voice reminders: `~/homeai-data/reminders.json`
|
||||
- Whisper models: `~/models/whisper/`
|
||||
- Ollama models: managed by Ollama at `~/.ollama/models/`
|
||||
- ComfyUI models: `~/ComfyUI/models/`
|
||||
- Voice reference audio: `~/voices/`
|
||||
- Gitea repos root: `~/gitea/`
|
||||
- Music Assistant (Pi): `~/docker/selbina/music-assistant/` on 10.0.0.199
|
||||
- Skills user guide: `homeai-agent/SKILLS_GUIDE.md`
|
||||
|
||||
---
|
||||
|
||||
@@ -188,6 +203,8 @@ This works for both ESP32/HA pipeline and dashboard chat.
|
||||
- ESP32-S3-BOX-3 units are dumb satellites — all intelligence stays on Mac Mini
|
||||
- The character JSON schema (from Character Manager) should be treated as a versioned spec; pipeline components read from it, never hardcode personality values
|
||||
- OpenClaw skills are the primary extension mechanism — new capabilities = new skills
|
||||
- Prefer local models; cloud API keys (Anthropic, OpenAI) are fallback only
|
||||
- Primary LLM is Claude Sonnet 4 (Anthropic API); local Ollama models are available as fallback
|
||||
- Launchd plists are symlinked from repo source to ~/Library/LaunchAgents/ — edit source, then bootout/bootstrap to reload
|
||||
- Music Assistant runs on Pi (10.0.0.199), not Mac Mini — needs host networking for Chromecast mDNS discovery
|
||||
- VTube Studio API bridge should be a standalone OpenClaw skill with clear event interface
|
||||
- mem0 memory store should be backed up as part of regular Gitea commits
|
||||
|
||||
Reference in New Issue
Block a user