aodhan/homeai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: memory v2, prompt styles, Dream/GAZE integration, Wyoming TTS fix

Browse Source 
SQLite + sqlite-vec replaces JSON memory files with semantic search,
follow-up injection, privacy levels, and lifecycle management.

Six prompt styles (quick/standard/creative/roleplayer/game-master/storyteller)
with per-style Claude model tiering (Haiku/Sonnet/Opus), temperature control,
and section stripping. Characters can set default style and per-style overrides.

Dream character import and GAZE character linking in the dashboard editor
with auto-populated fields, cover image resolution, and preset assignment.

Bridge: session isolation (conversation_id / 12h satellite buckets),
model routing refactor, PUT/DELETE support, memory REST endpoints.

Dashboard: mobile-responsive sidebar, retry button, style picker in chat,
follow-up banner, memory lifecycle/privacy UI, cloud model options in editor.

Wyoming TTS: upgraded to v1.8.0 for HA 1.7.2 compatibility.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Aodhan Collins
2026-03-24 22:31:04 +00:00
parent c3bae6fdc0
commit 56580a2cb2
34 changed files with 2891 additions and 467 deletions
Download Patch File Download Diff File Expand all files Collapse all files

64
CLAUDE.md
View File

@@ -18,14 +18,14 @@ A self-hosted, always-on personal AI assistant running on a **Mac Mini M4 Pro (6
| Storage | 1TB SSD |
| Network | Gigabit Ethernet |
Primary LLM is Claude Sonnet 4 via Anthropic API. Local Ollama models available as fallback. All other inference (STT, TTS, image gen) runs locally.
Primary LLMs are Claude 4.5/4.6 family via Anthropic API (Haiku for quick, Sonnet for standard, Opus for creative/RP). Local Ollama models available as fallback. All other inference (STT, TTS, image gen) runs locally.
---
## Core Stack
### AI & LLM
- **Claude Sonnet 4** — primary LLM via Anthropic API (`anthropic/claude-sonnet-4-20250514`), used for all agent interactions
- **Claude 4.5/4.6 family** — primary LLMs via Anthropic API, tiered per prompt style: Haiku 4.5 (quick commands), Sonnet 4.6 (standard/creative), Opus 4.6 (roleplay/storytelling)
- **Ollama** — local LLM runtime (fallback models: Llama 3.3 70B, Qwen 3.5 35B-A3B, Qwen 2.5 7B)
- **Model keep-warm daemon** — `preload-models.sh` runs as a loop, checks every 5 min, re-pins evicted models with `keep_alive=-1`. Keeps `qwen2.5:7b` (small/fast) and `$HOMEAI_MEDIUM_MODEL` (default: `qwen3.5:35b-a3b`) always loaded in VRAM. Medium model is configurable via env var for per-persona model assignment.
- **Open WebUI** — browser-based chat interface, runs as Docker container
@@ -53,13 +53,16 @@ Primary LLM is Claude Sonnet 4 via Anthropic API. Local Ollama models available
- **OpenClaw** — primary AI agent layer; receives voice commands, calls tools, manages personality
- **OpenClaw Skills** — 13 skills total: home-assistant, image-generation, voice-assistant, vtube-studio, memory, service-monitor, character, routine, music, workflow, gitea, calendar, mode
- **n8n** — visual workflow automation (Docker), chains AI actions
- **Character Memory System** — two-tier JSON-based memories (personal per-character + general shared), injected into LLM system prompt with budget truncation
- **Public/Private Mode** — routes requests to local Ollama (private) or cloud LLMs (public) with per-category overrides via `active-mode.json`. Default primary model is Claude Sonnet 4.
- **Character Memory System** — SQLite + sqlite-vec semantic search (personal per-character + general shared + follow-ups), injected into LLM system prompt with context-aware retrieval
- **Prompt Styles** — 6 styles (quick, standard, creative, roleplayer, game-master, storyteller) with per-style model routing, temperature, and section stripping. JSON templates in `homeai-agent/prompt-styles/`
- **Public/Private Mode** — routes requests to local Ollama (private) or cloud LLMs (public) with per-category overrides via `active-mode.json`. Default primary model is Claude Sonnet 4.6, with per-style model tiering (Haiku/Sonnet/Opus).
### Character & Personality
- **Character Schema v2** — JSON spec with background, dialogue_style, appearance, skills, gaze_presets (v1 auto-migrated)
- **Character Schema v2** — JSON spec with background, dialogue_style, appearance, skills, gaze_presets, dream_id, gaze_character, prompt style overrides (v1 auto-migrated)
- **HomeAI Dashboard** — unified web app: character editor, chat, memory manager, service dashboard
- **Dream** — character management service (http://10.0.0.101:3000), REST API for character CRUD with GAZE integration for cover images
- **Character MCP Server** — LLM-assisted character creation via Fandom wiki/Wikipedia lookup (Docker)
- **GAZE** — image generation service (http://10.0.0.101:5782), REST API for presets, characters, and job-based image generation
- Character config stored as JSON files in `~/homeai-data/characters/`, consumed by bridge for system prompt construction
### Visual Representation
@@ -97,7 +100,7 @@ ESP32-S3-BOX-3 (room)
→ Bridge resolves character (satellite_id → character mapping)
→ Bridge builds system prompt (profile + memories) and writes TTS config to state file
→ Bridge checks active-mode.json for model routing (private=local, public=cloud)
→ OpenClaw CLI → LLM generates response (Claude Sonnet 4 default, Ollama fallback)
→ OpenClaw CLI → LLM generates response (Claude Haiku/Sonnet/Opus per style, Ollama fallback)
→ Response dispatched:
→ Wyoming TTS reads state file → routes to Kokoro (local) or ElevenLabs (cloud)
→ Audio sent back to ESP32-S3-BOX-3 (spoken response)
@@ -130,15 +133,32 @@ Each character is a JSON file in `~/homeai-data/characters/` with:
- **Profile fields** — background, appearance, dialogue_style, skills array
- **TTS config** — engine (kokoro/elevenlabs), kokoro_voice, elevenlabs_voice_id, elevenlabs_model, speed
- **GAZE presets** — array of `{preset, trigger}` for image generation styles
- **Dream link** — `dream_id` for syncing character data from Dream service
- **GAZE link** — `gaze_character` for auto-assigned cover image and presets
- **Prompt style config** — `default_prompt_style`, `prompt_style_overrides` for per-style tuning
- **Custom prompt rules** — trigger/response overrides for specific contexts
### Memory System
Two-tier memory stored as JSON in `~/homeai-data/memories/`:
- **Personal memories** (`personal/{character_id}.json`) — per-character, about user interactions
- **General memories** (`general.json`) — shared operational knowledge (tool usage, device info, routines)
SQLite + sqlite-vec database at `~/homeai-data/memories/memories.db`:
- **Personal memories** — per-character, semantic/episodic/relational/opinion types
- **General memories** — shared operational knowledge (character_id = "general")
- **Follow-ups** — LLM-driven questions injected into system prompt, auto-resolve after 2 surfacings or 48h
- **Privacy levels** — public, sensitive, local_only (local_only excluded from cloud model requests)
- **Semantic search** — sentence-transformers all-MiniLM-L6-v2 embeddings (384 dims) for context-aware retrieval
- Core module: `homeai-agent/memory_store.py` (imported by bridge + memory-ctl skill)
Memories are injected into the system prompt by the bridge with budget truncation (personal: 4000 chars, general: 3000 chars, newest first).
### Prompt Styles
Six response styles in `homeai-agent/prompt-styles/`, each a JSON template with model, temperature, and instructions:
- **quick** — Claude Haiku 4.5, low temp, brief responses, strips profile sections
- **standard** — Claude Sonnet 4.6, balanced
- **creative** — Claude Sonnet 4.6, higher temp, elaborative
- **roleplayer** — Claude Opus 4.6, full personality injection
- **game-master** — Claude Opus 4.6, narrative-focused
- **storyteller** — Claude Opus 4.6, story-centric
Style selection: dashboard chat has a style picker; characters can set `default_prompt_style`; satellites use the global active style. Bridge resolves model per style → group → mode → default.
### TTS Voice Routing
@@ -160,11 +180,12 @@ This works for both ESP32/HA pipeline and dashboard chat.
6. **Character system** — schema v2, dashboard editor, memory system, per-character TTS routing ✅
7. **OpenClaw skills expansion** — 9 new skills (memory, monitor, character, routine, music, workflow, gitea, calendar, mode) + public/private mode routing ✅
8. **Music Assistant** — deployed on Pi (10.0.0.199:8095), Spotify + SMB + Chromecast players ✅
9. **Animated visual** — PNG/GIF character visual for the web assistant (initial visual layer)
10. **Android app** — companion app for mobile access to the assistant
11. **ComfyUI** — image generation online, character-consistent model workflows
12. **Extended integrations** — Snapcast, code-server
13. **Polish** — Authelia, Tailscale hardening, iOS widgets
9. **Memory v2 + Prompt Styles + Dream/GAZE** — SQLite memory with semantic search, 6 prompt styles with model tiering, Dream character import, GAZE character linking ✅
10. **Animated visual** — PNG/GIF character visual for the web assistant (initial visual layer)
11. **Android app** — companion app for mobile access to the assistant
12. **ComfyUI** — image generation online, character-consistent model workflows
13. **Extended integrations** — Snapcast, code-server
14. **Polish** — Authelia, Tailscale hardening, iOS widgets
### Stretch Goals
- **Live2D / VTube Studio** — full Live2D model with WebSocket API bridge (requires learning Live2D tooling)
@@ -180,7 +201,10 @@ This works for both ESP32/HA pipeline and dashboard chat.
- OpenClaw workspace tools: `~/.openclaw/workspace/TOOLS.md`
- OpenClaw config: `~/.openclaw/openclaw.json`
- Character configs: `~/homeai-data/characters/`
- Character memories: `~/homeai-data/memories/`
- Character memories DB: `~/homeai-data/memories/memories.db`
- Memory store module: `homeai-agent/memory_store.py`
- Prompt style templates: `homeai-agent/prompt-styles/`
- Active prompt style: `~/homeai-data/active-prompt-style.json`
- Conversation history: `~/homeai-data/conversations/`
- Active TTS state: `~/homeai-data/active-tts-voice.json`
- Active mode state: `~/homeai-data/active-mode.json`
@@ -194,6 +218,8 @@ This works for both ESP32/HA pipeline and dashboard chat.
- Gitea repos root: `~/gitea/`
- Music Assistant (Pi): `~/docker/selbina/music-assistant/` on 10.0.0.199
- Skills user guide: `homeai-agent/SKILLS_GUIDE.md`
- Dream service: `http://10.0.0.101:3000` (character management, REST API)
- GAZE service: `http://10.0.0.101:5782` (image generation, REST API)
---
@@ -203,8 +229,10 @@ This works for both ESP32/HA pipeline and dashboard chat.
- ESP32-S3-BOX-3 units are dumb satellites — all intelligence stays on Mac Mini
- The character JSON schema (from Character Manager) should be treated as a versioned spec; pipeline components read from it, never hardcode personality values
- OpenClaw skills are the primary extension mechanism — new capabilities = new skills
- Primary LLM is Claude Sonnet 4 (Anthropic API); local Ollama models are available as fallback
- Primary LLMs are Claude 4.5/4.6 family (Anthropic API) with per-style tiering; local Ollama models are available as fallback
- Launchd plists are symlinked from repo source to ~/Library/LaunchAgents/ — edit source, then bootout/bootstrap to reload
- Music Assistant runs on Pi (10.0.0.199), not Mac Mini — needs host networking for Chromecast mDNS discovery
- VTube Studio API bridge should be a standalone OpenClaw skill with clear event interface
- mem0 memory store should be backed up as part of regular Gitea commits
- Memory DB (`memories.db`) should be backed up as part of regular Gitea commits
- Dream characters can be linked to GAZE characters for cover image fallback and cross-referencing
- Prompt style selection hierarchy: explicit user pick → character default → global active style