# GAZE — Character Browser: LLM Development Guide ## What This Project Is GAZE is a Flask web app for managing AI image generation assets and generating images via ComfyUI. It is a **personal creative tool** for organizing characters, outfits, actions, styles, scenes, and detailers — all of which map to Stable Diffusion LoRAs and prompt fragments — and generating images by wiring those assets into a ComfyUI workflow at runtime. The app is deployed locally, connects to a local ComfyUI instance at `http://127.0.0.1:8188`, and uses SQLite for persistence. LoRA and model files live on `/mnt/alexander/AITools/Image Models/`. --- ## Architecture ### Entry Point - `app.py` — Single-file Flask app (~4500+ lines). All routes, helpers, ComfyUI integration, LLM calls, and sync functions live here. - `models.py` — SQLAlchemy models only. - `comfy_workflow.json` — ComfyUI workflow template with placeholder strings. ### Database SQLite at `instance/database.db`, managed by Flask-SQLAlchemy. The DB is a cache of the JSON files on disk — the JSON files are the source of truth. **Models**: `Character`, `Look`, `Outfit`, `Action`, `Style`, `Scene`, `Detailer`, `Checkpoint`, `Settings` All category models (except Settings and Checkpoint) share this pattern: - `{entity}_id` — canonical ID (from JSON, often matches filename without extension) - `slug` — URL-safe version of the ID (alphanumeric + underscores only, via `re.sub(r'[^a-zA-Z0-9_]', '', id)`) - `name` — display name - `filename` — original JSON filename - `data` — full JSON blob (SQLAlchemy JSON column) - `default_fields` — list of `section::key` strings saved as the user's preferred prompt fields - `image_path` — relative path under `static/uploads/` ### Data Flow: JSON → DB → Prompt → ComfyUI 1. **JSON files** in `data/{characters,clothing,actions,styles,scenes,detailers,looks}/` are loaded by `sync_*()` functions into SQLite. 2. At generation time, `build_prompt(data, selected_fields, default_fields, active_outfit)` converts the character JSON blob into `{"main": ..., "face": ..., "hand": ...}` prompt strings. 3. `_prepare_workflow(workflow, character, prompts, ...)` wires prompts and LoRAs into the loaded `comfy_workflow.json`. 4. `queue_prompt(workflow, client_id)` POSTs the workflow to ComfyUI's `/prompt` endpoint. 5. The app polls `get_history(prompt_id)` and retrieves the image via `get_image(filename, subfolder, type)`. --- ## ComfyUI Workflow Node Map The workflow (`comfy_workflow.json`) uses string node IDs. These are the critical nodes: | Node | Role | |------|------| | `3` | Main KSampler | | `4` | Checkpoint loader | | `5` | Empty latent (width/height) | | `6` | Positive prompt — contains `{{POSITIVE_PROMPT}}` placeholder | | `7` | Negative prompt | | `8` | VAE decode | | `9` | Save image | | `11` | Face ADetailer | | `13` | Hand ADetailer | | `14` | Face detailer prompt — contains `{{FACE_PROMPT}}` placeholder | | `15` | Hand detailer prompt — contains `{{HAND_PROMPT}}` placeholder | | `16` | Character LoRA (or Look LoRA when a Look is active) | | `17` | Outfit LoRA | | `18` | Action LoRA | | `19` | Style / Detailer / Scene LoRA (priority: style > detailer > scene) | LoRA nodes chain: `4 → 16 → 17 → 18 → 19`. Unused LoRA nodes are bypassed by pointing `model_source`/`clip_source` directly to the prior node. All model/clip consumers (nodes 3, 6, 7, 11, 13, 14, 15) are wired to the final `model_source`/`clip_source` at the end of `_prepare_workflow`. --- ## Key Helpers ### `build_prompt(data, selected_fields, default_fields, active_outfit)` Converts a character (or combined) data dict into `{"main", "face", "hand"}` prompt strings. Field selection priority: 1. `selected_fields` (from form submission) — if non-empty, use it exclusively 2. `default_fields` (saved in DB per character) — used if no form selection 3. Select all (fallback) Fields are addressed as `"section::key"` strings (e.g. `"identity::hair"`, `"wardrobe::top"`, `"special::name"`, `"lora::lora_triggers"`). Wardrobe format: Characters support a **nested** format where `wardrobe` is a dict of outfit names → outfit dicts (e.g. `{"default": {...}, "swimsuit": {...}}`). Legacy characters have a flat `wardrobe` dict. `Character.get_active_wardrobe()` handles both. ### `build_extras_prompt(actions, outfits, scenes, styles, detailers)` Used by the Generator page. Combines prompt text from all checked items across categories into a single string. LoRA triggers, wardrobe fields, scene fields, style fields, detailer prompts — all concatenated. ### `_prepare_workflow(workflow, character, prompts, checkpoint, custom_negative, outfit, action, style, detailer, scene, width, height, checkpoint_data, look)` The core workflow wiring function. Mutates the loaded workflow dict in place and returns it. Key behaviours: - Replaces `{{POSITIVE_PROMPT}}`, `{{FACE_PROMPT}}`, `{{HAND_PROMPT}}` in node inputs. - If `look` is provided, uses the Look's LoRA in node 16 instead of the character's LoRA, and prepends the Look's negative to node 7. - Chains LoRA nodes dynamically — only active LoRAs participate in the chain. - Randomises seeds for nodes 3, 11, 13. - Applies checkpoint-specific settings (steps, cfg, sampler, VAE, base prompts) via `_apply_checkpoint_settings`. - Runs `_cross_dedup_prompts` on nodes 6 and 7 as the final step before returning. ### `_cross_dedup_prompts(positive, negative)` Cross-deduplicates tags between the positive and negative prompt strings. For each tag present in both, repeatedly removes the first occurrence from each side until the tag exists on only one side (or neither). Equal counts cancel completely; any excess on one side is retained. This allows deliberate overrides: adding a tag twice in positive while it appears once in negative leaves one copy in positive. Called as the last step of `_prepare_workflow`, after `_apply_checkpoint_settings` has added `base_positive`/`base_negative`, so it operates on fully-assembled prompts. ### `_IDENTITY_KEYS` / `_WARDROBE_KEYS` (module-level constants) Lists of canonical field names for the `identity` and `wardrobe` sections. Used by `_ensure_character_fields()` to avoid hard-coding key lists in every route. ### `_resolve_character(character_slug)` Returns a `Character` ORM object for a given slug string. Handles the `"__random__"` sentinel by selecting a random character. Returns `None` if `character_slug` is falsy or no match is found. Every route that accepts an optional character dropdown (outfit, action, style, scene, detailer, checkpoint, look generate routes) uses this instead of an inline if/elif block. ### `_ensure_character_fields(character, selected_fields, include_wardrobe=True, include_defaults=False)` Mutates `selected_fields` in place, appending any populated identity, wardrobe, and optional defaults keys that are not already present. Ensures `"special::name"` is always included. Called in every secondary-category generate route immediately after `_resolve_character()` to guarantee the character's essential fields are sent to `build_prompt`. ### `_append_background(prompts, character=None)` Appends a `" simple background"` tag (or plain `"simple background"` if no primary color) to `prompts['main']`. Called in outfit, action, style, detailer, and checkpoint generate routes instead of repeating the same inline string construction. ### `_make_finalize(category, slug, db_model_class=None, action=None)` Factory function that returns a `_finalize(comfy_prompt_id, job)` callback closure. The closure: 1. Calls `get_history()` and `get_image()` to retrieve the generated image from ComfyUI. 2. Saves the image to `static/uploads///gen_.png`. 3. Sets `job['result']` with `image_url` and `relative_path`. 4. If `db_model_class` is provided **and** (`action` is `None` or `action == 'replace'`), updates the ORM object's `image_path` and commits. All `generate` routes pass a `_make_finalize(...)` call as the finalize argument to `_enqueue_job()` instead of defining an inline closure. ### `_prune_job_history(max_age_seconds=3600)` Removes entries from `_job_history` that are in a terminal state (`done`, `failed`, `removed`) and older than `max_age_seconds`. Called at the end of every worker loop iteration to prevent unbounded memory growth. ### `_queue_generation(character, action, selected_fields, client_id)` Convenience wrapper for character detail page generation. Loads workflow, calls `build_prompt`, calls `_prepare_workflow`, calls `queue_prompt`. ### `_queue_detailer_generation(detailer_obj, character, selected_fields, client_id, action, extra_positive, extra_negative)` Generation helper for the detailer detail page. Merges the detailer's `prompt` list (flattened — `prompt` may be a list or string) and LoRA triggers into the character data before calling `build_prompt`. Appends `extra_positive` to `prompts["main"]` and passes `extra_negative` as `custom_negative` to `_prepare_workflow`. Accepts an optional `action` (Action model object) passed through to `_prepare_workflow` for node 18 LoRA wiring. ### `_get_default_checkpoint()` Returns `(checkpoint_path, checkpoint_data)` from the Flask session's `default_checkpoint` key. The global default checkpoint is set via the navbar dropdown and stored server-side in the filesystem session. ### `call_llm(prompt, system_prompt)` OpenAI-compatible chat completion call supporting: - **OpenRouter** (cloud, configured API key + model) - **Ollama** / **LMStudio** (local, configured base URL + model) Implements a tool-calling loop (up to 10 turns) using `DANBOORU_TOOLS` (search_tags, validate_tags, suggest_tags) via an MCP Docker container (`danbooru-mcp:latest`). If the provider rejects tools (HTTP 400), retries without tools. --- ## JSON Data Schemas ### Character (`data/characters/*.json`) ```json { "character_id": "tifa_lockhart", "character_name": "Tifa Lockhart", "identity": { "base_specs": "", "hair": "", "eyes": "", "hands": "", "arms": "", "torso": "", "pelvis": "", "legs": "", "feet": "", "extra": "" }, "defaults": { "expression": "", "pose": "", "scene": "" }, "wardrobe": { "default": { "full_body": "", "headwear": "", "top": "", "bottom": "", "legwear": "", "footwear": "", "hands": "", "gloves": "", "accessories": "" } }, "styles": { "aesthetic": "", "primary_color": "", "secondary_color": "", "tertiary_color": "" }, "lora": { "lora_name": "Illustrious/Looks/tifa.safetensors", "lora_weight": 0.8, "lora_triggers": "" }, "tags": [], "participants": { "orientation": "1F", "solo_focus": "true" } } ``` `participants` is optional; when absent, `(solo:1.2)` is injected. `orientation` is parsed by `parse_orientation()` into Danbooru tags (`1girl`, `hetero`, etc.). ### Outfit (`data/clothing/*.json`) ```json { "outfit_id": "french_maid_01", "outfit_name": "French Maid", "wardrobe": { "full_body": "", "headwear": "", "top": "", "bottom": "", "legwear": "", "footwear": "", "hands": "", "accessories": "" }, "lora": { "lora_name": "Illustrious/Clothing/maid.safetensors", "lora_weight": 0.8, "lora_triggers": "" }, "tags": [] } ``` ### Action (`data/actions/*.json`) ```json { "action_id": "sitting", "action_name": "Sitting", "action": { "full_body": "", "additional": "", "head": "", "eyes": "", "arms": "", "hands": "" }, "lora": { "lora_name": "", "lora_weight": 1.0, "lora_triggers": "" }, "tags": [] } ``` ### Scene (`data/scenes/*.json`) ```json { "scene_id": "beach", "scene_name": "Beach", "scene": { "background": "", "foreground": "", "furniture": "", "colors": "", "lighting": "", "theme": "" }, "lora": { "lora_name": "", "lora_weight": 1.0, "lora_triggers": "" }, "tags": [] } ``` ### Style (`data/styles/*.json`) ```json { "style_id": "watercolor", "style_name": "Watercolor", "style": { "artist_name": "", "artistic_style": "" }, "lora": { "lora_name": "", "lora_weight": 1.0, "lora_triggers": "" } } ``` ### Detailer (`data/detailers/*.json`) ```json { "detailer_id": "detailed_skin", "detailer_name": "Detailed Skin", "prompt": ["detailed skin", "pores"], "focus": { "face": true, "hands": true }, "lora": { "lora_name": "", "lora_weight": 1.0, "lora_triggers": "" } } ``` ### Look (`data/looks/*.json`) ```json { "look_id": "tifa_casual", "look_name": "Tifa Casual", "character_id": "tifa_lockhart", "positive": "casual clothes, jeans", "negative": "revealing", "lora": { "lora_name": "Illustrious/Looks/tifa_casual.safetensors", "lora_weight": 0.85, "lora_triggers": "" }, "tags": [] } ``` Looks occupy LoRA node 16, overriding the character's own LoRA. The Look's `negative` is prepended to the workflow's negative prompt. ### Checkpoint (`data/checkpoints/*.json`) ```json { "checkpoint_path": "Illustrious/model.safetensors", "checkpoint_name": "Model Display Name", "base_positive": "anime", "base_negative": "text, logo", "steps": 25, "cfg": 5, "sampler_name": "euler_ancestral", "scheduler": "normal", "vae": "integrated" } ``` Checkpoint JSONs are keyed by `checkpoint_path`. If no JSON exists for a discovered model file, `_default_checkpoint_data()` provides defaults. --- ## URL Routes ### Characters - `GET /` — character gallery (index) - `GET /character/` — character detail with generation UI - `POST /character//generate` — queue generation (AJAX or form); returns `{"job_id": ...}` - `POST /character//replace_cover_from_preview` — promote preview to cover - `GET/POST /character//edit` — edit character data - `POST /character//upload` — upload cover image - `POST /character//save_defaults` — save default field selection - `POST /character//outfit/switch|add|delete|rename` — manage per-character wardrobe outfits - `GET/POST /create` — create new character (blank or LLM-generated) - `POST /rescan` — sync DB from JSON files ### Category Pattern (Outfits, Actions, Styles, Scenes, Detailers) Each category follows the same URL pattern: - `GET //` — gallery - `GET //` — detail + generation UI - `POST ///generate` — queue generation; returns `{"job_id": ...}` - `POST ///replace_cover_from_preview` - `GET/POST ///edit` - `POST ///upload` - `POST ///save_defaults` - `POST ///clone` — duplicate entry - `POST ///save_json` — save raw JSON (from modal editor) - `POST //rescan` - `POST //bulk_create` — LLM-generate entries from LoRA files on disk ### Looks - `GET /looks` — gallery - `GET /look/` — detail - `GET/POST /look//edit` - `POST /look//generate` — queue generation; returns `{"job_id": ...}` - `POST /look//replace_cover_from_preview` - `GET/POST /look/create` - `POST /looks/rescan` ### Generator (Mix & Match) - `GET/POST /generator` — freeform generator with multi-select accordion UI - `POST /generator/preview_prompt` — AJAX: preview composed prompt without generating ### Checkpoints - `GET /checkpoints` — gallery - `GET /checkpoint/` — detail + generation settings editor - `POST /checkpoint//save_json` - `POST /checkpoints/rescan` ### Job Queue API All generation routes use the background job queue. Frontend polls: - `GET /api/queue//status` — returns `{"status": "pending"|"running"|"done"|"failed", "result": {...}}` Image retrieval is handled server-side by the `_make_finalize()` callback; there are no separate client-facing finalize routes. ### Utilities - `POST /set_default_checkpoint` — save default checkpoint to session and persist to `comfy_workflow.json` - `GET /get_missing_{characters,outfits,actions,scenes,styles,detailers,looks,checkpoints}` — AJAX: list items without cover images (sorted by display name) - `POST /generate_missing` — batch generate covers for all characters missing one (uses job queue) - `POST /clear_all_covers` / `clear_all_{outfit,action,scene,style,detailer,look,checkpoint}_covers` - `GET /gallery` — global image gallery browsing `static/uploads/` - `GET/POST /settings` — LLM provider configuration - `POST /resource///delete` — soft (JSON only) or hard (JSON + safetensors) delete --- ## Frontend - Bootstrap 5.3 (CDN). Custom styles in `static/style.css`. - All templates extend `templates/layout.html`. The base layout provides: - `{% block content %}` — main page content - `{% block scripts %}` — additional JS at end of body - Navbar with links to all sections - Global default checkpoint selector (saves to session via AJAX) - Resource delete modal (soft/hard) shared across gallery pages - `initJsonEditor(saveUrl)` — shared JSON editor modal (simple form + raw textarea tabs) - Context processors inject `all_checkpoints`, `default_checkpoint_path`, and `COMFYUI_WS_URL` into every template. - **No `{% block head %}` exists** in layout.html — do not try to use it. - Generation is async: JS submits the form via AJAX (`X-Requested-With: XMLHttpRequest`), receives a `{"job_id": ...}` response, then polls `/api/queue//status` every ~1.5 seconds until `status == "done"`. The server-side worker handles all ComfyUI polling and image saving via the `_make_finalize()` callback. There are no client-facing finalize HTTP routes. - **Batch generation** (library pages): Uses a two-phase pattern: 1. **Queue phase**: All jobs are submitted upfront via sequential fetch calls, collecting job IDs 2. **Poll phase**: All jobs are polled concurrently via `Promise.all()`, updating UI as each completes 3. **Progress tracking**: Displays currently processing items in real-time using a `Set` to track active jobs 4. **Sorting**: All batch operations sort items by display `name` (not `filename`) for better UX --- ## LLM Integration ### System Prompts Text files in `data/prompts/` define JSON output schemas for LLM-generated entries: - `character_system.txt` — character JSON schema - `outfit_system.txt` — outfit JSON schema - `action_system.txt`, `scene_system.txt`, `style_system.txt`, `detailer_system.txt`, `look_system.txt`, `checkpoint_system.txt` Used by: character/outfit/action/scene/style create forms, and bulk_create routes. ### Danbooru MCP Tools The LLM loop in `call_llm()` provides three tools via a Docker-based MCP server (`danbooru-mcp:latest`): - `search_tags(query, limit, category)` — prefix search - `validate_tags(tags)` — exact-match validation - `suggest_tags(partial, limit, category)` — autocomplete The LLM uses these to verify and discover correct Danbooru-compatible tags for prompts. All system prompts (`character_system.txt`, `outfit_system.txt`, `action_system.txt`, `scene_system.txt`, `style_system.txt`, `detailer_system.txt`, `look_system.txt`, `checkpoint_system.txt`) instruct the LLM to use these tools before finalising any tag values. `checkpoint_system.txt` applies them specifically to the `base_positive` and `base_negative` fields. --- ## LoRA File Paths LoRA filenames in JSON are stored as paths relative to ComfyUI's `models/lora/` root: | Category | Path prefix | Example | |----------|-------------|---------| | Character / Look | `Illustrious/Looks/` | `Illustrious/Looks/tifa_v2.safetensors` | | Outfit | `Illustrious/Clothing/` | `Illustrious/Clothing/maid.safetensors` | | Action | `Illustrious/Poses/` | `Illustrious/Poses/sitting.safetensors` | | Style | `Illustrious/Styles/` | `Illustrious/Styles/watercolor.safetensors` | | Detailer | `Illustrious/Detailers/` | `Illustrious/Detailers/skin.safetensors` | | Scene | `Illustrious/Backgrounds/` | `Illustrious/Backgrounds/beach.safetensors` | Checkpoint paths: `Illustrious/.safetensors` or `Noob/.safetensors`. Absolute paths on disk: - Checkpoints: `/mnt/alexander/AITools/Image Models/Stable-diffusion/{Illustrious,Noob}/` - LoRAs: `/mnt/alexander/AITools/Image Models/lora/Illustrious/{Looks,Clothing,Poses,Styles,Detailers,Backgrounds}/` --- ## Adding a New Category To add a new content category (e.g. "Poses" as a separate concept from Actions), the pattern is: 1. **Model** (`models.py`): Add a new SQLAlchemy model with the standard fields. 2. **Sync function** (`app.py`): Add `sync_newcategory()` following the pattern of `sync_outfits()`. 3. **Data directory**: Add `app.config['NEWCATEGORY_DIR'] = 'data/newcategory'`. 4. **Routes**: Implement index, detail, edit, generate, finalize_generation, replace_cover_from_preview, upload, save_defaults, clone, rescan routes. Follow the outfit/scene pattern exactly. 5. **Templates**: Create `templates/newcategory/{index,detail,edit,create}.html` extending `layout.html`. 6. **Nav**: Add link to navbar in `templates/layout.html`. 7. **`with_appcontext`**: Call `sync_newcategory()` in the `with app.app_context()` block at the bottom of `app.py`. 8. **Generator page**: Add to `generator()` route, `build_extras_prompt()`, and `templates/generator.html` accordion. --- ## Session Keys The Flask filesystem session stores: - `default_checkpoint` — checkpoint_path string for the global default - `prefs_{slug}` — selected_fields list for character detail page - `preview_{slug}` — relative image path of last character preview - `prefs_outfit_{slug}`, `preview_outfit_{slug}`, `char_outfit_{slug}` — outfit detail state - `prefs_action_{slug}`, `preview_action_{slug}`, `char_action_{slug}` — action detail state - `prefs_scene_{slug}`, `preview_scene_{slug}`, `char_scene_{slug}` — scene detail state - `prefs_detailer_{slug}`, `preview_detailer_{slug}`, `char_detailer_{slug}`, `action_detailer_{slug}`, `extra_pos_detailer_{slug}`, `extra_neg_detailer_{slug}` — detailer detail state (selected fields, preview image, character, action LoRA, extra positive prompt, extra negative prompt) - `prefs_style_{slug}`, `preview_style_{slug}`, `char_style_{slug}` — style detail state - `prefs_look_{slug}`, `preview_look_{slug}` — look detail state --- ## Running the App ### Directly (development) ```bash cd /mnt/alexander/Projects/character-browser source venv/bin/activate python app.py ``` The app runs in debug mode on port 5000 by default. ComfyUI must be running at `http://127.0.0.1:8188`. The DB is initialised and all sync functions are called inside `with app.app_context():` at the bottom of `app.py` before `app.run()`. ### Docker ```bash docker compose up -d ``` The compose file (`docker-compose.yml`) runs two services: - **`danbooru-mcp`** — built from `https://git.liveaodh.com/aodhan/danbooru-mcp.git`; the MCP tag-search container used by `call_llm()`. - **`app`** — the Flask app, exposed on host port **5782** → container port 5000. Key environment variables set by compose: - `COMFYUI_URL=http://10.0.0.200:8188` — points at ComfyUI on the Docker host network. - `SKIP_MCP_AUTOSTART=true` — disables the app's built-in danbooru-mcp launch logic (compose manages it). Volumes mounted into the app container: - `./data`, `./static/uploads`, `./instance`, `./flask_session` — persistent app data. - `/Volumes/ImageModels:/ImageModels:ro` — model files for checkpoint/LoRA scanning (**requires Docker Desktop file sharing enabled for `/Volumes/ImageModels`**). - `/var/run/docker.sock` — Docker socket so the app can exec danbooru-mcp tool containers. --- ## Common Pitfalls - **SQLAlchemy JSON mutation**: After modifying a JSON column dict in place, always call `flag_modified(obj, "data")` or the change won't be detected. - **Dual write**: Every edit route writes back to both the DB (`db.session.commit()`) and the JSON file on disk. Both must be kept in sync. - **Slug generation**: `re.sub(r'[^a-zA-Z0-9_]', '', id)` — note this removes hyphens and dots, not just replaces them. Character IDs like `yuna_(ff10)` become slug `yunaffx10`. This is intentional. - **Checkpoint slugs use underscore replacement**: `re.sub(r'[^a-zA-Z0-9_]', '_', ...)` (replaces with `_`, not removes) to preserve readability in paths. - **LoRA chaining**: If a LoRA node has no LoRA (name is empty/None), the node is skipped and `model_source`/`clip_source` pass through unchanged. Do not set the node inputs for skipped nodes. - **AJAX detection**: `request.headers.get('X-Requested-With') == 'XMLHttpRequest'` determines whether to return JSON or redirect. - **Session must be marked modified for JSON responses**: After setting session values in AJAX-responding routes, set `session.modified = True`. - **Detailer `prompt` is a list**: The `prompt` field in detailer JSON is stored as a list of strings (e.g. `["detailed skin", "pores"]`), not a plain string. When merging into `tags` for `build_prompt`, use `extend` for lists and `append` for strings — never append the list object itself or `", ".join()` will fail on the nested list item. - **`_make_finalize` action semantics**: Pass `action=None` when the route should always update the DB cover (e.g. batch generate, checkpoint generate). Pass `action=request.form.get('action')` for routes that support both "preview" (no DB update) and "replace" (update DB). The factory skips the DB write when `action` is truthy and not `"replace"`.