Major refactor: deduplicate routes, sync, JS, and fix bugs

- Extract 8 common route patterns into factory functions in routes/shared.py
  (favourite, upload, replace cover, save defaults, clone, save JSON,
  get missing, clear covers) — removes ~1,100 lines across 9 route files
- Extract generic _sync_category() in sync.py — 7 sync functions become
  one-liner wrappers, removing ~350 lines
- Extract shared detail page JS into static/js/detail-common.js — all 9
  detail templates now call initDetailPage() with minimal config
- Extract layout inline JS into static/js/layout-utils.js (~185 lines)
- Extract library toolbar JS into static/js/library-toolbar.js
- Fix finalize missing-image bug: raise RuntimeError instead of logging
  warning so job is marked failed
- Fix missing scheduler default in _default_checkpoint_data()
- Fix N+1 query in Character.get_available_outfits() with batch IN query
- Convert all print() to logger across services and routes
- Add missing tags display to styles, scenes, detailers, checkpoints detail
- Update delete buttons to use trash.png icon with solid red background
- Update CLAUDE.md to reflect new architecture

Net reduction: ~1,600 lines

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Aodhan Collins
2026-03-21 23:06:58 +00:00
parent ed9a7b4b11
commit 55ff58aba6
42 changed files with 1493 additions and 3105 deletions

View File

@@ -24,12 +24,13 @@ services/
prompts.py # Prompt building + dedup (build_prompt, build_extras_prompt)
llm.py # LLM integration + MCP tool calls (call_llm, load_prompt)
mcp.py # MCP/Docker server lifecycle (ensure_mcp_server_running)
sync.py # All sync_*() functions + preset resolution helpers
sync.py # Generic _sync_category() + sync_characters/sync_checkpoints + preset resolution helpers
job_queue.py # Background job queue (_enqueue_job, _make_finalize, worker thread)
file_io.py # LoRA/checkpoint scanning, file helpers
generation.py # Shared generation logic (generate_from_preset)
routes/
__init__.py # register_routes(app) — imports and calls all route modules
shared.py # Factory functions for common routes (favourite, upload, clone, save_json, etc.)
characters.py # Character CRUD + generation + outfit management
outfits.py # Outfit routes
actions.py # Action routes
@@ -48,6 +49,10 @@ routes/
api.py # REST API v1 (preset generation, auth)
regenerate.py # Tag regeneration (single + bulk, via LLM queue)
search.py # Global search across resources and gallery images
static/js/
detail-common.js # Shared detail page JS (initDetailPage: preview, generation, batch, endless, JSON editor)
layout-utils.js # Extracted from layout.html (confirmResourceDelete, regenerateTags, initJsonEditor)
library-toolbar.js # Library page toolbar (batch generate, clear covers, missing items)
```
### Dependency Graph
@@ -67,16 +72,19 @@ app.py
│ ├── file_io.py ← models, utils
│ └── generation.py ← prompts, workflow, job_queue, sync, models
└── routes/
├── All route modules ← services/*, utils, models
── (routes never import from other routes)
├── shared.py ← models, utils, services (lazy-init to avoid circular imports)
── All route modules ← services/*, utils, models, shared
└── (routes never import from other routes except shared.py)
```
**No circular imports**: routes → services → utils/models. Services never import routes. Utils never imports services.
**No circular imports**: routes → services → utils/models. Services never import routes. Utils never imports services. `routes/shared.py` uses `_init_models()` lazy initialization to avoid circular imports with model classes.
### Route Registration Pattern
Routes use a `register_routes(app)` closure pattern — each route module defines a function that receives the Flask `app` object and registers routes via `@app.route()` closures. This preserves all existing `url_for()` endpoint names without requiring Blueprint prefixes. Helper functions used only by routes in that module are defined inside `register_routes()` before the routes that reference them.
Common routes (favourite, upload, replace cover, save defaults, clone, save JSON, get missing, clear covers) are registered via factory functions in `routes/shared.py`. Each route module calls `register_common_routes(app, 'category_name')` as the first line of its `register_routes()`. The `CATEGORY_CONFIG` dict in `shared.py` maps each category to its model, URL prefix, endpoint names, and directory paths.
### Database
SQLite at `instance/database.db`, managed by Flask-SQLAlchemy. The DB is a cache of the JSON files on disk — the JSON files are the source of truth.
@@ -186,8 +194,9 @@ Two independent queues with separate worker threads:
### `services/sync.py` — Data Synchronization
- **`sync_characters()`, `sync_outfits()`, `sync_actions()`, etc.** — Load JSON files from `data/` directories into SQLite. One function per category.
- **`_sync_nsfw_from_tags(entity, data)`** — Reads `data['tags']['nsfw']` and sets `entity.is_nsfw`. Called in every sync function on both create and update paths.
- **`_sync_category(config_key, model_class, id_field, name_field, extra_fn=None, sync_nsfw=True)`** — Generic sync function handling 80% shared logic. 7 sync functions are one-liner wrappers.
- **`sync_characters()`** / **`sync_checkpoints()`** — Custom sync functions (special ID handling, filesystem scan).
- **`_sync_nsfw_from_tags(entity, data)`** — Reads `data['tags']['nsfw']` and sets `entity.is_nsfw`. Called by `_sync_category` and custom sync functions.
- **`_resolve_preset_entity(type, id)`** / **`_resolve_preset_fields(preset_data)`** — Preset resolution helpers.
### `services/file_io.py` — File & DB Helpers
@@ -424,7 +433,10 @@ Image retrieval is handled server-side by the `_make_finalize()` callback; there
- Navbar with links to all sections
- Global default checkpoint selector (saves to session via AJAX)
- Resource delete modal (soft/hard) shared across gallery pages
- `initJsonEditor(saveUrl)` — shared JSON editor modal (simple form + raw textarea tabs)
- **Shared JS files** (extracted from inline scripts):
- `static/js/detail-common.js``initDetailPage(options)`: favourite toggle, selectPreview, waitForJob, AJAX form submit + polling, addToPreviewGallery, batch generation, endless mode, JSON editor init. All 9 detail templates call this with minimal config.
- `static/js/layout-utils.js``confirmResourceDelete(mode)`, `regenerateTags(category, slug)`, `initJsonEditor(saveUrl)`
- `static/js/library-toolbar.js` — Library page toolbar (batch generate, clear covers, missing items)
- Context processors inject `all_checkpoints`, `default_checkpoint_path`, and `COMFYUI_WS_URL` into every template. The `random_gen_image(category, slug)` template global returns a random image path from `static/uploads/<category>/<slug>/` for use as a fallback cover when `image_path` is not set.
- **No `{% block head %}` exists** in layout.html — do not try to use it.
- Generation is async: JS submits the form via AJAX (`X-Requested-With: XMLHttpRequest`), receives a `{"job_id": ...}` response, then polls `/api/queue/<job_id>/status` every ~1.5 seconds until `status == "done"`. The server-side worker handles all ComfyUI polling and image saving via the `_make_finalize()` callback. There are no client-facing finalize HTTP routes.
@@ -522,14 +534,15 @@ Absolute paths on disk:
To add a new content category (e.g. "Poses" as a separate concept from Actions), the pattern is:
1. **Model** (`models.py`): Add a new SQLAlchemy model with the standard fields.
2. **Sync function** (`services/sync.py`): Add `sync_newcategory()` following the pattern of `sync_outfits()`.
2. **Sync function** (`services/sync.py`): Add `sync_newcategory()` as a one-liner using `_sync_category()` (or custom if needed).
3. **Data directory** (`app.py`): Add `app.config['NEWCATEGORY_DIR'] = 'data/newcategory'`.
4. **Routes** (`routes/newcategory.py`): Create a new route module with a `register_routes(app)` function. Implement index, detail, edit, generate, replace_cover_from_preview, upload, save_defaults, clone, rescan routes. Follow `routes/outfits.py` or `routes/scenes.py` exactly.
5. **Route registration** (`routes/__init__.py`): Import and call `newcategory.register_routes(app)`.
6. **Templates**: Create `templates/newcategory/{index,detail,edit,create}.html` extending `layout.html`.
7. **Nav**: Add link to navbar in `templates/layout.html`.
8. **Startup** (`app.py`): Import and call `sync_newcategory()` in the `with app.app_context()` block.
9. **Generator page**: Add to `routes/generator.py`, `services/prompts.py` `build_extras_prompt()`, and `templates/generator.html` accordion.
4. **Shared routes** (`routes/shared.py`): Add entry to `CATEGORY_CONFIG` dict with model, url_prefix, detail_endpoint, config_dir, category_folder, id_field, name_field, endpoint_prefix.
5. **Routes** (`routes/newcategory.py`): Create a new route module with a `register_routes(app)` function. Call `register_common_routes(app, 'newcategory')` first, then implement category-specific routes (index, detail, edit, generate, rescan). Follow `routes/outfits.py` or `routes/scenes.py` exactly.
6. **Route registration** (`routes/__init__.py`): Import and call `newcategory.register_routes(app)`.
7. **Templates**: Create `templates/newcategory/{index,detail,edit,create}.html` extending `layout.html`. Detail template should call `initDetailPage({...})` from `detail-common.js`.
8. **Nav**: Add link to navbar in `templates/layout.html`.
9. **Startup** (`app.py`): Import and call `sync_newcategory()` in the `with app.app_context()` block.
10. **Generator page**: Add to `routes/generator.py`, `services/prompts.py` `build_extras_prompt()`, and `templates/generator.html` accordion.
---