character-browser/plans/APP_REFACTOR.md

# APP_REFACTOR.md — Split app.py into Modules

## Goal

Split the 8,478-line `app.py` into a clean module structure using Flask Blueprints for routes and plain Python modules for services/utilities. The JSON files and DB remain the source of truth — no data migration needed.

---

## Target Structure

```
app.py                          # ~80 lines: Flask init, config, register blueprints, startup
models.py                       # Unchanged
utils.py                        # Pure helpers (no Flask/DB deps beyond current_app)
services/
  __init__.py
  comfyui.py                    # ComfyUI HTTP client
  workflow.py                   # Workflow building + checkpoint settings
  prompts.py                    # Prompt building + dedup
  llm.py                        # LLM integration + MCP tool calls
  mcp.py                        # MCP/Docker server lifecycle
  sync.py                       # All sync_*() functions
  job_queue.py                  # Background job queue + worker thread
  file_io.py                    # LoRA/checkpoint scanning, file uploads, image saving
routes/
  __init__.py                   # register_blueprints() helper
  characters.py                 # Character CRUD + generate + outfit management
  outfits.py                    # Outfit routes
  actions.py                    # Action routes
  styles.py                     # Style routes
  scenes.py                     # Scene routes
  detailers.py                  # Detailer routes
  checkpoints.py                # Checkpoint routes
  looks.py                      # Look routes
  presets.py                    # Preset routes
  generator.py                  # Generator mix-and-match page
  gallery.py                    # Gallery browsing + image deletion
  settings.py                   # Settings page + status API + context processors
  strengths.py                  # Strengths gallery system
  transfer.py                   # Resource transfer system
  queue_api.py                  # /api/queue/* endpoints
```

---

## Module Contents — Exact Function Mapping

### `app.py` (entry point, ~80 lines)

Keep only:
- Flask app creation, config, extensions (SQLAlchemy, Session)
- Logging setup
- `from routes import register_blueprints; register_blueprints(app)`
- `with app.app_context():` block (DB init, migrations, sync calls, worker start)

### `utils.py`

| Function | Current Line | Notes |
|----------|-------------|-------|
| `parse_orientation()` | 768 | Pure logic |
| `_resolve_lora_weight()` | 833 | Pure logic |
| `allowed_file()` | 765 | Pure logic |
| `_IDENTITY_KEYS` | 850 | Constant |
| `_WARDROBE_KEYS` | 851 | Constant |
| `ALLOWED_EXTENSIONS` | 728 | Constant |
| `_LORA_DEFAULTS` | 730 | Constant |

### `services/job_queue.py`

| Function/Global | Current Line | Notes |
|----------------|-------------|-------|
| `_job_queue_lock` | 75 | Global |
| `_job_queue` | 76 | Global |
| `_job_history` | 77 | Global |
| `_queue_worker_event` | 78 | Global |
| `_enqueue_job()` | 81 | Needs `comfyui.queue_prompt`, `comfyui.get_history` |
| `_queue_worker()` | 102 | The background thread loop |
| `_make_finalize()` | 227 | Needs `comfyui.get_history`, `comfyui.get_image`, DB models |
| `_prune_job_history()` | 292 | Pure logic on globals |

### `services/comfyui.py`

| Function | Current Line | Notes |
|----------|-------------|-------|
| `queue_prompt()` | 1083 | HTTP POST to ComfyUI |
| `get_history()` | 1116 | HTTP GET + polling |
| `get_image()` | 1148 | HTTP GET for image bytes |
| `_ensure_checkpoint_loaded()` | 1056 | HTTP POST to load checkpoint |

All use `current_app.config['COMFYUI_URL']` — pass URL as param or read from config.

### `services/workflow.py`

| Function | Current Line | Notes |
|----------|-------------|-------|
| `_prepare_workflow()` | 3119 | Core workflow wiring |
| `_apply_checkpoint_settings()` | 6445 | Checkpoint-specific settings |
| `_log_workflow_prompts()` | 3030 | Logging helper |
| `_build_style_workflow()` | 5089 | Style-specific workflow builder |
| `_build_checkpoint_workflow()` | 6501 | Checkpoint-specific workflow builder |
| `_queue_scene_generation()` | 5603 | Scene generation helper |
| `_queue_detailer_generation()` | 6091 | Detailer generation helper |
| `_get_default_checkpoint()` | 3273 | Reads session + DB |

### `services/prompts.py`

| Function | Current Line | Notes |
|----------|-------------|-------|
| `build_prompt()` | 933 | Core prompt builder |
| `build_extras_prompt()` | 2104 | Generator page prompt builder |
| `_dedup_tags()` | 797 | Tag deduplication |
| `_cross_dedup_prompts()` | 808 | Cross-dedup positive/negative |
| `_build_strengths_prompts()` | 7577 | Strengths-specific prompt builder |
| `_get_character_data_without_lora()` | 7563 | Helper for strengths |
| `_resolve_character()` | 853 | Resolve slug → Character |
| `_ensure_character_fields()` | 861 | Mutate selected_fields |
| `_append_background()` | 889 | Add background tag |

### `services/llm.py`

| Function/Constant | Current Line | Notes |
|-------------------|-------------|-------|
| `DANBOORU_TOOLS` | 1844 | Tool definitions constant |
| `call_mcp_tool()` | 1904 | Sync MCP tool call |
| `load_prompt()` | 1911 | Load system prompt from file |
| `call_llm()` | 1918 | LLM chat completion + tool loop |

### `services/mcp.py`

| Function | Current Line | Notes |
|----------|-------------|-------|
| `_ensure_mcp_repo()` | 423 | Clone danbooru-mcp repo |
| `ensure_mcp_server_running()` | 457 | Start danbooru-mcp container |
| `_ensure_character_mcp_repo()` | 496 | Clone character-mcp repo |
| `ensure_character_mcp_server_running()` | 530 | Start character-mcp container |

### `services/sync.py`

| Function | Current Line | Notes |
|----------|-------------|-------|
| `sync_characters()` | 1158 | |
| `sync_outfits()` | 1218 | |
| `ensure_default_outfit()` | 1276 | |
| `sync_looks()` | 1360 | |
| `sync_presets()` | 1415 | |
| `sync_actions()` | 1534 | |
| `sync_styles()` | 1591 | |
| `sync_detailers()` | 1648 | |
| `sync_scenes()` | 1705 | |
| `sync_checkpoints()` | 1776 | |
| `_default_checkpoint_data()` | 1762 | |
| `_resolve_preset_entity()` | 1484 | Used by preset routes too |
| `_resolve_preset_fields()` | 1494 | Used by preset routes too |
| `_PRESET_ENTITY_MAP` | 1472 | Constant |

### `services/file_io.py`

| Function | Current Line | Notes |
|----------|-------------|-------|
| `get_available_loras()` | 739 | Reads Settings + filesystem |
| `get_available_checkpoints()` | 750 | Reads Settings + filesystem |
| `_count_look_assignments()` | 895 | DB query |
| `_count_outfit_lora_assignments()` | 907 | DB query |
| `_scan_gallery_images()` | 7233 | Filesystem scan |
| `_enrich_with_names()` | 7276 | DB lookups for gallery |
| `_parse_comfy_png_metadata()` | 7340 | PNG metadata parsing |

### `routes/__init__.py`

```python
def register_blueprints(app):
    from routes.characters import bp as characters_bp
    from routes.outfits import bp as outfits_bp
    # ... etc for all route modules
    app.register_blueprint(characters_bp)
    app.register_blueprint(outfits_bp)
    # ...
```

### `routes/characters.py`

| Route | Current Line |
|-------|-------------|
| `GET /` (index) | 2093 |
| `POST /rescan` | 2098 |
| `GET /character/<slug>` | 2289 |
| `GET/POST /character/<slug>/transfer` | 2305 |
| `GET/POST /create` | 2465 |
| `GET/POST /character/<slug>/edit` | 2718 |
| `POST /character/<slug>/outfit/*` (6 routes) | 2800–2988 |
| `POST /character/<slug>/upload` | 2989 |
| `POST /character/<slug>/replace_cover_from_preview` | 3018 |
| `POST /character/<slug>/generate` | 3344 |
| `POST /character/<slug>/save_defaults` | 3379 |
| `GET /get_missing_characters` | 3303 |
| `POST /clear_all_covers` | 3308 |
| `POST /generate_missing` | 3316 |

### `routes/outfits.py`

| Route | Current Line |
|-------|-------------|
| `GET /outfits` | 3844 |
| `POST /outfits/rescan` | 3850 |
| `POST /outfits/bulk_create` | 3856 |
| `GET /outfit/<slug>` | 3966 |
| `GET/POST /outfit/<slug>/edit` | 3991 |
| `POST /outfit/<slug>/upload` | 4062 |
| `POST /outfit/<slug>/generate` | 4091 |
| `POST /outfit/<slug>/replace_cover_from_preview` | 4187 |
| `GET/POST /outfit/create` | 4199 |
| `POST /outfit/<slug>/save_defaults` | 4321 |
| `POST /outfit/<slug>/clone` | 4330 |
| `POST /outfit/<slug>/save_json` | 4380 |
| Helper: `_get_linked_characters_for_outfit()` | 3956 |

### `routes/actions.py`

| Route | Current Line |
|-------|-------------|
| `GET /actions` | 4398 |
| `POST /actions/rescan` | 4403 |
| `GET /action/<slug>` | 4409 |
| `GET/POST /action/<slug>/edit` | 4430 |
| `POST /action/<slug>/upload` | 4501 |
| `POST /action/<slug>/generate` | 4530 |
| `POST /action/<slug>/replace_cover_from_preview` | 4706 |
| `POST /action/<slug>/save_defaults` | 4718 |
| `POST /actions/bulk_create` | 4727 |
| `GET/POST /action/create` | 4831 |
| `POST /action/<slug>/clone` | 4905 |
| `POST /action/<slug>/save_json` | 4947 |
| `GET /get_missing_actions` | 3401 |
| `POST /clear_all_action_covers` | 3406 |

### `routes/styles.py`

| Route | Current Line |
|-------|-------------|
| `GET /styles` | 4965 |
| `POST /styles/rescan` | 4970 |
| `GET /style/<slug>` | 4976 |
| `GET/POST /style/<slug>/edit` | 4997 |
| `POST /style/<slug>/upload` | 5060 |
| `POST /style/<slug>/generate` | 5142 |
| `POST /style/<slug>/save_defaults` | 5185 |
| `POST /style/<slug>/replace_cover_from_preview` | 5194 |
| `GET /get_missing_styles` | 5206 |
| `POST /clear_all_style_covers` | 5224 |
| `POST /styles/generate_missing` | 5232 |
| `POST /styles/bulk_create` | 5262 |
| `GET/POST /style/create` | 5358 |
| `POST /style/<slug>/clone` | 5412 |
| `POST /style/<slug>/save_json` | 5453 |

### `routes/scenes.py`

| Route | Current Line |
|-------|-------------|
| `GET /scenes` | 5471 |
| `POST /scenes/rescan` | 5476 |
| `GET /scene/<slug>` | 5482 |
| `GET/POST /scene/<slug>/edit` | 5503 |
| `POST /scene/<slug>/upload` | 5574 |
| `POST /scene/<slug>/generate` | 5680 |
| `POST /scene/<slug>/save_defaults` | 5721 |
| `POST /scene/<slug>/replace_cover_from_preview` | 5730 |
| `POST /scenes/bulk_create` | 5742 |
| `GET/POST /scene/create` | 5844 |
| `POST /scene/<slug>/clone` | 5902 |
| `POST /scene/<slug>/save_json` | 5943 |
| `GET /get_missing_scenes` | 3414 |
| `POST /clear_all_scene_covers` | 3419 |

### `routes/detailers.py`

| Route | Current Line |
|-------|-------------|
| `GET /detailers` | 5961 |
| `POST /detailers/rescan` | 5966 |
| `GET /detailer/<slug>` | 5972 |
| `GET/POST /detailer/<slug>/edit` | 5999 |
| `POST /detailer/<slug>/upload` | 6062 |
| `POST /detailer/<slug>/generate` | 6154 |
| `POST /detailer/<slug>/save_defaults` | 6206 |
| `POST /detailer/<slug>/replace_cover_from_preview` | 6215 |
| `POST /detailer/<slug>/save_json` | 6227 |
| `POST /detailers/bulk_create` | 6243 |
| `GET/POST /detailer/create` | 6340 |
| `GET /get_missing_detailers` | 5211 |
| `POST /clear_all_detailer_covers` | 5216 |

### `routes/checkpoints.py`

| Route | Current Line |
|-------|-------------|
| `GET /checkpoints` | 6396 |
| `POST /checkpoints/rescan` | 6401 |
| `GET /checkpoint/<slug>` | 6407 |
| `POST /checkpoint/<slug>/upload` | 6425 |
| `POST /checkpoint/<slug>/generate` | 6531 |
| `POST /checkpoint/<slug>/replace_cover_from_preview` | 6558 |
| `POST /checkpoint/<slug>/save_json` | 6570 |
| `GET /get_missing_checkpoints` | 6586 |
| `POST /clear_all_checkpoint_covers` | 6591 |
| `POST /checkpoints/bulk_create` | 6598 |

### `routes/looks.py`

| Route | Current Line |
|-------|-------------|
| `GET /looks` | 6702 |
| `POST /looks/rescan` | 6708 |
| `GET /look/<slug>` | 6714 |
| `GET/POST /look/<slug>/edit` | 6748 |
| `POST /look/<slug>/upload` | 6803 |
| `POST /look/<slug>/generate` | 6820 |
| `POST /look/<slug>/replace_cover_from_preview` | 6898 |
| `POST /look/<slug>/save_defaults` | 6910 |
| `POST /look/<slug>/generate_character` | 6919 |
| `POST /look/<slug>/save_json` | 7043 |
| `GET/POST /look/create` | 7060 |
| `GET /get_missing_looks` | 7103 |
| `POST /clear_all_look_covers` | 7108 |
| `POST /looks/bulk_create` | 7116 |

### `routes/presets.py`

| Route | Current Line |
|-------|-------------|
| `GET /presets` | 3429 |
| `GET /preset/<slug>` | 3435 |
| `POST /preset/<slug>/generate` | 3442 |
| `POST /preset/<slug>/replace_cover_from_preview` | 3595 |
| `POST /preset/<slug>/upload` | 3608 |
| `GET/POST /preset/<slug>/edit` | 3628 |
| `POST /preset/<slug>/save_json` | 3710 |
| `POST /preset/<slug>/clone` | 3728 |
| `POST /presets/rescan` | 3757 |
| `GET/POST /preset/create` | 3764 |
| `GET /get_missing_presets` | 3836 |

### `routes/generator.py`

| Route | Current Line |
|-------|-------------|
| `GET/POST /generator` | 2168 |
| `POST /generator/preview_prompt` | 2256 |

### `routes/gallery.py`

| Route | Current Line |
|-------|-------------|
| `GET /gallery` | 7296 |
| `GET /gallery/prompt-data` | 7414 |
| `POST /gallery/delete` | 7434 |
| `POST /resource/<category>/<slug>/delete` | 7457 |

Constants: `GALLERY_CATEGORIES`, `_MODEL_MAP`

### `routes/settings.py`

| Route/Function | Current Line |
|----------------|-------------|
| `GET/POST /settings` | 2066 |
| `POST /set_default_checkpoint` | 588 |
| `GET /api/status/comfyui` | 625 |
| `GET /api/comfyui/loaded_checkpoint` | 638 |
| `GET /api/status/mcp` | 659 |
| `GET /api/status/llm` | 674 |
| `GET /api/status/character-mcp` | 712 |
| `POST /get_openrouter_models` | 2035 |
| `POST /get_local_models` | 2051 |
| Context processor: `inject_comfyui_ws_url()` | 569 |
| Context processor: `inject_default_checkpoint()` | 582 |

### `routes/strengths.py`

| Route | Current Line |
|-------|-------------|
| `POST /strengths/<category>/<slug>/generate` | 7796 |
| `GET /strengths/<category>/<slug>/list` | 7892 |
| `POST /strengths/<category>/<slug>/clear` | 7917 |
| `POST /strengths/<category>/<slug>/save_range` | 7939 |

Constants: `_STRENGTHS_MODEL_MAP`, `_CATEGORY_LORA_NODES`, `_STRENGTHS_DATA_DIRS`
Helper: `_prepare_strengths_workflow()` (7690)

### `routes/transfer.py`

| Route | Current Line |
|-------|-------------|
| `GET/POST /resource/<category>/<slug>/transfer` | 8118 |

Constants: `_RESOURCE_TRANSFER_MAP`, `_TRANSFER_TARGET_CATEGORIES`
Helper: `_create_minimal_template()` (8045)

### `routes/queue_api.py`

| Route | Current Line |
|-------|-------------|
| `GET /api/queue` | 306 |
| `GET /api/queue/count` | 323 |
| `POST /api/queue/<job_id>/remove` | 331 |
| `POST /api/queue/<job_id>/pause` | 348 |
| `POST /api/queue/clear` | 365 |
| `GET /api/queue/<job_id>/status` | 396 |

---

## Dependency Graph

```
app.py
  ├── models.py (unchanged)
  ├── utils.py (no deps except stdlib)
  ├── services/
  │   ├── comfyui.py        ← utils (for config)
  │   ├── prompts.py        ← utils, models
  │   ├── workflow.py       ← comfyui, prompts, utils, models
  │   ├── llm.py            ← mcp (for tool calls)
  │   ├── mcp.py            ← (stdlib only: subprocess, docker)
  │   ├── sync.py           ← models, utils
  │   ├── job_queue.py      ← comfyui, models
  │   └── file_io.py        ← models, utils
  └── routes/
      ├── All route modules  ← services/*, utils, models
      └── (routes never import from other routes)
```

**No circular imports**: routes → services → utils/models. Services never import routes. Utils never imports services.

---

## Migration Phases

### Phase 1 — Extract services (no route changes)

**Order matters** — extract leaf dependencies first:

1. **`utils.py`** — Pure constants and helpers. Zero risk. Cut `parse_orientation`, `_resolve_lora_weight`, `allowed_file`, `ALLOWED_EXTENSIONS`, `_LORA_DEFAULTS`, `_IDENTITY_KEYS`, `_WARDROBE_KEYS` out of app.py. Add imports in app.py to keep everything working.

2. **`services/comfyui.py`** — `queue_prompt`, `get_history`, `get_image`, `_ensure_checkpoint_loaded`. These only use `requests` + config URL. Accept `comfyui_url` as parameter or read `current_app.config`.

3. **`services/mcp.py`** — `_ensure_mcp_repo`, `ensure_mcp_server_running`, `_ensure_character_mcp_repo`, `ensure_character_mcp_server_running`. Only uses `subprocess`/`os`/`logging`.

4. **`services/llm.py`** — `DANBOORU_TOOLS`, `call_mcp_tool`, `load_prompt`, `call_llm`. Depends on `services/mcp` for tool calls and `models.Settings` for config.

5. **`services/prompts.py`** — `build_prompt`, `build_extras_prompt`, `_dedup_tags`, `_cross_dedup_prompts`, `_resolve_character`, `_ensure_character_fields`, `_append_background`, `_build_strengths_prompts`, `_get_character_data_without_lora`. Depends on `utils` and `models`.

6. **`services/workflow.py`** — `_prepare_workflow`, `_apply_checkpoint_settings`, `_log_workflow_prompts`, `_build_style_workflow`, `_build_checkpoint_workflow`, `_queue_scene_generation`, `_queue_detailer_generation`, `_get_default_checkpoint`. Depends on `comfyui`, `prompts`, `utils`, `models`.

7. **`services/sync.py`** — All `sync_*()` + `_default_checkpoint_data`, `_resolve_preset_entity`, `_resolve_preset_fields`, `_PRESET_ENTITY_MAP`. Depends on `models`, `utils`.

8. **`services/file_io.py`** — `get_available_loras`, `get_available_checkpoints`, `_count_look_assignments`, `_count_outfit_lora_assignments`, `_scan_gallery_images`, `_enrich_with_names`, `_parse_comfy_png_metadata`. Depends on `models`, `utils`.

9. **`services/job_queue.py`** — Queue globals, `_enqueue_job`, `_queue_worker`, `_make_finalize`, `_prune_job_history`. Depends on `comfyui`, `models`. Extract last because the worker thread references many things.

**After Phase 1**: `app.py` still has all routes, but imports helpers from `services/*` and `utils`. Each service module is independently testable. The app should work identically.

### Phase 2 — Extract routes into Blueprints

**Order**: Start with the smallest/most isolated, work toward characters (largest).

1. **`routes/queue_api.py`** — 6 routes, only depends on `job_queue` globals. Blueprint prefix: none (keeps `/api/queue/*`).

2. **`routes/settings.py`** — Settings page + status APIs + context processors. Register context processors on the app via `app.context_processor` in `register_blueprints()`.

3. **`routes/gallery.py`** — Gallery + resource delete. Depends on `file_io`, `models`.

4. **`routes/transfer.py`** — Transfer system. Self-contained with its own constants.

5. **`routes/strengths.py`** — Strengths system. Self-contained with its own constants + workflow helper.

6. **`routes/generator.py`** — Generator page. Depends on `prompts.build_extras_prompt`.

7. **`routes/checkpoints.py`** — Smallest category. Good test case for the category pattern.

8. **`routes/presets.py`** — Preset CRUD. Depends on `sync._resolve_preset_*`.

9. **`routes/looks.py`** — Look CRUD + generate_character.

10. **`routes/detailers.py`** — Detailer CRUD.

11. **`routes/scenes.py`** — Scene CRUD.

12. **`routes/styles.py`** — Style CRUD.

13. **`routes/actions.py`** — Action CRUD.

14. **`routes/outfits.py`** — Outfit CRUD + linked characters helper.

15. **`routes/characters.py`** — Character CRUD + outfit management + transfer. Largest blueprint (~1300 lines). Do last since it has the most cross-cutting concerns.

**After Phase 2**: `app.py` is ~80 lines. All routes live in blueprints. Full functionality preserved.

### Phase 3 — Verification & cleanup

1. Run the app, test every page manually (index, detail, generate, edit, clone, delete for each category).
2. Test batch generation, generator page, gallery, settings, strengths, transfer.
3. Remove any dead imports from `app.py`.
4. Update `CLAUDE.md` to reflect new file structure.

---

## Blueprint Pattern Template

Each route blueprint follows this pattern:

```python
from flask import Blueprint, render_template, request, redirect, url_for, flash, session, jsonify
from models import db, Outfit  # only what's needed
from services.workflow import _prepare_workflow
from services.prompts import build_prompt, _resolve_character, _ensure_character_fields, _append_background
from services.job_queue import _enqueue_job, _make_finalize
from services.file_io import get_available_loras
from services.llm import call_llm, load_prompt
from utils import allowed_file

import logging
logger = logging.getLogger('gaze')

bp = Blueprint('outfits', __name__)

@bp.route('/outfits')
def outfits_index():
    ...
```

---

## Risk Mitigation

- **Circular imports**: Enforced by the dependency graph — routes → services → utils/models. If a service needs something from another service, import it at function level if needed.
- **`current_app` vs `app`**: Routes already use `request`, `session`, etc. which are context-local. Services that need app config use `current_app.config[...]` inside function bodies (not at module level).
- **Thread safety**: `job_queue.py` keeps the same threading globals. The worker thread is started in `app.py`'s startup block, same as before.
- **Session access**: Only route functions access `session`. Services that currently read session (like `_get_default_checkpoint`) will stay in `services/workflow.py` and import `session` from flask — this is fine since they're only called from within request context.
- **Testing**: After each phase-1 extraction, verify the app starts and the affected functionality works before proceeding to the next module.

---

## Lines of Code Estimate (per module)

| Module | Approx Lines |
|--------|-------------|
| `app.py` (final) | ~100 |
| `utils.py` | ~120 |
| `services/comfyui.py` | ~120 |
| `services/mcp.py` | ~150 |
| `services/llm.py` | ~200 |
| `services/prompts.py` | ~350 |
| `services/workflow.py` | ~400 |
| `services/sync.py` | ~800 |
| `services/job_queue.py` | ~250 |
| `services/file_io.py` | ~250 |
| `routes/characters.py` | ~1300 |
| `routes/outfits.py` | ~550 |
| `routes/actions.py` | ~570 |
| `routes/styles.py` | ~500 |
| `routes/scenes.py` | ~500 |
| `routes/detailers.py` | ~450 |
| `routes/checkpoints.py` | ~350 |
| `routes/looks.py` | ~550 |
| `routes/presets.py` | ~450 |
| `routes/generator.py` | ~150 |
| `routes/gallery.py` | ~300 |
| `routes/settings.py` | ~250 |
| `routes/strengths.py` | ~400 |
| `routes/transfer.py` | ~200 |
| `routes/queue_api.py` | ~120 |
| **Total** | **~8,430** |