From a71c95c8caf7e570c3f484ce1a53b7acb5ef2006 Mon Sep 17 00:00:00 2001 From: skal Date: Wed, 25 Mar 2026 06:25:53 +0100 Subject: feat(cnn_v3/tools): embed default weights in HTML tool; add --html export flag MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - cnn_v3/tools/weights.js: new file — base64-encoded cnn_v3_weights.bin + cnn_v3_film_mlp.bin; loaded at startup so the tool works without dropping files - tester.js: preload() falls back to embedded weights.js constants when fetch fails; logs "Loaded embedded" vs "Preloaded" to distinguish the two paths - index.html: load weights.js before tester.js - export_cnn_v3_weights.py: add --html / --html-output flags that call update_weights_js() to regenerate weights.js after a training run - HOW_TO_CNN.md: update pipeline diagram, §3 export commands, §7 HTML tool section (file table, workflow, weights.js description), Appendix A handoff(Gemini): weights.js now the canonical source for HTML tool defaults; regenerate with `uv run export_cnn_v3_weights.py --output ... --html` --- cnn_v3/docs/HOW_TO_CNN.md | 102 +++++++++++++++++++++++++++------------------- 1 file changed, 60 insertions(+), 42 deletions(-) (limited to 'cnn_v3/docs/HOW_TO_CNN.md') diff --git a/cnn_v3/docs/HOW_TO_CNN.md b/cnn_v3/docs/HOW_TO_CNN.md index 4966a61..624deaa 100644 --- a/cnn_v3/docs/HOW_TO_CNN.md +++ b/cnn_v3/docs/HOW_TO_CNN.md @@ -58,13 +58,13 @@ Input: 20-channel G-buffer feature textures (rgba32uint) ``` photos/Blender → pack → dataset/ → train_cnn_v3.py → checkpoint.pth │ - export_cnn_v3_weights.py - ┌─────────┴──────────┐ - cnn_v3_weights.bin cnn_v3_film_mlp.bin - │ - CNNv3Effect::upload_weights() - │ - demo / HTML tool + export_cnn_v3_weights.py [--html] + ┌──────────┴────────────┬──────────────┐ + cnn_v3_weights.bin cnn_v3_film_mlp.bin weights.js + │ (HTML tool + CNNv3Effect::upload_weights() defaults) + │ + demo ``` --- @@ -284,29 +284,36 @@ The U-Net conv weights and FiLM MLP train **jointly** in a single run. No separa ### Prerequisites +`train_cnn_v3.py` and `export_cnn_v3_weights.py` carry inline `uv` dependency metadata +(`# /// script`). Use `uv run` — no manual `pip install` needed: + ```bash -pip install torch torchvision pillow numpy opencv-python cd cnn_v3/training +uv run train_cnn_v3.py --input dataset/ --epochs 1 --patch-size 32 --detector random ``` -**With `uv` (no pip needed):** dependencies are declared inline in `train_cnn_v3.py` -and installed automatically on first run: +**Without `uv` (manual pip):** ```bash +pip install torch torchvision pillow numpy opencv-python cd cnn_v3/training -uv run train_cnn_v3.py --input dataset/ --epochs 1 --patch-size 32 --detector random +python3 train_cnn_v3.py ... ``` +The pack scripts (`pack_photo_sample.py`, `pack_blender_sample.py`) and +`gen_test_vectors.py` do **not** have uv metadata — run them with `python3` directly +(they only need `numpy`, `pillow`, and optionally `openexr`). + ### Quick-start commands **Smoke test — 1 epoch, validates end-to-end without GPU:** ```bash -python3 train_cnn_v3.py --input dataset/ --epochs 1 \ +uv run train_cnn_v3.py --input dataset/ --epochs 1 \ --patch-size 32 --detector random ``` **Standard photo training (patch-based):** ```bash -python3 train_cnn_v3.py \ +uv run train_cnn_v3.py \ --input dataset/ \ --input-mode simple \ --epochs 200 @@ -314,7 +321,7 @@ python3 train_cnn_v3.py \ **Blender G-buffer training:** ```bash -python3 train_cnn_v3.py \ +uv run train_cnn_v3.py \ --input dataset/ \ --input-mode full \ --epochs 200 @@ -322,7 +329,7 @@ python3 train_cnn_v3.py \ **Full-image mode (better global coherence, slower):** ```bash -python3 train_cnn_v3.py \ +uv run train_cnn_v3.py \ --input dataset/ \ --input-mode full \ --full-image --image-size 256 \ @@ -340,7 +347,8 @@ python3 train_cnn_v3.py \ | `--lr F` | 1e-3 | Reduce to 1e-4 if loss oscillates or NaN | | `--patch-size N` | 64 | Smaller = faster epoch, less spatial context | | `--patches-per-image N` | 256 | Reduce for small datasets | -| `--detector` | `harris` | `random` for smoke tests; `shi-tomasi` as alternative | +| `--detector` | `harris` | `random` for smoke tests; also `shi-tomasi`, `fast`, `gradient` | +| `--patch-search-window N` | 0 | Search ±N px in target to find best alignment (grayscale MSE) per patch; 0=disabled. Use when source and target are not perfectly co-registered (e.g. photo + hand-painted target). Offsets cached at dataset init. | | `--channel-dropout-p F` | 0.3 | Lower if all samples have geometry (Blender only) | | `--full-image` | off | Resize full image instead of patch crops | | `--image-size N` | 256 | Resize target; only used with `--full-image` | @@ -454,14 +462,27 @@ The final checkpoint is always written even if `--checkpoint-every 0`. ## 3. Exporting Weights -Converts a trained `.pth` checkpoint to two raw binary files for the C++ runtime. +Converts a trained `.pth` checkpoint to two raw binary files for the C++ runtime, +and optionally updates the HTML tool's embedded defaults. +**Standard export (C++ runtime only):** ```bash cd cnn_v3/training -python3 export_cnn_v3_weights.py checkpoints/checkpoint_epoch_200.pth \ +uv run export_cnn_v3_weights.py checkpoints/checkpoint_epoch_200.pth \ --output ../../workspaces/main/weights/ ``` +**Export + update HTML tool defaults (`cnn_v3/tools/weights.js`):** +```bash +uv run export_cnn_v3_weights.py checkpoints/checkpoint_epoch_200.pth \ + --output ../../workspaces/main/weights/ \ + --html +``` + +`--html` base64-encodes both `.bin` files and rewrites `cnn_v3/tools/weights.js` +so the HTML tool loads the new weights as its embedded defaults at startup. +Use `--html-output PATH` to write to a different `weights.js` location. + Output files are registered in `workspaces/main/assets.txt` as: ``` WEIGHTS_CNN_V3, BINARY, weights/cnn_v3_weights.bin, "CNN v3 conv weights (f16, 3928 bytes)" @@ -636,7 +657,7 @@ Do not reference them from outside the effect unless debugging. ```bash cmake -B build -DCMAKE_BUILD_TYPE=Release -cmake --build build -j$(nproc) +cmake --build build -j4 ./build/demo ``` @@ -733,13 +754,14 @@ If results drift after shader edits, verify these invariants match the Python re ## 7. HTML WebGPU Tool -**Location:** `cnn_v3/tools/` — three files, no build step. +**Location:** `cnn_v3/tools/` — four files, no build step. | File | Lines | Contents | |------|-------|----------| -| `index.html` | 147 | HTML + CSS | -| `shaders.js` | 252 | WGSL shader constants, weight-offset constants | -| `tester.js` | 540 | `CNNv3Tester` class, event wiring | +| `index.html` | 168 | HTML + CSS | +| `shaders.js` | 312 | WGSL shader constants, weight-offset constants | +| `tester.js` | 913 | `CNNv3Tester` class, inference pipeline, layer viz | +| `weights.js` | 7 | Embedded default weights (base64); auto-generated by `--html` | ### Usage @@ -750,32 +772,27 @@ python3 -m http.server 8080 # Open: http://localhost:8080/cnn_v3/tools/ ``` -Or on macOS with Chrome: +Weights are **loaded automatically at startup** from `weights.js` (embedded base64). +If the tool is served from the repo root, it also tries to fetch the latest +`workspaces/main/weights/*.bin` over HTTP and uses those if available. +Use the **↺ Reload** button to re-fetch after updating weights on disk. + +To update the embedded defaults after a training run, use `--html` (§3): ```bash -open -a "Google Chrome" --args --allow-file-access-from-files -open cnn_v3/tools/index.html +uv run export_cnn_v3_weights.py checkpoints/checkpoint.pth \ + --output ../../workspaces/main/weights/ --html ``` ### Workflow -1. **Drop `cnn_v3_weights.bin`** onto the left "weights" drop zone. -2. **Drop a PNG or video** onto the centre canvas → CNN runs immediately. -3. _(Optional)_ **Drop `cnn_v3_film_mlp.bin`** → FiLM sliders become active. -4. Adjust **beat_phase / beat_norm / audio_int / style_p0 / style_p1** sliders → reruns on change. -5. Click layer buttons (**Feat · Enc0 · Enc1 · BN · Dec1 · Output**) in the right panel to inspect activations. -6. **Save PNG** to export the current output. +1. **Drop a PNG or video** onto the canvas → CNN runs immediately (weights pre-loaded). +2. Adjust **beat_phase / beat_norm / audio_int / style_p0 / style_p1** sliders. +3. Click layer buttons (**Feat · Enc0 · Enc1 · BN · Dec1 · Output**) to inspect activations. +4. **Save PNG** to export the current output. +5. _(Optional)_ Drop updated `.bin` files onto the left panel to override embedded weights. Keyboard: `[SPACE]` toggle original · `[D]` diff×10. -### Input files - -| File | Format | Notes | -|------|--------|-------| -| `cnn_v3_weights.bin` | raw u32 (no header) | 982 u32 = 1964 f16 = ~3.9 KB | -| `cnn_v3_film_mlp.bin` | raw f32 | 776 f32 = 3.1 KB; optional — identity FiLM used if absent | - -Both produced by `export_cnn_v3_weights.py` (§3). - ### Texture chain | Texture | Format | Size | @@ -816,7 +833,7 @@ all geometric channels (normal, depth, depth_grad, mat_id, prev) = 0. | `cnn_v3/training/pack_photo_sample.py` | Photo → zeroed-geometry sample directory | | `cnn_v3/training/cnn_v3_utils.py` | Dataset class, feature assembly, channel dropout, salient-point detection | | `cnn_v3/training/train_cnn_v3.py` | CNNv3 model definition, training loop, CLI | -| `cnn_v3/training/export_cnn_v3_weights.py` | Checkpoint → `cnn_v3_weights.bin` + `cnn_v3_film_mlp.bin` | +| `cnn_v3/training/export_cnn_v3_weights.py` | Checkpoint → `cnn_v3_weights.bin` + `cnn_v3_film_mlp.bin`; `--html` rewrites `weights.js` | | `cnn_v3/training/gen_test_vectors.py` | NumPy reference forward pass + C header generator | | `cnn_v3/test_vectors.h` | Compiled-in test vectors (auto-generated, do not edit) | | `cnn_v3/src/cnn_v3_effect.h` | C++ class, Params structs, `CNNv3FiLMParams` API | @@ -827,6 +844,7 @@ all geometric channels (normal, depth, depth_grad, mat_id, prev) = 0. | `cnn_v3/tools/index.html` | HTML tool — UI shell + CSS | | `cnn_v3/tools/shaders.js` | HTML tool — inline WGSL shaders + weight-offset constants | | `cnn_v3/tools/tester.js` | HTML tool — CNNv3Tester class, inference pipeline, layer viz | +| `cnn_v3/tools/weights.js` | HTML tool — embedded default weights (base64, auto-generated) | | `cnn_v2/tools/cnn_v2_test/index.html` | HTML tool reference pattern (v2) | --- -- cgit v1.2.3