1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
|
# To-Do List
**High-level task tracker.** See design docs for details. Completed: `doc/COMPLETED.md`
---
## Priority 1: Spectral Brush Editor (Task #5) [IN PROGRESS]
Procedural spectrogram tool: 50-100× compression (5 KB .spec → ~100 bytes C++).
**Design:** `doc/SPECTRAL_BRUSH_2.md` (MQ-based v2)
---
## Priority 2: Test Infrastructure Maintenance [ONGOING]
**Status:** 35/35 tests passing
**Outstanding TODOs:**
1. **test_fft.cc:87** - Investigate FFT-DCT algorithm discrepancy
## Priority 4: Audio System Enhancements [LOW PRIORITY]
1. **`synth.cc`: use `ola_decode()` from `src/audio/ola.h`** — the OLA decode logic in
`synth_render()` is currently inlined for frame-by-frame lazy decoding. Refactor to
call `ola_decode()` for consistency with `spectool` and the test (requires decoupling
the per-frame lazy path, e.g. decode a full block on demand then serve samples).
2. **GPU-Accelerated PCM Synthesis:**
- Compute shader for direct PCM generation (bypass spectrogram)
- Write to compute buffer, readback to synth
---
## Priority 4: 3D System Enhancements (Task #18)
Pipeline for importing complex 3D scenes to replace hardcoded geometry.
**Status:** C++ object data loading complete. Shader SDF integration pending.
---
## Priority 4: WGSL Modularization (Task #50) [RECURRENT]
Ongoing shader code hygiene for granular, reusable snippets.
---
## Priority 4: Wine/Windows Black Screen
`demo64k.exe` runs under Wine (wgpu-native v27, Vulkan/MoltenVK) but shows a black window — no visuals rendered. Audio and timeline progress correctly. GPU device/adapter init succeeds.
**Likely causes to investigate:**
- Swapchain format mismatch (Wine Vulkan may prefer BGRA8 over RGBA8)
- Surface present failing silently (check `WGPUSurfaceGetCurrentTexture` status)
- Render pass output not reaching the surface (missing present call or wrong texture view)
**To reproduce:** `./scripts/run_win.sh` — window opens, stays black.
---
## CNN v3 — U-Net + FiLM [IN PROGRESS]
U-Net architecture with FiLM conditioning. Runtime style control via beat/audio.
Richer G-buffer input (normals, depth, material IDs). Per-pixel testability across
PyTorch / HTML WebGPU / C++ WebGPU.
**Design:** `cnn_v3/docs/CNN_V3.md`
**Phases:**
1. ✅ G-buffer: `GBufferEffect` integrated. SDF/shadow placeholder (shadow=1, transp=0).
2. ✅ Training infrastructure: `blender_export.py`, `pack_blender_sample.py`, `pack_photo_sample.py`
3. ✅ WGSL shaders: cnn_v3_common (snippet), enc0, enc1, bottleneck, dec1, dec0
4. ✅ C++ `CNNv3Effect`: 5 compute passes, FiLM uniform upload, `set_film_params()` API
- Params alignment fix: WGSL `vec3u` align=16 → C++ structs 64/96 bytes
- Weight offsets as explicit formulas (e.g. `20*4*9+4`)
- FiLM γ/β: identity defaults; real values require trained MLP (see below)
5. ✅ Parity validation: test vectors + `test_cnn_v3_parity.cc`. max_err=4.88e-4 (≤1/255).
- Key fix: intermediate nodes at fractional resolutions (W/2, W/4) via `NodeRegistry::default_width()/default_height()`
6. ✅ Training script: `train_cnn_v3.py` + `cnn_v3_utils.py` written
- ✅ `export_cnn_v3_weights.py` — convert trained `.pth` → `.bin` (f16)
7. ✅ Validation tools:
- `GBufViewEffect` — C++ 4×5 channel grid (all 20 G-buffer channels)
- Web tool "Load sample directory" — G-buffer pack → CNN inference → PSNR
- See `cnn_v3/docs/HOWTO.md` §9
**Next: run a real training pass**
- See `cnn_v3/docs/HOWTO.md` §3 for training commands
## Future: CNN v3 "2D Mode" (G-buffer-free)
Allow `CNNv3Effect` to run on a plain screen buffer / photo without a real G-buffer.
Fake the missing feature vectors (normals, depth, material IDs, shadow, transp) from
the RGB input alone:
- normals: approximate from local luminance gradient (Sobel)
- depth: constant (e.g. 0.5) or estimated from a simple heuristic
- material IDs / shadow / transp: neutral defaults (e.g. 0)
This would let the effect be applied to any rendered frame (post-NTSC, post-Scratch, etc.)
without requiring a 3D G-buffer pass upstream, and enable training/inference on photos.
Implementation sketch:
- New `CNNv3Effect2D` subclass (or a mode flag) that synthesizes `feat_tex0`/`feat_tex1`
internally from a single `rgba8unorm` input, then runs the same 5-pass U-Net.
- Separate `gbuf_pack_2d.wgsl` compute shader that fills feat0/feat1 from a photo buffer.
## Future: CNN v2 8-bit Quantization
Reduce weights from f16 (~3.2 KB) to i8 (~1.6 KB).
**Requirements:** Quantization-aware training (QAT)
**Design:** `cnn_v2/docs/CNN_V2.md`
---
## Future: Size Optimization (64k Target)
- Task #22: Windows Native Platform (Win32)
- Task #28: Spectrogram Quantization
- Task #34: Full STL Removal
- Task #35: CRT Replacement
**Measure:** `./scripts/measure_size.sh`
---
**Backlog:** `doc/BACKLOG.md` for untriaged ideas
|
