diff options
Diffstat (limited to 'doc/CNN.md')
| -rw-r--r-- | doc/CNN.md | 40 |
1 files changed, 34 insertions, 6 deletions
@@ -1,11 +1,15 @@ # Convolutional Neural Net Shader (CNN) post-processing +**Status:** ✅ Foundation implemented (single-layer, expandable to multi-pass) + ## Idea Have the input 3d scene be processed by a multi-layer CNN trained on the side. Input: some rendered scene. Output: 'stylized' scene with CNN post-processing. +**See `doc/CNN_EFFECT.md` for implementation details, usage, and API reference.** + ## Shader implementation ### input / output @@ -36,16 +40,40 @@ we need 3 or 4 layer ? Several different shaders for each layer. Ping-pong for input/output texture buffer between each layers? -## Training +## Implementation Status + +**Completed:** +- ✅ Modular WGSL shader architecture (6 snippet files) +- ✅ CNNEffect C++ class (single-layer rendering) +- ✅ ShaderComposer integration (#include resolution) +- ✅ Asset registration (7 new shader assets) +- ✅ Test coverage (test_demo_effects.cc) +- ✅ Placeholder identity weights for testing + +**Size:** ~3-4 KB shader code + ~2-4 KB weights = **5-8 KB total** + +**Pending:** +- ⏳ Training script (`scripts/train_cnn.py`) to generate real weights +- ⏳ Multi-layer rendering with ping-pong textures +- ⏳ Weight quantization for size optimization + +--- + +## Training (To Be Implemented) The layer weight/bias data are hard-coded in the shaders. -Need training with external python script. -File: CNN.py contains an example of what the training script could be. -Just an example, doesn't match our requirement yet. +Training workflow: + +1. Prepare image pairs (before: raw render, after: target style) +2. Run `python scripts/train_cnn.py --input scene.png --target stylized.png` +3. Script generates `cnn_weights_generated.wgsl` +4. Rebuild: `cmake --build build -j4` + +**Reference:** File `CNN.py` contains training example (needs adaptation). Need a repository of reference image pairs (before/after) for training and validation. -Each input image is randomly sampled into 3x3 patch of (r,g,b,1/z) input samples. +Each input image is randomly sampled into 3×3 patch of (r,g,b,1/z) input samples. And trained to match the (r,g,b,a) output. -Training generates the .wgsl code for layers' shaders, and the c++ code for the post-processing 'Effect'. +Training generates the .wgsl code for layers' shaders. |