# Convolutional Neural Net Shader (CNN) post-processing

## Idea

Have the input 3d scene be processed by a multi-layer CNN trained on the side.
Input: some rendered scene.
Output: 'stylized' scene with CNN post-processing.

## Shader implementation

### input / output

Need 1 texture buffer per CNN layer.
Input (r,g,b,1/z) for layer 0 (render 3d scene), or output from layer N-1 for layer N.
output: (r,g,b, alpha). Don't need the 1/z information (can be fetched from input)

### size of one layer

Notation:
S: the number of input samples from layer N-1.
Example: 3x3 input -> S = 3x3 = 9. 

Each S samples is 4 values (r,g,b, w=1/z).

Each sample is processed by a mat4 matrix. 4 input => 4 output.

Weight matrix = S x mat4

Final bias: 4 values.

WGSL code example: See file CNN.shader

### Layers

we need 3 or 4 layer ?
Several different shaders for each layer.
Ping-pong for input/output texture buffer between each layers?

## Training

The layer weight/bias data are hard-coded in the shaders.
Need training with external python script.
File: CNN.py contains an example of what the training script could be.
Just an example, doesn't match our requirement yet.

Need a repository of reference image pairs (before/after) for training and validation.
Each input image is randomly sampled into 3x3 patch of (r,g,b,1/z) input samples.
And trained to match the (r,g,b,a) output.

Training generates the .wgsl code for layers' shaders, and the c++ code for the post-processing 'Effect'.