diff options
Diffstat (limited to 'cnn_v3/docs')
| -rw-r--r-- | cnn_v3/docs/HOWTO.md | 138 |
1 files changed, 111 insertions, 27 deletions
diff --git a/cnn_v3/docs/HOWTO.md b/cnn_v3/docs/HOWTO.md index 08979e7..5c5cc2a 100644 --- a/cnn_v3/docs/HOWTO.md +++ b/cnn_v3/docs/HOWTO.md @@ -22,57 +22,141 @@ It rasterizes proxy geometry to MRT G-buffer textures and packs them into two ### Adding to a Sequence -`GBufferEffect` does not exist in `seq_compiler.py` as a named effect yet -(no `.seq` syntax integration for Phase 1). Wire it directly in C++ alongside -your scene code, or add it to the timeline when the full CNNv3Effect is ready. +Both `GBufferEffect` and `GBufViewEffect` are registered in `seq_compiler.py` +(`CLASS_TO_HEADER`) and can be wired directly in `timeline.seq`. -**C++ wiring example** (e.g. inside a Sequence or main.cc): +**Debug view (G-buffer → sink)**: +```seq +SEQUENCE 12.00 0 "cnn_v3_test" + NODE gbuf_feat0 gbuf_rgba32uint + NODE gbuf_feat1 gbuf_rgba32uint + EFFECT + GBufferEffect source -> gbuf_feat0 gbuf_feat1 0.00 8.00 + EFFECT + GBufViewEffect gbuf_feat0 gbuf_feat1 -> sink 0.00 8.00 +``` -```cpp -#include "../../cnn_v3/src/gbuffer_effect.h" +**Full CNN pipeline**: +```seq +SEQUENCE 12.00 0 "cnn_v3_test" + NODE gbuf_feat0 gbuf_rgba32uint + NODE gbuf_feat1 gbuf_rgba32uint + NODE cnn_v3_out gbuf_albedo + EFFECT + GBufferEffect source -> gbuf_feat0 gbuf_feat1 0.00 8.00 + EFFECT + CNNv3Effect gbuf_feat0 gbuf_feat1 -> cnn_v3_out 0.00 8.00 + EFFECT + Passthrough cnn_v3_out -> sink 0.00 8.00 +``` -// Allocate once alongside your scene -auto gbuf = std::make_shared<GBufferEffect>( - ctx, /*inputs=*/{"prev_cnn"}, // or any dummy node - /*outputs=*/{"gbuf_feat0", "gbuf_feat1"}, - /*start=*/0.0f, /*end=*/60.0f); +### Internal scene -gbuf->set_scene(&my_scene, &my_camera); +Call `set_scene()` once before the first render to populate the built-in demo +scene. No external `Scene` or `Camera` pointer is required — the effect owns +them. -// In render loop, call before CNN pass: -gbuf->render(encoder, params, nodes); -``` +**What `set_scene()` creates:** +- **20 small cubes** — random positions in [-2,2]×[-1.5,1.5]³, scale 0.1–0.25, + random colors. Each has a random rotation axis and speed; animated each frame + via `quat::from_axis(axis, time * speed)`. +- **4 pumping spheres** — at fixed world positions, base radii 0.25–0.35. + Scale driven by `audio_intensity`: `r = base_r * (1 + audio_intensity * 0.8)`. +- **Camera** — position (0, 2.5, 6), target (0, 0, 0), 45° FOV. + Aspect ratio updated each frame from `params.aspect_ratio`. +- **Two directional lights** (uploaded to `lights_uniform_`, ready for shadow pass): + - Key: warm white (1.0, 0.92, 0.78), direction `normalize(1, 2, 1)` (upper-right-front) + - Fill: cool blue (0.4, 0.45, 0.8 × 0.4), direction `normalize(-1, 1, -1)` (upper-left-back) ### Internal passes Each frame, `GBufferEffect::render()` executes: -1. **Pass 1 — MRT rasterization** (`gbuf_raster.wgsl`) +1. **Pass 1 — MRT rasterization** (`gbuf_raster.wgsl`) ✅ - Proxy box (36 verts) × N objects, instanced - MRT outputs: `gbuf_albedo` (rgba16float), `gbuf_normal_mat` (rgba16float) - Depth test + write into `gbuf_depth` (depth32float) + - `obj.type` written to `ObjectData.params.x` for future SDF branching + +2. **Pass 2 — SDF shadow raymarching** (`gbuf_shadow.wgsl`) ✅ + - See implementation plan below. -2. **Pass 2/3 — SDF + Lighting** — TODO (placeholder: shadow=1, transp=0) +3. **Pass 3 — Transparency** — TODO (deferred; transp=0 for opaque scenes) -3. **Pass 4 — Pack compute** (`gbuf_pack.wgsl`) +4. **Pass 4 — Pack compute** (`gbuf_pack.wgsl`) ✅ - Reads all G-buffer textures + `prev_cnn` input - Writes `feat_tex0` + `feat_tex1` (rgba32uint, 20 channels, 32 bytes/pixel) + - Shadow / transp nodes cleared to 1.0 / 0.0 via zero-draw render passes + until Pass 2/3 are implemented. ### Output node names -By default the outputs are named from the `outputs` vector passed to the -constructor. Use these names when binding the CNN effect input: +Outputs are named from the `outputs` vector passed to the constructor: ``` outputs[0] → feat_tex0 (rgba32uint: albedo.rgb, normal.xy, depth, depth_grad.xy) outputs[1] → feat_tex1 (rgba32uint: mat_id, prev.rgb, mip1.rgb, mip2.rgb, shadow, transp) ``` -### Scene data +--- + +## 1b. GBufferEffect — Implementation Plan (Pass 2: SDF Shadow) + +### What remains + +| Item | Status | Notes | +|------|--------|-------| +| Pass 1: MRT raster | ✅ Done | proxy box, all object types | +| Pass 4: Pack compute | ✅ Done | 20 channels packed | +| Internal scene + animation | ✅ Done | cubes + spheres + 2 lights | +| Pass 2: SDF shadow | ✅ Done | `gbuf_shadow.wgsl`, proxy-box SDF per object | +| Pass 3: Transparency | ❌ TODO | low priority, opaque scenes only | +| Phase 4: type-aware SDF | ✅ Done | switch on `obj.params.x` in `dfWithID` | + +### Pass 2: SDF shadow raymarching + +**New file: `cnn_v3/shaders/gbuf_shadow.wgsl`** — fullscreen render pass. + +Bind layout: + +| Binding | Type | Content | +|---------|------|---------| +| 0 | `uniform` | `GlobalUniforms` (`#include "common_uniforms"`) | +| 1 | `storage read` | `ObjectsBuffer` | +| 2 | `texture_depth_2d` | depth from Pass 1 | +| 3 | `sampler` (non-filtering) | depth load | +| 4 | `uniform` | `GBufLightsUniforms` (2 lights) | + +Algorithm per fragment: +1. Reconstruct world position from NDC depth + `globals.inv_view_proj` +2. For each object: `sdBox((inv_model * world_pos).xyz, vec3(1.0))` — proxy box in local space +3. For each light: offset ray origin by `0.02 * surface_normal`; march shadow ray toward `light.direction` +4. Soft shadow via `shadowWithStoredDistance()` from `render/raymarching_id` +5. Combine lights: `shadow = min(shadow_light0, shadow_light1)` +6. Discard fragments where depth == 1.0 (sky/background → shadow = 1.0) +7. Output shadow factor to RGBA8Unorm render target (`.r` = shadow) + +**C++ additions (`gbuffer_effect.h/.cc`):** +```cpp +RenderPipeline shadow_pipeline_; +void create_shadow_pipeline(); +``` +In `render()` between Pass 1 and the shadow/transp node clears: +- Build bind group (global_uniforms_buf_, objects_buf_, depth_view, sampler_, lights_uniform_) +- Run fullscreen triangle → `node_shadow_` color attachment +- Remove the `clear_node(node_shadow_, 1.0f)` placeholder once the pass is live + +**Register:** +- `cnn_v3/shaders/gbuf_shadow.wgsl` → `SHADER_GBUF_SHADOW` in `assets.txt` +- `extern const char* gbuf_shadow_wgsl;` in `gbuffer_effect.cc` + +### Phase 4: Object-type-aware SDF (optional) + +Branch on `obj.params.x` (populated since this commit) using `math/sdf_shapes`: + +| Type value | ObjectType | SDF | +|------------|-----------|-----| +| 0 | CUBE | `sdBox(local_p, vec3(1))` | +| 1 | SPHERE | `sdSphere(local_p, 1.0)` | +| 2 | PLANE | `sdPlane(local_p, vec3(0,1,0), obj.params.y)` | +| 3 | TORUS | `sdTorus(local_p, vec2(0.8, 0.2))` | -Call `set_scene(scene, camera)` before the first render. The effect uploads -`GlobalUniforms` (view-proj, camera pos, resolution) and `ObjectData` (model -matrix, color) to GPU storage buffers each frame. +Only worth adding after Pass 2 is validated visually. --- @@ -253,7 +337,7 @@ Test vectors generated by `cnn_v3/training/gen_test_vectors.py` (PyTorch referen | Phase | Status | Notes | |-------|--------|-------| | 1 — G-buffer (raster + pack) | ✅ Done | Integrated, 36/36 tests pass | -| 1 — G-buffer (SDF + shadow passes) | TODO | Placeholder: shadow=1, transp=0 | +| 1 — G-buffer (SDF shadow pass) | ✅ Done | `gbuf_shadow.wgsl`, proxy-box SDF | | 2 — Training infrastructure | ✅ Done | blender_export.py, pack_*_sample.py | | 3 — WGSL U-Net shaders | ✅ Done | 5 compute shaders + cnn_v3/common snippet | | 4 — C++ CNNv3Effect | ✅ Done | FiLM uniform upload, 36/36 tests pass | @@ -386,13 +470,13 @@ auto gview = std::make_shared<GBufViewEffect>(ctx, | Row | Col 0 | Col 1 | Col 2 | Col 3 | |-----|-------|-------|-------|-------| -| 0 | `alb.r` (red tint) | `alb.g` (green tint) | `alb.b` (blue tint) | `nrm.x` remap→[0,1] | +| 0 | `alb.r` | `alb.g` | `alb.b` | `nrm.x` remap→[0,1] | | 1 | `nrm.y` remap→[0,1] | `depth` (inverted) | `dzdx` ×20+0.5 | `dzdy` ×20+0.5 | | 2 | `mat_id` | `prev.r` | `prev.g` | `prev.b` | | 3 | `mip1.r` | `mip1.g` | `mip1.b` | `mip2.r` | | 4 | `mip2.g` | `mip2.b` | `shadow` | `transp` | -1-pixel gray grid lines separate cells. Dark background for out-of-range cells. +All channels displayed as grayscale. 1-pixel gray grid lines separate cells. Dark background for out-of-range cells. **Shader binding layout** (no sampler needed — integer texture): |