summaryrefslogtreecommitdiff
path: root/doc/CNN_V2.md
AgeCommit message (Collapse)Author
22 hoursReplace hard clamp with sigmoid activation in CNN v2skal
Fixes training collapse where p1/p2 channels saturate due to gradient blocking at clamp boundaries. Sigmoid provides smooth [0,1] mapping with continuous gradients. Changes: - Layer 0: clamp(x, 0, 1) → sigmoid(x) - Final layer: clamp(x, 0, 1) → sigmoid(x) - Middle layers: ReLU unchanged (already stable) Updated files: - training/train_cnn_v2.py: PyTorch model activations - workspaces/main/shaders/cnn_v2/cnn_v2_compute.wgsl: WGSL shader - tools/cnn_v2_test/index.html: HTML validation tool - doc/CNN_V2.md: Documentation Validation: - Build clean (no shader errors) - 34/36 tests pass (2 unrelated script tests fail) - 10-epoch training: loss 0.153 → 0.088 (good convergence) - cnn_test processes images successfully Breaking change: Old checkpoints trained with clamp() incompatible. Retrain from scratch required. handoff(Claude): CNN v2 sigmoid activation implemented and validated.
24 hoursFix CNN v2 static feature channel mapping (p4-p7 → channels 8-11)skal
Fixed bug in gen_identity_weights.py --p47 mode: static features p4-p7 (uv_x, uv_y, sin20_y, bias) are at input channels 8-11, not 4-7. Weight tensor layout: - Channels 0-3: Previous layer output (4D RGBA) - Channels 4-11: Static features (8D: p0-p7) Static features: - p0-p3 (channels 4-7): RGB+D from mip level - p4-p7 (channels 8-11): uv_x, uv_y, sin20_y, bias Updated: - training/gen_identity_weights.py: Change weights[i,i+4] to weights[i,i+8] - workspaces/main/weights/mix_p47.bin: Regenerated (not in repo) - doc/CNN_V2.md: Add Input Channel Mapping section with full layout table Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
25 hoursCNN v2: Alpha channel depth handling and layer visualizationskal
Training changes: - Changed p3 default depth from 0.0 to 1.0 (far plane semantics) - Extract depth from target alpha channel in both datasets - Consistent alpha-as-depth across training/validation Test tool enhancements (cnn_test): - Added load_depth_from_alpha() for R32Float depth texture - Fixed bind group layout for UnfilterableFloat sampling - Added --save-intermediates with per-channel grayscale composites - Each layer saved as 4x wide PNG (p0-p3 stacked horizontally) - Global layers_composite.png for vertical layer stack overview Investigation notes: - Static features p4-p7 ARE computed and bound correctly - Sin_20_y pattern visibility difference between tools under investigation - Binary weights timestamp (Feb 13 20:36) vs HTML tool (Feb 13 22:12) - Next: Update HTML tool with canonical binary weights handoff(Claude): HTML tool weights update pending - base64 encoded canonical weights ready in /tmp/weights_b64.txt for line 392 replacement. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
31 hoursCNN v2 training: Add --grayscale-loss option for luminance-based loss ↵skal
computation Add option to compute loss on grayscale (Y = 0.299*R + 0.587*G + 0.114*B) instead of full RGBA channels. Useful for training models that prioritize luminance accuracy over color accuracy. Changes: - training/train_cnn_v2.py: Add --grayscale-loss flag and grayscale conversion in loss computation - scripts/train_cnn_v2_full.sh: Add --grayscale-loss parameter support - doc/CNN_V2.md: Document grayscale loss in training configuration and checkpoint format - doc/HOWTO.md: Add usage examples for --grayscale-loss flag Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
32 hoursCNN v2: Change feature #6 from sin(10*x) to sin(20*y)skal
Update positional encoding to use vertical coordinate at higher frequency. Changes: - train_cnn_v2.py: sin10_x → sin20_y (computed from uv_y) - cnn_v2_static.wgsl: sin10_x → sin20_y (computed from uv_y) - index.html: sin10_x → sin20_y (STATIC_SHADER) - CNN_V2.md: Update feature descriptions and examples - CNN_V2_BINARY_FORMAT.md: Update static features documentation Feature vector: [p0, p1, p2, p3, uv_x, uv_y, sin20_y, bias] Rationale: Higher frequency (20 vs 10) + vertical axis provides better spatial discrimination for position encoding. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
32 hoursCNN v2: Add TODO for flexible feature layout in binary format v3skal
Document future enhancement for arbitrary feature vector layouts. Proposed feature descriptor in binary format v3: - Specify feature types, sources, and ordering - Enable runtime experimentation without shader recompilation - Examples: [R,G,B,dx,dy,uv_x,bias] or [mip1.r,mip2.g,laplacian,uv_x,sin20_x,bias] Added TODOs in: - CNN_V2_BINARY_FORMAT.md: Detailed proposal with struct layout - CNN_V2.md: Future extensions section - train_cnn_v2.py: compute_static_features() docstring - cnn_v2_static.wgsl: Shader header comment - cnn_v2_effect.cc: Version check comment Current limitation: Hardcoded [p0,p1,p2,p3,uv_x,uv_y,sin10_x,bias] layout. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
32 hoursDoc: Update CNN v2 docs for binary format v2 and mip-level supportskal
Updated documentation to reflect binary format v2 with mip_level field. Changes: - CNN_V2_BINARY_FORMAT.md: Document v2 (20-byte header) with mip_level, v1 backward compat - CNN_V2_WEB_TOOL.md: Document auto-detection of mip_level, UI updates - CNN_V2.md: Update overview with mip-level feature, training pipeline Binary format v2: - Header: 20 bytes (was 16) - New field: mip_level (u32) at offset 0x10 - Backward compatible: v1 loaders treat as mip_level=0 Documentation complete for full mip-level pipeline integration. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
32 hoursCNN v2: Add --mip-level option for parametric featuresskal
Add mip level control for p0-p3 features (0=original, 1=half, 2=quarter, 3=eighth). Uses pyrDown/pyrUp for proper Gaussian filtering during mip generation. Changes: - compute_static_features(): Accept mip_level param, generate mip via cv2 pyramid - PatchDataset/ImagePairDataset: Pass mip_level to feature computation - CLI: Add --mip-level arg with choices [0,1,2,3] - Save mip_level in checkpoint config for tracking - Doc updates: HOWTO.md and CNN_V2.md Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
34 hoursCNN v2 web tool: Fix layer naming and visualization bugsskal
- Align layer naming with codebase: Layer 0/1/2 (not Layer 1/2/3) - Split static features: Static 0-3 (p0-p3) and Static 4-7 (uv,sin,bias) - Fix Layer 2 not appearing: removed isOutput filter from layerOutputs - Fix canvas context switching: force clear before recreation - Disable static buttons in weights mode - Add ASCII pipeline diagram to CNN_V2.md Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
36 hoursDoc: Clarify CNN v2 training uses RGBA targetsskal
Updated CNN_V2.md to document that: - Model outputs 4 channels (RGBA) - Training targets preserve alpha from target images - Loss function compares all 4 channels Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
36 hoursCNN v2: Restore per-layer kernel sizes supportskal
Training: - train_cnn_v2.py: Accept --kernel-sizes as comma-separated list - CNNv2 model: Per-layer kernel sizes (e.g., [1,3,5]) - Single value replicates across layers (e.g., "3" → [3,3,3]) Export: - export_cnn_v2_weights.py: Backward compatible with old checkpoints - Handles both kernel_size (old) and kernel_sizes (new) format Documentation: - CNN_V2.md: Updated code examples and config format - HOWTO.md: Updated training examples to show comma-separated syntax Binary format: Already supports per-layer kernel sizes (no changes) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
36 hoursCNN v2: Refactor to uniform 12D→4D architectureskal
**Architecture changes:** - Static features (8D): p0-p3 (parametric) + uv_x, uv_y, sin(10×uv_x), bias - Input RGBD (4D): fed separately to all layers - All layers: uniform 12D→4D (4 prev/input + 8 static → 4 output) - Bias integrated in static features (bias=False in PyTorch) **Weight calculations:** - 3 layers × (12 × 3×3 × 4) = 1296 weights - f16: 2.6 KB (vs old variable arch: ~6.4 KB) **Updated files:** *Training (Python):* - train_cnn_v2.py: Uniform model, takes input_rgbd + static_features - export_cnn_v2_weights.py: Binary export for storage buffers - export_cnn_v2_shader.py: Per-layer shader export (debugging) *Shaders (WGSL):* - cnn_v2_static.wgsl: p0-p3 parametric features (mips/gradients) - cnn_v2_compute.wgsl: 12D input, 4D output, vec4 packing *Tools:* - HTML tool (cnn_v2_test): Updated for 12D→4D, layer visualization *Docs:* - CNN_V2.md: Updated architecture, training, validation sections - HOWTO.md: Reference HTML tool for validation *Removed:* - validate_cnn_v2.sh: Obsolete (used CNN v1 tool) All code consistent with bias=False (bias in static features as 1.0). handoff(Claude): CNN v2 architecture finalized and documented
37 hoursCNN v2 Web Tool: Unify layer terminology and add binary format specskal
- Rename 'Static (L0)' → 'Static' (clearer, less confusing) - Update channel labels: 'R/G/B/D' → 'Ch0 (R)/Ch1 (G)/Ch2 (B)/Ch3 (D)' - Add 'Layer' prefix in weights table for consistency - Document layer indexing: Static + Layer 1,2,3... (UI) ↔ weights.layers[0,1,2...] - Add explanatory notes about 7D input and 4-of-8 channel display - Create doc/CNN_V2_BINARY_FORMAT.md with complete .bin specification - Cross-reference spec in CNN_V2.md and CNN_V2_WEB_TOOL.md Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2 daystest_demo: Add beat-synchronized CNN post-processing with version selectionskal
- Add --cnn-version <1|2> flag to select between CNN v1 and v2 - Implement beat_phase modulation for dynamic blend in both CNN effects - Fix CNN v2 per-layer uniform buffer sharing (each layer needs own buffer) - Fix CNN v2 y-axis orientation to match render pass convention - Add Scene1Effect as base visual layer to test_demo timeline - Reorganize CNN v2 shaders into cnn_v2/ subdirectory - Update asset paths and documentation for new shader organization Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
3 daysUpdate docs and help messages for CNN v2 completionskal
Updated: - HOWTO.md: Complete pipeline, storage buffer, --validate mode - TODO.md: Mark CNN v2 complete, add QAT TODO - PROJECT_CONTEXT.md: Update Effects status - CNN_V2.md: Mark complete, add storage buffer notes - train_cnn_v2_full.sh: Add --help message All documentation now reflects: - Storage buffer architecture - Binary weight format - Live training progress - Validation-only mode - 8-bit quantization TODO
3 daysCNN v2: parametric static features - design docskal
Design document for CNN v2 with enhanced feature inputs: - 7D static features: RGBD + UV + sin encoding + bias - Per-layer configurable kernels (1×1, 3×3, 5×5) - Float16 weight storage (~6.4 KB vs 3.2 KB) - Multi-pass architecture with static feature compute Implementation plan: 1. Static features compute shader (RGBD + UV + sin + bias) 2. C++ effect class (CNNv2Effect) 3. Training pipeline (train_cnn_v2.py, export_cnn_v2_shader.py) 4. Validation tooling (validate_cnn_v2.sh) Files: - doc/CNN_V2.md: Complete technical design (architecture, training, export) - scripts/validate_cnn_v2.sh: End-to-end validation script - TODO.md: Add CNN v2 as Priority 2 task - doc/HOWTO.md: Add CNN v2 validation usage Target: <10 KB for 64k demo constraint handoff(Claude): CNN v2 design ready for implementation
generated by cgit v1.2.3 (git 2.51.0) at 2026-02-14 23:41:25 +0000