diff options
Diffstat (limited to 'cnn_v2/tools/cnn_v2_test')
| -rw-r--r-- | cnn_v2/tools/cnn_v2_test/README.md | 251 | ||||
| -rw-r--r-- | cnn_v2/tools/cnn_v2_test/index.html | 2014 |
2 files changed, 2265 insertions, 0 deletions
diff --git a/cnn_v2/tools/cnn_v2_test/README.md b/cnn_v2/tools/cnn_v2_test/README.md new file mode 100644 index 0000000..d41a00f --- /dev/null +++ b/cnn_v2/tools/cnn_v2_test/README.md @@ -0,0 +1,251 @@ +# CNN v2 Testing Tool + +WebGPU-based browser tool for testing trained CNN v2 weights. + +--- + +## Features + +- Drag-drop PNG images and `.bin` weights (or click to browse) +- Real-time CNN inference with WebGPU compute shaders +- View modes: CNN output, original input, difference (×10) +- Adjustable blend amount and depth +- Data-driven pipeline (supports variable layer count) +- GPU timing display +- **Left Panel:** Weights info + kernel visualization (1px/weight, all layers) +- **Right Panel:** Layer activation viewer with 4-channel split + 4× zoom + +--- + +## Requirements + +- Browser with WebGPU support: + - Chrome/Edge 113+ (enable `chrome://flags/#enable-unsafe-webgpu` if needed) + - Safari 18+ (macOS Ventura+) +- Trained CNN v2 weights in binary format (`.bin`) +- Test images (PNG format) + +--- + +## Usage + +### 1. Open Tool + +```bash +open tools/cnn_v2_test/index.html +``` + +Or use a local server to avoid CORS: +```bash +python3 -m http.server 8000 +# Open http://localhost:8000/tools/cnn_v2_test/ +``` + +### 2. Load Data + +1. **Drop `.bin` weights** into left sidebar zone (or click to browse) +2. **Drop PNG image** anywhere in center canvas area +3. CNN runs automatically when both loaded + +### 3. Layout + +**Left Sidebar:** +- Weights drop zone (click or drag-drop `.bin` files) +- Weights info panel (layer specs, ranges, file size) +- Weights visualization (click Layer 0/1/2 buttons) + - 1 pixel per weight, all input channels horizontally + - Output channels (Out 0-3) stacked vertically + +**Center Canvas:** +- Main output view (CNN result, original, or diff) +- Keyboard: `SPACE` = original, `D` = diff (×10) + +**Right Sidebar:** +- Layer selection buttons (Static 0-3/4-7, Layer 0/1/2) +- 4 small activation views (Ch0/1/2/3) in a row +- Large zoom view below (4× magnification, follows mouse) + +**Header Controls:** +- **Blend:** Mix between original (0.0) and CNN output (1.0) +- **Depth:** Uniform depth value for all pixels (0.0–1.0) +- **View:** Current display mode + +**Footer:** +- Status: GPU timing (ms), image dimensions, view mode +- Console: Timestamped event log (file loads, errors) + +--- + +## Preparing Test Data + +### Export Weights + +```bash +# From trained checkpoint +./training/export_cnn_v2_weights.py \ + checkpoints/checkpoint_epoch_100.pth \ + --output-weights tools/cnn_v2_test/test_weights.bin +``` + +Binary format: 16-byte header + 20 bytes per layer + f16 weights (~3.2 KB for 3-layer model) + +### Test Images + +Use training images or any PNG: +```bash +# Copy test image +cp training/input/test.png tools/cnn_v2_test/ +``` + +**Note:** Grayscale images automatically converted to RGB. + +--- + +## Validation + +### Visual Comparison + +Compare browser output with C++ tool: + +```bash +# Generate C++ output +./build/cnn_test training/input/test.png /tmp/cpp_output.png + +# Load same image in browser tool +# Visually compare outputs +``` + +### GPU Timing + +Expected performance: +- 512×512: ~1-2 ms (integrated GPU) +- 1024×1024: ~3-5 ms +- 1920×1080: ~5-8 ms + +Slower than expected? Check: +- WebGPU enabled in browser +- Dedicated GPU selected (if available) +- No background tabs consuming GPU + +--- + +## Troubleshooting + +### "WebGPU not supported" + +- Update browser to latest version +- Enable WebGPU flag: `chrome://flags/#enable-unsafe-webgpu` +- Try Safari 18+ (native WebGPU on macOS) + +### "Invalid .bin file" + +- Check magic number: `hexdump -C weights.bin | head` +- Should start with: `43 4e 4e 32` ('CNN2') +- Re-export weights: `./training/export_cnn_v2_weights.py` + +### Black output / incorrect colors + +- Check blend slider (set to 1.0 for full CNN output) +- Verify training converged (loss < 0.01) +- Compare with C++ tool output + +### Shader compilation errors + +Open browser console (F12) for detailed errors. Common issues: +- Image too large (>4096×4096 not tested) +- Unsupported texture format (rare on modern GPUs) + +--- + +## Architecture + +**Pipeline:** +1. **Static Features Pass** - Generate 8D features (RGBD, UV, sin, bias) +2. **CNN Layer Passes** - Compute N layers with ping-pong textures +3. **Display Pass** - Unpack and render with view mode + +**Textures:** +- Input: RGBA8 (original image) +- Depth: R32F (uniform depth) +- Static features: RGBA32Uint (8×f16 packed) +- Layer buffers: RGBA32Uint (ping-pong) + +**Data-Driven Execution:** +- Layer count read from binary header +- Per-layer params (kernel size, channels, offsets) from binary +- Single CNN shader dispatched N times + +--- + +## Implemented Features + +**✓ Weights Metadata Panel:** +- Layer descriptions (kernel size, channels, weight count) +- Weight statistics (min/max per layer) +- File size and layer count + +**✓ Weights Visualization:** +- Per-layer kernel heatmaps (1px/weight) +- All input channels displayed horizontally +- Output channels stacked vertically +- Normalized grayscale display + +**✓ Layer Activation Viewer:** +- Static features (8D split into 0-3 and 4-7 views) +- All CNN layer outputs (Layer 0/1/2...) +- 4-channel split view (grayscale per channel) +- Mouse-driven 4× zoom view + +## TODO + +**Future Enhancements:** +- Weight distribution histograms per layer +- Activation statistics (min/max/mean overlay) +- Side-by-side diff mode (browser vs C++ output) +- Export rendered layers as PNG + +--- + +## Extensions (v2+) + +Planned enhancements: + +**Variable Feature Count:** +- Binary v2: Add `num_features` to header +- Shader: Dynamic feature array or multiple textures + +**Multi-Scale Input (Mip Levels):** +- Uncomment mip bindings in static shader +- No binary format change needed + +**8-bit Quantized Weights:** +- Binary version bump (format field already present) +- Add quantization codepath in `get_weight()` function +- 2× size reduction (~1.6 KB) + +**Pre-defined Test Images:** +- Dropdown menu with training/input/*.png +- Requires local file server + +--- + +## Size + +- HTML structure: ~2 KB +- CSS styling: ~2 KB +- JavaScript logic: ~10 KB (includes zoom + weights viz) +- Static shader: ~1 KB +- CNN shader: ~3 KB +- Display shader: ~1 KB +- Layer viz shader: ~2 KB +- Zoom shader: ~1 KB +- **Total: ~22 KB** (single file, no dependencies) + +--- + +## See Also + +- `doc/CNN_V2.md` - Architecture and design +- `doc/HOWTO.md` - Training workflows +- `training/export_cnn_v2_weights.py` - Binary format +- `src/effects/cnn_v2_effect.cc` - C++ reference implementation diff --git a/cnn_v2/tools/cnn_v2_test/index.html b/cnn_v2/tools/cnn_v2_test/index.html new file mode 100644 index 0000000..84702d5 --- /dev/null +++ b/cnn_v2/tools/cnn_v2_test/index.html @@ -0,0 +1,2014 @@ +<!DOCTYPE html> +<html lang="en"> +<!-- + CNN v2 Testing Tool - WebGPU-based inference validator + + Architecture: + - Static features (8D): p0-p3 (parametric), uv_x, uv_y, sin(10*uv_x), bias (NOT a CNN layer) + - Layer 0: input RGBD (4D) + static (8D) = 12D → 4 channels + - Layer 1+: previous layer (4D) + static (8D) = 12D → 4 channels + - All CNN layers: uniform 12D input, 4D output (ping-pong buffer) + + Naming convention (matches train_cnn_v2.py / .wgsl / .cc): + - UI shows: "Static 0-3", "Static 4-7", "Layer 0", "Layer 1", "Layer 2" + - weights.layers[] array: Layer 0 = weights.layers[0], Layer 1 = weights.layers[1] + + Features: + - Input: PNG images or video files (MP4, WebM, etc.) + - Video playback: Play/Pause, frame-by-frame navigation (◄/► buttons) + - Video mode: Non-realtime processing (drops frames if CNN slower than playback) + - Side panel: .bin metadata display, weight statistics per layer + - Layer inspection: 4-channel grayscale split, intermediate layer visualization + - View modes: CNN output, original, diff (×10) + - Optimization: Layer viz updates only on pause/seek during video playback + + WGSL Shader Reuse: + - CNN_SHADER (inference), STATIC_SHADER, LAYER_VIZ_SHADER are inline for single-file deployment + - Can extract to .wgsl files for: better IDE support, testing, cross-tool reuse + - Tradeoff: extraction needs fetch() or build step, breaks single-file portability + - C++ sync: manual (WGSL ≠ GLSL) but logic identical +--> +<head> + <meta charset="UTF-8"> + <meta name="viewport" content="width=device-width, initial-scale=1.0"> + <title>CNN v2 Testing Tool</title> + <link rel="stylesheet" href="../common/style.css"> + <style> + body { + display: flex; + flex-direction: column; + height: 100vh; + } + .header { + padding: 16px; + border-bottom: 1px solid #404040; + gap: 24px; + } + h1 { font-size: 18px; } + .controls { + gap: 16px; + } + .control-group { + display: flex; + gap: 8px; + align-items: center; + } + .control-group label { font-size: 12px; } + input[type="range"] { width: 120px; } + input[type="number"] { width: 60px; padding: 4px; } + .drop-zone { + border: 3px dashed #606060; + padding: 20px; + text-align: center; + cursor: pointer; + transition: all 0.2s; + font-size: 13px; + font-weight: bold; + background: #252525; + border-radius: 6px; + color: #4a9eff; + } + button { + padding: 6px 12px; + font-size: 12px; + } + button:hover { border-color: #606060; background: #252525; } + video { display: none; } + .drop-zone:hover { border-color: #4a9eff; background: #2a3545; } + .drop-zone.active { border-color: #4a9eff; background: #1a2a3a; } + .drop-zone.error { border-color: #ff4a4a; background: #3a1a1a; } + .content { + flex: 1; + display: flex; + overflow: hidden; + gap: 1px; + background: #404040; + } + .left-sidebar { + width: 315px; + background: #2a2a2a; + overflow-y: auto; + display: flex; + flex-direction: column; + gap: 16px; + padding: 16px; + } + .main { + flex: 1; + display: flex; + justify-content: center; + align-items: center; + padding: 24px; + overflow: auto; + position: relative; + } + .video-controls-float { + position: absolute; + top: 16px; + left: 50%; + transform: translateX(-50%); + display: flex; + gap: 8px; + background: rgba(42, 42, 42, 0.95); + padding: 8px 12px; + border-radius: 4px; + border: 1px solid #404040; + z-index: 100; + } + .bottom-controls-float { + position: absolute; + bottom: 16px; + left: 50%; + transform: translateX(-50%); + display: flex; + gap: 16px; + align-items: center; + background: rgba(42, 42, 42, 0.95); + padding: 8px 16px; + border-radius: 4px; + border: 1px solid #404040; + z-index: 100; + } + .bottom-controls-float .control-group { + display: flex; + gap: 8px; + align-items: center; + } + .bottom-controls-float #videoControls { + display: flex; + gap: 8px; + align-items: center; + padding-right: 16px; + border-right: 1px solid #404040; + } + .main.drop-active::after { + content: 'Drop PNG/video here'; + position: absolute; + inset: 24px; + display: flex; + align-items: center; + justify-content: center; + border: 3px dashed #4a9eff; + background: rgba(74, 158, 255, 0.1); + font-size: 24px; + color: #4a9eff; + pointer-events: none; + z-index: 10; + } + .sidebar { + width: 400px; + background: #2a2a2a; + overflow-y: auto; + display: flex; + flex-direction: column; + gap: 16px; + padding: 16px; + } + .panel { + border-radius: 4px; + overflow: hidden; + } + .panel.collapsed .panel-content { + display: none; + } + .panel-header { + background: #1a1a1a; + padding: 8px 12px; + font-size: 12px; + font-weight: bold; + border-bottom: 1px solid #404040; + } + .panel-content { + padding: 12px; + font-size: 11px; + } + .panel-content table { + width: 100%; + border-collapse: collapse; + } + .panel-content th { + text-align: left; + padding: 4px; + font-size: 10px; + color: #808080; + border-bottom: 1px solid #404040; + } + .panel-content td { + padding: 4px; + font-size: 10px; + } + .panel-content tr:hover { + background: #1a1a1a; + } + .layer-buttons { + display: flex; + flex-wrap: wrap; + gap: 6px; + margin-bottom: 12px; + } + .layer-buttons button { + padding: 6px 12px; + font-size: 10px; + } + .layer-buttons button.active { + background: #4a9eff; + border-color: #4a9eff; + color: #1a1a1a; + } + .layer-buttons button:disabled:hover { + border-color: #404040; + background: #1a1a1a; + } + .layer-grid { + display: grid; + grid-template-columns: repeat(4, 1fr); + gap: 4px; + margin-bottom: 12px; + } + .layer-view { + aspect-ratio: 1; + background: #1a1a1a; + border: 1px solid #404040; + display: flex; + flex-direction: column; + overflow: hidden; + } + .layer-preview { + background: #1a1a1a; + border: 1px solid #404040; + display: flex; + flex-direction: column; + overflow: hidden; + margin-top: 8px; + } + .layer-preview canvas { + width: 100%; + height: 100%; + image-rendering: pixelated; + } + .layer-view.active { + border: 2px solid #ffffff; + } + .layer-view canvas { + cursor: pointer; + } + .layer-view-label { + background: #2a2a2a; + padding: 4px; + font-size: 9px; + text-align: center; + border-bottom: 1px solid #404040; + } + .layer-view canvas { + width: 100%; + height: 100%; + image-rendering: pixelated; + } + canvas { + max-width: 100%; + max-height: 100%; + image-rendering: pixelated; + box-shadow: 0 4px 12px rgba(0,0,0,0.5); + } + .footer { + background: #2a2a2a; + border-top: 1px solid #404040; + font-size: 11px; + display: flex; + flex-direction: column; + gap: 8px; + } + .footer-top { + padding: 12px 16px 0; + display: flex; + justify-content: space-between; + } + .status { color: #4a9eff; } + .shortcuts { color: #808080; } + .console { + background: #1a1a1a; + padding: 8px 16px; + font-family: 'Courier New', monospace; + font-size: 10px; + color: #808080; + max-height: 100px; + overflow-y: auto; + border-top: 1px solid #404040; + } + .console-line { margin: 2px 0; } + .console-line.error { color: #ff4a4a; } + .console-line.info { color: #4a9eff; } + </style> +</head> +<body> + <div class="header"> + <h1>CNN v2 Testing Tool</h1> + </div> + <video id="videoSource" muted loop></video> + <div class="content"> + <div class="left-sidebar"> + <input type="file" id="weightsFile" accept=".bin" style="display: none;"> + <div class="drop-zone" id="weightsDrop" onclick="document.getElementById('weightsFile').click()"> + Drop .bin Weights or Click to Browse + </div> + <div class="panel" id="weightsInfoPanel"> + <div class="panel-header">Weights Info</div> + <div class="panel-content" id="weightsInfo"> + <p style="color: #808080; text-align: center;">No weights loaded</p> + </div> + </div> + <div class="panel" id="weightsVizPanel" style="display: none;"> + <div class="panel-header">Weights Visualization</div> + <div class="panel-content" id="weightsViz"> + <div class="layer-buttons" id="weightsLayerButtons"></div> + <canvas id="weightsCanvas" style="width: 100%; image-rendering: pixelated; border: 1px solid #404040;"></canvas> + </div> + </div> + <div class="panel"> + <div class="panel-content"> + <label for="mipLevel" style="font-size: 11px;">Mip Level:</label> + <select id="mipLevel" style="width: 100%; background: #1a1a1a; color: #e0e0e0; border: 1px solid #404040; padding: 4px; margin-top: 4px;"> + <option value="0">Mip 0 (original)</option> + <option value="1">Mip 1 (half res)</option> + <option value="2">Mip 2 (quarter res)</option> + </select> + </div> + </div> + </div> + <div class="main" id="mainDrop"> + <div class="bottom-controls-float"> + <div id="videoControls"> + <button id="playPauseBtn" disabled>Play</button> + <button id="stepBackBtn" disabled>◄ Frame</button> + <button id="stepForwardBtn" disabled>Frame ►</button> + </div> + <div class="control-group"> + <label>Blend:</label> + <input type="range" id="blend" min="0" max="1" step="0.01" value="1.0"> + <span id="blendValue">1.0</span> + </div> + <div class="control-group"> + <label>Depth:</label> + <input type="range" id="depth" min="0" max="1" step="0.01" value="1.0"> + <span id="depthValue">1.0</span> + </div> + <button id="savePngBtn">Save PNG</button> + </div> + <canvas id="canvas"></canvas> + </div> + <div class="sidebar"> + <div class="panel" style="flex: 1; display: flex; flex-direction: column; min-height: 0;"> + <div class="panel-header">Layer Visualization</div> + <div class="panel-content" id="layerViz" style="flex: 1; overflow: hidden;"> + <p style="color: #808080; text-align: center;">Load image + weights</p> + </div> + </div> + </div> + </div> + <div class="footer"> + <div class="footer-top"> + <span class="status" id="status">Drop PNG/video anywhere to begin</span> + <span class="shortcuts">[SPACE] Original | [D] Diff (×10)</span> + </div> + <div class="console" id="console"></div> + </div> + + <script> +// ============================================================================ +// EMBEDDED WEIGHTS & CONSTANTS +// ============================================================================ + +// Default pre-trained weights (base64-encoded binary format) +// Version 2: 4 layers (3×3, 5×5, 3×3, 3×3), 2496 f16 weights, mip_level=2 +const DEFAULT_WEIGHTS_B64 = 'Q05OMgIAAAAEAAAAwAkAAAIAAAADAAAADAAAAAQAAAAAAAAAsAEAAAUAAAAMAAAABAAAALABAACwBAAAAwAAAAwAAAAEAAAAYAYAALABAAADAAAADAAAAAQAAAAQCAAAsAEAAAU3faplMDmtR7gnMLqt6bSrLM4RCa/En4q257kVsmWz57aSHJMxz6wILJC0tLdBriWww7IULUehCClCo60dBiu1nWqsf60ZKn6ktCWKjrswATSfLwQunzJjKKWkN6hxLTMwbS2DJvgvUjFDL1YsQDFFL78ysC5OL/cvxC2kJ6qh0i1BLH2rzCrcKFUoeixTqwwopjD+rXmewCY6sYUtXCwwsaKqGjBcqoykKigRJYStaqjMp+siPi1BLI+tGatfK5Ii6C1qLY0tYSGFKz4wpzNdH1QuJDKmMJi0lLVAs0y2Q7YWtY21fLXusf+n8LDSsaethK3drB4rtSROKYOrLK53qrqu0REYLEUuVy1qEqohDSzgqk4sDKKSKi0clKcVKvupJ69rKTmw8q7qptatQK7OsFUw5Z5JKJ4udSp9LLQeui87LbcxljEgJ6Iw75jDLfUvIjCxnh0g763Lq/ItMqzDqP0sXCRcqnkl9qDlJUStSyR8oTuwA616IrAnNqo5JS4qDKeILmahyaHZI48tryiajuEs0aghLBcuny+aovQpAhj6Kqkwdy+8MZ0wLzBvKBStsrRAKJez+raaKAotBiVSqZqyk7b2sHO1e7cJsfGmQLACpWizBLP9LnWxYLWoJPeb/CY5ISokXqynJ4qtG6K1qpesL6zGqYssIDJRpnErRi3RL9kh1zBFLPkdGSNvKtEuvyywmgilbC43LNovbywCKj4pFzEbMmMuly2gMFYscCgzliIomSqZnpSnyK3hJJKsAasgJGMrfCyNqXwpqaYNq14wiyzWLrSn/yLbqm+tnauOpkKtRKdCrBcYQS0dnGAveqeBrD8sMiGpLkAugzEaLM6lLzAkL5YydzYnqGo15zh2MuSwJK0nqxI04jZ5LAs2TjilNeSc3yANLecrCzBCprUvfjUHMWCuFrAkItyq/an0JSUnvKnrrAosv5CRrTGvQKesntuur6v2rsyxzbCAsHYn1y5GrAGsASYUmawrpSLooRSy86sBqmaxAq67sD0lJalOKxOtkqx8H+wqgygMLhup8SzNKZuhcafWKUKs567KI1opDCsoplatAykJpc+skavUrK4p2iznLlMqcig4Le6mDKiaJpIsMiOgLGOtQqI7sFGworKfsTOq86ZIlru0dLCEoMqq4KzsI6I2MzixMocqSym8MwQtT7Njqrwy26rEthe2nTGxL/Gq+az8MPg1Tq6EqXmslqyArkKs/S73MqEwmyuzrUUxejLhKYaw0yUlMzgxAZULsZ4rhq8ssgarCjDTrPop0ywBLswwjbT7MMAxdq2fsEC04DZoOIovG7G4LwM1gTNnKDsuEbByrzyxvLLBKJgkGDQANSMy66wVrM21ebURriAluK5quFa3wLBsK2wvaDU7OEg3RDGWKVUzpTfPNG+tbrGcr3ytRKosr7yuCbB2rV6gZq3msWmtjqvmoNurP6YXrOIpf6l/J2irl6/iqK2jy6MCLkkhjSDQoAWWACo1JrWjP6nvKvmthay+KJ6rUqoKqaatHKyJrUOarydBo5yu/CUaKFoxFCW1CNgpri2WK02kgqvYqkotwqlIrdiiEa1aKZ2tXa6mrkax4KkYKp2vcKgErYsi2RvbqWapU6EAnMyqtyPBpYwdZyVZkwGl1yhhJ2QBPaUJqMmMJJ54IikpcqmUHzmacCDzq1Cr3yR9n8aizKlWKFiogapBFlknrimnHmemDqbVKHciNRyII5AsxZ0+Lf0Xmyh7LMIqDS2KK9EkxyxRHKgp2iL9K0QfxCwGLLEuwiqrLcWob6xpppasp6+lotypGrC9qdmpPKUuplagES2cpSyrsSyHJTMi3Kk4KWAlSCaqKNMtR626rKaoj6koI1wqeivGI9cpuqQ9KQUkZyEJKOmquyW0JymirSjhprWgkBpKLFykzZyloWSrNKxrGaCtMi1MqL6t56lLqu+wbbTetYkqYDR1rB0wqir/sWQwNas8N9E4wq+9I6WwT6xuMDy1yC9tM/Kwka+btK8vJisnIJWeUa30LRkwDaqIsNqzWK9lLnEzKjEMqYMuWy8uMs0qI6xKLjcvxicEqYCv06zrrLusKK/lMeMz8CyCMmqxO7AtNpW38zFzL5i2Wq19tkCuBaTlt8Kv85Mlsg6wWLfgstutzDJVNAqZxCywrQgspDYOMS0mGbQCuf63QS7GJ4GsBLizuRS0mKyiKKMkBbLXseCufCr4qKUpah7Vqh8tV6eqLLQoGy1bMNEu6i4fMD4wZSvbjwOpmCBzLMmeJKddoYqkIic6qpqRY6nNqDiwIq5dqcmndqbnKnGkSCjmKBUsriySrHWsZyTaG7smSKxAIwolIi2zLX6unK5KqXCwKq03qyarcKWMqQmmd6tIodWtH6UvLg2tTadPJOOp2iGgny0ufyy+L7AvNClhpiEpC6qMqqMp7KTopJ4mmB2ylM6mrKhfKiQrTyiiKdGoQqjKJ6Umxip/qDiq/ChgKtmqIiwOr+CunZF7Kfot36poqkcthCx+Ksapg5T5pn0oNqOPq4osMSbSqQQmGqgXKhEl3yV1piyswazLK7QoQBTaqU8lIS13Ldch+qQqJ2AsPKfmp3Ink5Z2HhosR5z4qLIoGqkNLCct2Ck3KPGnUC0oJBQq7agOKyaq0qsqpAap8SylLg4qriy6M3MqKCtdKpMjSi86KigsGCz/n2erEyu7J/QRVCkpILUwcC35LI8qxiw6Knoq5jAAKo8wnieqLF0vVTAYMZw4Jyx2t/ayTjGWMoGzKbwus1w4QRxeJse1dTGSNJGwmCrEJV8uQKygKe4gjSqkrLeydiaMroS0FrQms8Uygi28qe2uXS2Ko4q1d7ZxszEpiDSBMoc0STWpNc0xJKSvrMWm6bCKsOC3CrEOJNC1Ga5Qubi7U6/+NRQ0AqnSuFoySDmKtJS0b7KcNAMmqi45IbMvGzjeMg2qSioPKVWtSK6EpaA1UTckMt2m16nwM5E2oDHBsZ+pniVpMc4vQy1epXkqHifBl7Mu36T/KzQorix4JAOmWyqJFVUqq67doiot2CxYME8i2JxVKhQt5ioYJsWp1KiSpL0lhq1JpWAgbCweKW2o1CrCIMsrcghkHUqW3hiTI5osYqMlB+WaLy0uKNUooKx4qdEezqRlJEapyKuUoEmoZyT7nqcoo6v3n4yqZaGcpNElwij3IkinQiAFIFQK2ygqIoKsiZxEI6ukqCf7KFSkgqSTqjEq8JZLJPufXKmFkaEj36lCKj2qURxfKkQouaqQhRIrGSmepKin7Cl8KEcuKI+ip4Evz6xIF0woVK/yHLyfLSj0ny+oWywSJHWmQaEomWos6ZTMpPWlY61pqLelZqYGpAidcyzQE5kneBr1pnQkJSwIqWYpIabdKA8oHKroGeCnYplOKzAmC51LJ0emp6o+rXAofCkCKV4w4x1sKCYjrKAgKa0r+BcPJDMmP6o2JW4pIqqtm4srTqgHlLWlsBBepaqrKq27rBat9aTlot8qkaw2o5sl76ivKDkjNyjzKKWY5KlHrQCr8SjxquarXqrlKB2xyyfZL1Sqq7LWpxA04zZwMkyvUiyHMig1ay+GJqenVq1Ao1awVLHQnrEqxTD/LO8kKB+NH1grfKsPsY6u+aIELLaj4LBmLBU0wDOlM8ksdKjbqPSqQykHJmYodC+WMcYuSCJ7psYvNDTaLqWw/qy7Myw4xjTnMIouQTV9OJ81YSlbLiIx3TVuMUcokrDzI0ow8CQQr9IvDyxsLnk0OTVhLmmobLAULN4zkyyZsGC0LK01L3Upw52Jroywlix0MCwr5qkQJkot9aWzsYuui66HrHykMa9ZsDet96yBqXWvXbAXsraxIqgpsVOvtq5frF+iZa2WqROwcaP+qX2w+aW3rxWpI7Bwrlqu5K0LrxexX7DUrfOvhK3QrUGwP7BrsY2tU6yWr8qkpK18rn2rHCbloYmfaqM1nfSr7Sn1qjuk2KT2qyem4KXJJ4MdxaidqPWsa58zKTSsoKXAJUymz6rJpv+oGKsOJo2hSicHqA4oOiiRmr4k0BxBq8Ui16jTKvyq7ijmqHcpZanhHnGfMikxIiEk7S4Yq90sfKWSoZyntKg/qh+nJiifnAyvlKeXJMIdViKeoxEjLKvZpXymAqkhraCofK5SnTGmLqdkq7mjYCD8qV0qQKo0qrUo+KsZKVSs0iaULFUI8qS0mlWtiiqbGBegACwBoAErhaW1qMwqHSxfKVKpp6x7poiweKxCrdkivK48sJewrKdArHYnqyhoHbUnsagYK58qSjAgMcUwsCt0K/4rLC7mJGwtvStOMFQu0SzuJQUsBTBMLswqcJyEnVQsESn3ox2z9ai/qFqwES7tKP0vSChMoqQwVzR4LKaT+y/NK06q2y0LIi2wHrIcKZuzsrSHn/6xkrPssAovJzEipEQiDbDjr3SqIis5LGIoOSm6p1apeqGGrtAqJzCIJRuptqrApiktWTAwMB4xQizXKoIgASFFsLwweTHbLdQtqyzXoKYtay3SLeOke6wgoPWr/SpFKUEmDacWptSoMChJKm6s6azkHe+mfzFKKyamfi6bK/wr5atPqEMxUTAlKSeueiRxoSQjQqxQLRavgauKriOssymXLZOooa97pFoufTSppqgoVq05tEg196yCsQIy7bEitAItJ7RgtUEzxjGML/QmEKIlrPgjPDFaoTYoPDFcJRavtK4XrKmsk6zjsCwsTa4UsPQs9jI/I3ct1C6cMV+b5y7wJZ0tYTF9MGojdS/oLTShziM/MVmnxC8FKJUwRCUxIz8wiS4QLWipLCCYq9EseabMKnEll6kPqIawRq+xGcgjyCkgqKed7SB6qZcr6CwJLW+st6ePq7WuHycUrhqsSq7zsKuZtimgCXCrmKkqnIGp4LHNsX2wnqyBsH2xIbDhpwCzra1ss44wTCypKDCyyK23LRiwYKKPMJmxcaqZKcshCCYipoyxNa1Nsbwozi1+MB8lQ5mtsDel3jDnlbutxiPzsWmp5SpTHaqys7EstauTPqoRsOosf6g3sLOgeaAfKUIsWi/BJdosUSzdMM4pSy3kpGM0DjWvLWw0cjR4MWWqQaYMLo2rZSijJjstZiFaLBadMq0TseyjYi0VGsQt8yo5oZCgti/HMLciM6r3KgMk8K6OqKup9q0srT0xcaWMMMwra67qrhSfsZ3GrrIj2a2+pqSvdrEcrRQ0IDhgMB+PCDWVM8qjnJ5ZKOmw4C0dMGyuG6DGMQUvrq+Oq4UsTSzHMRg2ibbXs+Axa7N5sAqqnSoerQUmky8oKIiuUjGsoBitdKy9q6iw661pqg4thKnpkYmt+a3gseypGp5Co22fM6YSKJap66hwopmsmqhlrCMkZyiLL4KnGKupKvUmyCQbLFUrbSZerKahlaRoqCYm5SqYKW0rcS8WrAUkzaMcGlqpRK3bnresXy18IXapEKqHKFssXKCpKMUrfamapf4tKjBiKJGoU54HK+8q5qq4qVuiZiy4JuEsTixNMFQnlSSIIw4k1KzxpbMlDqyKqz6gra4SpcOw3a3Vq+qqC6tOq22eORvnpC8hRadkka2q/K7HHUiowawpqPInLyA0qYMlsihUqGGkWCb7K1WdWK5Dr5EhnKv5KHKlXqYnJ/2l9i0YKUYuMzHxpyCs/ChMkPEtwanxoFQqJi3Uq7Mseq3arXskWKc5pOAc7CZcqCwc5w7qKO4f3iaKIDsq/KRgLpWsQqn5rYYkxCWPoU0bx6hzGdkkqibtofEoxy8GpUupSCTiKiwvpij7LbiulqkErXetejFkL2+upqtUp0OwiLAPsdCpxLIlrKOyQ7C2r3utIg0drZEl2y6oLkquoaX4rCysAa9GDRCwKrHDsNivAbHsqtioqiGvrqgJE66Kqw4rzKyDKgaomp6TK2EsDyc0oOSol6NZJkmsvyxorMss5pR0KBquEixPpjsgXCpsnXQocq2MrfGmoivvLBeacahmLROpe6kcGCSfdC03qL6i6yitHHohrxzqq4UiP6JMqF8qThOshWAVUqHupDsoohQuJSkv/ywqLiwlNjG7o++hxi3vIKmleCdyrH6wYatdsPWsjLCNol+sSTDpryCptbBDK+qs4zBpLGc0Nqc1rdo09jX5MqsrHi2xKOad8igwJxAoeSsiqgkqdChcLOYxJzGlMkAsUzCuKzskTjAOKhuplqjHqf8wzDKYIGefNDISqd8pIC23Ltwu7zC9KgMsQDL/JcgrryYzLJ0oTSoyqpkmLax+KuejVyqxr08ulZ2XpyQr5yxRsEMpwzD0KmEqoihRC6mwF6xOplwmjSSmpMep0SvhpOEndCluqLyvtCGgo3unOyy9IXKtmZ9yIK8hlqohrEUtxh0XKH0sGi18p6coHa3Tow6psqa/JRUMU6yiKbUoXigQpo2i7C18q3ur6CnWrSateC3/KY+jlCJ6o6qr+x8VJUkSFadyAgGpji0xraytBSd+rYksTqDAHQAtxSjkqMAmNqxhqNesEi5uKsqlFqo9Kg6seizOrdusAasErjmtoKv8rb8ph6cYLnMmcKlCLJ6pjiuIKpkpKK1UKvyq3RhVpZac+izlrYitWB+DrI4omKOZKikiZS1Fqicf+q25rJmsqKrYrNGt0JWRLWel2KfLqQ=='; + +// Reusable fullscreen quad vertex shader (2 triangles covering NDC) +const FULLSCREEN_QUAD_VS = ` +@vertex +fn vs_main(@builtin(vertex_index) idx: u32) -> @builtin(position) vec4<f32> { + var pos = array<vec2<f32>, 6>( + vec2<f32>(-1.0, -1.0), vec2<f32>(1.0, -1.0), vec2<f32>(-1.0, 1.0), + vec2<f32>(-1.0, 1.0), vec2<f32>(1.0, -1.0), vec2<f32>(1.0, 1.0) + ); + return vec4<f32>(pos[idx], 0.0, 1.0); +}`; + +// ============================================================================ +// WGSL SHADERS +// ============================================================================ + +// Static features: 7D parametric features (RGBD + UV + sin(10*uv_x) + bias) +const STATIC_SHADER = ` +@group(0) @binding(0) var input_tex: texture_2d<f32>; +@group(0) @binding(1) var linear_sampler: sampler; +@group(0) @binding(2) var depth_tex: texture_2d<f32>; +@group(0) @binding(3) var output_tex: texture_storage_2d<rgba32uint, write>; +@group(0) @binding(4) var<uniform> mip_level: u32; + +@compute @workgroup_size(8, 8) +fn main(@builtin(global_invocation_id) id: vec3<u32>) { + let coord = vec2<i32>(id.xy); + let dims = textureDimensions(input_tex); + if (coord.x >= i32(dims.x) || coord.y >= i32(dims.y)) { return; } + + // Use normalized UV coords with linear sampler (bilinear filtering) + let uv = (vec2<f32>(coord) + 0.5) / vec2<f32>(dims); + let rgba = textureSampleLevel(input_tex, linear_sampler, uv, f32(mip_level)); + + let p0 = rgba.r; + let p1 = rgba.g; + let p2 = rgba.b; + let p3 = textureLoad(depth_tex, coord, 0).r; + + let uv_x = f32(coord.x) / f32(dims.x); + let uv_y = f32(coord.y) / f32(dims.y); + let sin20_y = sin(20.0 * uv_y); + let bias = 1.0; + + let packed = vec4<u32>( + pack2x16float(vec2<f32>(p0, p1)), + pack2x16float(vec2<f32>(p2, p3)), + pack2x16float(vec2<f32>(uv_x, uv_y)), + pack2x16float(vec2<f32>(sin20_y, bias)) + ); + textureStore(output_tex, coord, packed); +}`; + +const CNN_SHADER = ` +struct LayerParams { + kernel_size: u32, + in_channels: u32, + out_channels: u32, + weight_offset: u32, + is_output_layer: u32, + blend_amount: f32, + is_layer_0: u32, +} + +@group(0) @binding(0) var static_features: texture_2d<u32>; +@group(0) @binding(1) var layer_input: texture_2d<u32>; +@group(0) @binding(2) var output_tex: texture_storage_2d<rgba32uint, write>; +@group(0) @binding(3) var<storage, read> weights_buffer: array<u32>; +@group(0) @binding(4) var<uniform> params: LayerParams; +@group(0) @binding(5) var original_input: texture_2d<f32>; + +fn unpack_static_features(coord: vec2<i32>) -> array<f32, 8> { + let packed = textureLoad(static_features, coord, 0); + let v0 = unpack2x16float(packed.x); + let v1 = unpack2x16float(packed.y); + let v2 = unpack2x16float(packed.z); + let v3 = unpack2x16float(packed.w); + return array<f32, 8>(v0.x, v0.y, v1.x, v1.y, v2.x, v2.y, v3.x, v3.y); +} + +fn unpack_layer_channels(coord: vec2<i32>) -> vec4<f32> { + let packed = textureLoad(layer_input, coord, 0); + let v0 = unpack2x16float(packed.x); + let v1 = unpack2x16float(packed.y); + return vec4<f32>(v0.x, v0.y, v1.x, v1.y); +} + +fn pack_channels(values: vec4<f32>) -> vec4<u32> { + return vec4<u32>( + pack2x16float(vec2<f32>(values.x, values.y)), + pack2x16float(vec2<f32>(values.z, values.w)), + 0u, + 0u + ); +} + +fn get_weight(idx: u32) -> f32 { + let pair_idx = idx / 2u; + let packed = weights_buffer[pair_idx]; + let unpacked = unpack2x16float(packed); + return select(unpacked.y, unpacked.x, (idx & 1u) == 0u); +} + +@compute @workgroup_size(8, 8) +fn main(@builtin(global_invocation_id) id: vec3<u32>) { + let coord = vec2<i32>(id.xy); + let dims = textureDimensions(static_features); + if (coord.x >= i32(dims.x) || coord.y >= i32(dims.y)) { return; } + + let kernel_size = params.kernel_size; + let in_channels = params.in_channels; // Always 12 (4 prev + 8 static) + let out_channels = params.out_channels; // Always 4 + let weight_offset = params.weight_offset; + let is_output = params.is_output_layer != 0u; + let kernel_radius = i32(kernel_size / 2u); + + let static_feat = unpack_static_features(coord); + + var output: vec4<f32> = vec4<f32>(0.0); + for (var c: u32 = 0u; c < 4u; c++) { + var sum: f32 = 0.0; + for (var ky: i32 = -kernel_radius; ky <= kernel_radius; ky++) { + for (var kx: i32 = -kernel_radius; kx <= kernel_radius; kx++) { + let sample_coord = coord + vec2<i32>(kx, ky); + let clamped = vec2<i32>( + clamp(sample_coord.x, 0, i32(dims.x) - 1), + clamp(sample_coord.y, 0, i32(dims.y) - 1) + ); + let static_local = unpack_static_features(clamped); + let layer_local = unpack_layer_channels(clamped); + + let ky_idx = u32(ky + kernel_radius); + let kx_idx = u32(kx + kernel_radius); + let spatial_idx = ky_idx * kernel_size + kx_idx; + + // Previous layer channels (4D) + for (var i: u32 = 0u; i < 4u; i++) { + let w_idx = weight_offset + + c * in_channels * kernel_size * kernel_size + + i * kernel_size * kernel_size + spatial_idx; + sum += get_weight(w_idx) * layer_local[i]; + } + + // Static features (8D) + for (var i: u32 = 0u; i < 8u; i++) { + let w_idx = weight_offset + + c * in_channels * kernel_size * kernel_size + + (4u + i) * kernel_size * kernel_size + spatial_idx; + sum += get_weight(w_idx) * static_local[i]; + } + } + } + + if (is_output || params.is_layer_0 != 0u) { + output[c] = 1.0 / (1.0 + exp(-sum)); // Sigmoid [0,1] + } else { + output[c] = max(0.0, sum); // ReLU + } + } + + if (is_output) { + let original = textureLoad(original_input, coord, 0).rgb; + let result_rgb = vec3<f32>(output.x, output.y, output.z); + let blended = mix(original, result_rgb, params.blend_amount); + output.x = blended.r; + output.y = blended.g; + output.z = blended.b; + } + + textureStore(output_tex, coord, pack_channels(output)); +}`; + +const DISPLAY_SHADER = ` +@group(0) @binding(0) var result_tex: texture_2d<u32>; +@group(0) @binding(1) var original_tex: texture_2d<f32>; +@group(0) @binding(2) var<uniform> mode: u32; + +@vertex +fn vs_main(@builtin(vertex_index) idx: u32) -> @builtin(position) vec4<f32> { + var pos = array<vec2<f32>, 6>( + vec2<f32>(-1.0, -1.0), vec2<f32>(1.0, -1.0), vec2<f32>(-1.0, 1.0), + vec2<f32>(-1.0, 1.0), vec2<f32>(1.0, -1.0), vec2<f32>(1.0, 1.0) + ); + return vec4<f32>(pos[idx], 0.0, 1.0); +} + +@fragment +fn fs_main(@builtin(position) pos: vec4<f32>) -> @location(0) vec4<f32> { + let coord = vec2<i32>(pos.xy); + let packed = textureLoad(result_tex, coord, 0); + let v0 = unpack2x16float(packed.x); + let v1 = unpack2x16float(packed.y); + let result = vec3<f32>(v0.x, v0.y, v1.x); + + if (mode == 0u) { + return vec4<f32>(result, 1.0); + } else if (mode == 1u) { + let original = textureLoad(original_tex, coord, 0).rgb; + return vec4<f32>(original, 1.0); + } else { + let original = textureLoad(original_tex, coord, 0).rgb; + let diff = abs(result - original) * 10.0; + return vec4<f32>(diff, 1.0); + } +}`; + +const LAYER_VIZ_SHADER = ` +@group(0) @binding(0) var layer_tex: texture_2d<u32>; +@group(0) @binding(1) var<uniform> viz_params: vec2<f32>; // x=channel_idx, y=scale + +@vertex +fn vs_main(@builtin(vertex_index) idx: u32) -> @builtin(position) vec4<f32> { + var pos = array<vec2<f32>, 6>( + vec2<f32>(-1.0, -1.0), vec2<f32>(1.0, -1.0), vec2<f32>(-1.0, 1.0), + vec2<f32>(-1.0, 1.0), vec2<f32>(1.0, -1.0), vec2<f32>(1.0, 1.0) + ); + return vec4<f32>(pos[idx], 0.0, 1.0); +} + +@fragment +fn fs_main(@builtin(position) pos: vec4<f32>) -> @location(0) vec4<f32> { + let coord = vec2<i32>(pos.xy); + let dims = textureDimensions(layer_tex); + + let channel = u32(viz_params.x); + + // DEBUG MODE 1: Texture coordinates (channel 10) + if (channel == 10u) { + let uv = vec2<f32>(f32(coord.x) / f32(dims.x), f32(coord.y) / f32(dims.y)); + return vec4<f32>(uv.x, uv.y, 0.0, 1.0); + } + + let packed = textureLoad(layer_tex, coord, 0); + + // DEBUG MODE 2: Raw packed data (channel 11) + if (channel == 11u) { + let raw_val = f32(packed.x) / 4294967295.0; + return vec4<f32>(raw_val, raw_val, raw_val, 1.0); + } + + let v0 = unpack2x16float(packed.x); + let v1 = unpack2x16float(packed.y); + let v2 = unpack2x16float(packed.z); + let v3 = unpack2x16float(packed.w); + + // DEBUG MODE 3: First unpacked value (channel 12) + if (channel == 12u) { + return vec4<f32>(v0.x, v0.x, v0.x, 1.0); + } + + var channels: array<f32, 8>; + channels[0] = v0.x; + channels[1] = v0.y; + channels[2] = v1.x; + channels[3] = v1.y; + channels[4] = v2.x; + channels[5] = v2.y; + channels[6] = v3.x; + channels[7] = v3.y; + + let scale = viz_params.y; + + let idx = min(channel, 7u); + let raw = channels[idx]; + + // Apply scale: multiply and clamp to [0, 1] + let val = clamp(raw * scale, 0.0, 1.0); + + return vec4<f32>(val, val, val, 1.0); +}`; + +class CNNTester { + constructor() { + this.canvas = document.getElementById('canvas'); + this.status = document.getElementById('status'); + this.console = document.getElementById('console'); + this.image = null; + this.video = document.getElementById('videoSource'); + this.weights = null; + this.viewMode = 0; + this.blendAmount = 1.0; + this.depth = 1.0; + this.currentLayerIdx = null; + this.currentChannelOffset = null; + this.isVideo = false; + this.fps = 30; + this.isProcessing = false; + this.mipLevel = 0; + this.selectedChannel = 0; + this.init(); + } + + log(msg, type = 'info') { + const line = document.createElement('div'); + line.className = `console-line ${type}`; + line.textContent = `[${new Date().toLocaleTimeString()}] ${msg}`; + this.console.appendChild(line); + this.console.scrollTop = this.console.scrollHeight; + } + + async init() { + if (!navigator.gpu) { + this.setStatus('WebGPU not supported', true); + this.log('WebGPU not supported in this browser', 'error'); + return; + } + + try { + this.adapter = await navigator.gpu.requestAdapter(); + this.device = await this.adapter.requestDevice(); + this.context = this.canvas.getContext('webgpu'); + this.format = navigator.gpu.getPreferredCanvasFormat(); + this.log('WebGPU initialized successfully'); + } catch (e) { + this.setStatus(`GPU init failed: ${e.message}`, true); + this.log(`GPU initialization failed: ${e.message}`, 'error'); + } + } + + setStatus(msg, isError = false) { + this.status.textContent = msg; + this.status.style.color = isError ? '#ff4a4a' : '#4a9eff'; + } + + // Get current source dimensions (video or image) + getDimensions() { + if (this.isVideo) { + return { width: this.video.videoWidth, height: this.video.videoHeight }; + } + return { width: this.image.width, height: this.image.height }; + } + + // Enable/disable video playback controls + setVideoControlsEnabled(enabled) { + ['playPauseBtn', 'stepBackBtn', 'stepForwardBtn'].forEach(id => + document.getElementById(id).disabled = !enabled + ); + } + + parseWeights(buffer) { + const view = new DataView(buffer); + const magic = view.getUint32(0, true); + if (magic !== 0x32_4E_4E_43) { + throw new Error('Invalid .bin file (bad magic)'); + } + + const version = view.getUint32(4, true); + const numLayers = view.getUint32(8, true); + const totalWeights = view.getUint32(12, true); + + // Version 2: added mip_level field (20-byte header) + let mipLevel = 0; + let headerSize = 16; + if (version === 2) { + mipLevel = view.getUint32(16, true); + headerSize = 20; + this.log(`Binary header: version=${version}, layers=${numLayers}, weights=${totalWeights}, mip_level=${mipLevel}`); + } else if (version === 1) { + this.log(`Binary header: version=${version}, layers=${numLayers}, weights=${totalWeights}`); + } else { + throw new Error(`Unsupported binary version: ${version}`); + } + + const layers = []; + for (let i = 0; i < numLayers; i++) { + const offset = headerSize + i * 20; + const layer = { + kernelSize: view.getUint32(offset, true), + inChannels: view.getUint32(offset + 4, true), + outChannels: view.getUint32(offset + 8, true), + weightOffset: view.getUint32(offset + 12, true), + weightCount: view.getUint32(offset + 16, true), + }; + layers.push(layer); + this.log(` Layer ${i}: ${layer.inChannels}→${layer.outChannels}, kernel=${layer.kernelSize}×${layer.kernelSize}, weights=${layer.weightCount}`); + } + + const weightsOffset = headerSize + numLayers * 20; + const weights = new Uint32Array(buffer.slice(weightsOffset)); + + // Calculate min/max per layer + for (let i = 0; i < numLayers; i++) { + const layer = layers[i]; + let min = Infinity, max = -Infinity; + const startIdx = layer.weightOffset; + const endIdx = startIdx + layer.weightCount; + + for (let j = startIdx; j < endIdx; j++) { + const pairIdx = Math.floor(j / 2); + const packed = weights[pairIdx]; + const unpacked = this.unpackF16(packed); + const val = (j % 2 === 0) ? unpacked[0] : unpacked[1]; + min = Math.min(min, val); + max = Math.max(max, val); + } + + layer.min = min; + layer.max = max; + this.log(` Layer ${i} range: [${min.toFixed(4)}, ${max.toFixed(4)}]`); + } + + let nonZero = 0; + for (let i = 0; i < weights.length; i++) { + if (weights[i] !== 0) nonZero++; + } + this.log(` Weight buffer: ${weights.length} u32 (${nonZero} non-zero)`); + + return { version, layers, weights, mipLevel, fileSize: buffer.byteLength }; + } + + unpackF16(packed) { + const lo = packed & 0xFFFF; + const hi = (packed >> 16) & 0xFFFF; + const toFloat = (bits) => { + const sign = (bits >> 15) & 1; + const exp = (bits >> 10) & 0x1F; + const frac = bits & 0x3FF; + if (exp === 0) return (sign ? -1 : 1) * Math.pow(2, -14) * (frac / 1024); + if (exp === 31) return frac ? NaN : (sign ? -Infinity : Infinity); + return (sign ? -1 : 1) * Math.pow(2, exp - 15) * (1 + frac / 1024); + }; + return [toFloat(lo), toFloat(hi)]; + } + + async loadImage(file) { + const img = await createImageBitmap(file); + this.image = img; + this.isVideo = false; + this.canvas.width = img.width; + this.canvas.height = img.height; + this.setVideoControlsEnabled(false); + this.log(`Loaded image: ${file.name} (${img.width}×${img.height})`); + if (this.weights) { + this.setStatus(`Ready: ${img.width}×${img.height}`); + this.run(); + } else { + this.setStatus(`Image loaded (${img.width}×${img.height}) - drop .bin weights to process`); + this.displayOriginal(); + } + } + + // Video loading: wait for metadata, then first frame decode (readyState≥2) + async loadVideo(file) { + return new Promise((resolve, reject) => { + this.video.src = URL.createObjectURL(file); + + this.video.onloadedmetadata = () => { + const w = this.video.videoWidth; + const h = this.video.videoHeight; + if (w === 0 || h === 0) { + reject(new Error('Video has invalid dimensions')); + return; + } + + this.isVideo = true; + this.canvas.width = w; + this.canvas.height = h; + this.fps = 30; + this.log(`Loaded video: ${file.name} (${w}×${h}, ${this.video.duration.toFixed(1)}s)`); + this.setVideoControlsEnabled(true); + + // Set up event handlers + this.video.onpause = () => { document.getElementById('playPauseBtn').textContent = 'Play'; }; + this.video.onplay = () => { document.getElementById('playPauseBtn').textContent = 'Pause'; this.playbackLoop(); }; + + // Wait for first frame to be decoded before displaying + const displayFirstFrame = () => { + this.video.onseeked = () => { if (!this.isProcessing) this.processVideoFrame(); }; + if (this.video.readyState >= 2) { // HAVE_CURRENT_DATA or better + if (this.weights) { + this.setStatus(`Ready: ${w}×${h}`); + this.processVideoFrame().then(() => resolve()); + } else { + this.setStatus(`Video loaded - drop .bin weights to process`); + this.displayOriginal(); + resolve(); + } + } else { + setTimeout(displayFirstFrame, 50); // Poll until frame ready + } + }; + + this.video.onseeked = displayFirstFrame; + this.video.currentTime = 0; + }; + + this.video.onerror = () => reject(new Error('Failed to load video')); + }); + } + + // Video playback loop (non-realtime, drops frames if CNN slow) + playbackLoop() { + if (this.video.paused || this.video.ended) return; + if (!this.isProcessing) this.processVideoFrame(); + requestAnimationFrame(() => this.playbackLoop()); + } + + // Process current video frame through CNN pipeline + async processVideoFrame() { + if (!this.weights || this.isProcessing) return; + this.isProcessing = true; + await this.run(); + this.isProcessing = false; + } + + // Video controls + togglePlayPause() { + this.video.paused ? this.video.play() : this.video.pause(); + } + + stepFrame(direction) { + if (!this.isVideo) return; + this.video.pause(); + this.video.currentTime = Math.max(0, Math.min(this.video.duration, + this.video.currentTime + direction / this.fps)); + } + + async loadWeights(file) { + const buffer = await file.arrayBuffer(); + this.weights = this.parseWeights(buffer); + this.weightsBuffer = buffer; + this.mipLevel = this.weights.mipLevel; // Set mip level from binary format + this.log(`Loaded weights: ${file.name} (${this.weights.layers.length} layers, ${(buffer.byteLength/1024).toFixed(1)} KB)`); + + // Update UI dropdown to reflect loaded mip level + const mipLevelSelect = document.getElementById('mipLevel'); + if (mipLevelSelect) { + mipLevelSelect.value = this.mipLevel.toString(); + } + + this.updateWeightsPanel(); + if (this.image) { + this.setStatus(`Ready: ${this.image.width}×${this.image.height}`); + this.run(); + } else { + this.setStatus('Weights loaded - drop PNG image to process'); + } + } + + updateWeightsPanel() { + const panel = document.getElementById('weightsInfo'); + const { version, layers, mipLevel, fileSize } = this.weights; + + let html = ` + <div style="margin-bottom: 12px;"> + <div><strong>File Size:</strong> ${(fileSize / 1024).toFixed(2)} KB</div> + <div><strong>Version:</strong> ${version}</div> + <div><strong>CNN Layers:</strong> ${layers.length}</div> + <div><strong>Mip Level:</strong> ${mipLevel} (p0-p3 features)</div> + <div style="font-size: 9px; color: #808080; margin-top: 4px;">Static features (input) + ${layers.length} conv layers</div> + </div> + <table> + <thead> + <tr> + <th>Layer</th> + <th>Size</th> + <th>Weights</th> + <th>Min</th> + <th>Max</th> + </tr> + </thead> + <tbody> + `; + + // Display layers as "Layer 0", "Layer 1", etc. (matching codebase convention) + for (let i = 0; i < layers.length; i++) { + const l = layers[i]; + html += ` + <tr> + <td>Layer ${i}</td> + <td>${l.inChannels}→${l.outChannels} (${l.kernelSize}×${l.kernelSize})</td> + <td>${l.weightCount}</td> + <td>${l.min.toFixed(3)}</td> + <td>${l.max.toFixed(3)}</td> + </tr> + `; + } + + html += ` + </tbody> + </table> + `; + + panel.innerHTML = html; + + // Show weights visualization panel and create layer buttons + const weightsVizPanel = document.getElementById('weightsVizPanel'); + weightsVizPanel.style.display = 'block'; + + const weightsLayerButtons = document.getElementById('weightsLayerButtons'); + let buttonsHtml = ''; + for (let i = 0; i < layers.length; i++) { + buttonsHtml += `<button onclick="tester.visualizeWeights(${i})" id="weightsBtn${i}">Layer ${i}</button>`; + } + weightsLayerButtons.innerHTML = buttonsHtml; + + // Auto-select first layer + this.visualizeWeights(0); + } + + generateMipmaps(texture, width, height) { + if (!this.mipmapPipeline) { + const mipmapShader = FULLSCREEN_QUAD_VS + ` + @group(0) @binding(0) var src: texture_2d<f32>; + @fragment + fn fs_main(@builtin(position) pos: vec4<f32>) -> @location(0) vec4<f32> { + let coord = vec2<i32>(i32(pos.x) * 2, i32(pos.y) * 2); + var sum = vec4<f32>(0.0); + for (var y: i32 = 0; y < 2; y++) { + for (var x: i32 = 0; x < 2; x++) { + sum += textureLoad(src, coord + vec2<i32>(x, y), 0); + } + } + return sum * 0.25; + } + `; + this.mipmapPipeline = this.device.createRenderPipeline({ + layout: 'auto', + vertex: { module: this.device.createShaderModule({ code: mipmapShader }), entryPoint: 'vs_main' }, + fragment: { + module: this.device.createShaderModule({ code: mipmapShader }), + entryPoint: 'fs_main', + targets: [{ format: 'rgba8unorm' }] + } + }); + } + + const encoder = this.device.createCommandEncoder(); + + for (let mip = 1; mip < 3; mip++) { + const mipWidth = Math.max(1, width >> mip); + const mipHeight = Math.max(1, height >> mip); + + const bindGroup = this.device.createBindGroup({ + layout: this.mipmapPipeline.getBindGroupLayout(0), + entries: [ + { binding: 0, resource: texture.createView({ baseMipLevel: mip - 1, mipLevelCount: 1 }) } + ] + }); + + const renderPass = encoder.beginRenderPass({ + colorAttachments: [{ + view: texture.createView({ baseMipLevel: mip, mipLevelCount: 1 }), + loadOp: 'clear', + storeOp: 'store' + }] + }); + + renderPass.setPipeline(this.mipmapPipeline); + renderPass.setBindGroup(0, bindGroup); + renderPass.setViewport(0, 0, mipWidth, mipHeight, 0, 1); + renderPass.draw(6); + renderPass.end(); + } + + this.device.queue.submit([encoder.finish()]); + } + + displayOriginal() { + const source = this.isVideo ? this.video : this.image; + if (!source || !this.device) return; + + const { width, height } = this.getDimensions(); + this.context.configure({ device: this.device, format: this.format }); + + const inputTex = this.device.createTexture({ + size: [width, height], + format: 'rgba8unorm', + usage: GPUTextureUsage.TEXTURE_BINDING | GPUTextureUsage.COPY_DST | GPUTextureUsage.RENDER_ATTACHMENT + }); + + this.device.queue.copyExternalImageToTexture( + { source: source }, + { texture: inputTex }, + [width, height] + ); + + const simpleShader = FULLSCREEN_QUAD_VS + ` + @group(0) @binding(0) var tex: texture_2d<f32>; + @fragment + fn fs_main(@builtin(position) pos: vec4<f32>) -> @location(0) vec4<f32> { + let coord = vec2<i32>(pos.xy); + return textureLoad(tex, coord, 0); + } + `; + + const pipeline = this.device.createRenderPipeline({ + layout: 'auto', + vertex: { module: this.device.createShaderModule({ code: simpleShader }), entryPoint: 'vs_main' }, + fragment: { + module: this.device.createShaderModule({ code: simpleShader }), + entryPoint: 'fs_main', + targets: [{ format: this.format }] + } + }); + + const bindGroup = this.device.createBindGroup({ + layout: pipeline.getBindGroupLayout(0), + entries: [{ binding: 0, resource: inputTex.createView() }] + }); + + const encoder = this.device.createCommandEncoder(); + const renderPass = encoder.beginRenderPass({ + colorAttachments: [{ + view: this.context.getCurrentTexture().createView(), + loadOp: 'clear', + storeOp: 'store' + }] + }); + renderPass.setPipeline(pipeline); + renderPass.setBindGroup(0, bindGroup); + renderPass.draw(6); + renderPass.end(); + + this.device.queue.submit([encoder.finish()]); + } + + // Run CNN inference pipeline on current source (image or video frame) + async run() { + const t0 = performance.now(); + const source = this.isVideo ? this.video : this.image; + if (!source) return; + const { width, height } = this.getDimensions(); + + this.context.configure({ device: this.device, format: this.format }); + + // Create persistent input texture for original view with mipmaps + if (this.inputTexture) this.inputTexture.destroy(); + this.inputTexture = this.device.createTexture({ + size: [width, height], + format: 'rgba8unorm', + mipLevelCount: 3, + usage: GPUTextureUsage.TEXTURE_BINDING | GPUTextureUsage.COPY_DST | GPUTextureUsage.RENDER_ATTACHMENT + }); + + this.device.queue.copyExternalImageToTexture( + { source: source }, + { texture: this.inputTexture, mipLevel: 0 }, + [width, height] + ); + + // Generate mipmaps + this.generateMipmaps(this.inputTexture, width, height); + + const staticTex = this.device.createTexture({ + size: [width, height], + format: 'rgba32uint', + usage: GPUTextureUsage.STORAGE_BINDING | GPUTextureUsage.TEXTURE_BINDING | GPUTextureUsage.COPY_SRC + }); + + // Create one texture per layer output (static + all CNN layers) + this.layerOutputs = []; + const numLayers = this.weights.layers.length + 1; // +1 for static features + const layerTextures = []; + for (let i = 0; i < numLayers; i++) { + layerTextures.push(this.device.createTexture({ + size: [width, height], + format: 'rgba32uint', + usage: GPUTextureUsage.STORAGE_BINDING | GPUTextureUsage.TEXTURE_BINDING | GPUTextureUsage.COPY_DST + })); + } + + // Ping-pong buffers for computation + const computeTextures = [ + this.device.createTexture({ + size: [width, height], + format: 'rgba32uint', + usage: GPUTextureUsage.STORAGE_BINDING | GPUTextureUsage.TEXTURE_BINDING | GPUTextureUsage.COPY_SRC + }), + this.device.createTexture({ + size: [width, height], + format: 'rgba32uint', + usage: GPUTextureUsage.STORAGE_BINDING | GPUTextureUsage.TEXTURE_BINDING | GPUTextureUsage.COPY_SRC + }) + ]; + + const weightsGPU = this.device.createBuffer({ + size: this.weightsBuffer.byteLength, + usage: GPUBufferUsage.STORAGE | GPUBufferUsage.COPY_DST + }); + this.device.queue.writeBuffer(weightsGPU, 0, this.weightsBuffer); + const staticPipeline = this.device.createComputePipeline({ + layout: 'auto', + compute: { module: this.device.createShaderModule({ code: STATIC_SHADER }), entryPoint: 'main' } + }); + + const cnnPipeline = this.device.createComputePipeline({ + layout: 'auto', + compute: { module: this.device.createShaderModule({ code: CNN_SHADER }), entryPoint: 'main' } + }); + + const displayPipeline = this.device.createRenderPipeline({ + layout: 'auto', + vertex: { module: this.device.createShaderModule({ code: DISPLAY_SHADER }), entryPoint: 'vs_main' }, + fragment: { + module: this.device.createShaderModule({ code: DISPLAY_SHADER }), + entryPoint: 'fs_main', + targets: [{ format: this.format }] + } + }); + + const encoder = this.device.createCommandEncoder(); + + const mipLevelBuffer = this.device.createBuffer({ + size: 4, + usage: GPUBufferUsage.UNIFORM | GPUBufferUsage.COPY_DST + }); + this.device.queue.writeBuffer(mipLevelBuffer, 0, new Uint32Array([this.mipLevel])); + + if (!this.pointSampler) { + this.pointSampler = this.device.createSampler({ + magFilter: 'linear', + minFilter: 'linear', + mipmapFilter: 'linear' + }); + } + + // Extract depth from alpha channel (or 1.0 if no alpha) + const depthTex = this.device.createTexture({ + size: [width, height, 1], + format: 'r32float', + usage: GPUTextureUsage.TEXTURE_BINDING | GPUTextureUsage.COPY_DST + }); + + // Read image data to extract alpha channel + const tempCanvas = document.createElement('canvas'); + tempCanvas.width = width; + tempCanvas.height = height; + const tempCtx = tempCanvas.getContext('2d'); + tempCtx.drawImage(source, 0, 0, width, height); + const imageData = tempCtx.getImageData(0, 0, width, height); + const pixels = imageData.data; + + // Extract alpha channel (RGBA format: every 4th byte) + const depthData = new Float32Array(width * height); + for (let i = 0; i < width * height; i++) { + depthData[i] = pixels[i * 4 + 3] / 255.0; // Alpha channel [0, 255] → [0, 1] + } + + this.device.queue.writeTexture( + { texture: depthTex }, + depthData, + { bytesPerRow: width * 4 }, + [width, height, 1] + ); + + const staticBG = this.device.createBindGroup({ + layout: staticPipeline.getBindGroupLayout(0), + entries: [ + { binding: 0, resource: this.inputTexture.createView() }, + { binding: 1, resource: this.pointSampler }, + { binding: 2, resource: depthTex.createView() }, // Depth from alpha (matches training) + { binding: 3, resource: staticTex.createView() }, + { binding: 4, resource: { buffer: mipLevelBuffer } } + ] + }); + + const staticPass = encoder.beginComputePass(); + staticPass.setPipeline(staticPipeline); + staticPass.setBindGroup(0, staticBG); + staticPass.dispatchWorkgroups(Math.ceil(width / 8), Math.ceil(height / 8)); + staticPass.end(); + + // Copy static features to persistent storage (visualization index 0, shown as Static 0-3 / Static 4-7) + encoder.copyTextureToTexture( + { texture: staticTex }, + { texture: layerTextures[0] }, + [width, height] + ); + this.layerOutputs.push(layerTextures[0]); + + let srcTex = staticTex; + let dstTex = computeTextures[0]; + + for (let i = 0; i < this.weights.layers.length; i++) { + const layer = this.weights.layers[i]; + const isOutput = i === this.weights.layers.length - 1; + + // Calculate absolute weight offset in f16 units (add header offset) + // Version 1: 4 u32 header, Version 2: 5 u32 header + const headerSizeU32 = (this.weights.version === 1) ? 4 : 5; + const headerOffsetU32 = headerSizeU32 + this.weights.layers.length * 5; // Header + layer info in u32 + const absoluteWeightOffset = headerOffsetU32 * 2 + layer.weightOffset; // Convert to f16 units + + const paramsData = new Uint32Array(7); + paramsData[0] = layer.kernelSize; + paramsData[1] = layer.inChannels; + paramsData[2] = layer.outChannels; + paramsData[3] = absoluteWeightOffset; // Use absolute offset + paramsData[4] = isOutput ? 1 : 0; + paramsData[6] = (i === 0) ? 1 : 0; // is_layer_0 flag + + const paramsView = new Float32Array(paramsData.buffer); + paramsView[5] = this.blendAmount; + + const paramsBuffer = this.device.createBuffer({ + size: 28, + usage: GPUBufferUsage.UNIFORM | GPUBufferUsage.COPY_DST + }); + this.device.queue.writeBuffer(paramsBuffer, 0, paramsData); + + const cnnBG = this.device.createBindGroup({ + layout: cnnPipeline.getBindGroupLayout(0), + entries: [ + { binding: 0, resource: layerTextures[0].createView() }, + { binding: 1, resource: srcTex.createView() }, + { binding: 2, resource: dstTex.createView() }, + { binding: 3, resource: { buffer: weightsGPU } }, + { binding: 4, resource: { buffer: paramsBuffer } }, + { binding: 5, resource: this.inputTexture.createView() } + ] + }); + + const cnnPass = encoder.beginComputePass(); + cnnPass.setPipeline(cnnPipeline); + cnnPass.setBindGroup(0, cnnBG); + cnnPass.dispatchWorkgroups(Math.ceil(width / 8), Math.ceil(height / 8)); + cnnPass.end(); + + [srcTex, dstTex] = [dstTex, srcTex]; + + // Copy CNN layer output to persistent storage for visualization + // i=0: Layer 0 → layerTextures[1] + // i=1: Layer 1 → layerTextures[2], etc. + encoder.copyTextureToTexture( + { texture: srcTex }, + { texture: layerTextures[i + 1] }, + [width, height] + ); + + // Always push layer outputs for visualization (including output layer) + this.layerOutputs.push(layerTextures[i + 1]); + } + + const modeBuffer = this.device.createBuffer({ + size: 4, + usage: GPUBufferUsage.UNIFORM | GPUBufferUsage.COPY_DST + }); + this.device.queue.writeBuffer(modeBuffer, 0, new Uint32Array([this.viewMode])); + + // Store result texture and display pipeline for view mode switching + this.resultTexture = srcTex; + this.displayPipeline = displayPipeline; + this.modeBuffer = modeBuffer; + + const displayBG = this.device.createBindGroup({ + layout: displayPipeline.getBindGroupLayout(0), + entries: [ + { binding: 0, resource: srcTex.createView() }, + { binding: 1, resource: this.inputTexture.createView() }, + { binding: 2, resource: { buffer: modeBuffer } } + ] + }); + this.displayBindGroup = displayBG; + + const renderPass = encoder.beginRenderPass({ + colorAttachments: [{ + view: this.context.getCurrentTexture().createView(), + loadOp: 'clear', + storeOp: 'store' + }] + }); + renderPass.setPipeline(displayPipeline); + renderPass.setBindGroup(0, displayBG); + renderPass.draw(6); + renderPass.end(); + + this.device.queue.submit([encoder.finish()]); + + // Wait for GPU to finish before visualizing layers + await this.device.queue.onSubmittedWorkDone(); + + const t1 = performance.now(); + const mode = ['CNN Output', 'Original', 'Diff (×10)'][this.viewMode]; + this.setStatus(`GPU: ${(t1-t0).toFixed(1)}ms | ${width}×${height} | ${mode}`); + this.log(`Completed in ${(t1-t0).toFixed(1)}ms`); + + // Update layer visualization panel + this.updateLayerVizPanel(); + } + + updateLayerVizPanel() { + const panel = document.getElementById('layerViz'); + + if (!this.layerOutputs || this.layerOutputs.length === 0) { + panel.innerHTML = '<p style="color: #808080; text-align: center;">No layers to visualize</p>'; + return; + } + + // Only rebuild panel structure if layer count changed + const needsRebuild = !this.lastLayerCount || this.lastLayerCount !== this.layerOutputs.length; + + if (needsRebuild) { + let html = '<div class="layer-buttons">'; + html += `<button onclick="tester.visualizeLayer(0, 0)" id="layerBtn0_0">Static 0-3</button>`; + html += `<button onclick="tester.visualizeLayer(0, 4)" id="layerBtn0_4">Static 4-7</button>`; + + for (let i = 1; i < this.layerOutputs.length; i++) { + const label = `Layer ${i - 1}`; + html += `<button onclick="tester.visualizeLayer(${i})" id="layerBtn${i}">${label}</button>`; + } + html += `<button onclick="tester.saveCompositedLayer()" style="margin-left: 20px; background: #28a745;">Save Composited</button>`; + html += '</div>'; + + html += '<div class="layer-grid" id="layerGrid"></div>'; + html += '<div class="layer-preview"><div class="layer-view-label" id="previewLabel">Ch0</div><canvas id="previewCanvas"></canvas></div>'; + + panel.innerHTML = html; + this.log(`Layer visualization ready: ${this.layerOutputs.length} layers`); + this.recreateCanvases(); + this.lastLayerCount = this.layerOutputs.length; + } + + // Update current visualization + if (this.currentLayerIdx !== null) { + this.visualizeLayer(this.currentLayerIdx, this.currentChannelOffset || 0); + } else { + this.visualizeLayer(0, 0); + } + } + + recreateCanvases() { + const grid = document.getElementById('layerGrid'); + if (!grid) return; + + // Force removal of old canvases to clear any WebGPU contexts + const oldCanvases = grid.querySelectorAll('canvas'); + oldCanvases.forEach(canvas => { + canvas.width = 0; + canvas.height = 0; + }); + + grid.innerHTML = ''; + for (let c = 0; c < 4; c++) { + const div = document.createElement('div'); + div.className = 'layer-view'; + div.innerHTML = ` + <div class="layer-view-label" id="channelLabel${c}">Ch ${c}</div> + <canvas id="layerCanvas${c}"></canvas> + `; + div.onclick = () => this.selectChannel(c); + grid.appendChild(div); + } + this.selectedChannel = 0; + } + + async visualizeLayer(layerIdx, channelOffset = 0) { + if (!this.layerOutputs || layerIdx >= this.layerOutputs.length) { + this.log(`Cannot visualize layer ${layerIdx}: no data`, 'error'); + return; + } + + // Store current selection + this.currentLayerIdx = layerIdx; + this.currentChannelOffset = channelOffset; + + // Update button states + document.querySelectorAll('.layer-buttons button').forEach(btn => btn.classList.remove('active')); + if (layerIdx === 0) { + // Static features + const btnId = `layerBtn0_${channelOffset}`; + const btn = document.getElementById(btnId); + if (btn) btn.classList.add('active'); + } else { + const btn = document.getElementById(`layerBtn${layerIdx}`); + if (btn) btn.classList.add('active'); + } + + const layerName = layerIdx === 0 ? `Static Features (${channelOffset}-${channelOffset + 3})` : `Layer ${layerIdx - 1}`; + const layerTex = this.layerOutputs[layerIdx]; + const { width, height } = this.getDimensions(); + + // Update channel labels based on layer type + // Static features (layerIdx=0): 8 channels split into two views + // CNN layers (layerIdx≥1): 4 channels per layer + const staticLabels = [ + ['Ch0 (p0)', 'Ch1 (p1)', 'Ch2 (p2)', 'Ch3 (p3)'], + ['Ch4 (uv_x)', 'Ch5 (uv_y)', 'Ch6 (sin10_x)', 'Ch7 (bias)'] + ]; + const channelLabels = layerIdx === 0 + ? staticLabels[channelOffset / 4] + : ['Ch0', 'Ch1', 'Ch2', 'Ch3']; + + for (let c = 0; c < 4; c++) { + const label = document.getElementById(`channelLabel${c}`); + if (label) label.textContent = channelLabels[c]; + } + + // Create layer viz pipeline if needed + if (!this.layerVizPipeline) { + this.layerVizPipeline = this.device.createRenderPipeline({ + layout: 'auto', + vertex: { + module: this.device.createShaderModule({ code: LAYER_VIZ_SHADER }), + entryPoint: 'vs_main' + }, + fragment: { + module: this.device.createShaderModule({ code: LAYER_VIZ_SHADER }), + entryPoint: 'fs_main', + targets: [{ format: this.format }] + } + }); + this.log('Created layer visualization pipeline'); + } + + // Render each channel to its canvas + for (let c = 0; c < 4; c++) { + const canvas = document.getElementById(`layerCanvas${c}`); + if (!canvas) { + this.log(`Canvas layerCanvas${c} not found`, 'error'); + continue; + } + + // Set canvas size BEFORE getting context + canvas.width = width; + canvas.height = height; + + const ctx = canvas.getContext('webgpu'); + if (!ctx) { + this.log(`Failed to get WebGPU context for channel ${c}`, 'error'); + continue; + } + + try { + ctx.configure({ device: this.device, format: this.format }); + } catch (e) { + this.log(`Failed to configure canvas ${c}: ${e.message}`, 'error'); + continue; + } + + const vizScale = 1.0; // Always 1.0, shader clamps to [0,1] + const paramsBuffer = this.device.createBuffer({ + size: 8, + usage: GPUBufferUsage.UNIFORM | GPUBufferUsage.COPY_DST + }); + // Use channel index with offset for static features + const actualChannel = channelOffset + c; + const paramsData = new Float32Array([actualChannel, vizScale]); + this.device.queue.writeBuffer(paramsBuffer, 0, paramsData); + + const bindGroup = this.device.createBindGroup({ + layout: this.layerVizPipeline.getBindGroupLayout(0), + entries: [ + { binding: 0, resource: layerTex.createView() }, + { binding: 1, resource: { buffer: paramsBuffer } } + ] + }); + + const encoder = this.device.createCommandEncoder(); + const renderPass = encoder.beginRenderPass({ + colorAttachments: [{ + view: ctx.getCurrentTexture().createView(), + loadOp: 'clear', + clearValue: { r: 1.0, g: 0.0, b: 1.0, a: 1.0 }, // Magenta clear for debugging + storeOp: 'store' + }] + }); + + renderPass.setPipeline(this.layerVizPipeline); + renderPass.setBindGroup(0, bindGroup); + renderPass.draw(6); + renderPass.end(); + + this.device.queue.submit([encoder.finish()]); + } + + // Wait for all renders to complete + await this.device.queue.onSubmittedWorkDone(); + + // Update active channel highlighting and preview + this.updateChannelSelection(); + await this.renderChannelPreview(); + } + + selectChannel(channelIdx) { + this.selectedChannel = channelIdx; + this.updateChannelSelection(); + this.renderChannelPreview(); + } + + updateChannelSelection() { + const grid = document.getElementById('layerGrid'); + if (!grid) return; + + const views = grid.querySelectorAll('.layer-view'); + views.forEach((view, idx) => { + view.classList.toggle('active', idx === this.selectedChannel); + }); + } + + async renderChannelPreview() { + const previewCanvas = document.getElementById('previewCanvas'); + const previewLabel = document.getElementById('previewLabel'); + if (!previewCanvas || !this.device) return; + + const { width, height } = this.getDimensions(); + previewCanvas.width = width; + previewCanvas.height = height; + + const ctx = previewCanvas.getContext('webgpu'); + if (!ctx) return; + + try { + ctx.configure({ device: this.device, format: this.format }); + } catch (e) { + return; + } + + // Update label + const channelLabel = document.getElementById(`channelLabel${this.selectedChannel}`); + if (channelLabel && previewLabel) { + previewLabel.textContent = channelLabel.textContent; + } + + // Render selected channel + const layerIdx = this.currentLayerIdx; + const channelOffset = this.currentChannelOffset; + const layerTex = this.layerOutputs[layerIdx]; + if (!layerTex) return; + + // Always 1.0, shader clamps to [0,1] - show exact layer values + const vizScale = 1.0; + const actualChannel = channelOffset + this.selectedChannel; + + const paramsBuffer = this.device.createBuffer({ + size: 8, + usage: GPUBufferUsage.UNIFORM | GPUBufferUsage.COPY_DST + }); + const paramsData = new Float32Array([actualChannel, vizScale]); + this.device.queue.writeBuffer(paramsBuffer, 0, paramsData); + + const bindGroup = this.device.createBindGroup({ + layout: this.layerVizPipeline.getBindGroupLayout(0), + entries: [ + { binding: 0, resource: layerTex.createView() }, + { binding: 1, resource: { buffer: paramsBuffer } } + ] + }); + + const encoder = this.device.createCommandEncoder(); + const renderPass = encoder.beginRenderPass({ + colorAttachments: [{ + view: ctx.getCurrentTexture().createView(), + loadOp: 'clear', + storeOp: 'store' + }] + }); + + renderPass.setPipeline(this.layerVizPipeline); + renderPass.setBindGroup(0, bindGroup); + renderPass.draw(6); + renderPass.end(); + + this.device.queue.submit([encoder.finish()]); + } + + visualizeWeights(cnnLayerIdx) { + const layer = this.weights.layers[cnnLayerIdx]; + if (!layer) { + this.log(`Layer ${cnnLayerIdx} not found`, 'error'); + return; + } + + // Update button states + document.querySelectorAll('#weightsLayerButtons button').forEach(btn => btn.classList.remove('active')); + const btn = document.getElementById(`weightsBtn${cnnLayerIdx}`); + if (btn) btn.classList.add('active'); + + const { kernelSize, inChannels, outChannels, weightOffset, min, max } = layer; + + const canvas = document.getElementById('weightsCanvas'); + const ctx = canvas.getContext('2d', { willReadFrequently: false }); + + // 1 pixel per weight, show all input channels horizontally + const width = inChannels * kernelSize; + const height = outChannels * kernelSize; + + canvas.width = width; + canvas.height = height; + + ctx.fillStyle = '#1a1a1a'; + ctx.fillRect(0, 0, width, height); + + // Stack output channels vertically + for (let outCh = 0; outCh < outChannels; outCh++) { + const yOffset = outCh * kernelSize; + + for (let inCh = 0; inCh < inChannels; inCh++) { + const xOffset = inCh * kernelSize; + + for (let ky = 0; ky < kernelSize; ky++) { + for (let kx = 0; kx < kernelSize; kx++) { + const spatialIdx = ky * kernelSize + kx; + const wIdx = weightOffset + + outCh * inChannels * kernelSize * kernelSize + + inCh * kernelSize * kernelSize + + spatialIdx; + + const weight = this.getWeightValue(wIdx); + const normalized = (weight - min) / (max - min); + const intensity = Math.floor(normalized * 255); + + ctx.fillStyle = `rgb(${intensity}, ${intensity}, ${intensity})`; + ctx.fillRect(xOffset + kx, yOffset + ky, 1, 1); + } + } + } + } + } + + getWeightValue(idx) { + const pairIdx = Math.floor(idx / 2); + const packed = this.weights.weights[pairIdx]; + const unpacked = this.unpackF16(packed); + return (idx % 2 === 0) ? unpacked[0] : unpacked[1]; + } + + toggleWeightsInfo() { + const panel = document.getElementById('weightsInfoPanel'); + const toggle = document.getElementById('weightsInfoToggle'); + panel.classList.toggle('collapsed'); + toggle.textContent = panel.classList.contains('collapsed') ? '▶' : '▼'; + } + + updateDisplay() { + if (!this.displayPipeline || !this.displayBindGroup) return; + + this.device.queue.writeBuffer(this.modeBuffer, 0, new Uint32Array([this.viewMode])); + + const encoder = this.device.createCommandEncoder(); + const renderPass = encoder.beginRenderPass({ + colorAttachments: [{ + view: this.context.getCurrentTexture().createView(), + loadOp: 'clear', + storeOp: 'store' + }] + }); + renderPass.setPipeline(this.displayPipeline); + renderPass.setBindGroup(0, this.displayBindGroup); + renderPass.draw(6); + renderPass.end(); + + this.device.queue.submit([encoder.finish()]); + } + + async savePNG() { + if (!this.image && !this.isVideo) { + this.log('No image loaded', 'error'); + return; + } + + if (!this.resultTexture) { + this.log('No result to save', 'error'); + return; + } + + try { + const { width, height } = this.getDimensions(); + + // GPU readback from result texture + const bytesPerRow = width * 16; // 4×u32 per pixel + const paddedBytesPerRow = Math.ceil(bytesPerRow / 256) * 256; + const bufferSize = paddedBytesPerRow * height; + + const stagingBuffer = this.device.createBuffer({ + size: bufferSize, + usage: GPUBufferUsage.COPY_DST | GPUBufferUsage.MAP_READ + }); + + const encoder = this.device.createCommandEncoder(); + encoder.copyTextureToBuffer( + { texture: this.resultTexture }, + { buffer: stagingBuffer, bytesPerRow: paddedBytesPerRow, rowsPerImage: height }, + { width, height, depthOrArrayLayers: 1 } + ); + this.device.queue.submit([encoder.finish()]); + + await stagingBuffer.mapAsync(GPUMapMode.READ); + const mapped = new Uint8Array(stagingBuffer.getMappedRange()); + + // Unpack f16 to RGBA8 + const pixels = new Uint8Array(width * height * 4); + for (let y = 0; y < height; y++) { + const rowOffset = y * paddedBytesPerRow; + for (let x = 0; x < width; x++) { + const pixelOffset = rowOffset + x * 16; + const data = new Uint32Array(mapped.buffer, mapped.byteOffset + pixelOffset, 4); + + // Unpack f16 (first 4 channels only) + const unpack = (u32, idx) => { + const h = (idx === 0) ? (u32 & 0xFFFF) : ((u32 >> 16) & 0xFFFF); + const sign = (h >> 15) & 1; + const exp = (h >> 10) & 0x1F; + const frac = h & 0x3FF; + if (exp === 0) return 0; + if (exp === 31) return sign ? 0 : 255; + const e = exp - 15; + const val = (1 + frac / 1024) * Math.pow(2, e); + return Math.max(0, Math.min(255, Math.round(val * 255))); + }; + + const outIdx = (y * width + x) * 4; + pixels[outIdx + 0] = unpack(data[0], 0); // R + pixels[outIdx + 1] = unpack(data[0], 1); // G + pixels[outIdx + 2] = unpack(data[1], 0); // B + pixels[outIdx + 3] = 255; // A + } + } + + stagingBuffer.unmap(); + stagingBuffer.destroy(); + + // Create blob from pixels + const canvas = document.createElement('canvas'); + canvas.width = width; + canvas.height = height; + const ctx = canvas.getContext('2d'); + const imageData = new ImageData(new Uint8ClampedArray(pixels), width, height); + ctx.putImageData(imageData, 0, 0); + + const blob = await new Promise(resolve => canvas.toBlob(resolve, 'image/png')); + const url = URL.createObjectURL(blob); + const a = document.createElement('a'); + const mode = ['cnn', 'original', 'diff'][this.viewMode]; + a.href = url; + a.download = `output_${width}x${height}_${mode}.png`; + a.click(); + URL.revokeObjectURL(url); + + this.log(`Saved PNG: ${a.download}`); + this.setStatus(`Saved: ${a.download}`); + } catch (err) { + this.log(`Failed to save PNG: ${err.message}`, 'error'); + this.setStatus(`Save failed: ${err.message}`, true); + } + } + + async saveCompositedLayer() { + if (!this.currentLayerIdx) { + this.log('No layer selected for compositing', 'error'); + return; + } + + try { + const canvases = []; + for (let i = 0; i < 4; i++) { + const canvas = document.getElementById(`layerCanvas${i}`); + if (!canvas) { + this.log(`Canvas layerCanvas${i} not found`, 'error'); + return; + } + canvases.push(canvas); + } + + const width = canvases[0].width; + const height = canvases[0].height; + const compositedWidth = width * 4; + + // Create composited canvas + const compositedCanvas = document.createElement('canvas'); + compositedCanvas.width = compositedWidth; + compositedCanvas.height = height; + const ctx = compositedCanvas.getContext('2d'); + + // Composite horizontally + for (let i = 0; i < 4; i++) { + ctx.drawImage(canvases[i], i * width, 0); + } + + // Convert to grayscale + const imageData = ctx.getImageData(0, 0, compositedWidth, height); + const pixels = imageData.data; + for (let i = 0; i < pixels.length; i += 4) { + const gray = 0.299 * pixels[i] + 0.587 * pixels[i + 1] + 0.114 * pixels[i + 2]; + pixels[i] = pixels[i + 1] = pixels[i + 2] = gray; + } + ctx.putImageData(imageData, 0, 0); + + // Save as PNG + const blob = await new Promise(resolve => compositedCanvas.toBlob(resolve, 'image/png')); + const url = URL.createObjectURL(blob); + const a = document.createElement('a'); + a.href = url; + a.download = `composited_layer${this.currentLayerIdx - 1}_${compositedWidth}x${height}.png`; + a.click(); + URL.revokeObjectURL(url); + + this.log(`Saved composited layer: ${a.download}`); + this.setStatus(`Saved: ${a.download}`); + } catch (err) { + this.log(`Failed to save composited layer: ${err.message}`, 'error'); + this.setStatus(`Compositing failed: ${err.message}`, true); + } + } +} + +const tester = new CNNTester(); + +// Load default weights on startup +(async () => { + try { + const binaryString = atob(DEFAULT_WEIGHTS_B64); + const bytes = new Uint8Array(binaryString.length); + for (let i = 0; i < binaryString.length; i++) { + bytes[i] = binaryString.charCodeAt(i); + } + await tester.loadWeights({ name: 'default.bin', arrayBuffer: () => Promise.resolve(bytes.buffer) }); + tester.log('Loaded default weights'); + } catch (err) { + tester.log(`Failed to load default weights: ${err.message}`, 'error'); + } +})(); + +function setupDropZone(id, callback) { + const zone = document.getElementById(id); + ['dragenter', 'dragover', 'dragleave', 'drop'].forEach(e => { + zone.addEventListener(e, ev => { ev.preventDefault(); ev.stopPropagation(); }); + }); + ['dragenter', 'dragover'].forEach(e => zone.addEventListener(e, () => zone.classList.add('active'))); + ['dragleave', 'drop'].forEach(e => zone.addEventListener(e, () => zone.classList.remove('active'))); + zone.addEventListener('drop', e => { + const file = e.dataTransfer.files[0]; + if (file) callback(file).catch(err => { + zone.classList.add('error'); + tester.setStatus(err.message, true); + tester.log(err.message, 'error'); + setTimeout(() => zone.classList.remove('error'), 2000); + }); + }); +} + +// Whole window drop for PNG images and videos +const mainArea = document.getElementById('mainDrop'); +['dragenter', 'dragover', 'dragleave', 'drop'].forEach(e => { + mainArea.addEventListener(e, ev => { ev.preventDefault(); ev.stopPropagation(); }); +}); +['dragenter', 'dragover'].forEach(e => mainArea.addEventListener(e, () => mainArea.classList.add('drop-active'))); +['dragleave', 'drop'].forEach(e => mainArea.addEventListener(e, () => mainArea.classList.remove('drop-active'))); +mainArea.addEventListener('drop', e => { + const file = e.dataTransfer.files[0]; + if (file) { + if (file.type.startsWith('image/')) { + tester.loadImage(file).catch(err => { + tester.setStatus(err.message, true); + tester.log(err.message, 'error'); + }); + } else if (file.type.startsWith('video/')) { + tester.loadVideo(file).catch(err => { + tester.setStatus(err.message, true); + tester.log(err.message, 'error'); + }); + } + } +}); + +// Weights drop zone +setupDropZone('weightsDrop', f => tester.loadWeights(f)); + +// Weights file input +document.getElementById('weightsFile').addEventListener('change', e => { + const file = e.target.files[0]; + if (file) { + tester.loadWeights(file).catch(err => { + tester.setStatus(err.message, true); + tester.log(err.message, 'error'); + }); + } +}); + +document.getElementById('blend').addEventListener('input', e => { + tester.blendAmount = parseFloat(e.target.value); + document.getElementById('blendValue').textContent = e.target.value; + if ((tester.image || tester.isVideo) && tester.weights) { + tester.log(`Blend changed to ${e.target.value}`); + tester.run(); + } +}); + +document.getElementById('depth').addEventListener('input', e => { + tester.depth = parseFloat(e.target.value); + document.getElementById('depthValue').textContent = e.target.value; + if ((tester.image || tester.isVideo) && tester.weights) tester.run(); +}); + +document.getElementById('mipLevel').addEventListener('change', e => { + tester.mipLevel = parseInt(e.target.value); + tester.log(`Mip level changed to ${e.target.value}`); + if ((tester.image || tester.isVideo) && tester.weights) tester.run(); +}); + +document.getElementById('playPauseBtn').addEventListener('click', () => tester.togglePlayPause()); +document.getElementById('stepBackBtn').addEventListener('click', () => tester.stepFrame(-1)); +document.getElementById('stepForwardBtn').addEventListener('click', () => tester.stepFrame(1)); +document.getElementById('savePngBtn').addEventListener('click', () => tester.savePNG()); + +document.addEventListener('keydown', e => { + if (e.code === 'Space') { + e.preventDefault(); + if (tester.viewMode === 1) { + tester.viewMode = 0; + } else { + tester.viewMode = 1; + } + const modeName = ['CNN Output', 'Original', 'Diff (×10)'][tester.viewMode]; + if ((tester.image || tester.isVideo) && tester.weights) { + tester.log(`View mode: ${modeName}`); + tester.updateDisplay(); + const width = tester.isVideo ? tester.video.videoWidth : tester.image.width; + const height = tester.isVideo ? tester.video.videoHeight : tester.image.height; + tester.setStatus(`${width}×${height} | ${modeName}`); + } + } else if (e.code === 'KeyD') { + e.preventDefault(); + if (tester.viewMode === 2) { + tester.viewMode = 0; + } else { + tester.viewMode = 2; + } + const modeName = ['CNN Output', 'Original', 'Diff (×10)'][tester.viewMode]; + if ((tester.image || tester.isVideo) && tester.weights) { + tester.log(`View mode: ${modeName}`); + tester.updateDisplay(); + const width = tester.isVideo ? tester.video.videoWidth : tester.image.width; + const height = tester.isVideo ? tester.video.videoHeight : tester.image.height; + tester.setStatus(`${width}×${height} | ${modeName}`); + } + } +}); + </script> +</body> +</html> |