diff options
Diffstat (limited to 'doc/SEQUENCE.md')
| -rw-r--r-- | doc/SEQUENCE.md | 221 |
1 files changed, 221 insertions, 0 deletions
diff --git a/doc/SEQUENCE.md b/doc/SEQUENCE.md new file mode 100644 index 0000000..76e19d4 --- /dev/null +++ b/doc/SEQUENCE.md @@ -0,0 +1,221 @@ +# Sequence: DAG-based Effect Routing + +**Status:** ✅ Operational + +Explicit node system with DAG effect routing. + +## Quick Start + +```bash +# Compile timeline +python3 tools/seq_compiler.py workspaces/main/timeline.seq --output src/generated/timeline.cc + +# Flatten mode (future: inline effects, no vtables) +python3 tools/seq_compiler.py timeline.seq --output timeline.cc --flatten +``` + +## Timeline Syntax + +``` +# BPM 120 (optional, currently ignored) + +SEQUENCE <start_time> <priority> ["name"] + # Node declarations (optional, auto-inferred as u8x4_norm) + NODE temp1 u8x4_norm + NODE depth depth24 + NODE gbuf_normal f16x8 + + # Asset dependencies (validated at compile-time) + ASSET shader_blur + + # Effect routing with priority modifier + EFFECT <+|=|-> EffectClass input1 input2 -> output1 output2 start end [params] +``` + +**Priority modifiers** (REQUIRED): +- `+` : Increment priority (foreground) +- `=` : Same priority as previous +- `-` : Decrement priority (background) + +**Node types**: `u8x4_norm` (default), `f32x4`, `f16x8`, `depth24`, `compute_f32` + +**Reserved nodes**: `source` (input), `sink` (output) + +### Examples + +**Simple chain:** +``` +SEQUENCE 0.0 0 "basic" + EFFECT + HeptagonEffect source -> temp1 0.0 10.0 + EFFECT + GaussianBlur temp1 -> sink 0.0 10.0 +``` + +**Multi-output (G-buffer):** +``` +SEQUENCE 0.0 0 "deferred" + NODE gbuf_n f16x8 + NODE gbuf_p f32x4 + NODE depth depth24 + + EFFECT + DeferredRender source depth -> gbuf_n gbuf_p 0.0 10.0 + EFFECT + Compose gbuf_n gbuf_p -> sink 0.0 10.0 +``` + +**Ping-pong (auto-optimized):** +``` +SEQUENCE 0.0 0 "blur" + # Compiler detects alternating pattern, aliases temp2 -> temp1 + EFFECT + BlurH source -> temp1 0.0 10.0 + EFFECT + BlurV temp1 -> temp2 0.0 10.0 + EFFECT + Sharpen temp2 -> sink 0.0 10.0 +``` + +## Architecture + +### Sequence Class + +```cpp +class Sequence { + NodeRegistry nodes_; // Typed texture management + std::vector<EffectDAGNode> effect_dag_; // Topologically sorted + UniformsSequenceParams params_; // Per-frame uniforms + + virtual void preprocess(float time, float beat_time, float beat_phase, float audio_intensity); + virtual void render_effects(WGPUCommandEncoder encoder); +}; +``` + +### Effect Class + +```cpp +class Effect { + std::vector<std::string> input_nodes_; + std::vector<std::string> output_nodes_; + + virtual void declare_nodes(NodeRegistry& registry) {} // Optional temp nodes + virtual void render(WGPUCommandEncoder encoder, + const UniformsSequenceParams& params, + NodeRegistry& nodes) = 0; +}; +``` + +### Node System + +**Types**: Match WGSL texture formats +- `U8X4_NORM`: RGBA8Unorm (default for source/sink/intermediate) +- `F32X4`: RGBA32Float (HDR, compute outputs) +- `F16X8`: 8-channel float16 (G-buffer normals/vectors) +- `DEPTH24`: Depth24Plus (3D rendering) +- `COMPUTE_F32`: Storage buffer (non-texture compute data) + +**Aliasing**: Compiler detects ping-pong patterns (Effect i writes A reads B, Effect i+1 writes B reads A) and aliases nodes to same backing texture. + +## Compiler Features + +**seq_compiler.py** generates optimized C++ from `.seq`: + +1. **DAG Validation**: Cycle detection, connectivity checks +2. **Topological Sort**: Execution order from dependencies +3. **Ping-pong Detection**: Automatic node aliasing +4. **Code Generation**: Sequence subclasses with node declarations and effect DAG + +**Output**: Single `.cc` file with: +- Multiple `Sequence` subclasses (one per SEQUENCE) +- `InitializeSequences()` - Registry initialization +- `GetActiveSequence(float time)` - Active sequence lookup +- `RenderTimeline()` - Encoder-based and surface-based variants + +## Creating Effects + +**For standard post-process:** +```cpp +class MyEffect : public Effect { + WGPURenderPipeline pipeline_; + UniformBuffer<UniformsSequenceParams> uniforms_; + + MyEffect(const GpuContext& ctx, const std::vector<std::string>& inputs, const std::vector<std::string>& outputs); + const std::vector<std::string>& outputs) + : Effect(ctx, inputs, outputs) { + uniforms_.init(ctx_.device); + pipeline_ = create_post_process_pipeline(ctx_.device, + WGPUTextureFormat_RGBA8Unorm, + my_shader_wgsl); + } + + void render(WGPUCommandEncoder encoder, const UniformsSequenceParams& params, + NodeRegistry& nodes) override { + WGPUTextureView input = nodes.get_view(input_nodes_[0]); + WGPUTextureView output = nodes.get_view(output_nodes_[0]); + uniforms_.update(ctx_.queue, params); + // ... render pass + } +}; +``` + +**For 3D effects with depth:** +```cpp +class My3DEffect : public Effect { + std::string depth_node_; + + My3DEffect(...) : Effect(...), depth_node_(outputs[0] + "_depth") {} + + void declare_nodes(NodeRegistry& registry) override { + registry.declare_node(depth_node_, NodeType::DEPTH24, -1, -1); + } + + void render(...) { + WGPUTextureView color = nodes.get_view(output_nodes_[0]); + WGPUTextureView depth = nodes.get_view(depth_node_); + // ... render pass with depth attachment + } +}; +``` + +## Uniform Access + +```cpp +params.time; // Physical seconds (constant speed) +params.beat_time; // Musical beats (tempo-scaled) +params.beat_phase; // Fractional beat 0.0-1.0 +params.audio_intensity; // Audio peak for beat sync +params.resolution; // vec2(width, height) +params.aspect_ratio; // width/height +``` + +## Status & Limitations + +**Implemented** ✅: +- DAG validation, topological sort, ping-pong optimization +- Multi-input/multi-output effects +- Node aliasing (compile-time optimization) +- 7 effects ported (Passthrough, Placeholder, GaussianBlur, Heptagon, Particles, RotatingCube, Hybrid3D) + +**Missing/Future** ❌: +- Flatten mode (--flatten generates same code as dev mode) +- BPM handling (parsed but ignored) +- GetDemoDuration() calculation (hardcoded 40.0f) +- Sequence-relative time (uses global time) +- Asset validation (not checked against AssetId enum) +- Node type compatibility checking + +**TODO**: +- Port remaining effects (10+ effects, CNN effects) +- Implement flatten mode (inline effects, direct members) +- Update HTML timeline editor for graph visualization + +## Migration Notes + +**Key Features**: +- V1: Implicit framebuffer ping-pong (framebuffer_a_ ↔ framebuffer_b_) +- Explicit node declarations with routing + +**Breaking Changes**: +- Effect base class standardized +- Constructor signature: `(GpuContext, inputs[], outputs[])` +- Render signature: Added `NodeRegistry& nodes` parameter +- No `is_post_process()` method (routing makes it explicit) + +**See Also**: +- `doc/EFFECT_WORKFLOW.md` - Step-by-step effect creation +- `tools/seq_compiler.py` - Compiler implementation +- `workspaces/main/timeline.seq` - Example timeline |