path: root/doc/SEQUENCE.md

# Sequence: DAG-based Effect Routing

**Status:** Operational

Explicit node system with DAG effect routing.

## Quick Start

```bash
# Compile timeline
python3 tools/seq_compiler.py workspaces/main/timeline.seq --output src/generated/timeline.cc
```

## Timeline Syntax

```
# BPM 120 (optional, currently ignored)

SEQUENCE <start_time> <priority> ["name"]
  # Node declarations (optional, auto-inferred as u8x4_norm)
  NODE temp1 u8x4_norm
  NODE depth depth24
  NODE gbuf_normal f16x8

  # Asset dependencies (validated at compile-time)
  ASSET shader_blur

  # Effect routing with priority modifier
  EFFECT <+|=|-> EffectClass input1 input2 -> output1 output2 start end [params]
```

**Priority modifiers** (REQUIRED):
- `+` : Increment priority (foreground)
- `=` : Same priority as previous
- `-` : Decrement priority (background)

**Node types**: `u8x4_norm` (default), `f32x4`, `f16x8`, `depth24`, `compute_f32`

**Reserved nodes**: `source` (input), `sink` (output)

**Effect times**: Absolute seconds. `seq_compiler` adds the sequence start offset automatically.

### Examples

**Simple chain:**
```
SEQUENCE 0.0 0 "basic"
  EFFECT + HeptagonEffect source -> temp1 0.0 10.0
  EFFECT + GaussianBlur temp1 -> sink 0.0 10.0
```

**Multi-output (G-buffer):**
```
SEQUENCE 0.0 0 "deferred"
  NODE gbuf_n f16x8
  NODE gbuf_p f32x4
  NODE depth depth24

  EFFECT + DeferredRender source depth -> gbuf_n gbuf_p 0.0 10.0
  EFFECT + Compose gbuf_n gbuf_p -> sink 0.0 10.0
```

**Ping-pong (auto-optimized):**
```
SEQUENCE 0.0 0 "blur"
  # Compiler detects alternating pattern, aliases temp2 -> temp1
  EFFECT + BlurH source -> temp1 0.0 10.0
  EFFECT + BlurV temp1 -> temp2 0.0 10.0
  EFFECT + Sharpen temp2 -> sink 0.0 10.0
```

## Architecture

### Sequence Class

```cpp
class Sequence {
  NodeRegistry nodes_;              // Typed texture management
  std::vector<EffectDAGNode> effect_dag_;  // Topologically sorted
  UniformsSequenceParams params_;   // Per-frame uniforms

  virtual void preprocess(float time, float beat_time, float beat_phase, float audio_intensity);
  virtual void render_effects(WGPUCommandEncoder encoder);
};
```

### Effect Class

```cpp
class Effect {
  std::vector<std::string> input_nodes_;
  std::vector<std::string> output_nodes_;

  // Optional: declare internal nodes (depth buffers, intermediate textures).
  virtual void declare_nodes(NodeRegistry& registry) {}

  // Required: render this effect for the current frame.
  virtual void render(WGPUCommandEncoder encoder,
                      const UniformsSequenceParams& params,
                      NodeRegistry& nodes) = 0;

  // Optional: called after ALL effects in the sequence have rendered.
  // Use for end-of-frame bookkeeping, e.g. copying temporal feedback buffers.
  // Default implementation is a no-op.
  virtual void post_render(WGPUCommandEncoder encoder, NodeRegistry& nodes) {}
};
```

### Frame execution order

Each frame, `Sequence::render_effects()` runs two passes over the DAG:

1. **Render pass** — `dispatch_render()` on every effect in topological order
2. **Post-render pass** — `post_render()` on every effect in the same order

This ordering guarantees that by the time any `post_render()` runs, all output
textures for the frame are fully written.  It is safe to read any node's texture
in `post_render()`.

### Temporal feedback pattern

DAG-based sequences cannot express read-after-write cycles within a single frame.
Use `post_render()` + a persistent internal node to implement temporal feedback
(e.g. CNN prev-frame input):

```cpp
class MyEffect : public Effect {
  std::string node_prev_;      // internal persistent texture
  std::string source_node_;    // node to capture at end of frame

 public:
  void set_source_node(const std::string& n) { source_node_ = n; }

  void declare_nodes(NodeRegistry& reg) override {
    // Use a NodeType whose format matches source_node_ and has CopyDst.
    reg.declare_node(node_prev_, NodeType::F16X8, -1, -1);
  }

  void render(...) override {
    // Read node_prev_ — contains source_node_ output from the *previous* frame.
    WGPUTextureView prev = nodes.get_view(node_prev_);
    // ... use prev
  }

  void post_render(WGPUCommandEncoder enc, NodeRegistry& nodes) override {
    if (source_node_.empty() || !nodes.has_node(source_node_)) return;
    // Copy this frame's output into node_prev_ for next frame.
    WGPUTexelCopyTextureInfo src = {.texture = nodes.get_texture(source_node_)};
    WGPUTexelCopyTextureInfo dst = {.texture = nodes.get_texture(node_prev_)};
    WGPUExtent3D ext = {(uint32_t)width_, (uint32_t)height_, 1};
    wgpuCommandEncoderCopyTextureToTexture(enc, &src, &dst, &ext);
  }
};
```

**Why not `input_nodes_[0]` / ping-pong as prev?**  The ping-pong alias makes
`source` equal to last frame's `sink` only when the effect is the first in the
sequence and no post-CNN effects overwrite `sink`.  `post_render()` is
unconditionally correct regardless of sequence structure.

**Current user**: `GBufferEffect` uses this pattern for `prev.rgb` (CNN temporal
feedback). `cnn_output_node_` is wired automatically via `wire_dag()` — no
manual `set_cnn_output_node()` call needed.

### DAG wiring (`wire_dag`)

```cpp
// Effect base class
virtual void wire_dag(const std::vector<EffectDAGNode>& dag) {}
```

Called once from `Sequence::init_effect_nodes()` after all `declare_nodes()`
calls, so the full DAG is visible.  Override to resolve inter-effect
dependencies that cannot be expressed through node names alone.

`GBufferEffect::wire_dag()` delegates to the base-class helper
`find_downstream_output(dag)`, then guards against wiring to `"sink"`:

```cpp
void GBufferEffect::wire_dag(const std::vector<EffectDAGNode>& dag) {
  const std::string out = find_downstream_output(dag);
  if (out != "sink") cnn_output_node_ = out;
}
```

`"sink"` is registered as an external view (`texture == nullptr`); copying
from it in `post_render` would crash.  When no CNN follows the G-buffer stage
(e.g. debug/deferred sequences), `cnn_output_node_` stays empty and
`post_render` is a no-op.

#### `Effect::find_downstream_output`

```cpp
// protected helper — call from wire_dag()
std::string find_downstream_output(const std::vector<EffectDAGNode>& dag) const;
```

Returns `output_nodes[0]` of the first direct downstream consumer in the DAG,
or `""` if none exists.  The helper is agnostic about node semantics — it is
the **caller's responsibility** to reject unsuitable results (e.g. `"sink"` or
any other external/terminal node whose texture is not owned by the registry).

`post_render` also null-checks the source texture as a belt-and-suspenders
guard:

```cpp
WGPUTexture src_tex = nodes.get_texture(cnn_output_node_);
if (!src_tex) return;  // external view — no owned texture to copy
```

### Node System

**Types**: Match WGSL texture formats
- `U8X4_NORM`: RGBA8Unorm — default for source/sink/intermediate; `COPY_SRC|COPY_DST`
- `F32X4`: RGBA32Float — HDR, compute outputs
- `F16X8`: 8-channel float16 — G-buffer normals/vectors
- `DEPTH24`: Depth24Plus — 3D rendering
- `COMPUTE_F32`: Storage buffer — non-texture compute data
- `GBUF_ALBEDO`: RGBA16Float — G-buffer albedo/normal MRT; `RENDER_ATTACHMENT|TEXTURE_BINDING|STORAGE_BINDING|COPY_SRC`
- `GBUF_DEPTH32`: Depth32Float — G-buffer depth; `RENDER_ATTACHMENT|TEXTURE_BINDING|COPY_SRC`
- `GBUF_R8`: RGBA8Unorm — G-buffer single-channel (shadow, transp); `STORAGE_BINDING|TEXTURE_BINDING|RENDER_ATTACHMENT`
- `GBUF_RGBA32UINT`: RGBA32Uint — packed feature textures (CNN v3 feat_tex0/1); `STORAGE_BINDING|TEXTURE_BINDING`

**`COPY_SRC|COPY_DST`** is required on any node used with `wgpuCommandEncoderCopyTextureToTexture`.
The `node_prev_` format **must match** the source texture format exactly —
`CopyTextureToTexture` requires identical formats.  `F16X8` (Rgba16Float,
`CopySrc|CopyDst`) matches `GBUF_ALBEDO` (CNNv3Effect output).  Use `U8X4_NORM`
only when the source is also Rgba8Unorm.

**Aliasing**: Compiler detects ping-pong patterns (Effect i writes A reads B, Effect i+1 writes B reads A) and aliases nodes to same backing texture.

## Compiler Features

**seq_compiler.py** generates optimized C++ from `.seq`:

1. **DAG Validation**: Cycle detection, connectivity checks
2. **Topological Sort**: Execution order from dependencies
3. **Ping-pong Detection**: Automatic node aliasing
4. **Code Generation**: Sequence subclasses with node declarations and effect DAG

**Output**: Single `.cc` file with:
- Multiple `Sequence` subclasses (one per SEQUENCE)
- `InitializeSequences()` - Registry initialization
- `GetActiveSequence(float time)` - Active sequence lookup
- `RenderTimeline()` - Encoder-based and surface-based variants

## Creating Effects

**For standard post-process:**
```cpp
class MyEffect : public Effect {
  WGPURenderPipeline pipeline_;
  UniformBuffer<UniformsSequenceParams> uniforms_;

  MyEffect(const GpuContext& ctx, const std::vector<std::string>& inputs,
           const std::vector<std::string>& outputs, float start, float end)
    : Effect(ctx, inputs, outputs, start, end) {
    uniforms_.init(ctx_.device);
    pipeline_ = create_post_process_pipeline(ctx_.device,
                                             WGPUTextureFormat_RGBA8Unorm,
                                             my_shader_wgsl);
  }

  void render(WGPUCommandEncoder encoder, const UniformsSequenceParams& params,
              NodeRegistry& nodes) override {
    WGPUTextureView input = nodes.get_view(input_nodes_[0]);
    WGPUTextureView output = nodes.get_view(output_nodes_[0]);
    uniforms_.update(ctx_.queue, params);
    // ... render pass
  }
};
```

**For 3D effects with depth:**
```cpp
class My3DEffect : public Effect {
  std::string depth_node_;

  My3DEffect(...) : Effect(...), depth_node_(outputs[0] + "_depth") {}

  void declare_nodes(NodeRegistry& registry) override {
    registry.declare_node(depth_node_, NodeType::DEPTH24, -1, -1);
  }

  void render(...) {
    WGPUTextureView color = nodes.get_view(output_nodes_[0]);
    WGPUTextureView depth = nodes.get_view(depth_node_);
    // ... render pass with depth attachment
  }
};
```

## Uniform Access

```cpp
params.time;            // Physical seconds (constant speed)
params.beat_time;       // Musical beats (tempo-scaled)
params.beat_phase;      // Fractional beat 0.0-1.0
params.audio_intensity; // Audio peak for beat sync
params.resolution;      // vec2(width, height)
params.aspect_ratio;    // width/height
```

## Status & Limitations

**Implemented:**
- DAG validation, topological sort, ping-pong optimization
- Multi-input/multi-output effects
- Node aliasing (compile-time optimization)
- 12 effects: Passthrough, Placeholder, GaussianBlur, Heptagon, Particles, RotatingCube, Hybrid3D, Flash, PeakMeter, Scene1, Scene2, Scratch

**Missing/Future:**
- Flatten mode (`--flatten` generates same code as dev mode)
- BPM handling (parsed but ignored)
- GetDemoDuration() calculation (hardcoded 40.0f)
- Asset validation (not checked against AssetId enum)
- Node type compatibility checking

## See Also

- `doc/EFFECT_WORKFLOW.md` - Step-by-step effect creation
- `tools/seq_compiler.py` - Compiler implementation
- `workspaces/main/timeline.seq` - Example timeline