path: root/tools/spectral_editor/README.md

# Spectral Brush Editor

A web-based tool for creating procedural audio by tracing spectrograms with parametric Bezier curves.

## Purpose

Replace large `.spec` binary assets with tiny procedural C++ code:
- **Before:** 5 KB binary `.spec` file
- **After:** ~100 bytes of C++ code calling `draw_bezier_curve()`

**Compression ratio:** 50-100×

## Features

### Core Functionality
- Load `.wav` or `.spec` files as reference
- Trace spectrograms with Bezier curves + vertical profiles
- Real-time audio preview (procedural vs. original)
- Undo/Redo support (50-action history)
- Export to `procedural_params.txt` (re-editable)
- Generate C++ code (copy-paste ready)
- **Live volume control** during playback
- **Reference opacity slider** for mixing original/procedural views
- **Per-curve volume control** for fine-tuning

### Profiles
- **Gaussian:** Smooth harmonic falloff (fully implemented)
- **Decaying Sinusoid:** Resonant/metallic texture (implemented)
- **Noise:** Random texture/grit with decay envelope (implemented)

### Visualization
- **Log-scale frequency axis** (20 Hz to 16 kHz) for better bass visibility
- **Logarithmic dB-scale intensity** for proper dynamic range display
- **Playhead indicator** showing current playback position
- **Mouse crosshair with tooltip** displaying frame and frequency
- **Real-time spectrum viewer** (bottom-right):
  - Shows frequency spectrum for frame under mouse (hover mode)
  - Shows frequency spectrum for current playback frame (playback mode)
  - Dual display: Reference (green) and Procedural (red) overlaid
  - Always visible for instant feedback

## Quick Start

1. **Open the editor:**
   ```bash
   open tools/spectral_editor/index.html
   ```
   (Or open in your browser via file:// protocol)

2. **Load a reference sound:**
   - Click "Load .wav/.spec" or press `Ctrl+O`
   - Select a `.wav` or `.spec` file

3. **Add a curve:**
   - Click "Add Curve" button
   - Click on canvas to place control points
   - Drag control points to adjust frequency and amplitude

4. **Adjust profile:**
   - Use "Sigma" slider to control width
   - Higher sigma = wider frequency spread

5. **Preview audio:**
   - Press `1` to play procedural sound
   - Press `2` to play original .wav
   - Press `Space` to stop

6. **Export:**
   - `Ctrl+S` → Save `procedural_params.txt` (re-editable)
   - `Ctrl+Shift+S` → Generate C++ code

## Keyboard Shortcuts

| Key | Action |
|-----|--------|
| **1** | Play procedural sound |
| **2** | Play original .wav |
| **Space** | Stop playback |
| **Delete** | Delete selected control point |
| **Esc** | Deselect all |
| **Ctrl+Z** | Undo |
| **Ctrl+Shift+Z** | Redo |
| **Ctrl+S** | Save procedural_params.txt |
| **Ctrl+Shift+S** | Generate C++ code |
| **Ctrl+O** | Open file |
| **?** | Show help |

## Mouse Controls

- **Click** on canvas: Place control point
- **Drag** control point: Adjust position (frame, frequency, amplitude)
- **Right-click** control point: Delete

## Workflow

### Example: Create a Kick Drum

1. Load a reference kick drum (e.g., `kick.wav`)
2. Add a curve
3. Place control points to trace the low-frequency punch:
   - Point 1: Frame 0, ~200 Hz, amplitude 0.9
   - Point 2: Frame 20, ~80 Hz, amplitude 0.7
   - Point 3: Frame 100, ~50 Hz, amplitude 0.0
4. Adjust sigma to ~30 (smooth falloff)
5. Press `1` to preview
6. Fine-tune control points
7. Export C++ code

### Generated C++ Code Example

```cpp
// Generated by Spectral Brush Editor
#include "audio/spectral_brush.h"

void gen_kick_procedural(float* spec, int dct_size, int num_frames) {
  // Curve 0
  {
    const float frames[] = {0.0f, 20.0f, 100.0f};
    const float freqs[] = {200.0f, 80.0f, 50.0f};
    const float amps[] = {0.900f, 0.700f, 0.000f};

    draw_bezier_curve(spec, dct_size, num_frames,
                      frames, freqs, amps, 3,
                      PROFILE_GAUSSIAN, 30.00f);
  }
}

// Usage in demo_assets.txt:
// KICK_PROC, PROC(gen_kick_procedural), NONE, "Procedural kick drum"
```

## File Formats

### procedural_params.txt (Re-editable)

Human-readable text format that can be loaded back into the editor:

```
# Spectral Brush Procedural Parameters
METADATA dct_size=512 num_frames=100 sample_rate=32000

CURVE bezier
  CONTROL_POINT 0 200.0 0.900
  CONTROL_POINT 20 80.0 0.700
  CONTROL_POINT 100 50.0 0.000
  PROFILE gaussian sigma=30.0
END_CURVE
```

### C++ Code (Ready to Compile)

Generated code using the spectral_brush runtime API. Copy-paste into `src/audio/procedural_samples.cc`.

## Technical Details

### Spectral Brush Primitive

A spectral brush consists of:

1. **Central Curve** (Bezier): Traces a path through time-frequency space
   - `{freq_bin, amplitude} = bezier(frame_number)`
   - Control points: `(frame, freq_hz, amplitude)`

2. **Vertical Profile**: Shapes the "brush stroke" around the central curve
   - Gaussian: `exp(-(dist² / σ²))`
   - Applied vertically at each frame

### Coordinate System

- **X-axis (Time):** Frame number (0 → num_frames)
- **Y-axis (Frequency):** Frequency in Hz (0 → 16 kHz for 32 kHz sample rate)
- **Amplitude:** Controlled by Y-position of control points (0.0-1.0)

### Audio Synthesis

1. Generate procedural spectrogram (DCT coefficients)
2. Apply IDCT to convert to time-domain audio
3. Use overlap-add with Hanning window
4. Play via Web Audio API (32 kHz sample rate)

## Development Status

### Phase 1: C++ Runtime ✅ COMPLETE
- Spectral brush primitive (Bezier + profiles)
- C++ implementation in `src/audio/spectral_brush.{h,cc}`
- Integration with asset system

### Phase 2: Web Editor ✅ COMPLETE (Milestone: February 6, 2026)
- Full-featured web-based editor
- Real-time audio preview and visualization
- Log-scale frequency display with dB-scale intensity
- All three profile types (Gaussian, Decaying Sinusoid, Noise)
- Live volume control and reference opacity mixing
- Real-time dual-spectrum viewer (reference + procedural)
- Export to `procedural_params.txt` and C++ code

### Phase 3: Advanced Features (TODO)

**High Priority:**
- **Effect Combination System:** How to combine effects (e.g., noise + Gaussian modulation)?
  - Layer-based compositing (add/multiply/subtract)
  - Profile modulation (noise modulated by Gaussian envelope)
  - Multi-pass rendering pipeline

- **Improved C++ Code Testing:**
  - Verify generated code compiles correctly
  - Test parameter ranges and edge cases
  - Add validation warnings in editor

- **Better Frequency Scale:**
  - Current log-scale is too bass-heavy
  - Investigate mu-law or similar perceptual scales
  - Allow user-configurable frequency mapping

- **Pre-defined Shape Library:**
  - Template curves for common sounds (kick, snare, hi-hat, bass)
  - One-click insertion with adjustable parameters
  - Save/load custom shape presets

**Future Enhancements:**
- Cubic Bezier interpolation (smoother curves)
- Multi-dimensional Bezier (vary decay, oscillation over time)
- Frequency snapping (snap to musical notes/scales)
- Load `procedural_params.txt` back into editor (re-editing)
- Frame cache optimization for faster rendering
- FFT-based DCT optimization (O(N log N) vs O(N²))

## Troubleshooting

**Q: Audio doesn't play**
- Check browser console for errors
- Ensure audio context initialized (some browsers require user interaction first)
- Try clicking canvas before pressing `1` or `2`

**Q: Canvas is blank**
- Make sure you loaded a reference file (`.wav` or `.spec`)
- Check console for file loading errors

**Q: Exported code doesn't compile**
- Ensure `spectral_brush.h/cc` is built and linked
- Verify `draw_bezier_curve()` function is available
- Check include paths in your build system

**Q: Generated sound doesn't match original**
- Adjust sigma (profile width)
- Add more control points for finer detail
- Use multiple curves for complex sounds

## Integration with Demo

1. Generate C++ code from editor
2. Copy code into `src/audio/procedural_samples.cc`
3. Add entry to `assets/final/demo_assets.txt`:
   ```
   SOUND_PROC, PROC(gen_procedural), NONE, "Procedural sound"
   ```
4. Rebuild demo
5. Use `AssetId::SOUND_PROC` in your code

## Browser Compatibility

- **Tested:** Chrome 90+, Firefox 88+, Edge 90+, Safari 14+
- **Requirements:** Web Audio API support
- **Recommended:** Desktop browser (mobile support limited)

## License

Part of the 64k demo project. See project LICENSE.