diff options
Diffstat (limited to 'tools/spectral_editor/README.md')
| -rw-r--r-- | tools/spectral_editor/README.md | 268 |
1 files changed, 268 insertions, 0 deletions
diff --git a/tools/spectral_editor/README.md b/tools/spectral_editor/README.md new file mode 100644 index 0000000..6bb3681 --- /dev/null +++ b/tools/spectral_editor/README.md @@ -0,0 +1,268 @@ +# Spectral Brush Editor + +A web-based tool for creating procedural audio by tracing spectrograms with parametric Bezier curves. + +## Purpose + +Replace large `.spec` binary assets with tiny procedural C++ code: +- **Before:** 5 KB binary `.spec` file +- **After:** ~100 bytes of C++ code calling `draw_bezier_curve()` + +**Compression ratio:** 50-100× + +## Features + +### Core Functionality +- Load `.wav` or `.spec` files as reference +- Trace spectrograms with Bezier curves + vertical profiles +- Real-time audio preview (procedural vs. original) +- Undo/Redo support (50-action history) +- Export to `procedural_params.txt` (re-editable) +- Generate C++ code (copy-paste ready) +- **Live volume control** during playback +- **Reference opacity slider** for mixing original/procedural views +- **Per-curve volume control** for fine-tuning + +### Profiles +- **Gaussian:** Smooth harmonic falloff (fully implemented) +- **Decaying Sinusoid:** Resonant/metallic texture (implemented) +- **Noise:** Random texture/grit with decay envelope (implemented) + +### Visualization +- **Log-scale frequency axis** (20 Hz to 16 kHz) for better bass visibility +- **Logarithmic dB-scale intensity** for proper dynamic range display +- **Playhead indicator** showing current playback position +- **Mouse crosshair with tooltip** displaying frame and frequency +- **Real-time spectrum viewer** (bottom-right): + - Shows frequency spectrum for frame under mouse (hover mode) + - Shows frequency spectrum for current playback frame (playback mode) + - Dual display: Reference (green) and Procedural (red) overlaid + - Always visible for instant feedback + +## Quick Start + +1. **Open the editor:** + ```bash + open tools/spectral_editor/index.html + ``` + (Or open in your browser via file:// protocol) + +2. **Load a reference sound:** + - Click "Load .wav/.spec" or press `Ctrl+O` + - Select a `.wav` or `.spec` file + +3. **Add a curve:** + - Click "Add Curve" button + - Click on canvas to place control points + - Drag control points to adjust frequency and amplitude + +4. **Adjust profile:** + - Use "Sigma" slider to control width + - Higher sigma = wider frequency spread + +5. **Preview audio:** + - Press `1` to play procedural sound + - Press `2` to play original .wav + - Press `Space` to stop + +6. **Export:** + - `Ctrl+S` → Save `procedural_params.txt` (re-editable) + - `Ctrl+Shift+S` → Generate C++ code + +## Keyboard Shortcuts + +| Key | Action | +|-----|--------| +| **1** | Play procedural sound | +| **2** | Play original .wav | +| **Space** | Stop playback | +| **Delete** | Delete selected control point | +| **Esc** | Deselect all | +| **Ctrl+Z** | Undo | +| **Ctrl+Shift+Z** | Redo | +| **Ctrl+S** | Save procedural_params.txt | +| **Ctrl+Shift+S** | Generate C++ code | +| **Ctrl+O** | Open file | +| **?** | Show help | + +## Mouse Controls + +- **Click** on canvas: Place control point +- **Drag** control point: Adjust position (frame, frequency, amplitude) +- **Right-click** control point: Delete + +## Workflow + +### Example: Create a Kick Drum + +1. Load a reference kick drum (e.g., `kick.wav`) +2. Add a curve +3. Place control points to trace the low-frequency punch: + - Point 1: Frame 0, ~200 Hz, amplitude 0.9 + - Point 2: Frame 20, ~80 Hz, amplitude 0.7 + - Point 3: Frame 100, ~50 Hz, amplitude 0.0 +4. Adjust sigma to ~30 (smooth falloff) +5. Press `1` to preview +6. Fine-tune control points +7. Export C++ code + +### Generated C++ Code Example + +```cpp +// Generated by Spectral Brush Editor +#include "audio/spectral_brush.h" + +void gen_kick_procedural(float* spec, int dct_size, int num_frames) { + // Curve 0 + { + const float frames[] = {0.0f, 20.0f, 100.0f}; + const float freqs[] = {200.0f, 80.0f, 50.0f}; + const float amps[] = {0.900f, 0.700f, 0.000f}; + + draw_bezier_curve(spec, dct_size, num_frames, + frames, freqs, amps, 3, + PROFILE_GAUSSIAN, 30.00f); + } +} + +// Usage in demo_assets.txt: +// KICK_PROC, PROC(gen_kick_procedural), NONE, "Procedural kick drum" +``` + +## File Formats + +### procedural_params.txt (Re-editable) + +Human-readable text format that can be loaded back into the editor: + +``` +# Spectral Brush Procedural Parameters +METADATA dct_size=512 num_frames=100 sample_rate=32000 + +CURVE bezier + CONTROL_POINT 0 200.0 0.900 + CONTROL_POINT 20 80.0 0.700 + CONTROL_POINT 100 50.0 0.000 + PROFILE gaussian sigma=30.0 +END_CURVE +``` + +### C++ Code (Ready to Compile) + +Generated code using the spectral_brush runtime API. Copy-paste into `src/audio/procedural_samples.cc`. + +## Technical Details + +### Spectral Brush Primitive + +A spectral brush consists of: + +1. **Central Curve** (Bezier): Traces a path through time-frequency space + - `{freq_bin, amplitude} = bezier(frame_number)` + - Control points: `(frame, freq_hz, amplitude)` + +2. **Vertical Profile**: Shapes the "brush stroke" around the central curve + - Gaussian: `exp(-(dist² / σ²))` + - Applied vertically at each frame + +### Coordinate System + +- **X-axis (Time):** Frame number (0 → num_frames) +- **Y-axis (Frequency):** Frequency in Hz (0 → 16 kHz for 32 kHz sample rate) +- **Amplitude:** Controlled by Y-position of control points (0.0-1.0) + +### Audio Synthesis + +1. Generate procedural spectrogram (DCT coefficients) +2. Apply IDCT to convert to time-domain audio +3. Use overlap-add with Hanning window +4. Play via Web Audio API (32 kHz sample rate) + +## Development Status + +### Phase 1: C++ Runtime ✅ COMPLETE +- Spectral brush primitive (Bezier + profiles) +- C++ implementation in `src/audio/spectral_brush.{h,cc}` +- Integration with asset system + +### Phase 2: Web Editor ✅ COMPLETE (Milestone: February 6, 2026) +- Full-featured web-based editor +- Real-time audio preview and visualization +- Log-scale frequency display with dB-scale intensity +- All three profile types (Gaussian, Decaying Sinusoid, Noise) +- Live volume control and reference opacity mixing +- Real-time dual-spectrum viewer (reference + procedural) +- Export to `procedural_params.txt` and C++ code + +### Phase 3: Advanced Features (TODO) + +**High Priority:** +- **Effect Combination System:** How to combine effects (e.g., noise + Gaussian modulation)? + - Layer-based compositing (add/multiply/subtract) + - Profile modulation (noise modulated by Gaussian envelope) + - Multi-pass rendering pipeline + +- **Improved C++ Code Testing:** + - Verify generated code compiles correctly + - Test parameter ranges and edge cases + - Add validation warnings in editor + +- **Better Frequency Scale:** + - Current log-scale is too bass-heavy + - Investigate mu-law or similar perceptual scales + - Allow user-configurable frequency mapping + +- **Pre-defined Shape Library:** + - Template curves for common sounds (kick, snare, hi-hat, bass) + - One-click insertion with adjustable parameters + - Save/load custom shape presets + +**Future Enhancements:** +- Cubic Bezier interpolation (smoother curves) +- Multi-dimensional Bezier (vary decay, oscillation over time) +- Frequency snapping (snap to musical notes/scales) +- Load `procedural_params.txt` back into editor (re-editing) +- Frame cache optimization for faster rendering +- FFT-based DCT optimization (O(N log N) vs O(N²)) + +## Troubleshooting + +**Q: Audio doesn't play** +- Check browser console for errors +- Ensure audio context initialized (some browsers require user interaction first) +- Try clicking canvas before pressing `1` or `2` + +**Q: Canvas is blank** +- Make sure you loaded a reference file (`.wav` or `.spec`) +- Check console for file loading errors + +**Q: Exported code doesn't compile** +- Ensure `spectral_brush.h/cc` is built and linked +- Verify `draw_bezier_curve()` function is available +- Check include paths in your build system + +**Q: Generated sound doesn't match original** +- Adjust sigma (profile width) +- Add more control points for finer detail +- Use multiple curves for complex sounds + +## Integration with Demo + +1. Generate C++ code from editor +2. Copy code into `src/audio/procedural_samples.cc` +3. Add entry to `assets/final/demo_assets.txt`: + ``` + SOUND_PROC, PROC(gen_procedural), NONE, "Procedural sound" + ``` +4. Rebuild demo +5. Use `AssetId::SOUND_PROC` in your code + +## Browser Compatibility + +- **Tested:** Chrome 90+, Firefox 88+, Edge 90+, Safari 14+ +- **Requirements:** Web Audio API support +- **Recommended:** Desktop browser (mobile support limited) + +## License + +Part of the 64k demo project. See project LICENSE. |
