1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
|
# MQ Spectral Editor
McAulay-Quatieri sinusoidal analysis and synthesis tool.
## Usage
```bash
open tools/mq_editor/index.html
```
1. Click **Open WAV** (or **⚗ Test WAV** for a built-in 440+660 Hz test signal)
2. Click **Extract Partials** (optional — Explore/Contour modes work immediately after load)
3. Press **1** to play synthesized, **2** to play original
## UI
- **Top bar:** title + loaded filename
- **Toolbar:** file, extraction, playback controls and parameters
- **Main view:** log-scale time-frequency spectrogram with partial trajectories
- **Right panel:** per-partial mode toggle (Sinusoid / Resonator), synth params, global checkboxes
- **Mini-spectrum** (bottom-right overlay): FFT slice at mouse/playhead time
- Blue/orange gradient: original spectrum; green/yellow: synthesized
- Green bars: detected spectral peaks
- Red lines (2px): partial f0 positions at current time
## Parameters
Parameters are grouped into four sections in the toolbar.
### STFT
| Param | Default | Description |
|-------|---------|-------------|
| **Hop** | 256 | STFT hop size in samples. Smaller = finer time resolution, more frames, slower. |
### Peak Detection
| Param | Default | Description |
|-------|---------|-------------|
| **Threshold (dB)** | −20 | Minimum spectral peak amplitude. Peaks below this are ignored. |
| **Prominence (dB)** | 1.0 | How much a peak must rise above its surrounding valley. Suppresses weak shoulders. |
| **f·Power** | off | Weight spectrum by `f × Power(f)` before detection. Boosts high-frequency peaks relative to low-frequency ones. |
### Tracking
| Param | Default | Description |
|-------|---------|-------------|
| **Birth** | 3 | Frames a candidate must persist before becoming a partial. Higher = fewer spurious bursts. |
| **Death** | 5 | Frames a partial can go unmatched before termination. Higher = bridges short gaps. |
| **Phase Wt** | 2.0 | Weight of phase prediction error in the peak-matching cost function. Higher = stricter phase coherence. |
| **Min Len** | 10 | Minimum frame count for a partial to survive after tracking. Discards very short partials. |
### Filter
| Param | Default | Description |
|-------|---------|-------------|
| **Keep %** | 100% | Retain only the strongest N% of extracted partials by peak amplitude. |
## Interactive Partial Creation
Three ways to add partials beyond automatic extraction:
### ⊕ Explore — Peak Tracking (key: `X`)
Hover the mouse over any spectral peak in the spectrogram. The tracker
snaps to the nearest detected peak and runs MQ tracking forward and
backward from the cursor position, showing an **orange dashed preview**.
Click to commit the partial.
- Works immediately after WAV load (no extraction required)
- Uses **Birth / Death / Phase Wt** params from the toolbar
- Good for: strong peaks missed by global extraction, fine-grained control
### ≋ Contour — Iso-Energy Tracking (key: `C`)
Hover over any region of the spectrogram. The tracker follows the
iso-energy contour at the energy level under the cursor (`E₀ = mag(t, f)`),
frame by frame, without requiring a spectral peak. Preview shown in **cyan**.
Click to commit.
- Designed for **broad, diffuse bass regions** where no peaks are detected
- Search window: ±15% in frequency; declares a miss if nearest bin deviates >15 dB
- Spread is auto-detected on commit (naturally comes out large for broad regions)
- Good for: bass smear, noisy resonances, sub-threshold energy
Both modes are mutually exclusive. `Escape` exits either mode.
Committed partials are prepended to the partial list with full undo support.
## Keyboard Shortcuts
| Key | Action |
|-----|--------|
| `1` | Play synthesized audio |
| `2` | Play original audio |
| `3` | Play selected partial alone |
| `E` | Extract Partials |
| `N` | New partial (flat 440 Hz, full duration) |
| `X` | Toggle ⊕ Explore mode (peak tracking) |
| `C` | Toggle ≋ Contour mode (iso-energy tracking) |
| `P` | Toggle raw peak overlay |
| `A` | Toggle mini-spectrum: original ↔ synthesized |
| `Esc` | Exit explore/contour mode · deselect partial |
| `Ctrl+Z` | Undo |
| `Ctrl+Y` / `Ctrl+Shift+Z` | Redo |
| Shift+scroll | Zoom time axis |
| Scroll | Pan time axis |
## Architecture
- `index.html` — UI, playback, extraction orchestration, keyboard shortcuts
- `editor.js` — Property panel and amplitude bezier editor for selected partials
- `fft.js` — Cooley-Tukey radix-2 FFT + STFT cache
- `mq_extract.js` — MQ algorithm: peak detection, forward tracking, backward expansion, bezier fitting
- `mq_synth.js` — Oscillator bank + two-pole resonator synthesis from extracted partials
- `viewer.js` — Visualization: coordinate API, spectrogram, partials, mini-spectrum, mouse/zoom
### viewer.js coordinate API
| Method | Description |
|--------|-------------|
| `timeToX(t)` | time → canvas X (current view) |
| `canvasToTime(x)` | canvas X → time |
| `freqToY(f)` | freq → canvas Y (log scale) |
| `canvasToFreq(y)` | canvas Y → freq |
| `freqLogNorm(f)` | freq → normalized [0..1] log position |
| `normToFreq(n)` | normalized [0..1] → freq |
| `normalizeDB(db, maxDB)` | dB → intensity [0..1] over 80 dB range |
| `partialColor(p)` | partial index → display color |
## Post-Synthesis Filters
Global LP/HP filters applied after the oscillator bank, before normalization.
Sliders in the **Synthesis** panel; display shows -3 dB cutoff frequency.
| Control | Filter | Formula | Bypass |
|---------|--------|---------|--------|
| **LP k1** | `y[n] = k1·x[n] + (1−k1)·y[n−1]` | `-3dB: cos(ω) = (2−2k−k²)/(2(1−k))` | k1 = 1.0 |
| **HP k2** | `y[n] = k2·(y[n−1] + x[n] − x[n−1])` | `-3dB from peak: cos(ω) = 2k/(1+k²)` | k2 = 1.0 |
Both default to 1.0 (bypass). Frequency display uses `audioBuffer.sampleRate` when loaded, falls back to 44100 Hz.
---
## Resonator Synthesis Mode
Each partial has a **per-partial synthesis mode** selectable in the **Synth** tab:
### Sinusoid (default)
Replica oscillator bank — direct additive sinusoids with optional spread/jitter/decay shaping.
### Resonator
Two-pole IIR bandpass resonator driven by band-limited noise:
```
y[n] = 2r·cos(ω₀)·y[n-1] − r²·y[n-2] + A(t)·√(1−r²)·noise[n]
```
- **ω₀** = `2π·f₀(t)/SR` — recomputed each sample from the freq Bezier curve (handles glides/vibrato)
- **A(t)** — amp Bezier curve scales excitation continuously
- **√(1−r²)** — power normalization, keeps output level ≈ sinusoidal mode at `gainComp = 1`
- **noise[n]** — deterministic per-partial LCG (reproducible renders)
**Parameters:**
| Param | Default | Range | Meaning |
|-------|---------|-------|---------|
| `r (pole)` | 0.995 | [0, 0.9999] | Pole radius. r→1 = narrow BW / long ring. r→0 = wide / fast decay. |
| `gain` | 1.0 | [0, ∞) | Output multiplier on top of power normalization. |
**Coefficient translation from spread:**
`r = exp(−π · BW / SR)` where `BW = f₀ · (spread_above + spread_below) / 2`.
For a partial at 440 Hz with `spread = 0.02`: `BW ≈ 8.8 Hz`, `r ≈ exp(−π·8.8/32000) ≈ 0.9991`.
**When to use:**
- Metallic / percussive partials with natural exponential decay
- Wide spectral peaks (large spread) where a bandpass filter is more physically accurate
- Comparing resonator vs. sinusoidal timbre on the same partial
**Note:** `RES` badge appears in the panel header when a partial is in resonator mode.
---
## Algorithm
1. **STFT:** Overlapping Hann windows, radix-2 FFT
2. **Peak Detection:** Local maxima above threshold + parabolic interpolation. Includes **Prominence Filtering** (rejects peaks not significantly higher than surroundings). Optional `f·Power(f)` weighting.
3. **Forward Tracking:** Phase-coherent birth/death/continuation. Configurable `Birth`, `Death`, `Phase Wt`, and `Min Len`. Uses velocity prediction and phase advance to resolve ambiguous peak matches.
4. **Backward Expansion:** Second pass extends each partial leftward to recover onset frames
5. **Bezier Fitting:** Cubic curves optimized via **Least-Squares** (minimizes error across all points).
## Implementation Status
- [x] Phase 1: MQ extraction + visualization
- [x] Log-scale spectrogram with power-law colormap
- [x] Zoom (shift+scroll) and pan (scroll)
- [x] Axis ticks, mouse tooltip
- [x] Partial tracking with candidate system
- [x] Mini-spectrum overlay with log-scale frequency axis
- [x] Overlay cursor (separate canvas, no main redraw on mousemove)
- [x] Backward pass for onset recovery
- [x] Phase 2: Synthesis preview
- [x] Replica oscillator bank (spread, jitter, phase integration)
- [x] Synthesis debug checkboxes (disable jitter/spread)
- [x] Synthesized STFT cache for FFT comparison
- [x] Phase 3: Editing UI
- [x] Partial selection with property panel (freq/amp/synth tabs)
- [x] Amplitude bezier drag editor
- [x] Synth params with jog sliders (decay, jitter, spread)
- [x] Auto-spread detection per partial and global
- [x] Mute / delete partials
- [x] Per-partial resonator synthesis mode (Synth tab toggle)
- [x] Global LP/HP post-synthesis filter sliders with Hz display
- [x] Explore mode: interactive peak tracking from mouse position (`X`)
- [x] Contour mode: iso-energy tracking for bass/diffuse regions (`C`)
- [ ] Phase 4: Export (.spec + C++ code generation)
## See Also
- Design doc: `doc/SPECTRAL_BRUSH_2.md`
|
