diff options
| author | skal <pascal.massimino@gmail.com> | 2026-02-15 09:16:06 +0100 |
|---|---|---|
| committer | skal <pascal.massimino@gmail.com> | 2026-02-15 09:16:06 +0100 |
| commit | e94ff513689c6477e9c756f58370e4f3490dd675 (patch) | |
| tree | c0b4deb886efe33e1bd242e0dbd639527656e86f /tools | |
| parent | c98286860885d1f025cd8cf9da699f174118ccba (diff) | |
feat(spectral-editor): add waveform intensity viewer for sample offset
Add interactive waveform timeline for determining SAMPLE OFFSET values:
Features:
- RMS envelope visualization (10ms windows, normalized)
- Uses synthesized PCM from spectrogram (not original WAV)
- Draggable offset marker with numeric input (0.001s precision)
- Snap-to-onset: auto-detects first transient (threshold 0.01)
- Copy button: generates `SAMPLE <name> OFFSET <seconds>` command
- Top panel (120px) with controls, z-indexed above spectrogram
Design rationale:
- Offset measured on procedural output (matches runtime behavior)
- Interactive workflow: load .spec → inspect → set offset → copy
- Supports tracker compile-time SAMPLE OFFSET feature
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Diffstat (limited to 'tools')
| -rw-r--r-- | tools/spectral_editor/README.md | 51 | ||||
| -rw-r--r-- | tools/spectral_editor/index.html | 29 | ||||
| -rw-r--r-- | tools/spectral_editor/script.js | 247 | ||||
| -rw-r--r-- | tools/spectral_editor/style.css | 85 |
4 files changed, 405 insertions, 7 deletions
diff --git a/tools/spectral_editor/README.md b/tools/spectral_editor/README.md index 6bb3681..95b4769 100644 --- a/tools/spectral_editor/README.md +++ b/tools/spectral_editor/README.md @@ -39,6 +39,17 @@ Replace large `.spec` binary assets with tiny procedural C++ code: - Dual display: Reference (green) and Procedural (red) overlaid - Always visible for instant feedback +### Sample Offset Tool +- **Waveform intensity timeline** (top panel): + - RMS envelope calculated from synthesized PCM (10ms windows) + - Visualizes procedural audio output, not original WAV + - Click or drag to set offset marker +- **Offset controls**: + - Numeric input (0.001s precision) + - Copy button: Generates `SAMPLE <name> OFFSET <seconds>` for .track files + - Snap-to-onset: Auto-detects first transient above threshold (0.01) +- **Use case:** Determine intrinsic sample offset for tracker SAMPLE declarations + ## Quick Start 1. **Open the editor:** @@ -153,6 +164,46 @@ Generated code using the spectral_brush runtime API. Copy-paste into `src/audio/ ## Technical Details +### Sample Offset Design + +**Problem:** Tracker system needs compile-time offsets to align procedural samples with beat grid. Requires manual inspection to identify where audio "truly" begins. + +**Solution:** Waveform intensity viewer that visualizes RMS envelope of synthesized audio. + +**Key Design Decisions:** + +1. **Use Synthesized PCM, Not Original WAV:** + - Offset applies to procedural output (what's heard in demo) + - Calls `spectrogramToAudio()` to generate PCM via IDCT + - Ensures offset measurement matches runtime behavior + +2. **RMS Envelope (10ms windows):** + - Smooths out sample-level noise + - Provides clear visual representation of attack envelope + - Normalized to max intensity for consistent display + +3. **Interactive Offset Marker:** + - Red dashed line draggable across timeline + - Click anywhere to jump to time position + - Direct feedback on offset value (4 decimal places) + +4. **Auto-Detection (Snap-to-Onset):** + - Simple threshold-based onset detection (|sample| > 0.01) + - Finds first significant transient + - Good starting point, can be manually refined + +5. **Copy Command Format:** + - Generates `SAMPLE <name> OFFSET <seconds>` for .track files + - Clipboard integration for fast workflow + - Infers sample name from loaded filename + +**Workflow:** +1. Load .spec file +2. Inspect waveform envelope (top panel) +3. Click/drag or snap-to-onset to set offset +4. Copy command → paste into .track file +5. Rebuild demo with sample offset baked in + ### Spectral Brush Primitive A spectral brush consists of: diff --git a/tools/spectral_editor/index.html b/tools/spectral_editor/index.html index f6c1def..75658ae 100644 --- a/tools/spectral_editor/index.html +++ b/tools/spectral_editor/index.html @@ -22,14 +22,29 @@ <div class="main-content"> <!-- Canvas area (left side, 80% width) --> <div class="canvas-container"> - <canvas id="spectrogramCanvas"></canvas> - <div id="canvasOverlay" class="canvas-overlay"> - <p>Load a .wav or .spec file to begin</p> - <p class="hint">Click "Load .wav/.spec" button or press Ctrl+O</p> + <!-- Waveform intensity timeline (top, fixed height) --> + <div class="waveform-container"> + <canvas id="waveformCanvas"></canvas> + <div class="waveform-controls"> + <label>Offset:</label> + <input type="number" id="offsetValue" class="offset-input" min="0" max="10" value="0" step="0.001"> + <span>s</span> + <button id="copyOffsetBtn" class="btn-copy" title="Copy SAMPLE line">📋</button> + <button id="snapToOnsetBtn" class="btn-snap" title="Snap to onset">🎯</button> + </div> </div> - <!-- Mini spectrum viewer (bottom-right overlay) --> - <div id="spectrumViewer" class="spectrum-viewer"> - <canvas id="spectrumCanvas" width="200" height="100"></canvas> + + <!-- Spectrogram canvas (below waveform) --> + <div class="spectrogram-wrapper"> + <canvas id="spectrogramCanvas"></canvas> + <div id="canvasOverlay" class="canvas-overlay"> + <p>Load a .wav or .spec file to begin</p> + <p class="hint">Click "Load .wav/.spec" button or press Ctrl+O</p> + </div> + <!-- Mini spectrum viewer (bottom-right overlay) --> + <div id="spectrumViewer" class="spectrum-viewer"> + <canvas id="spectrumCanvas" width="200" height="100"></canvas> + </div> </div> </div> diff --git a/tools/spectral_editor/script.js b/tools/spectral_editor/script.js index 70e55ec..4fff23c 100644 --- a/tools/spectral_editor/script.js +++ b/tools/spectral_editor/script.js @@ -18,6 +18,8 @@ const state = { referenceSpectrogram: null, // Float32Array or null referenceDctSize: DCT_SIZE, referenceNumFrames: 0, + originalAudioData: null, // Float32Array of raw audio samples + audioSampleRate: SAMPLE_RATE, // Procedural curves curves: [], // Array of {id, controlPoints: [{frame, freqHz, amplitude}], profile: {type, param1, param2}} @@ -52,6 +54,10 @@ const state = { mouseFrame: 0, mouseFreq: 0, + // Sample offset + sampleOffset: 0.0, // Offset in seconds + isDraggingOffset: false, + // Undo/Redo history: [], historyIndex: -1, @@ -64,6 +70,7 @@ const state = { document.addEventListener('DOMContentLoaded', () => { initCanvas(); + initWaveformCanvas(); initUI(); initKeyboardShortcuts(); initAudioContext(); @@ -101,6 +108,26 @@ function initCanvas() { canvas.addEventListener('wheel', onCanvasWheel, { passive: false }); } +function initWaveformCanvas() { + const canvas = document.getElementById('waveformCanvas'); + const container = canvas.parentElement; + + const resizeCanvas = () => { + canvas.width = container.clientWidth; + canvas.height = 100; // Fixed height for waveform display + renderWaveform(); + }; + + window.addEventListener('resize', resizeCanvas); + resizeCanvas(); + + // Mouse handlers for offset marker dragging + canvas.addEventListener('mousedown', onWaveformMouseDown); + canvas.addEventListener('mousemove', onWaveformMouseMove); + canvas.addEventListener('mouseup', onWaveformMouseUp); + canvas.addEventListener('mouseleave', onWaveformMouseUp); +} + function initUI() { // File loading document.getElementById('loadWavBtn').addEventListener('click', () => { @@ -132,6 +159,11 @@ function initUI() { document.getElementById('playOriginalBtn').addEventListener('click', () => playAudio('original')); document.getElementById('stopBtn').addEventListener('click', stopAudio); + // Offset controls + document.getElementById('offsetValue').addEventListener('input', onOffsetChanged); + document.getElementById('copyOffsetBtn').addEventListener('click', copyOffsetCommand); + document.getElementById('snapToOnsetBtn').addEventListener('click', snapToOnset); + // Action buttons document.getElementById('undoBtn').addEventListener('click', undo); document.getElementById('redoBtn').addEventListener('click', redo); @@ -417,10 +449,21 @@ function onReferenceLoaded(fileName) { state.viewportOffsetX = 0; // Reset pan state.viewportOffsetY = 0; + // Reset offset + state.sampleOffset = 0.0; + updateOffsetUI(); + + // Update offset input max based on spectrogram duration + const hopSize = state.referenceDctSize / 2; + const audioLength = state.referenceNumFrames * hopSize + state.referenceDctSize; + const duration = audioLength / SAMPLE_RATE; + document.getElementById('offsetValue').max = duration.toFixed(3); + updateCurveUI(); updateUndoRedoButtons(); render(); drawSpectrumViewer(); // Show initial spectrum + renderWaveform(); // Render waveform intensity view } // ============================================================================ @@ -1710,3 +1753,207 @@ function showHelp() { function hideHelp() { document.getElementById('helpModal').style.display = 'none'; } + +// ============================================================================ +// Waveform Intensity Timeline +// ============================================================================ + +function renderWaveform() { + const canvas = document.getElementById('waveformCanvas'); + if (!canvas) return; + + const ctx = canvas.getContext('2d'); + const width = canvas.width; + const height = canvas.height; + + // Clear canvas + ctx.fillStyle = '#252526'; + ctx.fillRect(0, 0, width, height); + + // If no spectrogram loaded, show message + if (!state.referenceSpectrogram) { + ctx.fillStyle = '#858585'; + ctx.font = '12px sans-serif'; + ctx.textAlign = 'center'; + ctx.fillText('Load .spec file to view intensity envelope', width / 2, height / 2); + return; + } + + // Synthesize audio from reference spectrogram + const audioData = spectrogramToAudio( + state.referenceSpectrogram, + state.referenceDctSize, + state.referenceNumFrames + ); + const sampleRate = SAMPLE_RATE; + const duration = audioData.length / sampleRate; + const windowSize = Math.floor(sampleRate * 0.01); // 10ms windows + const numWindows = Math.ceil(audioData.length / windowSize); + const intensityEnvelope = new Float32Array(numWindows); + + for (let i = 0; i < numWindows; ++i) { + const start = i * windowSize; + const end = Math.min(start + windowSize, audioData.length); + let sum = 0; + for (let j = start; j < end; ++j) { + sum += audioData[j] * audioData[j]; + } + intensityEnvelope[i] = Math.sqrt(sum / (end - start)); + } + + // Find max for normalization + const maxIntensity = Math.max(...intensityEnvelope); + + // Draw intensity curve + ctx.strokeStyle = '#4ec9b0'; + ctx.lineWidth = 2; + ctx.beginPath(); + + for (let i = 0; i < numWindows; ++i) { + const x = (i / numWindows) * width; + const normalized = maxIntensity > 0 ? intensityEnvelope[i] / maxIntensity : 0; + const y = height - (normalized * (height - 20)) - 10; + + if (i === 0) { + ctx.moveTo(x, y); + } else { + ctx.lineTo(x, y); + } + } + ctx.stroke(); + + // Draw offset marker + const offsetX = (state.sampleOffset / duration) * width; + ctx.strokeStyle = '#ff6b6b'; + ctx.lineWidth = 2; + ctx.setLineDash([5, 5]); + ctx.beginPath(); + ctx.moveTo(offsetX, 0); + ctx.lineTo(offsetX, height); + ctx.stroke(); + ctx.setLineDash([]); + + // Draw time labels + ctx.fillStyle = '#858585'; + ctx.font = '10px monospace'; + ctx.textAlign = 'left'; + const numLabels = 5; + for (let i = 0; i <= numLabels; ++i) { + const t = (i / numLabels) * duration; + const x = (i / numLabels) * width; + ctx.fillText(t.toFixed(2) + 's', x + 2, height - 2); + } +} + +function onWaveformMouseDown(e) { + const canvas = document.getElementById('waveformCanvas'); + const rect = canvas.getBoundingClientRect(); + const x = e.clientX - rect.left; + const width = canvas.width; + + if (!state.referenceSpectrogram) return; + + // Calculate duration from spectrogram + const hopSize = state.referenceDctSize / 2; + const audioLength = state.referenceNumFrames * hopSize + state.referenceDctSize; + const duration = audioLength / SAMPLE_RATE; + const offsetX = (state.sampleOffset / duration) * width; + + // Check if clicking near offset marker (within 5 pixels) + if (Math.abs(x - offsetX) < 5) { + state.isDraggingOffset = true; + } else { + // Click anywhere to set offset + state.sampleOffset = (x / width) * duration; + updateOffsetUI(); + renderWaveform(); + } +} + +function onWaveformMouseMove(e) { + if (!state.isDraggingOffset || !state.referenceSpectrogram) return; + + const canvas = document.getElementById('waveformCanvas'); + const rect = canvas.getBoundingClientRect(); + const x = Math.max(0, Math.min(e.clientX - rect.left, canvas.width)); + + // Calculate duration from spectrogram + const hopSize = state.referenceDctSize / 2; + const audioLength = state.referenceNumFrames * hopSize + state.referenceDctSize; + const duration = audioLength / SAMPLE_RATE; + + state.sampleOffset = (x / canvas.width) * duration; + updateOffsetUI(); + renderWaveform(); +} + +function onWaveformMouseUp() { + state.isDraggingOffset = false; +} + +function onOffsetChanged(e) { + const value = parseFloat(e.target.value); + if (!isNaN(value) && value >= 0) { + state.sampleOffset = value; + renderWaveform(); + } +} + +function updateOffsetUI() { + document.getElementById('offsetValue').value = state.sampleOffset.toFixed(4); +} + +function copyOffsetCommand() { + if (!state.referenceSpectrogram) { + alert('No spectrogram loaded'); + return; + } + + // Extract filename from current file info (if available) + const fileInfo = document.getElementById('fileInfo').textContent; + const sampleName = fileInfo ? fileInfo.split('.')[0] : 'sample_name'; + + const command = `SAMPLE ${sampleName} OFFSET ${state.sampleOffset.toFixed(4)}`; + + navigator.clipboard.writeText(command).then(() => { + // Visual feedback + const btn = document.getElementById('copyOffsetBtn'); + const originalText = btn.textContent; + btn.textContent = '✓'; + setTimeout(() => { + btn.textContent = originalText; + }, 1000); + }).catch(err => { + alert('Failed to copy to clipboard: ' + err); + }); +} + +function snapToOnset() { + if (!state.referenceSpectrogram) { + alert('No spectrogram loaded'); + return; + } + + // Synthesize audio from spectrogram + const audioData = spectrogramToAudio( + state.referenceSpectrogram, + state.referenceDctSize, + state.referenceNumFrames + ); + const sampleRate = SAMPLE_RATE; + + // Simple onset detection: find first sample above threshold + const threshold = 0.01; // Adjust as needed + let onsetSample = 0; + + for (let i = 0; i < audioData.length; ++i) { + if (Math.abs(audioData[i]) > threshold) { + onsetSample = i; + break; + } + } + + state.sampleOffset = onsetSample / sampleRate; + updateOffsetUI(); + renderWaveform(); +} diff --git a/tools/spectral_editor/style.css b/tools/spectral_editor/style.css index fa71d1d..48f7463 100644 --- a/tools/spectral_editor/style.css +++ b/tools/spectral_editor/style.css @@ -60,6 +60,8 @@ header h1 { position: relative; background: #1e1e1e; border-right: 1px solid #3e3e42; + display: flex; + flex-direction: column; } #spectrogramCanvas { @@ -506,3 +508,86 @@ header h1 { ::-webkit-scrollbar-thumb:hover { background: #4e4e52; } + +/* Waveform intensity viewer */ +.waveform-container { + position: relative; + height: 120px; + background: #252526; + border-bottom: 1px solid #3e3e42; + display: flex; + flex-direction: column; + flex-shrink: 0; + z-index: 10; +} + +#waveformCanvas { + flex: 1; + width: 100%; + display: block; + cursor: pointer; +} + +.waveform-controls { + display: flex; + align-items: center; + gap: 8px; + padding: 8px 12px; + background: #2d2d30; + border-top: 1px solid #3e3e42; + font-size: 12px; +} + +.waveform-controls label { + color: #cccccc; + font-weight: 500; +} + +.offset-input { + width: 80px; + padding: 4px 8px; + background: #3c3c3c; + border: 1px solid #3e3e42; + border-radius: 3px; + color: #d4d4d4; + font-size: 12px; + font-family: 'Consolas', 'Monaco', monospace; +} + +.offset-input:focus { + outline: none; + border-color: #007acc; +} + +.btn-copy, .btn-snap { + padding: 4px 8px; + background: #0e639c; + border: none; + border-radius: 3px; + color: white; + font-size: 12px; + cursor: pointer; + transition: background 0.2s; +} + +.btn-copy:hover, .btn-snap:hover { + background: #1177bb; +} + +.btn-copy:active, .btn-snap:active { + background: #0d5a8f; +} + +.spectrogram-wrapper { + flex: 1; + position: relative; + overflow: hidden; + z-index: 1; +} + +#spectrogramCanvas { + width: 100%; + height: 100%; + display: block; + cursor: crosshair; +} |
