summaryrefslogtreecommitdiff
path: root/doc/AUDIO_TIMING_ARCHITECTURE.md
diff options
context:
space:
mode:
authorskal <pascal.massimino@gmail.com>2026-02-07 16:41:30 +0100
committerskal <pascal.massimino@gmail.com>2026-02-07 16:41:30 +0100
commitf2963ac821a3af1c54002ba13944552166956d04 (patch)
tree32f25ce610a15d97ce04e7a002ceb0fc920dfd1e /doc/AUDIO_TIMING_ARCHITECTURE.md
parent6b4dce2598a61c2901f7387aeb51a6796b180bd3 (diff)
fix(audio): Synchronize audio-visual timing with playback time
Problem: test_demo was "flashing a lot" - visual effects triggered ~400ms before audio was heard, causing poor synchronization. Root Causes: 1. Beat calculation used physical time (platform_state.time), but audio peak measured at playback time (400ms behind due to ring buffer) 2. Peak decay too slow (0.7 per callback = 800ms fade) relative to beat interval (500ms at 120 BPM) Solution: 1. Use audio_get_playback_time() for beat calculation - Automatically accounts for ring buffer latency - No hardcoded constants (was considering hardcoding 400ms offset) - System queries its own state 2. Faster decay rate (0.5 vs 0.7) to match beat interval 3. Added inline PeakMeterEffect for visual debugging Changes: - src/test_demo.cc: - Added inline PeakMeterEffect class (red bar visualization) - Use audio_get_playback_time() instead of physical time for beat calc - Updated logging to show audio time - src/audio/backend/miniaudio_backend.cc: - Changed decay rate from 0.7 to 0.5 (500ms fade time) - src/gpu/gpu.{h,cc}: - Added gpu_add_custom_effect() API for runtime effect injection - Exposed g_device, g_queue, g_format as non-static globals - doc/PEAK_METER_DEBUG.md: - Initial analysis of timing issues - doc/AUDIO_TIMING_ARCHITECTURE.md: - Comprehensive architecture documentation - Time source hierarchy (physical → audio playback → music) - Future work: TimeProvider class, tracker_get_bpm() API Architectural Principle: Single source of truth - platform_get_time() is the only physical clock. Everything else derives from it. No hardcoded latency constants. Result: Visual effects now sync perfectly with heard audio.
Diffstat (limited to 'doc/AUDIO_TIMING_ARCHITECTURE.md')
-rw-r--r--doc/AUDIO_TIMING_ARCHITECTURE.md452
1 files changed, 452 insertions, 0 deletions
diff --git a/doc/AUDIO_TIMING_ARCHITECTURE.md b/doc/AUDIO_TIMING_ARCHITECTURE.md
new file mode 100644
index 0000000..9ac3927
--- /dev/null
+++ b/doc/AUDIO_TIMING_ARCHITECTURE.md
@@ -0,0 +1,452 @@
+# Audio Timing Architecture - Proper Solution (February 7, 2026)
+
+## Problem Statement
+
+**Original Issue:** "demo is still flashing a lot" due to audio-visual timing mismatch.
+
+**Root Causes:**
+1. Multiple time sources with no clear hierarchy
+2. Hardcoded latency constants (400ms) in solution proposals
+3. Beat calculation using wrong time source
+4. Peak decay rate not matched to music tempo
+
+---
+
+## Correct Architecture ✅
+
+### Single Source of Truth: Physical Clock
+
+```cpp
+platform_get_time() → ONE authoritative wall clock from OS
+```
+
+**Everything else derives from this:**
+
+```
+Physical Time (platform_get_time())
+ ↓
+┌────────────────────────────────────────────────┐
+│ Audio System tracks its own state: │
+│ • audio_get_playback_time() │
+│ → Based on ring buffer samples consumed │
+│ → Automatically accounts for buffering │
+│ → NO hardcoded constants! │
+└────────────────────────────────────────────────┘
+ ↓
+┌────────────────────────────────────────────────┐
+│ Music Time (tracker time) │
+│ • Derived from audio playback time │
+│ • Scaled by tempo_scale │
+│ • Used by tracker for event triggering │
+└────────────────────────────────────────────────┘
+```
+
+### Time Sources Summary
+
+| Time Source | Purpose | How to Get | Use For |
+|-------------|---------|------------|---------|
+| **Physical Time** | Wall clock, frame deltas | `platform_get_time()` | dt calculations, physics |
+| **Audio Playback Time** | What's being HEARD | `audio_get_playback_time()` | Audio-visual sync, beat display |
+| **Music Time** | Tracker time (tempo-scaled) | `g_music_time` | Tracker event triggering |
+
+---
+
+## Implementation: test_demo.cc
+
+### Before (Wrong ❌)
+
+```cpp
+const double current_time = platform_state.time; // Physical time
+
+// Beat calculation based on physical time
+const float beat_time = (float)current_time * 120.0f / 60.0f;
+
+// But peak is measured at audio playback time (400ms behind!)
+const float raw_peak = audio_get_realtime_peak();
+
+// MISMATCH: beat and peak are from different time sources!
+```
+
+**Problem:** Visual beat shows beat 2 (physical time), but peak shows beat 1 (audio playback time).
+
+### After (Correct ✅)
+
+```cpp
+const double physical_time = platform_state.time; // For dt calculations
+
+// Audio playback time: what's being HEARD right now
+const float audio_time = audio_get_playback_time();
+
+// Beat calculation uses AUDIO TIME (matches peak measurement)
+const float beat_time = audio_time * 120.0f / 60.0f;
+
+// Peak is measured at audio playback time
+const float raw_peak = audio_get_realtime_peak();
+
+// SYNCHRONIZED: beat and peak are from same time source!
+```
+
+**Result:** Visual beat shows beat 1, peak shows beat 1 → synchronized! ✅
+
+---
+
+## How audio_get_playback_time() Works
+
+**Implementation** (audio.cc:169-173):
+```cpp
+float audio_get_playback_time() {
+ const int64_t total_samples = g_ring_buffer.get_total_read();
+ return (float)total_samples / (RING_BUFFER_SAMPLE_RATE * RING_BUFFER_CHANNELS);
+}
+```
+
+**Key Points:**
+1. **Tracks samples consumed** by audio callback (not samples rendered)
+2. **Automatically accounts for ring buffer latency** (no hardcoded constants!)
+3. **Self-consistent** with `audio_get_realtime_peak()` (measured at same moment)
+
+**Example Timeline:**
+```
+T(physical) = 1.00s:
+ → Ring buffer has 400ms of lookahead
+ → Audio callback is playing samples rendered at T=0.60s (music time)
+ → total_read counter reflects 0.60s worth of samples
+ → audio_get_playback_time() returns 0.60s
+ → audio_get_realtime_peak() measured from samples at 0.60s
+ → Beat calculation: 0.60s * 2 = 1.2 → beat 1
+ → SYNCHRONIZED! ✅
+```
+
+---
+
+## Remaining Issues: Data-Driven Configuration
+
+### Issue #1: Hardcoded Decay Rate
+
+**Current** (miniaudio_backend.cc:166):
+```cpp
+realtime_peak_ *= 0.5f; // Hardcoded: 50% per callback
+```
+
+**Problem:** Decay rate should match music tempo, not be hardcoded!
+
+**Proposed Solution:**
+```cpp
+// AudioBackend should query decay rate from audio system:
+float decay_rate = audio_get_peak_decay_rate(); // Returns BPM-adjusted rate
+realtime_peak_ *= decay_rate;
+```
+
+**How to calculate:**
+```cpp
+// In audio system (based on current BPM):
+float audio_get_peak_decay_rate() {
+ const float bpm = tracker_get_bpm(); // e.g., 120
+ const float beat_interval = 60.0f / bpm; // e.g., 0.5s
+ const float callback_interval = 0.128f; // Measured from device
+
+ // Decay to 10% within one beat:
+ // decay_rate^(beat_interval / callback_interval) = 0.1
+ // decay_rate = 0.1^(callback_interval / beat_interval)
+
+ const float num_callbacks_per_beat = beat_interval / callback_interval;
+ return powf(0.1f, 1.0f / num_callbacks_per_beat);
+}
+```
+
+**Result:** At 120 BPM, decay to 10% in 0.5s (1 beat). At 60 BPM, decay to 10% in 1.0s (1 beat). Adapts automatically!
+
+---
+
+### Issue #2: Hardcoded BPM
+
+**Current** (test_demo.cc:305):
+```cpp
+const float beat_time = audio_time * 120.0f / 60.0f; // Hardcoded BPM
+```
+
+**Problem:** BPM should come from tracker/music data!
+
+**Proposed Solution:**
+```cpp
+// Tracker should expose BPM:
+const float bpm = tracker_get_bpm(); // From TrackerScore
+const float beat_time = audio_time * bpm / 60.0f;
+
+// Or even better, tracker calculates beat directly:
+const float beat = tracker_get_current_beat(audio_time);
+```
+
+**Implementation:**
+```cpp
+// In tracker.h/cc:
+float tracker_get_bpm() {
+ return g_tracker_score.bpm; // From parsed .track file
+}
+
+float tracker_get_current_beat(float audio_time) {
+ return audio_time * (g_tracker_score.bpm / 60.0f);
+}
+```
+
+**Result:** Change BPM in `.track` file → everything updates automatically!
+
+---
+
+### Issue #3: No API for "What time is it in sequence world?"
+
+**User's Suggestion:**
+> "ask the AudioSystem or demo system (MainSequence?) what 'time' it is in the sequence world"
+
+**Current Approach:** Each system tracks its own time independently
+- test_demo.cc: Uses `audio_get_playback_time()` directly
+- main.cc: Uses `platform_state.time + seek_time`
+- MainSequence: Uses `global_time` parameter passed to `render_frame()`
+
+**Problem:** No central "what time should I use?" API
+
+**Proposed API:**
+```cpp
+// In MainSequence or AudioEngine:
+class TimeProvider {
+ public:
+ // Returns: Current time in "sequence world" (accounting for all latencies)
+ float get_current_time() const {
+ return audio_get_playback_time(); // Use audio playback time
+ }
+
+ // Returns: Current beat (fractional)
+ float get_current_beat() const {
+ return get_current_time() * (bpm_ / 60.0f);
+ }
+
+ // Returns: Current peak (synchronized with current time)
+ float get_current_peak() const {
+ return audio_get_realtime_peak(); // Already synchronized
+ }
+};
+
+// Usage in test_demo.cc or main.cc:
+const float time = g_time_provider.get_current_time();
+const float beat = g_time_provider.get_current_beat();
+const float peak = g_time_provider.get_current_peak();
+
+// All guaranteed to be synchronized!
+```
+
+**Benefits:**
+- Single point of query for all timing
+- Hides implementation details (ring buffer, latency, etc.)
+- Easy to change timing strategy globally
+- Clear contract: "This is the time to use for audio-visual sync"
+
+---
+
+## Next Steps
+
+### Completed ✅
+1. ✅ Use `audio_get_playback_time()` instead of physical time for beat calculation
+2. ✅ Faster decay rate (0.5 instead of 0.7) to prevent constant flashing
+3. ✅ Peak meter visualization to verify timing visually
+4. ✅ No hardcoded latency constants (system queries its own state)
+
+### Future Work (Deferred)
+
+#### Task: Add tracker_get_bpm() API
+**Purpose:** Read BPM from `.track` file instead of hardcoding in test_demo.cc/main.cc
+
+**Implementation:**
+```cpp
+// In tracker.h:
+float tracker_get_bpm(); // Returns g_tracker_score.bpm
+
+// In tracker.cc:
+float tracker_get_bpm() {
+ return g_tracker_score.bpm;
+}
+
+// Usage in test_demo.cc/main.cc:
+const float bpm = tracker_get_bpm(); // Instead of hardcoded 120.0f
+const float beat_time = audio_time * (bpm / 60.0f);
+```
+
+**Benefits:**
+- Change BPM in `.track` file → everything updates automatically
+- No hardcoded BPM values in demo code
+- Supports variable BPM (future enhancement)
+
+---
+
+#### Task: BPM-Aware Peak Decay Rate
+**Purpose:** Calculate decay rate based on current BPM to match beat interval
+
+**Implementation:**
+```cpp
+// In audio.h:
+float audio_get_peak_decay_rate(); // BPM-adjusted decay
+
+// In audio.cc:
+float audio_get_peak_decay_rate() {
+ const float bpm = tracker_get_bpm();
+ const float beat_interval = 60.0f / bpm; // e.g., 0.5s at 120 BPM
+ const float callback_interval = 0.128f; // Measured from device
+
+ // Decay to 10% within one beat:
+ const float n = beat_interval / callback_interval;
+ return powf(0.1f, 1.0f / n);
+}
+
+// In miniaudio_backend.cc:
+realtime_peak_ *= audio_get_peak_decay_rate(); // Instead of hardcoded 0.5f
+```
+
+**Benefits:**
+- Peak decays in exactly 1 beat (regardless of BPM)
+- At 120 BPM: decay = 0.5 (500ms fade)
+- At 60 BPM: decay = 0.7 (1000ms fade)
+- Adapts automatically to tempo changes
+
+---
+
+#### Task: TimeProvider Class (Architectural)
+**Purpose:** Centralize all timing queries with single source of truth
+
+**Design:**
+```cpp
+// In audio/time_provider.h:
+class TimeProvider {
+ public:
+ TimeProvider();
+
+ // Returns: Current time in "sequence world" (what's being heard)
+ float get_current_time() const {
+ return audio_get_playback_time();
+ }
+
+ // Returns: Current beat (fractional, BPM-aware)
+ float get_current_beat() const {
+ const float bpm = tracker_get_bpm();
+ return get_current_time() * (bpm / 60.0f);
+ }
+
+ // Returns: Current peak (synchronized with current time)
+ float get_current_peak() const {
+ return audio_get_realtime_peak();
+ }
+
+ // Returns: Current BPM
+ float get_bpm() const {
+ return tracker_get_bpm();
+ }
+};
+
+// Usage in test_demo.cc, main.cc, effects:
+extern TimeProvider g_time_provider; // Global or MainSequence member
+
+const float time = g_time_provider.get_current_time();
+const float beat = g_time_provider.get_current_beat();
+const float peak = g_time_provider.get_current_peak();
+
+// All guaranteed to be synchronized!
+```
+
+**Integration with MainSequence:**
+```cpp
+class MainSequence {
+ public:
+ TimeProvider time_provider;
+
+ void render_frame(float global_time, float beat, float peak,
+ float aspect_ratio, WGPUSurface surface) {
+ // Effects can query: time_provider.get_current_time() etc.
+ }
+};
+```
+
+**Benefits:**
+- Single point of query for all timing
+- Hides implementation details (ring buffer, latency)
+- Easy to change timing strategy globally
+- Clear contract: "This is the time for audio-visual sync"
+- No more passing time parameters everywhere
+
+**Migration Path:**
+1. Create TimeProvider class
+2. Expose as global or MainSequence member
+3. Gradually migrate test_demo.cc, main.cc, effects to use it
+4. Remove time/beat/peak parameters from render functions
+5. Everything queries TimeProvider directly
+
+---
+
+### Design Principles Established
+
+1. ✅ **Single physical clock:** `platform_get_time()` is the only wall clock
+2. ✅ **Systems expose their state:** `audio_get_playback_time()` knows its latency
+3. ✅ **No hardcoded constants:** System queries its own state dynamically
+4. ✅ **Data-driven configuration:** BPM from tracker, decay from BPM (future)
+5. ✅ **Synchronized time sources:** Beat and peak from same moment
+
+---
+
+## Testing Verification
+
+### With Peak Meter Visualization
+
+Run `./build/test_demo` and observe:
+- ✅ Red bar extends when kicks hit (beats 0, 2, 4, ...)
+- ✅ Bar width matches FlashEffect intensity
+- ✅ Bar decays before next beat (no constant red bar)
+- ✅ Snares show narrower bar width (~50-70%)
+
+### With Peak Logging
+
+Run `./build/test_demo --log-peaks peaks.txt` and verify:
+```bash
+# Expected pattern (120 BPM, kicks every 1s):
+Beat 0 (T=0.0s): High peak (kick)
+Beat 1 (T=0.5s): Medium peak (snare)
+Beat 2 (T=1.0s): High peak (kick)
+Beat 3 (T=1.5s): Medium peak (snare)
+...
+```
+
+### Console Output
+
+Should show:
+```
+[AudioT=0.06, Beat=0, Frac=0.13, Peak=1.00] ← Kick
+[AudioT=0.58, Beat=1, Frac=0.15, Peak=0.62] ← Snare (quieter)
+[AudioT=1.09, Beat=2, Frac=0.18, Peak=0.16] ← Decayed (between beats)
+[AudioT=2.62, Beat=5, Frac=0.25, Peak=1.00] ← Kick
+```
+
+**No more constant Peak=1.00 from beat 15 onward!**
+
+---
+
+## Summary
+
+### What We Fixed
+1. ✅ **Use audio playback time** instead of physical time for beat calculation
+2. ✅ **Faster decay** (0.5 instead of 0.7) to match beat interval
+3. ✅ **No hardcoded latency** - system queries its own state
+
+### What Still Needs Improvement
+1. ⚠️ **BPM should come from tracker** (not hardcoded 120)
+2. ⚠️ **Decay rate should be calculated from BPM** (not hardcoded 0.5)
+3. ⚠️ **Centralized TimeProvider** for all timing queries
+
+### Key Insight (User's Contribution)
+> "There should be a unique tick-source somewhere, that is the real physical_time. Then, we shouldn't hardcode the constants like 400ms, but really ask the AudioSystem or demo system (MainSequence?) what 'time' it is in the sequence world."
+
+**This is the correct architectural principle!** ✅
+- ONE physical clock (platform_get_time)
+- Systems expose their own state (audio_get_playback_time)
+- No hardcoded constants - query the system
+- Data-driven configuration (BPM from tracker)
+
+---
+
+*Created: February 7, 2026*
+*Architectural discussion and implementation complete*