diff options
| author | skal <pascal.massimino@gmail.com> | 2026-02-05 19:13:34 +0100 |
|---|---|---|
| committer | skal <pascal.massimino@gmail.com> | 2026-02-05 19:13:34 +0100 |
| commit | 798fc6d471a70ed930e5b1fc084818cb337ca5b1 (patch) | |
| tree | bf89015f733bc7c52e4b9c37593669cb69ae8bd8 /doc/AUDIO_LIFECYCLE_REFACTOR.md | |
| parent | 4a3f7a2c379a3e9554e720685e03842180b021ce (diff) | |
feat(audio): Implement AudioEngine and SpectrogramResourceManager (Task #56 Phase 1)
Implements Phase 1 of the audio lifecycle refactor to eliminate initialization
order dependencies between synth and tracker.
New Components:
1. SpectrogramResourceManager (src/audio/spectrogram_resource_manager.{h,cc})
- Centralized resource loading and ownership
- Lazy loading: resources registered but not loaded until needed
- Handles both asset spectrograms and procedural notes
- Clear ownership: assets borrowed, procedurals owned
- Optional cache eviction under DEMO_ENABLE_CACHE_EVICTION flag
2. AudioEngine (src/audio/audio_engine.{h,cc})
- Unified audio subsystem manager
- Single initialization point eliminates order dependencies
- Manages synth, tracker, and resource manager lifecycle
- Timeline seeking API for debugging (!STRIP_ALL)
- Clean API: init(), shutdown(), reset(), seek()
Features:
- Lazy loading strategy with manual preload API
- Reset functionality for timeline seeking
- Zero impact on production builds
- Debug-only seeking support
Testing:
- Comprehensive test suite (test_audio_engine.cc)
- Tests lifecycle, resource loading, reset, seeking
- All 20 tests passing (100% pass rate)
Bug Fixes:
- Fixed infinite recursion in AudioEngine::tracker_reset()
Integration:
- Added to CMakeLists.txt audio library
- No changes to existing code (backward compatible)
Binary Size Impact: ~700 bytes (within budget)
Next: Phase 2 (Test Migration) - Update existing tests to use AudioEngine
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Diffstat (limited to 'doc/AUDIO_LIFECYCLE_REFACTOR.md')
| -rw-r--r-- | doc/AUDIO_LIFECYCLE_REFACTOR.md | 447 |
1 files changed, 437 insertions, 10 deletions
diff --git a/doc/AUDIO_LIFECYCLE_REFACTOR.md b/doc/AUDIO_LIFECYCLE_REFACTOR.md index 1d73bb1..70aaff6 100644 --- a/doc/AUDIO_LIFECYCLE_REFACTOR.md +++ b/doc/AUDIO_LIFECYCLE_REFACTOR.md @@ -575,30 +575,49 @@ class Tracker { - Spreads load cost over time (not all at init) - Can load asynchronously in background thread (if needed) -### Cache Eviction Policy (Optional) +### Cache Eviction Policy (Optional, Compile-Time Flag) -**For very long demos (>5 minutes) or memory-constrained systems:** +**Demo knows its timeline:** Effects can explicitly release resources when done. +**Explicit Resource Management (Preferred):** ```cpp +class Effect { + public: + void on_effect_end() override { + // Effect knows it won't need these samples again + resource_mgr_->release(SAMPLE_EXPLOSION); + resource_mgr_->release(SAMPLE_AMBIENT_WIND); + } +}; + +// In sequence compiler: +EFFECT ExplosionEffect 10.0 12.0 // Uses SAMPLE_EXPLOSION +// After t=12.0, explosion sample can be evicted +``` + +**Automatic Eviction (Under `ENABLE_CACHE_EVICTION` flag):** +```cpp +#if defined(ENABLE_CACHE_EVICTION) void try_evict_lru() { if (get_cache_memory_usage() > MAX_CACHE_SIZE) { - // Find least recently used entry int lru_id = find_lru_entry(); - - // Only evict if not used recently if (current_time() - cache_[lru_id].last_access_time > 10.0f) { evict(lru_id); } } } +#endif /* ENABLE_CACHE_EVICTION */ ``` -**Eviction strategies:** -1. **LRU** (Least Recently Used) - evict oldest access -2. **Time-based** - evict samples >10s since last use -3. **Timeline hints** - demo tells cache "won't need sample X again" +**Eviction strategies (all optional):** +1. **Explicit** (recommended) - Effect calls `release()` when done +2. **LRU** (fallback) - Evict least recently used if memory tight +3. **Timeline hints** - Sequence compiler marks "last use" of samples -**For 64k demo:** Probably not needed (short duration, small sample count) +**Recommendation:** +- Production: Explicit release only (demo knows timeline) +- Development: Enable LRU eviction for safety (`-DENABLE_CACHE_EVICTION`) +- Final 64k build: Disable eviction (cache everything, simple code) --- @@ -690,6 +709,12 @@ class AssetManagerAdapter : public IAssetProvider { - [ ] Implement `AudioEngine::init()` / `shutdown()` / `reset()` - [ ] Add delegation methods for synth/tracker APIs - [ ] Integrate ResourceManager into AudioEngine +- [ ] **Implement `AudioEngine::seek()` for timeline scrubbing** + - [ ] Add `Synth::reset()` and `Synth::set_time()` + - [ ] Add `Tracker::reset()` and `Tracker::set_time()` + - [ ] Add `Tracker::update_silent()` for fast-forward + - [ ] Implement pre-warming for target time range + - [ ] Add seeking tests (forward, backward, edge cases) - [ ] Write unit tests for `AudioEngine` lifecycle with resources ### Phase 2: Test Migration (3-5 days) @@ -814,6 +839,304 @@ class SpectrogramResourceManager { --- +## Timeline Seeking & Scrubbing (Critical for Development) + +### Requirement: Robust Seeking for Debugging + +**Need to support:** +- Jump forward: t=5s → t=45s (skip to specific scene) +- Rewind: t=45s → t=5s (go back to debug earlier issue) +- Fast-forward: Simulate t=0-30s without audio (load samples, update state) +- Frame-by-frame: t → t+0.016s (precise debugging) + +**Properties:** +- ✅ Robust: Works for any timeline position +- ✅ Correct: Audio/visuals match target time exactly +- ⚠️ Performance: Doesn't need to be fast (debugging tool) +- ✅ Pre-warming: Must load samples for target time range + +### Implementation: AudioEngine::seek() + +```cpp +class AudioEngine { + public: + // Seek to arbitrary point in timeline + void seek(float target_time) { + #if !defined(STRIP_ALL) // Only in debug builds + + // 1. Reset synth state (clear all active voices) + synth_.reset(); + + // 2. Reset tracker state (re-scan patterns from scratch) + tracker_.reset(); + tracker_.set_time(target_time); + + // 3. Pre-warm samples for target time range + float prewarm_start = std::max(0.0f, target_time - 1.0f); + float prewarm_end = target_time + 2.0f; + tracker_.prewarm_lookahead(prewarm_start, prewarm_end); + + // 4. Simulate tracker up to target time (no audio rendering) + // This ensures tracker internal state is correct + const float dt = 0.1f; // Simulate in 0.1s steps + for (float t = 0.0f; t < target_time; t += dt) { + tracker_.update_silent(t); // Update state only, no audio + } + + // 5. Final update at exact target time + tracker_.update(target_time); + + // 6. Notify effects of seek event (optional) + sequence_.on_seek(target_time); + + current_time_ = target_time; + + #endif /* !defined(STRIP_ALL) */ + } + + // Fast-forward: Simulate time range without rendering + void fast_forward(float from_time, float to_time) { + const float dt = 0.1f; + for (float t = from_time; t < to_time; t += dt) { + tracker_.update_silent(t); + sequence_.update_silent(t, dt); // Update effects without rendering + } + } + + private: + float current_time_ = 0.0f; +}; +``` + +### Tracker Support for Seeking + +```cpp +class Tracker { + public: + void reset() { + // Clear all active patterns + for (auto& pattern : active_patterns_) { + pattern.active = false; + } + last_trigger_idx_ = 0; + } + + void set_time(float time) { + current_music_time_ = time; + + // Re-scan score to find patterns that should be active at this time + for (uint32_t i = 0; i < score_->num_triggers; ++i) { + const auto& trigger = score_->triggers[i]; + + if (trigger.time_sec <= time) { + // This pattern should be active (or was active) + // Determine if still active based on pattern duration + const auto& pattern = patterns_[trigger.pattern_id]; + float pattern_end = trigger.time_sec + get_pattern_duration(pattern); + + if (time < pattern_end) { + // Pattern is currently active, add it + activate_pattern(trigger.pattern_id, trigger.time_sec); + + // Fast-forward pattern to current beat + update_pattern_to_time(trigger.pattern_id, time); + } + + last_trigger_idx_ = i + 1; + } + } + } + + void update_silent(float time) { + // Update tracker state without triggering voices + // Used for fast-forward and seeking + update_internal(time, /*trigger_audio=*/false); + } + + private: + void update_pattern_to_time(int pattern_id, float target_time) { + // Advance pattern's event index to match target time + ActivePattern& active = find_active_pattern(pattern_id); + const TrackerPattern& pattern = patterns_[pattern_id]; + + float elapsed = target_time - active.start_music_time; + float beat_duration = 60.0f / score_->bpm; + float elapsed_beats = elapsed / beat_duration; + + // Advance event index past all events before target time + while (active.next_event_idx < pattern.num_events) { + if (pattern.events[active.next_event_idx].beat > elapsed_beats) { + break; + } + active.next_event_idx++; + } + } +}; +``` + +### Synth Support for Seeking + +```cpp +class Synth { + public: + void reset() { + // Clear all active voices (stop all sound immediately) + for (auto& voice : voices_) { + voice.active = false; + } + + // Reset time tracking + #if !defined(STRIP_ALL) + g_elapsed_time_sec = 0.0f; + #endif + } + + void set_time(float time) { + #if !defined(STRIP_ALL) + g_elapsed_time_sec = time; + #endif + } +}; +``` + +### Effect Sequence Support for Seeking + +```cpp +class Sequence { + public: + void on_seek(float target_time) { + // Notify all effects that might be active at target time + for (auto& layer : layers_) { + for (auto& entry : layer.effects) { + if (entry.start_time <= target_time && target_time < entry.end_time) { + // Effect is active at target time + entry.effect->on_seek(target_time); + } else if (target_time < entry.start_time) { + // Effect hasn't started yet, reset it + entry.effect->on_reset(); + } + } + } + } +}; + +class Effect { + public: + virtual void on_seek(float time) { + // Optional: Reset effect state to match target time + // Example: Particle system re-simulates to target time + } + + virtual void on_reset() { + // Optional: Clear all effect state + } +}; +``` + +### Usage Examples + +**Jump to specific scene:** +```cpp +// User presses 'J' key to jump to scene 3 (at t=45s) +if (key_pressed('J')) { + audio_engine.seek(45.0f); + camera.set_position(scene3_camera_pos); // Sync visuals too +} +``` + +**Rewind to beginning:** +```cpp +if (key_pressed('R')) { + audio_engine.seek(0.0f); +} +``` + +**Frame-by-frame debugging:** +```cpp +if (key_pressed('N')) { // Next frame + float dt = 1.0f / 60.0f; + audio_engine.update(current_time, dt); + current_time += dt; +} + +if (key_pressed('P')) { // Previous frame + float dt = 1.0f / 60.0f; + audio_engine.seek(std::max(0.0f, current_time - dt)); +} +``` + +**Fast-forward to load all samples:** +```cpp +// Pre-load all samples by fast-forwarding through entire demo +void preload_all_samples() { + audio_engine.fast_forward(0.0f, demo_duration); + audio_engine.seek(0.0f); // Reset to beginning +} +``` + +### Seeking Guarantees + +**Invariants after `seek(t)`:** +1. ✅ Tracker state matches time `t` (patterns active, event indices correct) +2. ✅ Synth has no active voices (silent state) +3. ✅ Samples for range [t-1s, t+2s] are pre-warmed +4. ✅ Next `update(t)` will trigger correct events +5. ✅ Audio output matches what would have played at time `t` + +**Edge Cases Handled:** +- Seek to t=0: Reset everything (initial state) +- Seek beyond demo end: Clamp to valid range +- Seek to same time twice: No-op (already there) +- Seek while audio playing: Stop all voices first + +### Performance Characteristics + +**Seek cost:** +- Synth reset: O(MAX_VOICES) ~ 64 voices → <1ms +- Tracker reset: O(MAX_PATTERNS) ~ 20 patterns → <1ms +- Pattern re-scan: O(num_triggers) ~ 100 triggers → <1ms +- Pre-warming: O(samples_in_range) ~ 5 samples → 5-10ms +- Total: **~10-20ms** (acceptable for debugging) + +**Optimization (if needed):** +- Cache tracker state snapshots every 5s +- Seek to nearest snapshot, then fast-forward delta +- Reduces re-scan cost for long demos + +### Testing Seeking Robustness + +```cpp +void test_seeking() { + AudioEngine engine; + engine.init(); + + // Test 1: Forward seek + engine.seek(10.0f); + assert(engine.get_time() == 10.0f); + + // Test 2: Backward seek (rewind) + engine.seek(5.0f); + assert(engine.get_time() == 5.0f); + + // Test 3: Seek to beginning + engine.seek(0.0f); + assert(engine.get_time() == 0.0f); + + // Test 4: Multiple seeks in sequence + engine.seek(20.0f); + engine.seek(30.0f); + engine.seek(15.0f); + assert(engine.get_time() == 15.0f); + + // Test 5: Verify samples loaded correctly after seek + engine.seek(45.0f); + const auto* spec = engine.get_resource_manager().get_spectrogram(SAMPLE_ENDING); + assert(spec != nullptr); // Sample for ending should be loaded +} +``` + +--- + ## FAQ ### Q: How does this handle procedural vs asset spectrograms? @@ -866,6 +1189,44 @@ class AudioEngine { - Cleared automatically on `engine.reset()` - Testable (can verify cache contents) +### Q: How does seeking/scrubbing work for debugging? + +**A:** Robust seek API that resets state and pre-warms samples: + +```cpp +audio_engine.seek(45.0f); // Jump to t=45s + +// What happens internally: +// 1. synth.reset() - Clear all active voices +// 2. tracker.reset() - Clear active patterns +// 3. tracker.set_time(45.0) - Re-scan score, activate patterns at t=45 +// 4. prewarm(43-47s) - Load samples for target range +// 5. update(45.0) - Trigger events at exact time +``` + +**Supported operations:** +- **Jump forward:** `seek(45.0f)` - Skip to specific scene +- **Rewind:** `seek(5.0f)` - Go back to earlier time +- **Fast-forward:** `fast_forward(0, 30)` - Simulate without audio +- **Frame-by-frame:** `seek(t + 0.016)` - Step through frames + +**Performance:** ~10-20ms per seek (acceptable for debugging) + +**Guarantees:** +- ✅ Audio/visuals match target time exactly +- ✅ Tracker state correct (patterns active, event indices) +- ✅ Samples pre-warmed (no stutter on next update) +- ✅ Works for any timeline position (0 → end) + +**Development workflow:** +```cpp +// Press 'J' to jump to problematic scene at t=37s +if (key_pressed('J')) { + audio_engine.seek(37.0f); + // Audio and visuals now at t=37s, ready to debug +} +``` + ### Q: Why not load all samples at init? **A:** Lazy loading with pre-warming is more efficient: @@ -908,6 +1269,72 @@ When assets are compressed, decompress only when needed: --- +## Compile-Time Configuration Flags + +**Optional features controlled by CMake flags:** + +```cmake +# CMakeLists.txt +option(DEMO_ENABLE_CACHE_EVICTION "Enable automatic cache eviction (LRU)" OFF) +option(DEMO_ENABLE_TIMELINE_SEEKING "Enable timeline seeking for debugging" ON) +option(DEMO_ENABLE_ASYNC_LOADING "Enable background thread pre-warming" OFF) +``` + +**Usage in code:** + +```cpp +// Cache eviction (optional, for long demos or low memory) +#if defined(DEMO_ENABLE_CACHE_EVICTION) +void SpectrogramResourceManager::try_evict_lru() { + // ... LRU eviction logic +} +#endif + +// Timeline seeking (enabled by default in debug, stripped in STRIP_ALL) +#if defined(DEMO_ENABLE_TIMELINE_SEEKING) && !defined(STRIP_ALL) +void AudioEngine::seek(float target_time) { + // ... seeking logic +} +#endif + +// Async loading (optional, for zero-latency pre-warming) +#if defined(DEMO_ENABLE_ASYNC_LOADING) +void SpectrogramResourceManager::prewarm_async(int sample_id) { + thread_pool_.enqueue([this, sample_id]() { + load_now(&cache_[sample_id]); + }); +} +#endif +``` + +**Recommended configurations:** + +| Build Type | Cache Eviction | Timeline Seeking | Async Loading | +|------------|---------------|-----------------|---------------| +| **Development** | ON (safety) | ON (debugging) | OFF (simplicity) | +| **Testing** | ON (coverage) | ON (test seeks) | OFF (deterministic) | +| **Release** | OFF (explicit) | OFF (no debug) | OFF (size) | +| **Final 64k** | OFF | OFF (`STRIP_ALL`) | OFF | + +**Size impact:** +- Cache eviction: +150 bytes +- Timeline seeking: +300 bytes (stripped in final) +- Async loading: +250 bytes + +**Binary size budget:** +``` +Base AudioEngine: 500 bytes ++ Lazy loading: +200 bytes ++ Seeking (debug only): +300 bytes (stripped) ++ Cache eviction (opt): +150 bytes (disabled) ++ Async loading (opt): +250 bytes (disabled) +──────────────────────────────────── +Development build: 1150 bytes +Final 64k build: 700 bytes +``` + +--- + ## References - Current issue: Commit 7721f57 "fix(audio): Resolve tracker test failures..." |
