# Audio System Lifecycle Refactor Plan ## Problem Statement The current audio system has a fragile initialization order dependency between `synth_init()` and `tracker_init()`: - `synth_init()` clears ALL registered spectrograms (global state reset) - `tracker_init()` registers spectrograms with the synth - If called in wrong order → spectrograms cleared → no audio / test failures **Current Workarounds:** - Tests must call `synth_init()` before `tracker_init()` - `tracker_init()` re-registers everything on every call (memory overhead) - Hidden bugs when `audio_init()` internally calls `synth_init()` **Why This is Bad:** - Brittle: Easy to break by calling init functions in wrong order - Non-obvious: No compile-time or runtime checks for correct usage - Wasteful: Re-allocates and re-registers on every init - Hard to test: Tests must know internal implementation details - Prevents composition: Can't have multiple tracker instances --- ## Design Goals 1. **Order Independence:** Init functions should work in any order 2. **Explicit Ownership:** Clear who owns what resources 3. **Testability:** Easy to mock, inject, and test components 4. **Memory Safety:** No leaks, no dangling pointers 5. **Size Conscious:** Minimal overhead for 64k binary goal --- ## Proposed Solutions (3 Options) ### Option A: Unified AudioEngine Class (Recommended) **Concept:** Single top-level class that manages synth and tracker as private members. ```cpp class AudioEngine { public: // Lifecycle void init(); void shutdown(); void reset(); // Clear all state // Synth interface (delegates to private synth_) int register_spectrogram(const Spectrogram* spec); void trigger_voice(int spec_id, float volume, float pan); void render(float* output, int num_frames); // Tracker interface (delegates to private tracker_) void load_music_data(const TrackerScore* score); void update_tracker(float music_time); private: Synth synth_; // Embedded instance Tracker tracker_; // Embedded instance bool initialized_ = false; }; ``` **Pros:** - Single initialization point: `engine.init()` sets up everything - Order independence: Internal init sequence is correct by design - Clear ownership: Engine owns both synth and tracker - Easy to test: Mock the entire engine or inject test data **Cons:** - Larger refactor: Need to update all callsites - Increased coupling: Synth and tracker bundled together - Slightly larger binary (vtable overhead if using virtual methods) **Implementation Steps:** 1. Create `AudioEngine` class in `src/audio/audio_engine.h/cc` 2. Move synth and tracker global state into class members 3. Update `audio.cc` to use `AudioEngine` internally 4. Refactor tests to use `AudioEngine` API 5. Gradually migrate production code 6. Remove old `synth_init()` / `tracker_init()` globals --- ### Option B: Registration Handle System **Concept:** Tracker holds registration handles and manages synth resources explicitly. ```cpp class Tracker { public: void init(Synth* synth); // Takes synth pointer, registers spectrograms void shutdown(); // Unregisters spectrograms void update(float music_time); private: Synth* synth_ = nullptr; std::vector registered_spec_ids_; // Tracks what we registered }; // Usage: Synth synth; synth.init(); Tracker tracker; tracker.init(&synth); // Registers spectrograms // Later, if synth is reset: synth.reset(); tracker.re_register(); // Tracker knows what to re-register ``` **Pros:** - Explicit dependencies: Tracker knows it depends on Synth - Order enforced by API: Can't init tracker without synth pointer - Resource cleanup: Tracker can unregister on shutdown - Smaller refactor: Only changes tracker implementation **Cons:** - Still has global synth state (just less fragile) - Tracker needs to track registered IDs (extra bookkeeping) - Multiple trackers would need careful coordination **Implementation Steps:** 1. Add `Synth* synth_` member to Tracker 2. Change `tracker_init()` to `tracker_init(Synth* synth)` 3. Store registered spec IDs in tracker 4. Add `tracker_re_register()` to handle synth resets 5. Update all callsites to pass synth pointer --- ### Option C: Synth Resource Manager with Reference Counting **Concept:** Spectrograms are ref-counted, auto-cleanup when no refs remain. ```cpp class Synth { public: // Returns a handle that auto-releases on destruction SpectrogramHandle register_spectrogram(const Spectrogram* spec); void trigger_voice(const SpectrogramHandle& handle, float volume, float pan); }; class SpectrogramHandle { public: ~SpectrogramHandle() { if (synth_) synth_->unregister(id_); } SpectrogramHandle(const SpectrogramHandle&); // Ref count++ int id() const { return id_; } private: Synth* synth_; int id_; }; ``` **Pros:** - RAII: Automatic cleanup, no leaks - Safe against synth resets: Handles become invalid automatically - Modern C++ design: Follows std::shared_ptr pattern **Cons:** - Complex implementation: Ref counting overhead - Larger binary: More code for handle management - Doesn't solve init order (still need synth before tracker) **Implementation Steps:** 1. Create `SpectrogramHandle` class with ref counting 2. Update `synth_register_spectrogram()` to return handle 3. Add `synth_unregister()` internal API 4. Update tracker to store handles instead of IDs 5. Test thoroughly (ref counting bugs are subtle) --- ## Recommended Approach: Option A (AudioEngine) + Resource Manager **Why Option A?** - Most robust: Completely solves initialization order issues - Future-proof: Easy to extend with more audio subsystems - Testable: Single point of dependency injection - Size acceptable: ~500 bytes overhead (acceptable for robustness) ### Component Separation: Addressing Asset/Procedural Spectrograms **Current Problem:** Tracker currently handles 3 distinct responsibilities: 1. Loading assets via `GetAsset()` (AssetManager dependency) 2. Generating procedural spectrograms (owns allocated memory) 3. Pattern sequencing (triggers events at correct times) **Proposed Architecture:** ``` ┌─────────────────┐ │ AudioEngine │ │ (Facade) │ └────────┬────────┘ │ ┌─────────────────┼─────────────────┐ │ │ │ ┌──────▼──────┐ ┌─────▼──────┐ ┌─────▼────────┐ │ Synth │ │ Tracker │ │ Resource │ │ │ │ │ │ Manager │ │ (Playback) │ │ (Sequence) │ │ (Loading) │ └─────────────┘ └────────────┘ └──────┬───────┘ │ ┌──────────┼──────────┐ │ │ ┌──────▼──────┐ ┌──────▼──────┐ │ AssetManager│ │ Procedural │ │ (Assets) │ │ Generator │ └─────────────┘ └─────────────┘ ``` **SpectrogramResourceManager Responsibilities:** - Load asset spectrograms from AssetManager - Generate procedural spectrograms from NoteParams - Own all allocated memory (consistent ownership) - Provide unified interface for both types - Cache resources to avoid re-generation **Migration Path (Incremental):** 1. Create `AudioEngine` class alongside existing globals 2. Update tests to use `AudioEngine` first (validate correctness) 3. Add `audio_engine.cc` to build, mark old functions deprecated 4. Migrate `main.cc` and production code 5. Remove old `synth_init()` / `tracker_init()` in final cleanup **Estimated Effort:** - Week 1: Design + implement AudioEngine class - Week 2: Migrate tests + validate - Week 3: Migrate production code - Week 4: Cleanup + documentation --- ## Alternative: Keep Current Design with Better Documentation If refactor is too costly, improve current system: 1. **Add Assertions:** ```cpp void tracker_init() { assert(synth_is_initialized() && "Must call synth_init() first!"); // ... rest of init } ``` 2. **Add State Tracking:** ```cpp static bool g_synth_initialized = false; void synth_init() { // ... init code g_synth_initialized = true; } ``` 3. **Document Init Order:** - Update `CONTRIBUTING.md` with explicit init order requirements - Add comments to every init function - Create example code snippets **Pros:** Zero refactor cost **Cons:** Doesn't solve underlying fragility --- ## Decision Matrix | Criterion | Option A (Engine) | Option B (Handles) | Option C (RefCount) | Status Quo | |-----------|-------------------|-------------------|-------------------|------------| | Order Independence | ✅ Full | ⚠️ Partial | ❌ No | ❌ No | | Testability | ✅ Excellent | ✅ Good | ⚠️ Moderate | ❌ Poor | | Memory Safety | ✅ Full | ✅ Good | ✅ Excellent | ⚠️ Manual | | Refactor Effort | 🔴 High | 🟡 Medium | 🔴 High | 🟢 None | | Binary Size Impact | +500B | +200B | +1KB | 0B | | Future Extensibility | ✅ High | ⚠️ Medium | ⚠️ Medium | ❌ Low | **Recommendation:** Option A for production, Status Quo improvements for short-term --- ## Detailed Design: SpectrogramResourceManager ### Interface ```cpp class SpectrogramResourceManager { public: // Lifecycle void init(); void shutdown(); // Frees all owned memory // Lazy loading API (loads on first access) const Spectrogram* get_or_load_asset(AssetId asset_id); const Spectrogram* get_or_generate_procedural(int sample_id, const NoteParams& params); // Explicit loading (for pre-warming if needed) int preload_asset(AssetId asset_id); int preload_procedural(int sample_id, const NoteParams& params); // Query API const Spectrogram* get_spectrogram(int resource_id) const; bool is_loaded(int resource_id) const; // Cache management void clear_cache(); int get_cache_size() const; private: struct Resource { Spectrogram spec; float* owned_data; // nullptr if asset (not owned) AssetId asset_id; // ASSET_LAST_ID if procedural bool is_procedural; }; Resource resources_[MAX_SPECTROGRAMS]; int next_slot_ = 0; }; ``` ### Usage in AudioEngine (Lazy Loading) ```cpp class AudioEngine { public: void init() { synth_.init(); resource_mgr_.init(); tracker_.init(&synth_, &resource_mgr_); } void load_music_data(const TrackerScore* score, const NoteParams* samples, const AssetId* sample_assets, uint32_t sample_count) { // Register sample metadata (but don't load yet!) for (uint32_t i = 0; i < sample_count; ++i) { if (sample_assets[i] != AssetId::ASSET_LAST_ID) { resource_mgr_.register_asset(i, sample_assets[i]); // Metadata only } else { resource_mgr_.register_procedural(i, samples[i]); // Metadata only } } tracker_.load_score(score); } void update(float music_time, float dt) { // Pre-warm samples needed in next 2 seconds tracker_.prewarm_lookahead(music_time, music_time + 2.0f); // Update tracker (triggers events) tracker_.update(music_time); } void trigger_sample(int sample_id, float volume, float pan) { // Load on-demand if not cached const Spectrogram* spec = resource_mgr_.get_or_load(sample_id); // Register with synth (lazy registration) int synth_id = get_or_register_synth_id(sample_id, spec); // Trigger voice synth_.trigger_voice(synth_id, volume, pan); } private: int get_or_register_synth_id(int sample_id, const Spectrogram* spec) { if (sample_to_synth_id_[sample_id] == -1) { // First use: register with synth sample_to_synth_id_[sample_id] = synth_.register_spectrogram(spec); } return sample_to_synth_id_[sample_id]; } Synth synth_; Tracker tracker_; SpectrogramResourceManager resource_mgr_; int sample_to_synth_id_[256] = {-1}; // -1 = not registered yet }; ``` **Timeline:** ``` t=0.0s: init() - No samples loaded (instant startup) t=0.0s: load_music_data() - Register metadata only (~0ms) t=0.0s: update(0.0) - Pre-warm samples for t=0-2s (load 3-5 samples) t=0.0s: Pattern triggers → trigger_sample() → Instant (pre-warmed) t=1.0s: update(1.0) - Pre-warm samples for t=1-3s (load 2-3 more) t=2.0s: update(2.0) - Pre-warm samples for t=2-4s (load 2-3 more) ... Total samples loaded by t=10s: Maybe 10-12 (not all 19) ``` ### Memory Ownership Rules **Clear Ownership:** 1. **Assets:** AssetManager owns, ResourceManager borrows (pointer only) 2. **Procedurals:** ResourceManager owns (allocated with `new[]`, freed in `shutdown()`) 3. **Synth:** Only stores pointers, never owns data **Lifetime Guarantees:** - Assets live until program exit (static data) - Procedurals live until `ResourceManager::shutdown()` - Synth references valid as long as ResourceManager exists --- ## Lazy Loading & On-Demand Strategy ### Problem with Eager Loading **Current plan says:** "Load all resources ONCE at init" **Issues:** - Long startup delay (load all 19 samples before demo starts) - Wasted memory (samples used only once stay loaded) - No benefit for streaming or late-game content **Example:** Demo is 60s, sample only used at t=45s → why load it at t=0? ### Proposed: Lazy Loading with Cache **Strategy:** 1. **Load on first trigger** - Don't load until sample is actually needed 2. **Cache aggressively** - Keep loaded samples in memory (demo is short, ~60s) 3. **Optional pre-warming** - Load predicted samples 1-2 seconds ahead 4. **Eviction policy** - Unload samples that won't be used again (optional) ### Implementation: Two-Tier Cache ```cpp class SpectrogramResourceManager { private: struct CacheEntry { Spectrogram spec; float* owned_data; // nullptr if not loaded yet AssetId asset_id; NoteParams proc_params; // For procedurals enum State { UNLOADED, // Not loaded yet LOADING, // Load in progress (async) LOADED, // Ready to use EVICTED // Was loaded, now evicted }; State state = UNLOADED; float last_access_time; // For LRU eviction }; CacheEntry cache_[MAX_SPECTROGRAMS]; public: // Lazy loading (loads if not cached) const Spectrogram* get_or_load(int sample_id) { CacheEntry& entry = cache_[sample_id]; if (entry.state == LOADED) { entry.last_access_time = current_time(); return &entry.spec; // Cache hit! } if (entry.state == UNLOADED || entry.state == EVICTED) { load_now(&entry); // Load synchronously entry.state = LOADED; } return &entry.spec; } private: void load_now(CacheEntry* entry) { if (entry->asset_id != ASSET_LAST_ID) { // ASSET: Borrow pointer from AssetManager size_t size; const uint8_t* data = GetAsset(entry->asset_id, &size); #if defined(SUPPORT_COMPRESSED_ASSETS) // ON-DEMAND DECOMPRESSION if (is_compressed(data)) { entry->owned_data = decompress_asset(data, size); entry->spec.spectral_data_a = entry->owned_data; } else { entry->spec.spectral_data_a = (const float*)(data + sizeof(SpecHeader)); } #else // Direct pointer (no decompression) entry->spec.spectral_data_a = (const float*)(data + sizeof(SpecHeader)); #endif } else { // PROCEDURAL: Generate on-demand int frames; std::vector data = generate_note_spectrogram(entry->proc_params, &frames); entry->owned_data = new float[data.size()]; memcpy(entry->owned_data, data.data(), data.size() * sizeof(float)); entry->spec.spectral_data_a = entry->owned_data; entry->spec.num_frames = frames; } } }; ``` ### On-Demand Decompression (Future) When assets are compressed (Task #27), decompress lazily: ```cpp const Spectrogram* get_or_load(int sample_id) { CacheEntry& entry = cache_[sample_id]; if (entry.state == LOADED) { return &entry.spec; // Already decompressed } // Load compressed asset const uint8_t* compressed = GetAsset(entry->asset_id, &size); // Decompress into owned buffer entry->owned_data = decompress_zlib(compressed, size, &decompressed_size); entry->spec.spectral_data_a = entry->owned_data; entry->state = LOADED; return &entry.spec; } void evict(int sample_id) { // Free decompressed data, can re-decompress from asset later delete[] cache_[sample_id].owned_data; cache_[sample_id].owned_data = nullptr; cache_[sample_id].state = EVICTED; } ``` ### Pre-warming Strategy **Idea:** Predict samples needed in next 1-2 seconds, load ahead of time ```cpp class Tracker { public: void update(float music_time) { // 1. Trigger patterns at current time trigger_patterns_at(music_time); // 2. Pre-warm samples needed in next 2 seconds prewarm_lookahead(music_time, music_time + 2.0f); } private: void prewarm_lookahead(float start_time, float end_time) { // Scan upcoming pattern triggers for (const auto& trigger : upcoming_triggers_) { if (trigger.time > start_time && trigger.time < end_time) { const TrackerPattern& pattern = get_pattern(trigger.pattern_id); // Load all samples used in this pattern for (const auto& event : pattern.events) { resource_mgr_->preload(event.sample_id); // Async or sync } } } } }; ``` **Benefits:** - No loading stutter when pattern triggers - Spreads load cost over time (not all at init) - Can load asynchronously in background thread (if needed) ### Cache Eviction Policy (Optional, Compile-Time Flag) **Demo knows its timeline:** Effects can explicitly release resources when done. **Explicit Resource Management (Preferred):** ```cpp class Effect { public: void on_effect_end() override { // Effect knows it won't need these samples again resource_mgr_->release(SAMPLE_EXPLOSION); resource_mgr_->release(SAMPLE_AMBIENT_WIND); } }; // In sequence compiler: EFFECT ExplosionEffect 10.0 12.0 // Uses SAMPLE_EXPLOSION // After t=12.0, explosion sample can be evicted ``` **Automatic Eviction (Under `ENABLE_CACHE_EVICTION` flag):** ```cpp #if defined(ENABLE_CACHE_EVICTION) void try_evict_lru() { if (get_cache_memory_usage() > MAX_CACHE_SIZE) { int lru_id = find_lru_entry(); if (current_time() - cache_[lru_id].last_access_time > 10.0f) { evict(lru_id); } } } #endif /* ENABLE_CACHE_EVICTION */ ``` **Eviction strategies (all optional):** 1. **Explicit** (recommended) - Effect calls `release()` when done 2. **LRU** (fallback) - Evict least recently used if memory tight 3. **Timeline hints** - Sequence compiler marks "last use" of samples **Recommendation:** - Production: Explicit release only (demo knows timeline) - Development: Enable LRU eviction for safety (`-DENABLE_CACHE_EVICTION`) - Final 64k build: Disable eviction (cache everything, simple code) --- ### Comparison: Eager vs Lazy Loading | Aspect | Eager (Load All at Init) | Lazy (Load on Demand) | Lazy + Pre-warm | |--------|-------------------------|----------------------|-----------------| | **Startup Time** | ❌ Slow (load all 19) | ✅ Fast (load 0) | ✅ Fast (load 0) | | **First Trigger Latency** | ✅ Instant (cached) | ❌ Stutter (load now) | ✅ Instant (pre-loaded) | | **Memory Usage** | ❌ High (all loaded) | ✅ Low (only used) | ⚠️ Medium (active + lookahead) | | **Complexity** | ✅ Simple | ⚠️ Medium | ❌ Complex | | **Best For** | Short demos, few samples | Long demos, many samples | **Recommended for 64k** | **Recommendation:** **Lazy + Pre-warm (1-2s lookahead)** - Fast startup (no eager loading) - No stutter (pre-warming prevents load-on-trigger) - Memory efficient (only loads active + upcoming samples) --- ### Benefits Over Current Design | Aspect | Current (Tracker-owned) | Proposed (ResourceManager) | |--------|------------------------|---------------------------| | **Ownership** | Split (assets borrowed, procedurals owned) | Centralized (clear rules) | | **Coupling** | Tracker → AssetManager + gen | ResourceManager → AssetManager + gen | | **Testability** | Hard (global GetAsset) | Easy (inject mock manager) | | **Reusability** | Tracker-specific | Usable by any audio component | | **Memory Safety** | Manual (delete[] in init) | RAII in shutdown() | ### Integration with AssetManager **Option 1: Direct Dependency (Simple)** ```cpp int SpectrogramResourceManager::load_asset_spectrogram(AssetId id) { size_t size; const uint8_t* data = GetAsset(id, &size); // Direct call // ... parse and store } ``` **Option 2: Injected Interface (Testable)** ```cpp class IAssetProvider { public: virtual const uint8_t* get_asset(AssetId id, size_t* size) = 0; }; class SpectrogramResourceManager { public: void set_asset_provider(IAssetProvider* provider) { asset_provider_ = provider; } int load_asset_spectrogram(AssetId id) { const uint8_t* data = asset_provider_->get_asset(id, &size); // ... parse and store } private: IAssetProvider* asset_provider_ = nullptr; }; // Production: Adapter for AssetManager class AssetManagerAdapter : public IAssetProvider { public: const uint8_t* get_asset(AssetId id, size_t* size) override { return GetAsset(id, size); // Delegate to global } }; ``` **Recommendation:** Option 1 for now (simpler), Option 2 if we refactor AssetManager too --- ## Implementation Plan (Option A) ### Phase 1: Design & Prototype (5-7 days) - [ ] Create `src/audio/spectrogram_resource_manager.h/cc` - [ ] Implement lazy loading cache (UNLOADED → LOADED states) - [ ] Add `get_or_load()` API for on-demand loading - [ ] Add `register_*()` API for metadata-only registration - [ ] Implement resource loading (assets + procedurals) - [ ] Add memory ownership tracking - [ ] Write unit tests for lazy loading behavior - [ ] Create `src/audio/audio_engine.h/cc` with class interface - [ ] Implement `AudioEngine::init()` / `shutdown()` / `reset()` - [ ] Add delegation methods for synth/tracker APIs - [ ] Integrate ResourceManager into AudioEngine - [ ] **Implement `AudioEngine::seek()` for timeline scrubbing** - [ ] Add `Synth::reset()` and `Synth::set_time()` - [ ] Add `Tracker::reset()` and `Tracker::set_time()` - [ ] Add `Tracker::update_silent()` for fast-forward - [ ] Implement pre-warming for target time range - [ ] Add seeking tests (forward, backward, edge cases) - [ ] Write unit tests for `AudioEngine` lifecycle with resources ### Phase 2: Test Migration (3-5 days) - [ ] Update `test_tracker.cc` to use `AudioEngine` - [ ] Update `test_tracker_timing.cc` - [ ] Update `test_variable_tempo.cc` - [ ] Update `test_wav_dump.cc` - [ ] Ensure 100% test pass rate ### Phase 3: Production Integration (5-7 days) - [ ] Update `audio.cc` to use `AudioEngine` internally - [ ] Update `main.cc` demo loop - [ ] Update all effect classes that use audio - [ ] Add backwards compatibility shims (temporary) ### Phase 4: Cleanup (2-3 days) - [ ] Remove old `synth_init()` / `tracker_init()` functions - [ ] Remove global synth/tracker state - [ ] Update documentation (HOWTO.md, CONTRIBUTING.md) - [ ] Remove compatibility shims ### Phase 5: Optimization (Optional, 2-3 days) - [ ] Profile binary size impact - [ ] Optimize if >1KB overhead - [ ] Consider `#if !defined(STRIP_ALL)` for test-only features - [ ] **Optional:** Add async loading (background thread for pre-warming) - Load samples in background while demo runs - Requires mutex for cache access - Benefit: Zero stutter even for large samples - Cost: +200-300 bytes for threading code --- ## Future: Async Loading (Post-MVP) **Idea:** Pre-warm samples in background thread to avoid any stutter ```cpp class SpectrogramResourceManager { public: void prewarm_async(int sample_id) { if (cache_[sample_id].state == UNLOADED) { cache_[sample_id].state = LOADING; // Enqueue background load thread_pool_.enqueue([this, sample_id]() { load_now(&cache_[sample_id]); std::lock_guard lock(cache_mutex_); cache_[sample_id].state = LOADED; }); } } const Spectrogram* get_or_load(int sample_id) { std::lock_guard lock(cache_mutex_); CacheEntry& entry = cache_[sample_id]; if (entry.state == LOADED) { return &entry.spec; // Ready! } if (entry.state == LOADING) { // Wait for background load to finish wait_for_load(sample_id); } else { // Not started, load synchronously (fallback) load_now(&entry); entry.state = LOADED; } return &entry.spec; } private: std::mutex cache_mutex_; ThreadPool thread_pool_; }; ``` **Benefits:** - Zero latency: All loads happen in background - Demo runs at full framerate during loading **Costs:** - +200-300 bytes for threading code - Complexity: Need mutexes, thread management - May not be needed for 64k (samples are small, load quickly) **Recommendation:** Only implement if profiling shows stutter issues --- ## Success Criteria 1. **No Init Order Failures:** Tests pass regardless of call order 2. **Memory Clean:** No leaks detected by sanitizers 3. **Size Budget:** <1KB binary size increase 4. **API Simplicity:** Single `engine.init()` call for users 5. **Test Coverage:** 100% coverage of lifecycle edge cases --- ## Open Questions 1. Should `AudioEngine` be a singleton or instance-based? - **Proposal:** Instance-based for testability, but provide global getter for convenience 2. How to handle legacy code during migration? - **Proposal:** Keep old functions as deprecated wrappers for 1-2 releases 3. Should synth and tracker be members or pointers? - **Proposal:** Direct members (smaller, faster, simpler ownership) 4. What about backend abstraction (MiniaudioBackend, MockBackend)? - **Proposal:** AudioEngine owns backend pointer, injected via `set_backend()` --- ## Timeline Seeking & Scrubbing (Critical for Development) ### Requirement: Robust Seeking for Debugging **Need to support:** - Jump forward: t=5s → t=45s (skip to specific scene) - Rewind: t=45s → t=5s (go back to debug earlier issue) - Fast-forward: Simulate t=0-30s without audio (load samples, update state) - Frame-by-frame: t → t+0.016s (precise debugging) **Properties:** - ✅ Robust: Works for any timeline position - ✅ Correct: Audio/visuals match target time exactly - ⚠️ Performance: Doesn't need to be fast (debugging tool) - ✅ Pre-warming: Must load samples for target time range ### Implementation: AudioEngine::seek() ```cpp class AudioEngine { public: // Seek to arbitrary point in timeline void seek(float target_time) { #if !defined(STRIP_ALL) // Only in debug builds // 1. Reset synth state (clear all active voices) synth_.reset(); // 2. Reset tracker state (re-scan patterns from scratch) tracker_.reset(); tracker_.set_time(target_time); // 3. Pre-warm samples for target time range float prewarm_start = std::max(0.0f, target_time - 1.0f); float prewarm_end = target_time + 2.0f; tracker_.prewarm_lookahead(prewarm_start, prewarm_end); // 4. Simulate tracker up to target time (no audio rendering) // This ensures tracker internal state is correct const float dt = 0.1f; // Simulate in 0.1s steps for (float t = 0.0f; t < target_time; t += dt) { tracker_.update_silent(t); // Update state only, no audio } // 5. Final update at exact target time tracker_.update(target_time); // 6. Notify effects of seek event (optional) sequence_.on_seek(target_time); current_time_ = target_time; #endif /* !defined(STRIP_ALL) */ } // Fast-forward: Simulate time range without rendering void fast_forward(float from_time, float to_time) { const float dt = 0.1f; for (float t = from_time; t < to_time; t += dt) { tracker_.update_silent(t); sequence_.update_silent(t, dt); // Update effects without rendering } } private: float current_time_ = 0.0f; }; ``` ### Tracker Support for Seeking ```cpp class Tracker { public: void reset() { // Clear all active patterns for (auto& pattern : active_patterns_) { pattern.active = false; } last_trigger_idx_ = 0; } void set_time(float time) { current_music_time_ = time; // Re-scan score to find patterns that should be active at this time for (uint32_t i = 0; i < score_->num_triggers; ++i) { const auto& trigger = score_->triggers[i]; if (trigger.time_sec <= time) { // This pattern should be active (or was active) // Determine if still active based on pattern duration const auto& pattern = patterns_[trigger.pattern_id]; float pattern_end = trigger.time_sec + get_pattern_duration(pattern); if (time < pattern_end) { // Pattern is currently active, add it activate_pattern(trigger.pattern_id, trigger.time_sec); // Fast-forward pattern to current beat update_pattern_to_time(trigger.pattern_id, time); } last_trigger_idx_ = i + 1; } } } void update_silent(float time) { // Update tracker state without triggering voices // Used for fast-forward and seeking update_internal(time, /*trigger_audio=*/false); } private: void update_pattern_to_time(int pattern_id, float target_time) { // Advance pattern's event index to match target time ActivePattern& active = find_active_pattern(pattern_id); const TrackerPattern& pattern = patterns_[pattern_id]; float elapsed = target_time - active.start_music_time; float beat_duration = 60.0f / score_->bpm; float elapsed_beats = elapsed / beat_duration; // Advance event index past all events before target time while (active.next_event_idx < pattern.num_events) { if (pattern.events[active.next_event_idx].beat > elapsed_beats) { break; } active.next_event_idx++; } } }; ``` ### Synth Support for Seeking ```cpp class Synth { public: void reset() { // Clear all active voices (stop all sound immediately) for (auto& voice : voices_) { voice.active = false; } // Reset time tracking #if !defined(STRIP_ALL) g_elapsed_time_sec = 0.0f; #endif } void set_time(float time) { #if !defined(STRIP_ALL) g_elapsed_time_sec = time; #endif } }; ``` ### Effect Sequence Support for Seeking ```cpp class Sequence { public: void on_seek(float target_time) { // Notify all effects that might be active at target time for (auto& layer : layers_) { for (auto& entry : layer.effects) { if (entry.start_time <= target_time && target_time < entry.end_time) { // Effect is active at target time entry.effect->on_seek(target_time); } else if (target_time < entry.start_time) { // Effect hasn't started yet, reset it entry.effect->on_reset(); } } } } }; class Effect { public: virtual void on_seek(float time) { // Optional: Reset effect state to match target time // Example: Particle system re-simulates to target time } virtual void on_reset() { // Optional: Clear all effect state } }; ``` ### Usage Examples **Jump to specific scene:** ```cpp // User presses 'J' key to jump to scene 3 (at t=45s) if (key_pressed('J')) { audio_engine.seek(45.0f); camera.set_position(scene3_camera_pos); // Sync visuals too } ``` **Rewind to beginning:** ```cpp if (key_pressed('R')) { audio_engine.seek(0.0f); } ``` **Frame-by-frame debugging:** ```cpp if (key_pressed('N')) { // Next frame float dt = 1.0f / 60.0f; audio_engine.update(current_time, dt); current_time += dt; } if (key_pressed('P')) { // Previous frame float dt = 1.0f / 60.0f; audio_engine.seek(std::max(0.0f, current_time - dt)); } ``` **Fast-forward to load all samples:** ```cpp // Pre-load all samples by fast-forwarding through entire demo void preload_all_samples() { audio_engine.fast_forward(0.0f, demo_duration); audio_engine.seek(0.0f); // Reset to beginning } ``` ### Seeking Guarantees **Invariants after `seek(t)`:** 1. ✅ Tracker state matches time `t` (patterns active, event indices correct) 2. ✅ Synth has no active voices (silent state) 3. ✅ Samples for range [t-1s, t+2s] are pre-warmed 4. ✅ Next `update(t)` will trigger correct events 5. ✅ Audio output matches what would have played at time `t` **Edge Cases Handled:** - Seek to t=0: Reset everything (initial state) - Seek beyond demo end: Clamp to valid range - Seek to same time twice: No-op (already there) - Seek while audio playing: Stop all voices first ### Performance Characteristics **Seek cost:** - Synth reset: O(MAX_VOICES) ~ 64 voices → <1ms - Tracker reset: O(MAX_PATTERNS) ~ 20 patterns → <1ms - Pattern re-scan: O(num_triggers) ~ 100 triggers → <1ms - Pre-warming: O(samples_in_range) ~ 5 samples → 5-10ms - Total: **~10-20ms** (acceptable for debugging) **Optimization (if needed):** - Cache tracker state snapshots every 5s - Seek to nearest snapshot, then fast-forward delta - Reduces re-scan cost for long demos ### Testing Seeking Robustness ```cpp void test_seeking() { AudioEngine engine; engine.init(); // Test 1: Forward seek engine.seek(10.0f); assert(engine.get_time() == 10.0f); // Test 2: Backward seek (rewind) engine.seek(5.0f); assert(engine.get_time() == 5.0f); // Test 3: Seek to beginning engine.seek(0.0f); assert(engine.get_time() == 0.0f); // Test 4: Multiple seeks in sequence engine.seek(20.0f); engine.seek(30.0f); engine.seek(15.0f); assert(engine.get_time() == 15.0f); // Test 5: Verify samples loaded correctly after seek engine.seek(45.0f); const auto* spec = engine.get_resource_manager().get_spectrogram(SAMPLE_ENDING); assert(spec != nullptr); // Sample for ending should be loaded } ``` --- ## FAQ ### Q: How does this handle procedural vs asset spectrograms? **A:** Via a dedicated `SpectrogramResourceManager`: - **Assets:** Loaded from AssetManager, ResourceManager stores pointer only (doesn't own) - **Procedurals:** Generated via `generate_note_spectrogram()`, ResourceManager owns memory - **Unified Interface:** Both types returned as `const Spectrogram*`, caller doesn't care about source **Key Insight:** Separating resource loading from pattern sequencing allows: - Tracker focuses on timing/triggering (single responsibility) - Resources loaded once at init (no per-frame overhead) - Easy to add new spectrogram sources (streaming, compressed, etc.) ### Q: Does AudioEngine depend on AssetManager? **A:** Indirectly, via ResourceManager: ``` AudioEngine → ResourceManager → AssetManager (via GetAsset) ↘ Tracker (no asset dependency) ↘ Synth (no asset dependency) ``` **Benefits:** - Only one component knows about assets (ResourceManager) - Easy to mock for testing (inject fake ResourceManager) - If AssetManager is refactored, only ResourceManager needs updating ### Q: What happens to the current caching mechanism? **A:** Improved and centralized: **Current (Tracker-owned):** ```cpp static int g_sample_synth_cache[256]; // Maps sample_id → synth_id ``` **Proposed (AudioEngine-owned):** ```cpp class AudioEngine { private: int sample_to_synth_id_[256]; // Clearer ownership }; ``` **Why better?** - Cache lifetime matches engine lifetime (no stale data) - Cleared automatically on `engine.reset()` - Testable (can verify cache contents) ### Q: How does seeking/scrubbing work for debugging? **A:** Robust seek API that resets state and pre-warms samples: ```cpp audio_engine.seek(45.0f); // Jump to t=45s // What happens internally: // 1. synth.reset() - Clear all active voices // 2. tracker.reset() - Clear active patterns // 3. tracker.set_time(45.0) - Re-scan score, activate patterns at t=45 // 4. prewarm(43-47s) - Load samples for target range // 5. update(45.0) - Trigger events at exact time ``` **Supported operations:** - **Jump forward:** `seek(45.0f)` - Skip to specific scene - **Rewind:** `seek(5.0f)` - Go back to earlier time - **Fast-forward:** `fast_forward(0, 30)` - Simulate without audio - **Frame-by-frame:** `seek(t + 0.016)` - Step through frames **Performance:** ~10-20ms per seek (acceptable for debugging) **Guarantees:** - ✅ Audio/visuals match target time exactly - ✅ Tracker state correct (patterns active, event indices) - ✅ Samples pre-warmed (no stutter on next update) - ✅ Works for any timeline position (0 → end) **Development workflow:** ```cpp // Press 'J' to jump to problematic scene at t=37s if (key_pressed('J')) { audio_engine.seek(37.0f); // Audio and visuals now at t=37s, ready to debug } ``` ### Q: Why not load all samples at init? **A:** Lazy loading with pre-warming is more efficient: **Problems with eager loading:** - Slow startup: Load all 19 samples before t=0 (~50-100ms delay) - Wasted memory: Sample used only at t=45s stays loaded from t=0 - No benefit: 60s demo doesn't need all samples loaded upfront **Lazy + Pre-warm strategy:** ``` t=0.0s: Load 0 samples (instant startup) t=0.0s: Pre-warm 3-5 samples for t=0-2s range (background) t=1.0s: Pre-warm 2-3 more for t=1-3s range By t=10s: Only ~10 samples loaded (not all 19) ``` **Benefits:** - ✅ Instant startup (no initial loading) - ✅ No trigger stutter (pre-warming prevents load-on-access) - ✅ Memory efficient (only active + upcoming samples) - ✅ Spreads load cost over time **Future: On-demand decompression (Task #27)** When assets are compressed, decompress only when needed: - Compressed asset in binary: 2KB - Decompressed in memory: 8KB (4x larger) - Only decompress samples actually used (save memory) ### Q: How does this affect binary size? **Estimated overhead:** - `SpectrogramResourceManager` (basic): ~300 bytes - `SpectrogramResourceManager` (with lazy loading): ~500 bytes - `AudioEngine` wrapper: ~200 bytes - Total: **~700 bytes** (1.1% of 64KB budget) **Savings from removing global state:** ~100 bytes **Net impact:** **~600 bytes** (acceptable for robustness + lazy loading) --- ## Compile-Time Configuration Flags **Optional features controlled by CMake flags:** ```cmake # CMakeLists.txt option(DEMO_ENABLE_CACHE_EVICTION "Enable automatic cache eviction (LRU)" OFF) option(DEMO_ENABLE_TIMELINE_SEEKING "Enable timeline seeking for debugging" ON) option(DEMO_ENABLE_ASYNC_LOADING "Enable background thread pre-warming" OFF) ``` **Usage in code:** ```cpp // Cache eviction (optional, for long demos or low memory) #if defined(DEMO_ENABLE_CACHE_EVICTION) void SpectrogramResourceManager::try_evict_lru() { // ... LRU eviction logic } #endif // Timeline seeking (enabled by default in debug, stripped in STRIP_ALL) #if defined(DEMO_ENABLE_TIMELINE_SEEKING) && !defined(STRIP_ALL) void AudioEngine::seek(float target_time) { // ... seeking logic } #endif // Async loading (optional, for zero-latency pre-warming) #if defined(DEMO_ENABLE_ASYNC_LOADING) void SpectrogramResourceManager::prewarm_async(int sample_id) { thread_pool_.enqueue([this, sample_id]() { load_now(&cache_[sample_id]); }); } #endif ``` **Recommended configurations:** | Build Type | Cache Eviction | Timeline Seeking | Async Loading | |------------|---------------|-----------------|---------------| | **Development** | ON (safety) | ON (debugging) | OFF (simplicity) | | **Testing** | ON (coverage) | ON (test seeks) | OFF (deterministic) | | **Release** | OFF (explicit) | OFF (no debug) | OFF (size) | | **Final 64k** | OFF | OFF (`STRIP_ALL`) | OFF | **Size impact:** - Cache eviction: +150 bytes - Timeline seeking: +300 bytes (stripped in final) - Async loading: +250 bytes **Binary size budget:** ``` Base AudioEngine: 500 bytes + Lazy loading: +200 bytes + Seeking (debug only): +300 bytes (stripped) + Cache eviction (opt): +150 bytes (disabled) + Async loading (opt): +250 bytes (disabled) ──────────────────────────────────── Development build: 1150 bytes Final 64k build: 700 bytes ``` --- ## References - Current issue: Commit 7721f57 "fix(audio): Resolve tracker test failures..." - Related: `doc/TRACKER.md`, `doc/ASSET_SYSTEM.md` - Design patterns: Facade pattern, Dependency Injection, Resource Manager pattern