From a9a151a4fdcd46f4737abe98c654c1ec619ef425 Mon Sep 17 00:00:00 2001 From: skal Date: Sat, 7 Feb 2026 15:34:46 +0100 Subject: docs: Reorganize documentation with tiered hierarchy for context optimization MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Major documentation reorganization to reduce AI agent context size by ~58% and establish sustainable maintenance practices. ## File Moves (Root → doc/) - Move COMPLETED.md (new), HANDOFF*.md, *_ANALYSIS.md, *_SUMMARY.md to doc/ - Keep only 5 essential files in root: CLAUDE.md, GEMINI.md, PROJECT_CONTEXT.md, TODO.md, README.md - Result: Clean root directory with clear project essentials ## New Documentation - doc/CONTEXT_MAINTENANCE.md: Comprehensive guide for keeping context clean - 4-tier hierarchy (Critical/Technical/Design/Archive) - Maintenance schedules (after milestones, monthly, on-demand) - Size targets, red flags, workflows - Monthly checklist template - doc/COMPLETED.md: Historical archive of completed milestones - Moved "Recently Completed" sections from TODO.md and PROJECT_CONTEXT.md - Detailed completion history (February 4-7, 2026) - Frees up ~200 lines from active context ## Agent Config Updates - CLAUDE.md: Restructured with 4-tier hierarchy - Tier 1: Critical (always loaded) - 3 files - Tier 2: Technical (always loaded) - 3 files - Tier 3: Design (on-demand) - 9 files - Tier 4: Archive (rarely) - 10 files - Clear usage instructions for on-demand loading - GEMINI.md: Same tier structure + Gemini-specific state snapshot - Consistent with CLAUDE.md hierarchy - Preserved agent-specific context ## Content Optimization - PROJECT_CONTEXT.md: Removed verbose milestones (~160 lines) - Replaced with concise "Current Status" summary - Points to COMPLETED.md for history - TODO.md: Removed Task #51 detailed plan (~200 lines) - Marked Task #51 as completed - Kept only active/next tasks ## Impact - Context size: 70K → 29K tokens (58% reduction) - Root directory: 15 → 5 files (67% cleaner) - Tier 1-2 files: 7,329 words (well under 10K target) - Documented maintenance process for sustainability ## Files Changed Modified: CLAUDE.md, GEMINI.md, PROJECT_CONTEXT.md, TODO.md New: doc/COMPLETED.md, doc/CONTEXT_MAINTENANCE.md Moved: 10 technical docs from root to doc/ --- doc/CONTEXT_MAINTENANCE.md | 332 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 332 insertions(+) create mode 100644 doc/CONTEXT_MAINTENANCE.md (limited to 'doc/CONTEXT_MAINTENANCE.md') diff --git a/doc/CONTEXT_MAINTENANCE.md b/doc/CONTEXT_MAINTENANCE.md new file mode 100644 index 0000000..c228cd7 --- /dev/null +++ b/doc/CONTEXT_MAINTENANCE.md @@ -0,0 +1,332 @@ +# Context Maintenance Guide + +This document describes how to maintain clean, efficient context for AI agents (Claude, Gemini) working on this project. + +## Philosophy + +**Goal**: Keep AI context focused on **actionable, current information** while archiving **historical details** for reference. + +**Principle**: The context should answer "What am I working on **now**?" not "What did we do 6 months ago?" + +--- + +## File Organization Hierarchy + +### Tier 1: Critical (Always Loaded) +These files are essential for every session and should remain lean: + +- **CLAUDE.md** / **GEMINI.md** - Agent configuration (keep to ~30 lines) +- **PROJECT_CONTEXT.md** - High-level project overview (keep to ~200 lines) + - Current status summary (not detailed milestones) + - Architectural overview (brief) + - Next priorities (not detailed plans) +- **TODO.md** - Active tasks only (keep to ~300 lines) + - Current and next tasks + - Brief attack plans (not 200-line implementation details) + - Mark completed tasks and reference COMPLETED.md +- **README.md** - Quick start guide + +### Tier 2: Technical Reference (Always Loaded) +Essential for daily work: + +- **doc/HOWTO.md** - Usage instructions, common commands +- **doc/CONTRIBUTING.md** - Coding guidelines, commit policies +- **doc/AI_RULES.md** - Agent-specific rules + +### Tier 3: Design Docs (Load On-Demand) +Load these only when working on specific subsystems: + +- **doc/ASSET_SYSTEM.md** - Asset pipeline details +- **doc/BUILD.md** - Build system details +- **doc/3D.md** - 3D rendering architecture +- **doc/SPEC_EDITOR.md** - Spectral editor design +- **doc/TRACKER.md** - Audio tracker system +- **doc/PROCEDURAL.md** - Procedural generation +- **doc/ANALYSIS_VARIABLE_TEMPO_V2.md** - Tempo system analysis + +### Tier 4: Historical Archive (Load Rarely) +Reference material for understanding past decisions: + +- **doc/COMPLETED.md** - Detailed completion history +- **doc/HANDOFF*.md** - Agent handoff documents +- **doc/*_ANALYSIS.md** - Technical investigations +- **doc/*_SUMMARY.md** - Task/feature summaries + +--- + +## Maintenance Schedule + +### After Major Milestone (Immediately) +**Trigger**: Completed a multi-week task or significant feature + +**Actions**: +1. ✅ Move detailed completion notes from TODO.md to COMPLETED.md +2. ✅ Update PROJECT_CONTEXT.md "Current Status" (remove old "Recently Completed") +3. ✅ Archive detailed implementation plans from TODO.md +4. ✅ Update "Next Up" section in PROJECT_CONTEXT.md + +**Time**: 10-15 minutes + +### Monthly Review (1st of Month) +**Trigger**: Calendar-based + +**Actions**: +1. ✅ Archive tasks completed >30 days ago from TODO.md → COMPLETED.md +2. ✅ Review PROJECT_CONTEXT.md for outdated "Recently Completed" items +3. ✅ Check for temporary analysis docs in root directory → move to doc/ +4. ✅ Verify CLAUDE.md/GEMINI.md only load Tier 1-2 files by default + +**Time**: 15-20 minutes + +### On-Demand (When Context Feels Slow) +**Trigger**: AI responses taking longer, context bloat suspected + +**Actions**: +1. ✅ Run token analysis: `wc -w CLAUDE.md PROJECT_CONTEXT.md TODO.md doc/*.md | sort -rn` +2. ✅ Check for oversized files (>500 lines in Tier 1-2) +3. ✅ Aggressively archive verbose sections +4. ✅ Consider splitting large design docs + +**Time**: 30-45 minutes + +--- + +## What to Archive + +### ✅ ALWAYS Archive: +- Detailed implementation plans for completed tasks (>100 lines) +- Verbose milestone descriptions with step-by-step breakdowns +- Analysis documents for resolved issues +- Deprecated design docs (keep note in main doc pointing to archive) +- Old handoff documents (>1 month old) + +### ⚠️ SUMMARIZE Before Archiving: +- Major architectural decisions (keep 2-3 line summary) +- Critical bug resolutions (keep root cause + solution) +- Performance optimizations (keep before/after metrics) + +### ❌ NEVER Archive: +- Current task priorities +- Essential coding guidelines (CONTRIBUTING.md, AI_RULES.md) +- Quick-start instructions (README.md, HOWTO.md) +- Active architectural constraints + +--- + +## Red Flags (Time to Clean Up) + +### File Size Red Flags: +- PROJECT_CONTEXT.md >500 lines +- TODO.md >800 lines +- Any Tier 1-2 file >1000 lines + +### Content Red Flags: +- "Recently Completed" section >100 lines in PROJECT_CONTEXT.md +- TODO.md contains tasks marked `[DONE]` for >30 days +- Detailed implementation plans (>50 lines) for completed tasks in TODO.md +- Analysis documents in root directory (should be in doc/) + +### Context Red Flags: +- AI responses noticeably slower +- Token usage >50K on initial load +- Repeated "let me read..." for basic project info + +--- + +## Archiving Workflow + +### 1. Identify Content to Archive +```bash +# Find large files +find . -name "*.md" -type f -exec wc -l {} + | sort -rn | head -10 + +# Find old completed tasks in TODO.md +grep -n "\[x\].*202[0-9]-[0-9][0-9]-[0-9][0-9]" TODO.md +``` + +### 2. Move to COMPLETED.md +```markdown +## Recently Completed (YYYY-MM-DD) + +- [x] **Task Name**: Brief summary + - Key points (2-3 bullets max) + - Result: One-line outcome +``` + +### 3. Update Source Files +- Replace verbose sections with: `**Note**: Detailed history in doc/COMPLETED.md` +- Update cross-references +- Keep high-level summaries + +### 4. Update Agent Configs +Update CLAUDE.md and GEMINI.md if files moved: +```markdown +# Design docs (load on-demand when needed) +# Use: "read @doc/COMPLETED.md" for detailed history +``` + +### 5. Verify +```bash +# Check context size +cat CLAUDE.md PROJECT_CONTEXT.md TODO.md | wc -w +# Target: <10,000 words for Tier 1 files + +# Verify no broken references +grep -r "@doc/" CLAUDE.md GEMINI.md PROJECT_CONTEXT.md TODO.md +``` + +--- + +## Common Mistakes to Avoid + +### ❌ Don't: +- Archive current priorities or active tasks +- Remove essential architectural context +- Break cross-references without updating +- Move files without updating CLAUDE.md/GEMINI.md +- Delete content (archive instead) +- Keep outdated "Recently Completed" sections >3 months + +### ✅ Do: +- Preserve key technical insights (in summaries) +- Update file references after moving docs +- Keep audit trail (note where content moved) +- Maintain searchable keywords (for finding archives) +- Regular small cleanups (better than big purges) + +--- + +## Quick Reference: File Moves + +When moving files from root to doc/: + +```bash +# 1. Move file (use git mv if tracked) +git mv OLDFILE.md doc/OLDFILE.md + +# 2. Update CLAUDE.md +# Change: @OLDFILE.md +# To: # Use: "read @doc/OLDFILE.md" when needed + +# 3. Update GEMINI.md (same as above) + +# 4. Update cross-references in other files +grep -r "OLDFILE.md" *.md doc/*.md + +# 5. Commit with clear message +git commit -m "docs: Move OLDFILE.md to doc/ for better organization" +``` + +--- + +## Context Size Targets + +### Optimal Targets: +- **CLAUDE.md**: <30 lines +- **PROJECT_CONTEXT.md**: <300 lines +- **TODO.md**: <500 lines +- **Total Tier 1-2**: <10,000 words (~40K tokens) + +### Warning Thresholds: +- **PROJECT_CONTEXT.md**: >500 lines +- **TODO.md**: >800 lines +- **Total Tier 1-2**: >15,000 words (~60K tokens) + +### Critical (Clean Up Immediately): +- **PROJECT_CONTEXT.md**: >800 lines +- **TODO.md**: >1200 lines +- **Total Tier 1-2**: >20,000 words (~80K tokens) + +--- + +## Examples + +### Good: Lean TODO.md Entry +```markdown +- [ ] **Task #72: Implement Audio Compression** + - Goal: Reduce .spec file sizes by 50% + - Approach: Quantize to uint16_t, use RLE encoding + - See: doc/AUDIO_COMPRESSION_PLAN.md for details +``` + +### Bad: Verbose TODO.md Entry +```markdown +- [ ] **Task #72: Implement Audio Compression** + - Goal: Reduce .spec file sizes by 50% + - Background: Currently using float32 which is wasteful... + - Research: Investigated 5 compression algorithms... + - Step 1: Create compression interface + - Define CompressorBase class + - Add compress() and decompress() methods + - Implement factory pattern for algorithm selection + - Step 2: Implement RLE compressor + - [100+ more lines of detailed implementation] +``` + +### Good: Lean PROJECT_CONTEXT.md Status +```markdown +### Current Status +- Audio: Stable, 93% test coverage, variable tempo support +- 3D: Hybrid SDF/raster, BVH acceleration, physics system +- Shaders: Modular WGSL, comprehensive compilation tests +``` + +### Bad: Verbose PROJECT_CONTEXT.md Status +```markdown +### Recently Completed + +#### Audio Peak Measurement (February 7, 2026) +- **Real-Time Peak Fix**: Fixed critical bug where... [20 lines] +- **Peak Decay Optimization**: Changed decay from 0.95 to 0.7... [15 lines] +- **SilentBackend**: Created test backend with... [25 lines] +- [200+ more lines of detailed milestones] +``` + +--- + +## Questions? + +If unsure whether to archive something, ask: + +1. **Is this about work completed >30 days ago?** → Archive +2. **Is this a detailed implementation plan?** → Archive (keep 2-line summary) +3. **Is this needed for next week's work?** → Keep +4. **Is this >50 lines of historical detail?** → Archive +5. **Would a new contributor need this immediately?** → Keep if yes + +When in doubt, **archive with a pointer**. Better to have clean context and occasionally load an archive than to bloat every session. + +--- + +## Maintenance Checklist + +Copy this for monthly reviews: + +```markdown +## Context Maintenance - [Month YYYY] + +### File Sizes (before) +- [ ] PROJECT_CONTEXT.md: ___ lines +- [ ] TODO.md: ___ lines +- [ ] Tier 1-2 total: ___ words + +### Actions Taken +- [ ] Archived tasks >30 days old to COMPLETED.md +- [ ] Moved temp analysis docs to doc/ +- [ ] Updated PROJECT_CONTEXT.md current status +- [ ] Simplified TODO.md task descriptions +- [ ] Verified CLAUDE.md/GEMINI.md config +- [ ] Updated cross-references + +### File Sizes (after) +- [ ] PROJECT_CONTEXT.md: ___ lines (target: <300) +- [ ] TODO.md: ___ lines (target: <500) +- [ ] Tier 1-2 total: ___ words (target: <10K) + +### Notes +[Any issues encountered, files moved, etc.] +``` + +--- + +*Last updated: February 7, 2026* -- cgit v1.2.3