39948-vm/documentation/audio-playback.md

Audio Playback Feature - E2E Documentation

Overview

The Tour Builder Platform implements a comprehensive audio playback system supporting background audio on pages, element-triggered audio effects (hover/click sounds), and media player elements. Key features:

• User Interaction Unlock: All audio waits for first user click/tap before playing (consistent across all browsers)
• Sound Effects Layering: Hover/click effects play over background audio (no interruption)
• Media Player Ducking: AudioPlayer and VideoPlayer elements pause background audio when playing
• Constructor Modes: Edit mode silences audio; Interact mode enables full audio playback

---

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                      Audio Types                                 │
│  Background Audio    │    Element Audio Effects                  │
│  (page-level)        │    (hover/click sounds)                   │
└─────────────────────────────────────────────────────────────────┘
                              │
┌─────────────────────────────────────────────────────────────────┐
│                     Playback Hooks                               │
│  useBackgroundAudioPlayback │ useAudioEffects │ useAudioEventMgr│
└─────────────────────────────────────────────────────────────────┘
                              │
┌─────────────────────────────────────────────────────────────────┐
│                     Controller Layer                             │
│  backgroundAudioController (singleton for audio ducking)         │
└─────────────────────────────────────────────────────────────────┘
                              │
┌─────────────────────────────────────────────────────────────────┐
│                     Rendering                                    │
│  CanvasBackground │ UiElements (hover/click audio)              │
└─────────────────────────────────────────────────────────────────┘

---

Audio Types

1. Background Audio

Page-level ambient audio that plays behind all content with configurable playback settings.

// Using useBackgroundAudioPlayback hook for controlled playback
const { audioRef } = useBackgroundAudioPlayback({
  audioUrl: page.background_audio_url,
  autoplay: page.background_audio_autoplay ?? true,
  loop: page.background_audio_loop ?? true,
  startTime: page.background_audio_start_time ?? null,
  endTime: page.background_audio_end_time ?? null,
});

<audio ref={audioRef} src={audioUrl} preload="auto" hidden />

Configurable Properties (stored in tour_pages):

Property	Type	Default	Description
background_audio_autoplay	boolean	true	Start playback automatically
background_audio_loop	boolean	true	Loop continuously
background_audio_start_time	DECIMAL(10,1) | null	null	Start playback at this time (seconds)
background_audio_end_time	DECIMAL(10,1) | null	null	Stop/loop at this time (seconds)

DECIMAL Parsing (Critical): Sequelize DECIMAL fields return strings from the database (e.g., "2.5" not 2.5). In runtime code, these must be parsed with parseFloat(String(value)) before passing to the hook.

Note: When background_audio_end_time is set, looping is handled via JavaScript (timeupdate event) to properly seek back to startTime.

2. Element Audio Effects

Interactive audio triggered by user hover and click on UI elements.

// Using useAudioEffects hook
const { playClickAudio, stopAll } = useAudioEffects({
  hoverAudioUrl: element.hoverAudioUrl,
  clickAudioUrl: element.clickAudioUrl,
  volume: element.audioVolume ?? 1,
  isHovered,
  isActive,
  resolveUrl: preloadOrchestrator.getReadyBlobUrl,
});

Configurable Properties (stored in element's ui_schema_json):

Property	Type	Description
hoverAudioUrl	string	Audio to play on mouse hover
clickAudioUrl	string	Audio to play on click/tap
audioVolume	number (0-1)	Volume level (iOS ignores this)

---

User Interaction Unlock

All audio waits for first user interaction (click/tap) before playing. This ensures consistent behavior across all browsers (Safari, Chrome, Firefox).

backgroundAudioController

Module-level singleton that manages audio unlock and ducking:

// frontend/src/lib/backgroundAudioController.ts

class BackgroundAudioController {
  private audioElement: HTMLAudioElement | null = null;
  private waitingForInteraction = false;
  private hasUserInteracted = false;

  // Register background audio element
  register(audio: HTMLAudioElement | null): void;

  // Mark audio as waiting for interaction
  setWaitingForInteraction(waiting: boolean): void;

  // Called on first user click/tap - unlocks all audio
  notifyUserInteraction(): void;

  // Check if audio is unlocked
  hasInteracted(): boolean;

  // Ducking methods for media players
  notifyForegroundStart(): void;
  notifyForegroundEnd(): void;
}

Unlock Flow

┌──────────────────────────────────────────────────────────────┐
│  1. Page loads                                                │
│     └── backgroundAudioController.setWaitingForInteraction() │
│     └── All audio is silent                                  │
└──────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌──────────────────────────────────────────────────────────────┐
│  2. User clicks/taps anywhere on canvas                       │
│     └── handleCanvasInteraction() triggered                  │
│     └── backgroundAudioController.notifyUserInteraction()    │
└──────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌──────────────────────────────────────────────────────────────┐
│  3. Audio unlocked                                            │
│     └── Background audio starts playing                      │
│     └── Hover/click effects now work                         │
│     └── hasInteracted() returns true                         │
└──────────────────────────────────────────────────────────────┘

---

Audio Layering & Ducking

Different audio types have different behaviors:

Audio Type	Background Audio Behavior
Hover/click effects	Layers over (plays simultaneously)
AudioPlayer element	Pauses background (ducking)
VideoPlayer element	Pauses background (ducking)

Sound Effects (No Ducking)

Hover and click effects play over background audio without interruption:

// useAudioEffects.ts - no ducking calls
if (backgroundAudioController.hasInteracted()) {
  audio.play().catch(() => undefined);
}

Media Players (With Ducking)

AudioPlayer and VideoPlayer elements pause background audio:

// AudioPlayerElement.tsx / VideoPlayerElement.tsx
useEffect(() => {
  const audio = audioRef.current;
  if (!audio) return;

  const handlePlay = () => backgroundAudioController.notifyForegroundStart();
  const handlePause = () => backgroundAudioController.notifyForegroundEnd();
  const handleEnded = () => backgroundAudioController.notifyForegroundEnd();

  audio.addEventListener('play', handlePlay);
  audio.addEventListener('pause', handlePause);
  audio.addEventListener('ended', handleEnded);

  return () => { /* cleanup */ };
}, []);

Reference Counting

The controller uses reference counting for multiple media players:

notifyForegroundStart(); // count = 1, background pauses
notifyForegroundStart(); // count = 2, still paused
notifyForegroundEnd();   // count = 1, still paused
notifyForegroundEnd();   // count = 0, background resumes

---

Browser Autoplay Policy

All major browsers restrict audio autoplay. Our implementation handles this uniformly.

Browser Policies

Browser	Policy
Safari	Strictest - blocks ALL audio until user interaction
Chrome	Blocks until user interaction or high Media Engagement Index
Firefox	Blocks by default, user-configurable

Our Strategy: Always Wait for Interaction

Instead of trying autoplay and handling failures, we consistently wait for user interaction:

// useBackgroundAudioPlayback.ts
if (!shouldBlockAutoplay) {
  // Always wait for user interaction before playing
  backgroundAudioController.setWaitingForInteraction(true);
}

// RuntimePresentation.tsx & constructor.tsx
const handleCanvasInteraction = useCallback(() => {
  backgroundAudioController.notifyUserInteraction();
}, []);

<div onClick={handleCanvasInteraction} onTouchEnd={handleCanvasInteraction}>

Why onTouchEnd not onTouchStart? iOS Safari only unlocks audio when finger is LIFTED from screen.

Session-Scoped Play Once

When loop=false, audio only plays once per browser session:

// Module-level Set (cleared on browser refresh)
const playedAudios = new Set<string>();

// In useBackgroundAudioPlayback
if (!loop && playedAudios.has(audioStoragePath)) {
  // Skip autoplay - already played this session
  return { shouldBlockAutoplay: true };
}

// After audio plays
audioElement.onended = () => {
  playedAudios.add(audioStoragePath);
};

---

Constructor Mode

Edit vs Interact Mode

Mode	Audio Behavior
Edit Mode	All audio silenced (pauseAudio={true})
Interact Mode	Full audio playback (after user interaction)

The constructor passes pauseAudio={isConstructorEditMode} to CanvasBackground and only triggers handleCanvasInteraction when NOT in edit mode.

Audio Settings Editor

The Constructor provides UI controls for background audio settings via BackgroundSettingsEditor:

// frontend/src/components/Constructor/BackgroundSettingsEditor.tsx

// When type='audio', shows playback settings
<BackgroundSettingsEditor
  type="audio"
  value={backgroundAudioUrl}
  options={audioAssetOptions}
  onChange={setBackgroundAudioUrl}
  audioAutoplay={pageBackground.audioSettings.autoplay}
  audioLoop={pageBackground.audioSettings.loop}
  audioStartTime={pageBackground.audioSettings.startTime}
  audioEndTime={pageBackground.audioSettings.endTime}
  onAudioSettingsChange={setBackgroundAudioSettings}
/>

Settings UI

Setting	Control	Description
Autoplay	Checkbox	Start audio automatically on page load
Loop	Checkbox	Continuously loop audio
Start Time	Number input	Begin playback at specific time (seconds)
End Time	Number input	Stop/loop at specific time (seconds)

---

Runtime Mode (Presentations)

RuntimePresentation Component

Passes audio settings to CanvasBackground for playback:

// frontend/src/components/RuntimePresentation.tsx

// Extract audio URL from navigation state
const backgroundAudioUrl = navCurrentBgAudioUrl;

// Extract settings from selected page
const audioAutoplay = selectedPage?.background_audio_autoplay ?? true;
const audioLoop = selectedPage?.background_audio_loop ?? true;
const audioStartTime = selectedPage?.background_audio_start_time != null
  ? parseFloat(String(selectedPage.background_audio_start_time))
  : null;
const audioEndTime = selectedPage?.background_audio_end_time != null
  ? parseFloat(String(selectedPage.background_audio_end_time))
  : null;

// Pass to CanvasBackground
<CanvasBackground
  backgroundAudioUrl={backgroundAudioUrl}
  audioAutoplay={audioAutoplay}
  audioLoop={audioLoop}
  audioStartTime={audioStartTime}
  audioEndTime={audioEndTime}
  audioStoragePath={selectedPage?.background_audio_url}
/>

CanvasBackground Audio Integration

// frontend/src/components/Constructor/CanvasBackground.tsx

const { audioRef } = useBackgroundAudioPlayback({
  audioUrl: backgroundAudioUrl,
  audioStoragePath,
  autoplay: audioAutoplay,
  loop: audioLoop,
  startTime: audioStartTime,
  endTime: audioEndTime,
});

// Register for audio ducking
useEffect(() => {
  backgroundAudioController.register(audioRef.current);
  return () => backgroundAudioController.register(null);
}, [audioRef.current]);

// Render hidden audio element
{backgroundAudioUrl && (
  <audio ref={audioRef} src={backgroundAudioUrl} preload="auto" hidden />
)}

---

Hooks Reference

useBackgroundAudioPlayback

Manages background audio playback with start/end time control and ducking integration.

interface UseBackgroundAudioPlaybackOptions {
  audioUrl?: string;              // Audio URL (may be blob URL)
  audioStoragePath?: string;      // Original storage path for play-once tracking
  autoplay?: boolean;             // Default: true
  loop?: boolean;                 // Default: true
  startTime?: number | null;      // Start at time (seconds)
  endTime?: number | null;        // Stop/loop at time (seconds)
  paused?: boolean;               // External pause control
}

interface UseBackgroundAudioPlaybackResult {
  audioRef: RefObject<HTMLAudioElement | null>;
  shouldBlockAutoplay: boolean;   // True if already played this session
}

Location: frontend/src/hooks/useBackgroundAudioPlayback.ts (~220 LOC)

useAudioEventManager

Declarative event listener management for audio elements.

interface UseAudioEventManagerOptions {
  audioRef: RefObject<HTMLAudioElement | null>;
  enabled?: boolean;
  handlers: {
    onLoadedMetadata?: (event: Event) => void;
    onTimeUpdate?: (event: Event) => void;
    onEnded?: (event: Event) => void;
    onError?: (event: Event) => void;
    // ... other audio events
  };
}

Location: frontend/src/hooks/audio/useAudioEventManager.ts (~115 LOC)

useAudioEffects

Manages element hover/click audio (layers over background audio without ducking).

interface UseAudioEffectsOptions {
  hoverAudioUrl?: string;
  clickAudioUrl?: string;
  volume?: number;         // 0-1 (iOS ignores)
  isHovered: boolean;
  isActive: boolean;
  resolveUrl?: (url: string) => string;
  resetKey?: string;       // Reset on page navigation
}

interface UseAudioEffectsResult {
  playClickAudio: () => void;
  stopAll: () => void;
}

Location: frontend/src/hooks/useAudioEffects.ts (~302 LOC)

---

Database Schema

Audio settings are stored in the tour_pages table:

-- Added via migration 20260605000001-add-background-audio-settings.js
ALTER TABLE tour_pages
ADD COLUMN background_audio_autoplay BOOLEAN NOT NULL DEFAULT true,
ADD COLUMN background_audio_loop BOOLEAN NOT NULL DEFAULT true,
ADD COLUMN background_audio_start_time DECIMAL(10, 1),
ADD COLUMN background_audio_end_time DECIMAL(10, 1);

---

Key Files Reference

File	Location	LOC	Purpose
useBackgroundAudioPlayback.ts	frontend/src/hooks/	220	Background audio playback control
useAudioEventManager.ts	frontend/src/hooks/audio/	115	Audio event listener management
useAudioEffects.ts	frontend/src/hooks/	302	Element hover/click audio (no ducking)
backgroundAudioController.ts	frontend/src/lib/	81	Audio ducking coordination
BackgroundSettingsEditor.tsx	frontend/src/components/Constructor/	-	Audio settings UI
CanvasBackground.tsx	frontend/src/components/Constructor/	-	Audio element rendering
RuntimePresentation.tsx	frontend/src/components/	-	Runtime audio integration
constructor.tsx	frontend/src/pages/	-	Constructor audio integration
AudioPlayerElement.tsx	frontend/src/components/UiElements/elements/	77	Audio player with ducking
VideoPlayerElement.tsx	frontend/src/components/UiElements/elements/	91	Video player with ducking

---

Troubleshooting

Audio Won't Play

1. Verify user has interacted (click/tap) - check backgroundAudioController.hasInteracted()
2. In constructor, verify you're in Interact mode (not Edit mode)
3. Verify audio URL resolves correctly
4. Check CORS headers on audio source
5. Ensure audio format is supported (MP3/OGG/WAV)

Audio Ducking Not Working

1. Verify backgroundAudioController.register() is called in CanvasBackground
2. Check notifyForegroundStart/End calls in AudioPlayerElement and VideoPlayerElement
3. Ensure foreground audio count is balanced (start/end pairs)

Audio Doesn't Loop Correctly

1. Check if endTime is set (uses JS loop instead of native)
2. Verify startTime is valid for seeking
3. Ensure timeupdate event handler is attached

Memory Leaks

1. Verify blob URLs are revoked on cleanup
2. Check backgroundAudioController.register(null) on unmount
3. Ensure event listeners are removed via useAudioEventManager cleanup