500 lines
18 KiB
Markdown
500 lines
18 KiB
Markdown
# Audio Playback Feature - E2E Documentation
|
|
|
|
## Overview
|
|
|
|
The Tour Builder Platform implements a comprehensive audio playback system supporting background audio on pages, element-triggered audio effects (hover/click sounds), and media player elements. Key features:
|
|
|
|
- **User Interaction Unlock**: All audio waits for first user click/tap before playing (consistent across all browsers)
|
|
- **Sound Effects Layering**: Hover/click effects play over background audio (no interruption)
|
|
- **Media Player Ducking**: AudioPlayer and VideoPlayer elements pause background audio when playing
|
|
- **Constructor Modes**: Edit mode silences audio; Interact mode enables full audio playback
|
|
|
|
---
|
|
|
|
## Architecture
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ Audio Types │
|
|
│ Background Audio │ Element Audio Effects │
|
|
│ (page-level) │ (hover/click sounds) │
|
|
└─────────────────────────────────────────────────────────────────┘
|
|
│
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ Playback Hooks │
|
|
│ useBackgroundAudioPlayback │ useAudioEffects │ useAudioEventMgr│
|
|
└─────────────────────────────────────────────────────────────────┘
|
|
│
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ Controller Layer │
|
|
│ backgroundAudioController (singleton for audio ducking) │
|
|
└─────────────────────────────────────────────────────────────────┘
|
|
│
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ Rendering │
|
|
│ CanvasBackground │ UiElements (hover/click audio) │
|
|
└─────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
---
|
|
|
|
## Audio Types
|
|
|
|
### 1. Background Audio
|
|
|
|
Page-level ambient audio that plays behind all content with configurable playback settings.
|
|
|
|
```tsx
|
|
// Using useBackgroundAudioPlayback hook for controlled playback
|
|
const { audioRef } = useBackgroundAudioPlayback({
|
|
audioUrl: page.background_audio_url,
|
|
autoplay: page.background_audio_autoplay ?? true,
|
|
loop: page.background_audio_loop ?? true,
|
|
startTime: page.background_audio_start_time ?? null,
|
|
endTime: page.background_audio_end_time ?? null,
|
|
});
|
|
|
|
<audio ref={audioRef} src={audioUrl} preload="auto" hidden />
|
|
```
|
|
|
|
**Configurable Properties (stored in `tour_pages`):**
|
|
|
|
| Property | Type | Default | Description |
|
|
|----------|------|---------|-------------|
|
|
| `background_audio_autoplay` | boolean | true | Start playback automatically |
|
|
| `background_audio_loop` | boolean | true | Loop continuously |
|
|
| `background_audio_start_time` | DECIMAL(10,1) \| null | null | Start playback at this time (seconds) |
|
|
| `background_audio_end_time` | DECIMAL(10,1) \| null | null | Stop/loop at this time (seconds) |
|
|
|
|
**DECIMAL Parsing (Critical):** Sequelize DECIMAL fields return strings from the database (e.g., `"2.5"` not `2.5`). In runtime code, these must be parsed with `parseFloat(String(value))` before passing to the hook.
|
|
|
|
**Note:** When `background_audio_end_time` is set, looping is handled via JavaScript (`timeupdate` event) to properly seek back to `startTime`.
|
|
|
|
### 2. Element Audio Effects
|
|
|
|
Interactive audio triggered by user hover and click on UI elements.
|
|
|
|
```tsx
|
|
// Using useAudioEffects hook
|
|
const { playClickAudio, stopAll } = useAudioEffects({
|
|
hoverAudioUrl: element.hoverAudioUrl,
|
|
clickAudioUrl: element.clickAudioUrl,
|
|
volume: element.audioVolume ?? 1,
|
|
isHovered,
|
|
isActive,
|
|
resolveUrl: preloadOrchestrator.getReadyBlobUrl,
|
|
});
|
|
```
|
|
|
|
**Configurable Properties (stored in element's `ui_schema_json`):**
|
|
|
|
| Property | Type | Description |
|
|
|----------|------|-------------|
|
|
| `hoverAudioUrl` | string | Audio to play on mouse hover |
|
|
| `clickAudioUrl` | string | Audio to play on click/tap |
|
|
| `audioVolume` | number (0-1) | Volume level (iOS ignores this) |
|
|
|
|
---
|
|
|
|
## User Interaction Unlock
|
|
|
|
All audio waits for first user interaction (click/tap) before playing. This ensures consistent behavior across all browsers (Safari, Chrome, Firefox).
|
|
|
|
### backgroundAudioController
|
|
|
|
Module-level singleton that manages audio unlock and ducking:
|
|
|
|
```typescript
|
|
// frontend/src/lib/backgroundAudioController.ts
|
|
|
|
class BackgroundAudioController {
|
|
private audioElement: HTMLAudioElement | null = null;
|
|
private waitingForInteraction = false;
|
|
private hasUserInteracted = false;
|
|
|
|
// Register background audio element
|
|
register(audio: HTMLAudioElement | null): void;
|
|
|
|
// Mark audio as waiting for interaction
|
|
setWaitingForInteraction(waiting: boolean): void;
|
|
|
|
// Called on first user click/tap - unlocks all audio
|
|
notifyUserInteraction(): void;
|
|
|
|
// Check if audio is unlocked
|
|
hasInteracted(): boolean;
|
|
|
|
// Ducking methods for media players
|
|
notifyForegroundStart(): void;
|
|
notifyForegroundEnd(): void;
|
|
}
|
|
```
|
|
|
|
### Unlock Flow
|
|
|
|
```
|
|
┌──────────────────────────────────────────────────────────────┐
|
|
│ 1. Page loads │
|
|
│ └── backgroundAudioController.setWaitingForInteraction() │
|
|
│ └── All audio is silent │
|
|
└──────────────────────────────────────────────────────────────┘
|
|
│
|
|
▼
|
|
┌──────────────────────────────────────────────────────────────┐
|
|
│ 2. User clicks/taps anywhere on canvas │
|
|
│ └── handleCanvasInteraction() triggered │
|
|
│ └── backgroundAudioController.notifyUserInteraction() │
|
|
└──────────────────────────────────────────────────────────────┘
|
|
│
|
|
▼
|
|
┌──────────────────────────────────────────────────────────────┐
|
|
│ 3. Audio unlocked │
|
|
│ └── Background audio starts playing │
|
|
│ └── Hover/click effects now work │
|
|
│ └── hasInteracted() returns true │
|
|
└──────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
---
|
|
|
|
## Audio Layering & Ducking
|
|
|
|
Different audio types have different behaviors:
|
|
|
|
| Audio Type | Background Audio Behavior |
|
|
|------------|---------------------------|
|
|
| **Hover/click effects** | Layers over (plays simultaneously) |
|
|
| **AudioPlayer element** | Pauses background (ducking) |
|
|
| **VideoPlayer element** | Pauses background (ducking) |
|
|
|
|
### Sound Effects (No Ducking)
|
|
|
|
Hover and click effects play over background audio without interruption:
|
|
|
|
```typescript
|
|
// useAudioEffects.ts - no ducking calls
|
|
if (backgroundAudioController.hasInteracted()) {
|
|
audio.play().catch(() => undefined);
|
|
}
|
|
```
|
|
|
|
### Media Players (With Ducking)
|
|
|
|
AudioPlayer and VideoPlayer elements pause background audio:
|
|
|
|
```typescript
|
|
// AudioPlayerElement.tsx / VideoPlayerElement.tsx
|
|
useEffect(() => {
|
|
const audio = audioRef.current;
|
|
if (!audio) return;
|
|
|
|
const handlePlay = () => backgroundAudioController.notifyForegroundStart();
|
|
const handlePause = () => backgroundAudioController.notifyForegroundEnd();
|
|
const handleEnded = () => backgroundAudioController.notifyForegroundEnd();
|
|
|
|
audio.addEventListener('play', handlePlay);
|
|
audio.addEventListener('pause', handlePause);
|
|
audio.addEventListener('ended', handleEnded);
|
|
|
|
return () => { /* cleanup */ };
|
|
}, []);
|
|
```
|
|
|
|
### Reference Counting
|
|
|
|
The controller uses reference counting for multiple media players:
|
|
|
|
```typescript
|
|
notifyForegroundStart(); // count = 1, background pauses
|
|
notifyForegroundStart(); // count = 2, still paused
|
|
notifyForegroundEnd(); // count = 1, still paused
|
|
notifyForegroundEnd(); // count = 0, background resumes
|
|
```
|
|
|
|
---
|
|
|
|
## Browser Autoplay Policy
|
|
|
|
All major browsers restrict audio autoplay. Our implementation handles this uniformly.
|
|
|
|
### Browser Policies
|
|
|
|
| Browser | Policy |
|
|
|---------|--------|
|
|
| **Safari** | Strictest - blocks ALL audio until user interaction |
|
|
| **Chrome** | Blocks until user interaction or high Media Engagement Index |
|
|
| **Firefox** | Blocks by default, user-configurable |
|
|
|
|
### Our Strategy: Always Wait for Interaction
|
|
|
|
Instead of trying autoplay and handling failures, we consistently wait for user interaction:
|
|
|
|
```typescript
|
|
// useBackgroundAudioPlayback.ts
|
|
if (!shouldBlockAutoplay) {
|
|
// Always wait for user interaction before playing
|
|
backgroundAudioController.setWaitingForInteraction(true);
|
|
}
|
|
|
|
// RuntimePresentation.tsx & constructor.tsx
|
|
const handleCanvasInteraction = useCallback(() => {
|
|
backgroundAudioController.notifyUserInteraction();
|
|
}, []);
|
|
|
|
<div onClick={handleCanvasInteraction} onTouchEnd={handleCanvasInteraction}>
|
|
```
|
|
|
|
**Why `onTouchEnd` not `onTouchStart`?** iOS Safari only unlocks audio when finger is LIFTED from screen.
|
|
|
|
### Session-Scoped Play Once
|
|
|
|
When `loop=false`, audio only plays once per browser session:
|
|
|
|
```typescript
|
|
// Module-level Set (cleared on browser refresh)
|
|
const playedAudios = new Set<string>();
|
|
|
|
// In useBackgroundAudioPlayback
|
|
if (!loop && playedAudios.has(audioStoragePath)) {
|
|
// Skip autoplay - already played this session
|
|
return { shouldBlockAutoplay: true };
|
|
}
|
|
|
|
// After audio plays
|
|
audioElement.onended = () => {
|
|
playedAudios.add(audioStoragePath);
|
|
};
|
|
```
|
|
|
|
---
|
|
|
|
## Constructor Mode
|
|
|
|
### Edit vs Interact Mode
|
|
|
|
| Mode | Audio Behavior |
|
|
|------|---------------|
|
|
| **Edit Mode** | All audio silenced (`pauseAudio={true}`) |
|
|
| **Interact Mode** | Full audio playback (after user interaction) |
|
|
|
|
The constructor passes `pauseAudio={isConstructorEditMode}` to CanvasBackground and only triggers `handleCanvasInteraction` when NOT in edit mode.
|
|
|
|
### Audio Settings Editor
|
|
|
|
The Constructor provides UI controls for background audio settings via `BackgroundSettingsEditor`:
|
|
|
|
```tsx
|
|
// frontend/src/components/Constructor/BackgroundSettingsEditor.tsx
|
|
|
|
// When type='audio', shows playback settings
|
|
<BackgroundSettingsEditor
|
|
type="audio"
|
|
value={backgroundAudioUrl}
|
|
options={audioAssetOptions}
|
|
onChange={setBackgroundAudioUrl}
|
|
audioAutoplay={pageBackground.audioSettings.autoplay}
|
|
audioLoop={pageBackground.audioSettings.loop}
|
|
audioStartTime={pageBackground.audioSettings.startTime}
|
|
audioEndTime={pageBackground.audioSettings.endTime}
|
|
onAudioSettingsChange={setBackgroundAudioSettings}
|
|
/>
|
|
```
|
|
|
|
### Settings UI
|
|
|
|
| Setting | Control | Description |
|
|
|---------|---------|-------------|
|
|
| Autoplay | Checkbox | Start audio automatically on page load |
|
|
| Loop | Checkbox | Continuously loop audio |
|
|
| Start Time | Number input | Begin playback at specific time (seconds) |
|
|
| End Time | Number input | Stop/loop at specific time (seconds) |
|
|
|
|
---
|
|
|
|
## Runtime Mode (Presentations)
|
|
|
|
### RuntimePresentation Component
|
|
|
|
Passes audio settings to CanvasBackground for playback:
|
|
|
|
```tsx
|
|
// frontend/src/components/RuntimePresentation.tsx
|
|
|
|
// Extract audio URL from navigation state
|
|
const backgroundAudioUrl = navCurrentBgAudioUrl;
|
|
|
|
// Extract settings from selected page
|
|
const audioAutoplay = selectedPage?.background_audio_autoplay ?? true;
|
|
const audioLoop = selectedPage?.background_audio_loop ?? true;
|
|
const audioStartTime = selectedPage?.background_audio_start_time != null
|
|
? parseFloat(String(selectedPage.background_audio_start_time))
|
|
: null;
|
|
const audioEndTime = selectedPage?.background_audio_end_time != null
|
|
? parseFloat(String(selectedPage.background_audio_end_time))
|
|
: null;
|
|
|
|
// Pass to CanvasBackground
|
|
<CanvasBackground
|
|
backgroundAudioUrl={backgroundAudioUrl}
|
|
audioAutoplay={audioAutoplay}
|
|
audioLoop={audioLoop}
|
|
audioStartTime={audioStartTime}
|
|
audioEndTime={audioEndTime}
|
|
audioStoragePath={selectedPage?.background_audio_url}
|
|
/>
|
|
```
|
|
|
|
### CanvasBackground Audio Integration
|
|
|
|
```tsx
|
|
// frontend/src/components/Constructor/CanvasBackground.tsx
|
|
|
|
const { audioRef } = useBackgroundAudioPlayback({
|
|
audioUrl: backgroundAudioUrl,
|
|
audioStoragePath,
|
|
autoplay: audioAutoplay,
|
|
loop: audioLoop,
|
|
startTime: audioStartTime,
|
|
endTime: audioEndTime,
|
|
});
|
|
|
|
// Register for audio ducking
|
|
useEffect(() => {
|
|
backgroundAudioController.register(audioRef.current);
|
|
return () => backgroundAudioController.register(null);
|
|
}, [audioRef.current]);
|
|
|
|
// Render hidden audio element
|
|
{backgroundAudioUrl && (
|
|
<audio ref={audioRef} src={backgroundAudioUrl} preload="auto" hidden />
|
|
)}
|
|
```
|
|
|
|
---
|
|
|
|
## Hooks Reference
|
|
|
|
### useBackgroundAudioPlayback
|
|
|
|
Manages background audio playback with start/end time control and ducking integration.
|
|
|
|
```typescript
|
|
interface UseBackgroundAudioPlaybackOptions {
|
|
audioUrl?: string; // Audio URL (may be blob URL)
|
|
audioStoragePath?: string; // Original storage path for play-once tracking
|
|
autoplay?: boolean; // Default: true
|
|
loop?: boolean; // Default: true
|
|
startTime?: number | null; // Start at time (seconds)
|
|
endTime?: number | null; // Stop/loop at time (seconds)
|
|
paused?: boolean; // External pause control
|
|
}
|
|
|
|
interface UseBackgroundAudioPlaybackResult {
|
|
audioRef: RefObject<HTMLAudioElement | null>;
|
|
shouldBlockAutoplay: boolean; // True if already played this session
|
|
}
|
|
```
|
|
|
|
**Location:** `frontend/src/hooks/useBackgroundAudioPlayback.ts` (~220 LOC)
|
|
|
|
### useAudioEventManager
|
|
|
|
Declarative event listener management for audio elements.
|
|
|
|
```typescript
|
|
interface UseAudioEventManagerOptions {
|
|
audioRef: RefObject<HTMLAudioElement | null>;
|
|
enabled?: boolean;
|
|
handlers: {
|
|
onLoadedMetadata?: (event: Event) => void;
|
|
onTimeUpdate?: (event: Event) => void;
|
|
onEnded?: (event: Event) => void;
|
|
onError?: (event: Event) => void;
|
|
// ... other audio events
|
|
};
|
|
}
|
|
```
|
|
|
|
**Location:** `frontend/src/hooks/audio/useAudioEventManager.ts` (~115 LOC)
|
|
|
|
### useAudioEffects
|
|
|
|
Manages element hover/click audio (layers over background audio without ducking).
|
|
|
|
```typescript
|
|
interface UseAudioEffectsOptions {
|
|
hoverAudioUrl?: string;
|
|
clickAudioUrl?: string;
|
|
volume?: number; // 0-1 (iOS ignores)
|
|
isHovered: boolean;
|
|
isActive: boolean;
|
|
resolveUrl?: (url: string) => string;
|
|
resetKey?: string; // Reset on page navigation
|
|
}
|
|
|
|
interface UseAudioEffectsResult {
|
|
playClickAudio: () => void;
|
|
stopAll: () => void;
|
|
}
|
|
```
|
|
|
|
**Location:** `frontend/src/hooks/useAudioEffects.ts` (~302 LOC)
|
|
|
|
---
|
|
|
|
## Database Schema
|
|
|
|
Audio settings are stored in the `tour_pages` table:
|
|
|
|
```sql
|
|
-- Added via migration 20260605000001-add-background-audio-settings.js
|
|
ALTER TABLE tour_pages
|
|
ADD COLUMN background_audio_autoplay BOOLEAN NOT NULL DEFAULT true,
|
|
ADD COLUMN background_audio_loop BOOLEAN NOT NULL DEFAULT true,
|
|
ADD COLUMN background_audio_start_time DECIMAL(10, 1),
|
|
ADD COLUMN background_audio_end_time DECIMAL(10, 1);
|
|
```
|
|
|
|
---
|
|
|
|
## Key Files Reference
|
|
|
|
| File | Location | LOC | Purpose |
|
|
|------|----------|-----|---------|
|
|
| `useBackgroundAudioPlayback.ts` | `frontend/src/hooks/` | 220 | Background audio playback control |
|
|
| `useAudioEventManager.ts` | `frontend/src/hooks/audio/` | 115 | Audio event listener management |
|
|
| `useAudioEffects.ts` | `frontend/src/hooks/` | 302 | Element hover/click audio (no ducking) |
|
|
| `backgroundAudioController.ts` | `frontend/src/lib/` | 81 | Audio ducking coordination |
|
|
| `BackgroundSettingsEditor.tsx` | `frontend/src/components/Constructor/` | - | Audio settings UI |
|
|
| `CanvasBackground.tsx` | `frontend/src/components/Constructor/` | - | Audio element rendering |
|
|
| `RuntimePresentation.tsx` | `frontend/src/components/` | - | Runtime audio integration |
|
|
| `constructor.tsx` | `frontend/src/pages/` | - | Constructor audio integration |
|
|
| `AudioPlayerElement.tsx` | `frontend/src/components/UiElements/elements/` | 77 | Audio player with ducking |
|
|
| `VideoPlayerElement.tsx` | `frontend/src/components/UiElements/elements/` | 91 | Video player with ducking |
|
|
|
|
---
|
|
|
|
## Troubleshooting
|
|
|
|
### Audio Won't Play
|
|
1. Verify user has interacted (click/tap) - check `backgroundAudioController.hasInteracted()`
|
|
2. In constructor, verify you're in Interact mode (not Edit mode)
|
|
3. Verify audio URL resolves correctly
|
|
4. Check CORS headers on audio source
|
|
5. Ensure audio format is supported (MP3/OGG/WAV)
|
|
|
|
### Audio Ducking Not Working
|
|
1. Verify `backgroundAudioController.register()` is called in CanvasBackground
|
|
2. Check `notifyForegroundStart/End` calls in AudioPlayerElement and VideoPlayerElement
|
|
3. Ensure foreground audio count is balanced (start/end pairs)
|
|
|
|
### Audio Doesn't Loop Correctly
|
|
1. Check if `endTime` is set (uses JS loop instead of native)
|
|
2. Verify `startTime` is valid for seeking
|
|
3. Ensure `timeupdate` event handler is attached
|
|
|
|
### Memory Leaks
|
|
1. Verify blob URLs are revoked on cleanup
|
|
2. Check `backgroundAudioController.register(null)` on unmount
|
|
3. Ensure event listeners are removed via useAudioEventManager cleanup
|