# Audio Playback Feature - E2E Documentation

## Overview

The Tour Builder Platform implements a comprehensive audio playback system supporting background audio on pages, element-triggered audio effects (hover/click sounds), and media player elements. Key features:

- **User Interaction Unlock**: All audio waits for first user click/tap before playing (consistent across all browsers)
- **Sound Effects Layering**: Hover/click effects play over background audio (no interruption)
- **Media Player Ducking**: AudioPlayer and VideoPlayer elements pause background audio when playing
- **Constructor Modes**: Edit mode silences audio; Interact mode enables full audio playback

---

## Architecture

```
┌─────────────────────────────────────────────────────────────────┐
│                      Audio Types                                 │
│  Background Audio    │    Element Audio Effects                  │
│  (page-level)        │    (hover/click sounds)                   │
└─────────────────────────────────────────────────────────────────┘
                              │
┌─────────────────────────────────────────────────────────────────┐
│                     Playback Hooks                               │
│  useBackgroundAudioPlayback │ useAudioEffects │ useAudioEventMgr│
└─────────────────────────────────────────────────────────────────┘
                              │
┌─────────────────────────────────────────────────────────────────┐
│                     Controller Layer                             │
│  backgroundAudioController (singleton for audio ducking)         │
└─────────────────────────────────────────────────────────────────┘
                              │
┌─────────────────────────────────────────────────────────────────┐
│                     Rendering                                    │
│  CanvasBackground │ UiElements (hover/click audio)              │
└─────────────────────────────────────────────────────────────────┘
```

---

## Audio Types

### 1. Background Audio

Page-level ambient audio that plays behind all content with configurable playback settings.

```tsx
// Using useBackgroundAudioPlayback hook for controlled playback
const { audioRef } = useBackgroundAudioPlayback({
  audioUrl: page.background_audio_url,
  autoplay: page.background_audio_autoplay ?? true,
  loop: page.background_audio_loop ?? true,
  startTime: page.background_audio_start_time ?? null,
  endTime: page.background_audio_end_time ?? null,
});

<audio ref={audioRef} src={audioUrl} preload="auto" hidden />
```

**Configurable Properties (stored in `tour_pages`):**

| Property | Type | Default | Description |
|----------|------|---------|-------------|
| `background_audio_autoplay` | boolean | true | Start playback automatically |
| `background_audio_loop` | boolean | true | Loop continuously |
| `background_audio_start_time` | DECIMAL(10,1) \| null | null | Start playback at this time (seconds) |
| `background_audio_end_time` | DECIMAL(10,1) \| null | null | Stop/loop at this time (seconds) |

**DECIMAL Parsing (Critical):** Sequelize DECIMAL fields return strings from the database (e.g., `"2.5"` not `2.5`). In runtime code, these must be parsed with `parseFloat(String(value))` before passing to the hook.

**Note:** When `background_audio_end_time` is set, looping is handled via JavaScript (`timeupdate` event) to properly seek back to `startTime`.

### 2. Element Audio Effects

Interactive audio triggered by user hover and click on UI elements.

```tsx
// Using useAudioEffects hook
const { playClickAudio, stopAll } = useAudioEffects({
  hoverAudioUrl: element.hoverAudioUrl,
  clickAudioUrl: element.clickAudioUrl,
  volume: element.audioVolume ?? 1,
  isHovered,
  isActive,
  resolveUrl: preloadOrchestrator.getReadyBlobUrl,
});
```

**Configurable Properties (stored in element's `ui_schema_json`):**

| Property | Type | Description |
|----------|------|-------------|
| `hoverAudioUrl` | string | Audio to play on mouse hover |
| `clickAudioUrl` | string | Audio to play on click/tap |
| `audioVolume` | number (0-1) | Volume level (iOS ignores this) |

---

## User Interaction Unlock

All audio waits for first user interaction (click/tap) before playing. This ensures consistent behavior across all browsers (Safari, Chrome, Firefox).

### backgroundAudioController

Module-level singleton that manages audio unlock and ducking:

```typescript
// frontend/src/lib/backgroundAudioController.ts

class BackgroundAudioController {
  private audioElement: HTMLAudioElement | null = null;
  private waitingForInteraction = false;
  private hasUserInteracted = false;

  // Register background audio element
  register(audio: HTMLAudioElement | null): void;

  // Mark audio as waiting for interaction
  setWaitingForInteraction(waiting: boolean): void;

  // Called on first user click/tap - unlocks all audio
  notifyUserInteraction(): void;

  // Check if audio is unlocked
  hasInteracted(): boolean;

  // Ducking methods for media players
  notifyForegroundStart(): void;
  notifyForegroundEnd(): void;
}
```

### Unlock Flow

```
┌──────────────────────────────────────────────────────────────┐
│  1. Page loads                                                │
│     └── backgroundAudioController.setWaitingForInteraction() │
│     └── All audio is silent                                  │
└──────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌──────────────────────────────────────────────────────────────┐
│  2. User clicks/taps anywhere on canvas                       │
│     └── handleCanvasInteraction() triggered                  │
│     └── backgroundAudioController.notifyUserInteraction()    │
└──────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌──────────────────────────────────────────────────────────────┐
│  3. Audio unlocked                                            │
│     └── Background audio starts playing                      │
│     └── Hover/click effects now work                         │
│     └── hasInteracted() returns true                         │
└──────────────────────────────────────────────────────────────┘
```

---

## Audio Layering & Ducking

Different audio types have different behaviors:

| Audio Type | Background Audio Behavior |
|------------|---------------------------|
| **Hover/click effects** | Layers over (plays simultaneously) |
| **AudioPlayer element** | Pauses background (ducking) |
| **VideoPlayer element** | Pauses background (ducking) |

### Sound Effects (No Ducking)

Hover and click effects play over background audio without interruption:

```typescript
// useAudioEffects.ts - no ducking calls
if (backgroundAudioController.hasInteracted()) {
  audio.play().catch(() => undefined);
}
```

### Media Players (With Ducking)

AudioPlayer and VideoPlayer elements pause background audio:

```typescript
// AudioPlayerElement.tsx / VideoPlayerElement.tsx
useEffect(() => {
  const audio = audioRef.current;
  if (!audio) return;

  const handlePlay = () => backgroundAudioController.notifyForegroundStart();
  const handlePause = () => backgroundAudioController.notifyForegroundEnd();
  const handleEnded = () => backgroundAudioController.notifyForegroundEnd();

  audio.addEventListener('play', handlePlay);
  audio.addEventListener('pause', handlePause);
  audio.addEventListener('ended', handleEnded);

  return () => { /* cleanup */ };
}, []);
```

### Reference Counting

The controller uses reference counting for multiple media players:

```typescript
notifyForegroundStart(); // count = 1, background pauses
notifyForegroundStart(); // count = 2, still paused
notifyForegroundEnd();   // count = 1, still paused
notifyForegroundEnd();   // count = 0, background resumes
```

---

## Browser Autoplay Policy

All major browsers restrict audio autoplay. Our implementation handles this uniformly.

### Browser Policies

| Browser | Policy |
|---------|--------|
| **Safari** | Strictest - blocks ALL audio until user interaction |
| **Chrome** | Blocks until user interaction or high Media Engagement Index |
| **Firefox** | Blocks by default, user-configurable |

### Our Strategy: Always Wait for Interaction

Instead of trying autoplay and handling failures, we consistently wait for user interaction:

```typescript
// useBackgroundAudioPlayback.ts
if (!shouldBlockAutoplay) {
  // Always wait for user interaction before playing
  backgroundAudioController.setWaitingForInteraction(true);
}

// RuntimePresentation.tsx & constructor.tsx
const handleCanvasInteraction = useCallback(() => {
  backgroundAudioController.notifyUserInteraction();
}, []);

<div onClick={handleCanvasInteraction} onTouchEnd={handleCanvasInteraction}>
```

**Why `onTouchEnd` not `onTouchStart`?** iOS Safari only unlocks audio when finger is LIFTED from screen.

### Session-Scoped Play Once

When `loop=false`, audio only plays once per browser session:

```typescript
// Module-level Set (cleared on browser refresh)
const playedAudios = new Set<string>();

// In useBackgroundAudioPlayback
if (!loop && playedAudios.has(audioStoragePath)) {
  // Skip autoplay - already played this session
  return { shouldBlockAutoplay: true };
}

// After audio plays
audioElement.onended = () => {
  playedAudios.add(audioStoragePath);
};
```

---

## Constructor Mode

### Edit vs Interact Mode

| Mode | Audio Behavior |
|------|---------------|
| **Edit Mode** | All audio silenced (`pauseAudio={true}`) |
| **Interact Mode** | Full audio playback (after user interaction) |

The constructor passes `pauseAudio={isConstructorEditMode}` to CanvasBackground and only triggers `handleCanvasInteraction` when NOT in edit mode.

### Audio Settings Editor

The Constructor provides UI controls for background audio settings via `BackgroundSettingsEditor`:

```tsx
// frontend/src/components/Constructor/BackgroundSettingsEditor.tsx

// When type='audio', shows playback settings
<BackgroundSettingsEditor
  type="audio"
  value={backgroundAudioUrl}
  options={audioAssetOptions}
  onChange={setBackgroundAudioUrl}
  audioAutoplay={pageBackground.audioSettings.autoplay}
  audioLoop={pageBackground.audioSettings.loop}
  audioStartTime={pageBackground.audioSettings.startTime}
  audioEndTime={pageBackground.audioSettings.endTime}
  onAudioSettingsChange={setBackgroundAudioSettings}
/>
```

### Settings UI

| Setting | Control | Description |
|---------|---------|-------------|
| Autoplay | Checkbox | Start audio automatically on page load |
| Loop | Checkbox | Continuously loop audio |
| Start Time | Number input | Begin playback at specific time (seconds) |
| End Time | Number input | Stop/loop at specific time (seconds) |

---

## Runtime Mode (Presentations)

### RuntimePresentation Component

Passes audio settings to CanvasBackground for playback:

```tsx
// frontend/src/components/RuntimePresentation.tsx

// Extract audio URL from navigation state
const backgroundAudioUrl = navCurrentBgAudioUrl;

// Extract settings from selected page
const audioAutoplay = selectedPage?.background_audio_autoplay ?? true;
const audioLoop = selectedPage?.background_audio_loop ?? true;
const audioStartTime = selectedPage?.background_audio_start_time != null
  ? parseFloat(String(selectedPage.background_audio_start_time))
  : null;
const audioEndTime = selectedPage?.background_audio_end_time != null
  ? parseFloat(String(selectedPage.background_audio_end_time))
  : null;

// Pass to CanvasBackground
<CanvasBackground
  backgroundAudioUrl={backgroundAudioUrl}
  audioAutoplay={audioAutoplay}
  audioLoop={audioLoop}
  audioStartTime={audioStartTime}
  audioEndTime={audioEndTime}
  audioStoragePath={selectedPage?.background_audio_url}
/>
```

### CanvasBackground Audio Integration

```tsx
// frontend/src/components/Constructor/CanvasBackground.tsx

const { audioRef } = useBackgroundAudioPlayback({
  audioUrl: backgroundAudioUrl,
  audioStoragePath,
  autoplay: audioAutoplay,
  loop: audioLoop,
  startTime: audioStartTime,
  endTime: audioEndTime,
});

// Register for audio ducking
useEffect(() => {
  backgroundAudioController.register(audioRef.current);
  return () => backgroundAudioController.register(null);
}, [audioRef.current]);

// Render hidden audio element
{backgroundAudioUrl && (
  <audio ref={audioRef} src={backgroundAudioUrl} preload="auto" hidden />
)}
```

---

## Hooks Reference

### useBackgroundAudioPlayback

Manages background audio playback with start/end time control and ducking integration.

```typescript
interface UseBackgroundAudioPlaybackOptions {
  audioUrl?: string;              // Audio URL (may be blob URL)
  audioStoragePath?: string;      // Original storage path for play-once tracking
  autoplay?: boolean;             // Default: true
  loop?: boolean;                 // Default: true
  startTime?: number | null;      // Start at time (seconds)
  endTime?: number | null;        // Stop/loop at time (seconds)
  paused?: boolean;               // External pause control
}

interface UseBackgroundAudioPlaybackResult {
  audioRef: RefObject<HTMLAudioElement | null>;
  shouldBlockAutoplay: boolean;   // True if already played this session
}
```

**Location:** `frontend/src/hooks/useBackgroundAudioPlayback.ts` (~220 LOC)

### useAudioEventManager

Declarative event listener management for audio elements.

```typescript
interface UseAudioEventManagerOptions {
  audioRef: RefObject<HTMLAudioElement | null>;
  enabled?: boolean;
  handlers: {
    onLoadedMetadata?: (event: Event) => void;
    onTimeUpdate?: (event: Event) => void;
    onEnded?: (event: Event) => void;
    onError?: (event: Event) => void;
    // ... other audio events
  };
}
```

**Location:** `frontend/src/hooks/audio/useAudioEventManager.ts` (~115 LOC)

### useAudioEffects

Manages element hover/click audio (layers over background audio without ducking).

```typescript
interface UseAudioEffectsOptions {
  hoverAudioUrl?: string;
  clickAudioUrl?: string;
  volume?: number;         // 0-1 (iOS ignores)
  isHovered: boolean;
  isActive: boolean;
  resolveUrl?: (url: string) => string;
  resetKey?: string;       // Reset on page navigation
}

interface UseAudioEffectsResult {
  playClickAudio: () => void;
  stopAll: () => void;
}
```

**Location:** `frontend/src/hooks/useAudioEffects.ts` (~302 LOC)

---

## Database Schema

Audio settings are stored in the `tour_pages` table:

```sql
-- Added via migration 20260605000001-add-background-audio-settings.js
ALTER TABLE tour_pages
ADD COLUMN background_audio_autoplay BOOLEAN NOT NULL DEFAULT true,
ADD COLUMN background_audio_loop BOOLEAN NOT NULL DEFAULT true,
ADD COLUMN background_audio_start_time DECIMAL(10, 1),
ADD COLUMN background_audio_end_time DECIMAL(10, 1);
```

---

## Key Files Reference

| File | Location | LOC | Purpose |
|------|----------|-----|---------|
| `useBackgroundAudioPlayback.ts` | `frontend/src/hooks/` | 220 | Background audio playback control |
| `useAudioEventManager.ts` | `frontend/src/hooks/audio/` | 115 | Audio event listener management |
| `useAudioEffects.ts` | `frontend/src/hooks/` | 302 | Element hover/click audio (no ducking) |
| `backgroundAudioController.ts` | `frontend/src/lib/` | 81 | Audio ducking coordination |
| `BackgroundSettingsEditor.tsx` | `frontend/src/components/Constructor/` | - | Audio settings UI |
| `CanvasBackground.tsx` | `frontend/src/components/Constructor/` | - | Audio element rendering |
| `RuntimePresentation.tsx` | `frontend/src/components/` | - | Runtime audio integration |
| `constructor.tsx` | `frontend/src/pages/` | - | Constructor audio integration |
| `AudioPlayerElement.tsx` | `frontend/src/components/UiElements/elements/` | 77 | Audio player with ducking |
| `VideoPlayerElement.tsx` | `frontend/src/components/UiElements/elements/` | 91 | Video player with ducking |

---

## Troubleshooting

### Audio Won't Play
1. Verify user has interacted (click/tap) - check `backgroundAudioController.hasInteracted()`
2. In constructor, verify you're in Interact mode (not Edit mode)
3. Verify audio URL resolves correctly
4. Check CORS headers on audio source
5. Ensure audio format is supported (MP3/OGG/WAV)

### Audio Ducking Not Working
1. Verify `backgroundAudioController.register()` is called in CanvasBackground
2. Check `notifyForegroundStart/End` calls in AudioPlayerElement and VideoPlayerElement
3. Ensure foreground audio count is balanced (start/end pairs)

### Audio Doesn't Loop Correctly
1. Check if `endTime` is set (uses JS loop instead of native)
2. Verify `startTime` is valid for seeking
3. Ensure `timeupdate` event handler is attached

### Memory Leaks
1. Verify blob URLs are revoked on cleanup
2. Check `backgroundAudioController.register(null)` on unmount
3. Ensure event listeners are removed via useAudioEventManager cleanup