1078 lines
38 KiB
Markdown
1078 lines
38 KiB
Markdown
# Video Playback Feature - E2E Documentation
|
|
|
|
## Overview
|
|
|
|
The Tour Builder Platform implements a comprehensive video playback system supporting forward and reverse playback modes. The system handles background videos, video player elements, and transition animations between pages in both Constructor (editing) and Runtime (presentation) modes.
|
|
|
|
---
|
|
|
|
## Architecture
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ Video Types │
|
|
│ Background Videos │ Video Player Elements │ Transition Videos │
|
|
└─────────────────────────────────────────────────────────────────┘
|
|
│
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ Playback Hooks │
|
|
│ useTransitionPlayback │ useReversePlayback │ usePageSwitch │
|
|
│ useBackgroundVideoPlayback (background video time control) │
|
|
└─────────────────────────────────────────────────────────────────┘
|
|
│
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ Preloading Layer │
|
|
│ usePreloadOrchestrator │ getReadyBlobUrl (O(1)) │ S3 Presign │
|
|
└─────────────────────────────────────────────────────────────────┘
|
|
│
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ Utilities │
|
|
│ mediaDuration.ts │ assetUrl.ts │ StorageManager (Cache API) │
|
|
└─────────────────────────────────────────────────────────────────┘
|
|
│
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ Rendering │
|
|
│ Constructor Page │ RuntimePresentation │ Transition Overlay │
|
|
└─────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
---
|
|
|
|
## Video Types
|
|
|
|
### 1. Background Videos
|
|
|
|
Full-screen videos that play behind page content with configurable playback settings.
|
|
|
|
```tsx
|
|
// Using useBackgroundVideoPlayback hook for custom time control
|
|
const { videoRef, shouldBlockAutoplay } = useBackgroundVideoPlayback({
|
|
videoUrl: page.background_video_url,
|
|
autoplay: page.background_video_autoplay ?? true,
|
|
loop: page.background_video_loop ?? true,
|
|
muted: page.background_video_muted ?? true,
|
|
startTime: page.background_video_start_time ?? null,
|
|
endTime: page.background_video_end_time ?? null,
|
|
});
|
|
|
|
// Block autoplay if video already played this session (when loop=false)
|
|
const effectiveAutoplay = (page.background_video_autoplay ?? true) && !shouldBlockAutoplay;
|
|
|
|
<video
|
|
ref={videoRef}
|
|
src={page.background_video_url}
|
|
autoPlay={effectiveAutoplay}
|
|
loop={page.background_video_end_time == null ? (page.background_video_loop ?? true) : false}
|
|
muted={page.background_video_muted ?? true}
|
|
playsInline
|
|
className="absolute inset-0 w-full h-full object-cover"
|
|
/>
|
|
```
|
|
|
|
**Configurable Properties (stored in `tour_pages`):**
|
|
|
|
| Property | Type | Default | Description |
|
|
|----------|------|---------|-------------|
|
|
| `background_video_autoplay` | boolean | true | Start playback automatically |
|
|
| `background_video_loop` | boolean | true | Loop continuously |
|
|
| `background_video_muted` | boolean | true | Mute audio (required for autoplay policy) |
|
|
| `background_video_start_time` | DECIMAL(10,1) \| null | null | Start playback at this time (seconds, 0.1s precision) |
|
|
| `background_video_end_time` | DECIMAL(10,1) \| null | null | Stop/loop at this time (seconds, 0.1s precision) |
|
|
|
|
**DECIMAL Parsing (Critical):** Sequelize DECIMAL fields return strings from the database (e.g., `"2.5"` not `2.5`). In runtime code, these must be parsed with `parseFloat(String(value))` before passing to the hook.
|
|
|
|
**Note:** When `background_video_end_time` is set, native HTML5 `loop` attribute is disabled and looping is handled via JavaScript (`timeupdate` event) to properly seek back to `startTime`.
|
|
|
|
**Session-Scoped Play Once Behavior (when loop=false):**
|
|
When loop is disabled, videos only play once per browser session:
|
|
1. Video plays normally on first navigation to the page
|
|
2. When video ends, the video URL is added to a session-scoped Set (`playedVideos`)
|
|
3. On subsequent visits within the same session, autoplay is blocked and video shows last frame
|
|
4. Session tracking resets on browser refresh (module-level state, not persisted)
|
|
|
|
#### Global Sound Control and Autoplay Compatibility
|
|
|
|
Browsers block unmuted autoplay until the user interacts with the page. iOS WebKit is especially strict and can show native play overlays for unmuted videos. The platform handles this with a shared global mute state:
|
|
|
|
**Problem:** unmuted background audio/video, hover/click effects, and media-player sound cannot reliably autoplay before user interaction.
|
|
|
|
**Solution:**
|
|
1. **CSS Layer** - Hide native iOS video controls via `-webkit-media-controls-*` selectors in `main.css`
|
|
2. **Always Start Muted** - Presentation audio starts muted for autoplay compatibility
|
|
3. **Global Sound Control** - `backgroundAudioController` stores shared mute state; `useGlobalAudioMute` exposes it to React
|
|
4. **Custom Sound Button** - `useVideoSoundControl` drives the RuntimeControls sound button and decides when it should be visible
|
|
5. **webkit-playsinline** - Legacy attribute added to `<video>` for older iOS versions
|
|
|
|
```tsx
|
|
// useVideoSoundControl hook usage in RuntimePresentation
|
|
const soundControl = useVideoSoundControl({
|
|
pageHasSound: page.background_video_muted === false,
|
|
hasBackgroundVideo: Boolean(backgroundVideoUrl),
|
|
hasBackgroundAudio: Boolean(backgroundAudioUrl),
|
|
hasElementAudio,
|
|
});
|
|
|
|
// Background video follows global presentation mute state
|
|
<video
|
|
muted={soundControl.isMuted}
|
|
playsInline
|
|
webkit-playsinline=""
|
|
/>
|
|
|
|
// Sound toggle button in RuntimeControls
|
|
<RuntimeControls
|
|
showSoundButton={soundControl.showSoundButton}
|
|
isMuted={soundControl.isMuted}
|
|
onSoundToggle={soundControl.toggleSound}
|
|
/>
|
|
```
|
|
|
|
**Key Behaviors:**
|
|
| Behavior | Description |
|
|
|----------|-------------|
|
|
| Start muted | Presentation sound starts muted for browser autoplay compatibility |
|
|
| Sound button | Appears when the page has background video sound, background audio, hover/click audio, media players, or gallery/info-panel videos |
|
|
| Global scope | The same mute state controls background audio, background video, hover/click effects, audio/video player elements, and gallery/info-panel videos |
|
|
| Fullscreen gallery | RuntimeControls are hidden while the fullscreen gallery is open, so gallery videos render a small sound button connected to the same global mute state |
|
|
| User control | User can unmute via the sound button in RuntimeControls |
|
|
| Event isolation | Unmuting does not replay already-active hover/click effects; sound effects only play on their own fresh `false -> true` hover/active transition |
|
|
| UI chrome isolation | Runtime/global sound buttons stop event propagation so control clicks do not reach canvas or presentation elements |
|
|
|
|
**Files Involved:**
|
|
- `frontend/src/css/main.css` - CSS to hide native iOS controls
|
|
- `frontend/src/lib/backgroundAudioController.ts` - Shared interaction, ducking, and mute state controller
|
|
- `frontend/src/hooks/useGlobalAudioMute.ts` - React subscription hook for global mute state
|
|
- `frontend/src/hooks/useVideoSoundControl.ts` - RuntimeControls sound-button visibility and actions
|
|
- `frontend/src/components/Runtime/RuntimeControls.tsx` - Sound toggle button
|
|
- `frontend/src/components/Constructor/CanvasBackground.tsx` - Background video/audio rendering with `webkit-playsinline`
|
|
|
|
### 2. Video Player Elements
|
|
|
|
Interactive video elements with controls, placed in page UI schema.
|
|
|
|
```tsx
|
|
// frontend/src/components/UiElements/elements/VideoPlayerElement.tsx
|
|
<video
|
|
src={resolve(element.mediaUrl)}
|
|
controls
|
|
autoPlay={Boolean(element.mediaAutoplay)}
|
|
loop={Boolean(element.mediaLoop)}
|
|
muted={Boolean(element.mediaMuted)}
|
|
playsInline
|
|
/>
|
|
```
|
|
|
|
**Configurable Properties:**
|
|
| Property | Type | Description |
|
|
|----------|------|-------------|
|
|
| `mediaUrl` | string | Video asset URL |
|
|
| `mediaAutoplay` | boolean | Auto-start playback |
|
|
| `mediaLoop` | boolean | Loop continuously |
|
|
| `mediaMuted` | boolean | Mute audio |
|
|
|
|
**Note:** Controls are always shown. The component uses `resolveAssetPlaybackUrl()` for URL resolution.
|
|
|
|
### 3. Transition Videos
|
|
|
|
Animated videos that play during page navigation.
|
|
|
|
```typescript
|
|
interface TransitionVideo {
|
|
video_url: string; // Forward transition video
|
|
duration_sec: number; // Duration in seconds
|
|
supports_reverse: boolean; // Can play in reverse
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## Forward Playback
|
|
|
|
### How It Works
|
|
|
|
Forward playback uses native HTML5 video capabilities:
|
|
|
|
```
|
|
1. Set video.src = videoUrl
|
|
2. Set video.autoPlay = true (or call video.play())
|
|
3. Video plays from start to end
|
|
4. 'ended' event fires → trigger next action
|
|
```
|
|
|
|
### Implementation
|
|
|
|
```tsx
|
|
// Transition overlay forward playback
|
|
<video
|
|
ref={videoRef}
|
|
src={transitionVideoUrl}
|
|
autoPlay={!isReverse}
|
|
muted
|
|
playsInline
|
|
preload="auto"
|
|
onEnded={handleTransitionComplete}
|
|
/>
|
|
```
|
|
|
|
### Fallback Timer
|
|
|
|
If video fails to trigger `onEnded`, a fallback timer ensures navigation continues:
|
|
|
|
```typescript
|
|
const fallbackTimer = setTimeout(() => {
|
|
finishTransition();
|
|
}, transitionDuration * 1000 + 500); // duration + buffer
|
|
|
|
videoRef.current.onended = () => {
|
|
clearTimeout(fallbackTimer);
|
|
finishTransition();
|
|
};
|
|
```
|
|
|
|
---
|
|
|
|
## Reverse Playback
|
|
|
|
### Overview
|
|
|
|
Reverse playback is handled by `useReversePlayback` hook with two strategies:
|
|
|
|
```
|
|
┌─────────────────────┐
|
|
│ Start Reverse │
|
|
└─────────┬───────────┘
|
|
│
|
|
▼
|
|
┌─────────────────────┐ Supported? ┌─────────────────────┐
|
|
│ Try Native Reverse │ ───────────────────│ Use Native │
|
|
│ (playbackRate=-1) │ YES │ playbackRate=-1 │
|
|
└─────────┬───────────┘ └─────────────────────┘
|
|
│ NO
|
|
▼
|
|
┌─────────────────────┐
|
|
│ Frame-Stepping │
|
|
│ Fallback │
|
|
└─────────────────────┘
|
|
```
|
|
|
|
### Strategy 1: Native Reverse (Primary)
|
|
|
|
Uses browser's native reverse playback when supported (Chrome, Edge):
|
|
|
|
```typescript
|
|
const startNativeReverse = () => {
|
|
video.currentTime = video.duration;
|
|
video.playbackRate = -1;
|
|
video.play();
|
|
|
|
video.ontimeupdate = () => {
|
|
if (video.currentTime <= 0.05) {
|
|
video.currentTime = 0;
|
|
onComplete();
|
|
}
|
|
};
|
|
};
|
|
```
|
|
|
|
**Browser Support:**
|
|
| Browser | Native Reverse |
|
|
|---------|---------------|
|
|
| Chrome | ✅ Supported |
|
|
| Edge | ✅ Supported |
|
|
| Firefox | ❌ Use fallback |
|
|
| Safari | ❌ Use fallback |
|
|
|
|
### Strategy 2: Frame-Stepping Fallback
|
|
|
|
Manual frame-by-frame reverse for unsupported browsers:
|
|
|
|
```typescript
|
|
const startFrameStepping = () => {
|
|
const fps = isPreloaded ? 30 : 15; // Adaptive FPS
|
|
const frameInterval = 1000 / fps;
|
|
const frameStep = 1 / fps;
|
|
|
|
let lastFrameTime = 0;
|
|
|
|
const stepFrame = (timestamp: number) => {
|
|
if (timestamp - lastFrameTime >= frameInterval) {
|
|
video.currentTime = Math.max(0, video.currentTime - frameStep);
|
|
lastFrameTime = timestamp;
|
|
|
|
if (video.currentTime <= 0) {
|
|
onComplete();
|
|
return;
|
|
}
|
|
}
|
|
requestAnimationFrame(stepFrame);
|
|
};
|
|
|
|
requestAnimationFrame(stepFrame);
|
|
};
|
|
```
|
|
|
|
**Adaptive FPS:**
|
|
| Condition | FPS | Reason |
|
|
|-----------|-----|--------|
|
|
| Preloaded video | 30 | Smooth playback, data available |
|
|
| Streaming video | 15 | Conservative, bandwidth-aware |
|
|
|
|
### Buffering Handling
|
|
|
|
The system waits for sufficient buffered data before starting:
|
|
|
|
```typescript
|
|
const checkBuffered = () => {
|
|
const buffered = video.buffered;
|
|
if (buffered.length > 0) {
|
|
const bufferedEnd = buffered.end(buffered.length - 1);
|
|
const bufferedAmount = bufferedEnd - video.currentTime;
|
|
|
|
if (bufferedAmount >= 0.5) { // 0.5 seconds minimum
|
|
startReverse();
|
|
}
|
|
}
|
|
};
|
|
```
|
|
|
|
**Buffering Timeout:** 8 seconds before attempting with available buffer.
|
|
|
|
### Hook API
|
|
|
|
```typescript
|
|
const {
|
|
startReverse, // Start reverse playback (async)
|
|
stopReverse, // Cancel reverse playback
|
|
isReversing, // Currently playing in reverse
|
|
isBuffering, // Waiting for buffer
|
|
canUseNativeReverse, // Browser supports playbackRate = -1
|
|
} = useReversePlayback({
|
|
videoRef,
|
|
onComplete,
|
|
preloadedUrls?, // Optional: Set of preloaded video URLs
|
|
videoUrl?, // Optional: Current video URL for cache lookup
|
|
getCachedBlobUrl?, // Optional: Function to get cached blob URL
|
|
});
|
|
```
|
|
|
|
---
|
|
|
|
## Constructor Mode
|
|
|
|
### Video Preview System
|
|
|
|
The Constructor provides comprehensive video testing:
|
|
|
|
```typescript
|
|
interface TransitionPreview {
|
|
videoUrl: string; // Forward video URL
|
|
reverseVideoUrl?: string; // Optional separate reverse video
|
|
reverseMode: 'reverse'; // Playback mode indicator
|
|
}
|
|
```
|
|
|
|
### Video Loading Flow
|
|
|
|
```
|
|
1. User selects transition video
|
|
2. Fetch video as blob (better seeking)
|
|
3. Create blob URL
|
|
4. Monitor readyState events
|
|
5. Extract duration from metadata
|
|
6. Enable preview controls
|
|
```
|
|
|
|
### Event Monitoring
|
|
|
|
Constructor monitors comprehensive video events:
|
|
|
|
```typescript
|
|
// Ready state events
|
|
video.addEventListener('loadedmetadata', handleMetadata);
|
|
video.addEventListener('canplay', handleCanPlay);
|
|
video.addEventListener('canplaythrough', handleReady);
|
|
|
|
// Playback events
|
|
video.addEventListener('playing', handlePlaying);
|
|
video.addEventListener('ended', handleEnded);
|
|
|
|
// Error events
|
|
video.addEventListener('error', handleError);
|
|
video.addEventListener('abort', handleAbort);
|
|
video.addEventListener('stalled', handleStalled);
|
|
```
|
|
|
|
### Duration Extraction
|
|
|
|
```typescript
|
|
// frontend/src/lib/mediaDuration.ts (120 LOC)
|
|
|
|
export interface MediaDurationResult {
|
|
duration: number;
|
|
width?: number; // Only for video
|
|
height?: number; // Only for video
|
|
}
|
|
|
|
export function probeMediaDuration(
|
|
source: string | File, // URL string or File object
|
|
mediaType: 'video' | 'audio', // Media type to probe
|
|
timeoutMs = 10000, // Timeout in milliseconds
|
|
): Promise<MediaDurationResult | null> {
|
|
return new Promise((resolve) => {
|
|
const mediaElement = mediaType === 'video'
|
|
? document.createElement('video')
|
|
: document.createElement('audio');
|
|
|
|
mediaElement.preload = 'metadata';
|
|
|
|
mediaElement.onloadedmetadata = () => {
|
|
const result: MediaDurationResult = { duration: mediaElement.duration };
|
|
if (mediaType === 'video' && mediaElement instanceof HTMLVideoElement) {
|
|
result.width = mediaElement.videoWidth;
|
|
result.height = mediaElement.videoHeight;
|
|
}
|
|
cleanup();
|
|
resolve(result);
|
|
};
|
|
|
|
mediaElement.onerror = () => {
|
|
cleanup();
|
|
resolve(null); // Returns null on error, not reject
|
|
};
|
|
|
|
// Source can be URL string or File object
|
|
mediaElement.src = typeof source === 'string'
|
|
? source
|
|
: URL.createObjectURL(source);
|
|
|
|
setTimeout(() => {
|
|
cleanup();
|
|
resolve(null);
|
|
}, timeoutMs);
|
|
});
|
|
}
|
|
|
|
// Helper functions for MIME type detection
|
|
export function isVideoMimeType(mimeType: string | null | undefined): boolean {
|
|
if (!mimeType) return false;
|
|
return mimeType.startsWith('video/');
|
|
}
|
|
|
|
export function isAudioMimeType(mimeType: string | null | undefined): boolean {
|
|
if (!mimeType) return false;
|
|
return mimeType.startsWith('audio/');
|
|
}
|
|
```
|
|
|
|
### Preview Controls
|
|
|
|
| Control | Forward Mode | Reverse Mode |
|
|
|---------|-------------|--------------|
|
|
| Play | Native play() | startReverse() |
|
|
| Pause | Native pause() | stopReverse() |
|
|
| Progress | timeupdate event | Manual tracking |
|
|
|
|
---
|
|
|
|
## Runtime Mode (Presentations)
|
|
|
|
### RuntimePresentation Component
|
|
|
|
Handles video playback during tour presentations:
|
|
|
|
```tsx
|
|
// frontend/src/components/RuntimePresentation.tsx
|
|
|
|
// Background video rendering
|
|
{page.background_video_url && (
|
|
<video
|
|
src={resolveAssetUrl(page.background_video_url)}
|
|
autoPlay
|
|
loop
|
|
muted
|
|
playsInline
|
|
className="absolute inset-0 w-full h-full object-cover -z-10"
|
|
/>
|
|
)}
|
|
|
|
// Video player elements
|
|
{element.type === 'video_player' && (
|
|
<video
|
|
src={resolveAssetUrl(element.content_json.mediaUrl)}
|
|
controls={element.content_json.mediaControls}
|
|
autoPlay={element.content_json.autoPlay}
|
|
loop={element.content_json.loop}
|
|
muted={element.content_json.muted}
|
|
/>
|
|
)}
|
|
```
|
|
|
|
### Transition Overlay Flow
|
|
|
|
```
|
|
┌──────────────────────────────────────────────────────────────┐
|
|
│ 1. User clicks navigation element │
|
|
│ └── Resolve targetPageSlug → targetPageId │
|
|
│ └── startNavigation(targetPageId, element) │
|
|
└──────────────────────────────────────────────────────────────┘
|
|
│
|
|
▼
|
|
┌──────────────────────────────────────────────────────────────┐
|
|
│ 2. Determine transition type │
|
|
│ └── isReverse = isBack && linkData.supports_reverse │
|
|
└──────────────────────────────────────────────────────────────┘
|
|
│
|
|
▼
|
|
┌──────────────────────────────────────────────────────────────┐
|
|
│ 3. Set overlay transition state │
|
|
│ └── setOverlayTransition({ videoUrl, isReverse }) │
|
|
└──────────────────────────────────────────────────────────────┘
|
|
│
|
|
▼
|
|
┌──────────────────────────────────────────────────────────────┐
|
|
│ 4. Render transition overlay with video │
|
|
│ └── <TransitionOverlay video={...} /> │
|
|
└──────────────────────────────────────────────────────────────┘
|
|
│
|
|
┌───────────────────┴───────────────────┐
|
|
│ │
|
|
▼ ▼
|
|
┌──────────────────────┐ ┌──────────────────────┐
|
|
│ Forward Playback │ │ Reverse Playback │
|
|
│ - autoPlay={true} │ │ - useReversePlayback│
|
|
│ - Wait for onEnded │ │ - Wait for complete │
|
|
└──────────┬───────────┘ └──────────┬───────────┘
|
|
│ │
|
|
└──────────────────┬───────────────────┘
|
|
│
|
|
▼
|
|
┌──────────────────────────────────────────────────────────────┐
|
|
│ 5. Finish transition │
|
|
│ └── finishOverlayTransition() → switch pages │
|
|
└──────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
### Transition State Management
|
|
|
|
The `useTransitionPlayback` hook manages transition video playback with comprehensive state tracking:
|
|
|
|
```typescript
|
|
// frontend/src/hooks/useTransitionPlayback.ts
|
|
|
|
// Reverse playback modes
|
|
export type ReverseMode = 'none' | 'reverse' | 'separate';
|
|
|
|
// Transition configuration passed to the hook
|
|
export interface TransitionConfig {
|
|
videoUrl: string; // Forward transition video URL
|
|
storageKey?: string; // Raw storage path for cache lookup
|
|
reverseMode: ReverseMode; // How to handle reverse playback
|
|
reverseVideoUrl?: string; // Separate reverse video (for 'separate' mode)
|
|
durationSec?: number; // Expected duration in seconds
|
|
targetPageId?: string; // Target page for navigation
|
|
displayName?: string; // Human-readable name for logging
|
|
}
|
|
|
|
// Playback phases
|
|
export type PlaybackPhase =
|
|
| 'idle' // No transition active
|
|
| 'preparing' // Loading video, resolving URLs
|
|
| 'playing' // Forward playback in progress
|
|
| 'reversing' // Reverse playback in progress
|
|
| 'finishing' // Pre-decoding target page images
|
|
| 'completed'; // Transition finished, ready for page switch
|
|
|
|
// Hook options
|
|
export interface UseTransitionPlaybackOptions {
|
|
videoRef: RefObject<HTMLVideoElement | null>;
|
|
transition: TransitionConfig | null;
|
|
onComplete: (targetPageId?: string) => void;
|
|
onError?: (reason: string) => void;
|
|
|
|
timeouts?: {
|
|
playbackStartMs?: number; // Max wait for playback to start (default: 3000)
|
|
durationBufferMs?: number; // Buffer after expected duration (default: 200)
|
|
hardTimeoutMs?: number; // Absolute max timeout (default: 45000)
|
|
};
|
|
features?: {
|
|
useBlobUrl?: boolean; // Force blob URL loading
|
|
preDecodeImages?: boolean; // Pre-decode target page images
|
|
getTargetPageImages?: () => string[]; // Get images to pre-decode
|
|
};
|
|
preload?: {
|
|
preloadedUrls?: Set<string>;
|
|
getCachedBlobUrl?: (url: string) => Promise<string | null>;
|
|
getReadyBlobUrl?: (url: string) => string | null;
|
|
};
|
|
}
|
|
|
|
// Hook result
|
|
export interface UseTransitionPlaybackResult {
|
|
phase: PlaybackPhase; // Current playback phase
|
|
isBuffering: boolean; // Waiting for video data
|
|
isReversing: boolean; // Currently in reverse playback
|
|
cancel: () => void; // Cancel transition immediately
|
|
forceComplete: () => void; // Force transition to complete
|
|
}
|
|
```
|
|
|
|
**Default Timeouts:**
|
|
```typescript
|
|
const DEFAULT_TIMEOUTS = {
|
|
playbackStartMs: 3000, // 3 seconds to start playing
|
|
durationBufferMs: 200, // 200ms buffer after expected end
|
|
hardTimeoutMs: 45000, // 45 seconds absolute maximum
|
|
};
|
|
```
|
|
|
|
**Usage Example:**
|
|
```typescript
|
|
const { phase, isBuffering, cancel, forceComplete } = useTransitionPlayback({
|
|
videoRef,
|
|
transition: {
|
|
videoUrl: resolveAssetPlaybackUrl(element.transitionVideoUrl),
|
|
storageKey: element.transitionVideoUrl,
|
|
reverseMode: isBack ? 'reverse' : 'none',
|
|
durationSec: element.transitionDurationSec,
|
|
targetPageId: targetPage.id,
|
|
displayName: element.label,
|
|
},
|
|
onComplete: (targetPageId) => {
|
|
setCurrentPageId(targetPageId);
|
|
setOverlayTransition(null);
|
|
},
|
|
onError: (reason) => {
|
|
console.error('Transition failed:', reason);
|
|
// Navigate anyway to prevent user being stuck
|
|
setCurrentPageId(targetPageId);
|
|
setOverlayTransition(null);
|
|
},
|
|
preload: {
|
|
getReadyBlobUrl: preloadOrchestrator.getReadyBlobUrl,
|
|
getCachedBlobUrl: preloadOrchestrator.getCachedBlobUrl,
|
|
},
|
|
});
|
|
```
|
|
|
|
---
|
|
|
|
## Video URL Resolution
|
|
|
|
The `resolveAssetPlaybackUrl()` utility normalizes video URLs with presigned URL cache lookup:
|
|
|
|
```typescript
|
|
// frontend/src/lib/assetUrl.ts
|
|
export const resolveAssetPlaybackUrl = (value?: string): string => {
|
|
const normalized = String(value || '').trim();
|
|
if (!normalized) return '';
|
|
|
|
// Pass through special URLs
|
|
if (normalized.startsWith('data:') || normalized.startsWith('blob:'))
|
|
return normalized;
|
|
|
|
// Already an API file download URL
|
|
if (normalized.startsWith('/api/file/download')) return normalized;
|
|
|
|
// File download path (prepend API base)
|
|
if (normalized.startsWith('/file/download'))
|
|
return `${baseURLApi}${normalized}`;
|
|
|
|
// Full URLs pass through
|
|
if (normalized.startsWith('http://') || normalized.startsWith('https://'))
|
|
return normalized;
|
|
|
|
// Relative path - check presigned URL cache first (for fast S3 access)
|
|
const presigned = getPresignedUrl(normalized);
|
|
if (presigned) {
|
|
return presigned;
|
|
}
|
|
|
|
// Fallback to backend proxy
|
|
const normalizedPrivateUrl = normalized.replace(/^\/+/, '');
|
|
return `${baseURLApi}/file/download?privateUrl=${encodeURIComponent(normalizedPrivateUrl)}`;
|
|
};
|
|
```
|
|
|
|
**Presigned URL Priority:** The function first checks if a valid presigned S3 URL is cached for the storage key. If found and not expired, it returns the presigned URL for direct S3 access. Otherwise, it falls back to the backend proxy endpoint.
|
|
|
|
---
|
|
|
|
## Cache Integration
|
|
|
|
### S3 Presigned URL Preloading
|
|
|
|
Videos are preloaded directly from S3 using presigned URLs:
|
|
|
|
```
|
|
1. POST /api/file/presign { urls: [video1, video2, ...] } // Max 50 per batch
|
|
2. Download from S3 presigned URLs (1-hour expiry)
|
|
3. Store in Cache API (dual key: download URL + storage key)
|
|
4. Create blob URL: URL.createObjectURL(blob)
|
|
5. Store in readyBlobUrlsRef Map (dual key: download URL + storage key) for O(1) instant lookup
|
|
```
|
|
|
|
**Storage Key Mapping:** Videos are cached by canonical storage key (e.g., `assets/project/transition.mp4`) in addition to download URLs. This ensures cache hits even when presigned URL signatures change between sessions.
|
|
|
|
### Ready Blob URL Lookup (Primary)
|
|
|
|
For instant video playback, use `getReadyBlobUrl()` which returns pre-created blob URLs. Lookups prioritize **storage key** over resolved URLs for reliable cache hits:
|
|
|
|
```typescript
|
|
// 1. Primary: O(1) lookup by storage key (most reliable)
|
|
const readyUrl = preloadOrchestrator.getReadyBlobUrl(storageKey);
|
|
if (readyUrl) {
|
|
video.src = readyUrl; // Instant playback - no decode needed for video
|
|
return;
|
|
}
|
|
|
|
// 2. Fallback: O(1) lookup by resolved URL
|
|
const readyUrlByResolved = preloadOrchestrator.getReadyBlobUrl(resolvedUrl);
|
|
if (readyUrlByResolved) {
|
|
video.src = readyUrlByResolved;
|
|
}
|
|
```
|
|
|
|
**Why storage key first:** Presigned URL signatures change on each resolution (X-Amz-Signature differs). Storage keys (e.g., `assets/project/video.mp4`) are canonical and never change.
|
|
|
|
### Cached Blob URL Fallback
|
|
|
|
If ready URL not available, create blob URL from cache:
|
|
|
|
```typescript
|
|
const getCachedBlobUrl = async (url: string): Promise<string | null> => {
|
|
const cache = await caches.open('tour-builder-assets-v1');
|
|
const response = await cache.match(url);
|
|
|
|
if (response) {
|
|
const blob = await response.blob();
|
|
return URL.createObjectURL(blob);
|
|
}
|
|
|
|
return null;
|
|
};
|
|
```
|
|
|
|
### Usage in Reverse Playback
|
|
|
|
```typescript
|
|
// useReversePlayback attempts ready/cached URL for better seeking
|
|
// Uses storage key for reliable cache lookup
|
|
const setupReverse = async (storageKey: string) => {
|
|
// 1. Try instant ready blob URL by storage key (O(1) lookup, most reliable)
|
|
const readyUrl = preloadOrchestrator.getReadyBlobUrl(storageKey);
|
|
if (readyUrl) {
|
|
video.src = readyUrl;
|
|
isPreloaded = true;
|
|
} else {
|
|
// 2. Fallback: create blob URL from cache by storage key
|
|
const cachedUrl = await getCachedBlobUrl(storageKey);
|
|
if (cachedUrl) {
|
|
video.src = cachedUrl; // Better seeking with local blob
|
|
isPreloaded = true;
|
|
}
|
|
}
|
|
|
|
// Proceed with reverse playback
|
|
startReverse();
|
|
};
|
|
```
|
|
|
|
---
|
|
|
|
## Presigned URL Fallback Mechanism
|
|
|
|
The `useTransitionPlayback` hook includes automatic fallback when presigned S3 URLs fail (typically due to CORS issues):
|
|
|
|
```typescript
|
|
// Check if URL is a presigned S3 URL
|
|
function isPresignedUrl(url: string): boolean {
|
|
return url.includes('X-Amz-Signature=') || url.includes('x-amz-signature=');
|
|
}
|
|
|
|
// Convert presigned URL to proxy URL fallback
|
|
function getProxyUrlFallback(
|
|
presignedUrl: string,
|
|
originalStorageKey?: string,
|
|
): string | null {
|
|
// If we have the original storage key, use it directly
|
|
if (originalStorageKey && isRelativeStoragePath(originalStorageKey)) {
|
|
const normalizedPath = originalStorageKey.replace(/^\/+/, '');
|
|
return `${baseURLApi}/file/download?privateUrl=${encodeURIComponent(normalizedPath)}`;
|
|
}
|
|
|
|
// Try to extract path from presigned URL
|
|
try {
|
|
const url = new URL(presignedUrl);
|
|
const pathParts = url.pathname.split('/').filter(Boolean);
|
|
if (pathParts.length >= 2) {
|
|
const storagePath = pathParts.slice(1).join('/');
|
|
return `${baseURLApi}/file/download?privateUrl=${encodeURIComponent(storagePath)}`;
|
|
}
|
|
} catch {
|
|
// URL parsing failed
|
|
}
|
|
return null;
|
|
}
|
|
```
|
|
|
|
**Fallback Flow:**
|
|
```
|
|
1. Video error event fires on presigned URL
|
|
2. Check if current URL is a presigned S3 URL
|
|
3. If not already tried fallback:
|
|
a. Mark presigned URL as failed (future resolves use proxy)
|
|
b. Build proxy fallback URL from storage key
|
|
c. Retry video load with proxy URL
|
|
4. If fallback also fails → trigger error handler
|
|
```
|
|
|
|
This mechanism ensures video playback works even when S3 CORS is misconfigured or presigned URLs expire mid-session.
|
|
|
|
---
|
|
|
|
## Image Pre-decoding During Transitions
|
|
|
|
The `useTransitionPlayback` hook supports pre-decoding target page images during the `finishing` phase to ensure instant display after transition:
|
|
|
|
```typescript
|
|
// In useTransitionPlayback.ts
|
|
async function waitForImages(urls: string[], timeoutMs = 2000): Promise<void> {
|
|
if (urls.length === 0) return;
|
|
|
|
const decodePromises = urls.map(
|
|
(url) =>
|
|
new Promise<void>((resolve) => {
|
|
const img = new Image();
|
|
img.src = url;
|
|
if (typeof img.decode === 'function') {
|
|
img.decode()
|
|
.then(() => resolve())
|
|
.catch(() => resolve());
|
|
} else {
|
|
img.onload = () => resolve();
|
|
img.onerror = () => resolve();
|
|
}
|
|
}),
|
|
);
|
|
|
|
// Race against timeout to prevent blocking
|
|
await Promise.race([
|
|
Promise.all(decodePromises),
|
|
new Promise<void>((resolve) => setTimeout(resolve, timeoutMs)),
|
|
]);
|
|
}
|
|
```
|
|
|
|
**Usage in Transition Flow:**
|
|
```typescript
|
|
// Enable via features option
|
|
useTransitionPlayback({
|
|
// ...
|
|
features: {
|
|
preDecodeImages: true,
|
|
getTargetPageImages: () => {
|
|
// Return URLs of images on the target page
|
|
return targetPage.elements
|
|
.filter(el => el.type === 'image')
|
|
.map(el => resolveAssetUrl(el.imageUrl));
|
|
},
|
|
},
|
|
});
|
|
```
|
|
|
|
This ensures the target page appears instantly without visible image loading after the transition video completes.
|
|
|
|
---
|
|
|
|
## Comparison: Constructor vs Runtime
|
|
|
|
| Aspect | Constructor | Runtime |
|
|
|--------|-------------|---------|
|
|
| **Purpose** | Testing & preview | Presentation playback |
|
|
| **Controls** | Full manual control | Automated flow |
|
|
| **Logging** | Detailed debug logs | Silent operation |
|
|
| **Network Info** | Displayed in UI | Transparent |
|
|
| **Buffering** | Status shown | Hidden from user |
|
|
| **Error Handling** | Detailed messages | Fallback timers |
|
|
| **Duration Source** | Editable input | From transition data |
|
|
|
|
---
|
|
|
|
## Error Handling
|
|
|
|
### Reverse Playback Failures
|
|
|
|
```typescript
|
|
const handleReverseError = (error: Error) => {
|
|
console.error('Reverse playback failed:', error);
|
|
|
|
// Fallback chain:
|
|
// 1. Native reverse fails → Try frame-stepping
|
|
// 2. Frame-stepping fails → Use available buffer
|
|
// 3. Buffer timeout → Complete with partial playback
|
|
// 4. All fails → Skip transition, navigate directly
|
|
|
|
if (currentStrategy === 'native') {
|
|
switchToFrameStepping();
|
|
} else {
|
|
completeWithAvailableProgress();
|
|
}
|
|
};
|
|
```
|
|
|
|
### Forward Playback Failures
|
|
|
|
```typescript
|
|
const handleVideoError = (event: Event) => {
|
|
console.error('Video playback error:', event);
|
|
|
|
// Ensure navigation continues
|
|
clearTimeout(fallbackTimer);
|
|
finishTransition();
|
|
};
|
|
```
|
|
|
|
### Network Failures
|
|
|
|
```typescript
|
|
// Video stalls during loading
|
|
video.onstalled = () => {
|
|
console.warn('Video stalled, waiting for data...');
|
|
// Don't block - fallback timer will handle
|
|
};
|
|
|
|
video.onwaiting = () => {
|
|
setIsBuffering(true);
|
|
};
|
|
|
|
video.onplaying = () => {
|
|
setIsBuffering(false);
|
|
};
|
|
```
|
|
|
|
---
|
|
|
|
## Performance Optimizations
|
|
|
|
### 1. Adaptive Frame Rate
|
|
|
|
```typescript
|
|
// Adjust FPS based on video availability
|
|
const fps = isPreloaded ? 30 : 15;
|
|
```
|
|
|
|
### 2. Blob URL Caching
|
|
|
|
```typescript
|
|
// Use cached blob URLs for better seeking
|
|
const cachedUrl = await getCachedBlobUrl(videoUrl);
|
|
if (cachedUrl) {
|
|
video.src = cachedUrl;
|
|
}
|
|
```
|
|
|
|
### 3. Preload Priority
|
|
|
|
Videos receive priority based on context:
|
|
|
|
```typescript
|
|
const videoPriority = {
|
|
currentPage: 1000, // Current page assets
|
|
neighborBase: 500, // Neighbor page assets
|
|
transition: 150, // Highest type priority - needed immediately on click
|
|
image: 100, // Backgrounds load during transition
|
|
audio: 50,
|
|
video: 30, // Background videos can stream progressively
|
|
};
|
|
```
|
|
|
|
**Note:** `maxNeighborDepth` defaults to 1 (immediate neighbors only). Transition videos have the highest type priority (+150) because they're needed immediately when navigation is triggered.
|
|
|
|
### 4. Memory Management
|
|
|
|
```typescript
|
|
// Clean up blob URLs
|
|
useEffect(() => {
|
|
return () => {
|
|
if (blobUrl) {
|
|
URL.revokeObjectURL(blobUrl);
|
|
}
|
|
};
|
|
}, [blobUrl]);
|
|
|
|
// Cancel animation frames
|
|
useEffect(() => {
|
|
return () => {
|
|
if (animationFrameId) {
|
|
cancelAnimationFrame(animationFrameId);
|
|
}
|
|
};
|
|
}, []);
|
|
```
|
|
|
|
### 5. Event Listener Cleanup
|
|
|
|
```typescript
|
|
useEffect(() => {
|
|
const video = videoRef.current;
|
|
|
|
video.addEventListener('timeupdate', handleTimeUpdate);
|
|
video.addEventListener('ended', handleEnded);
|
|
|
|
return () => {
|
|
video.removeEventListener('timeupdate', handleTimeUpdate);
|
|
video.removeEventListener('ended', handleEnded);
|
|
};
|
|
}, []);
|
|
```
|
|
|
|
---
|
|
|
|
## Key Files Reference
|
|
|
|
| File | Location | LOC | Purpose |
|
|
|------|----------|-----|---------|
|
|
| `useTransitionPlayback.ts` | `frontend/src/hooks/` | 778 | Transition video playback coordination |
|
|
| `useReversePlayback.ts` | `frontend/src/hooks/` | 399 | Reverse playback logic |
|
|
| `useBackgroundVideoPlayback.ts` | `frontend/src/hooks/` | ~210 | Background video time control, session-scoped play-once when loop=false |
|
|
| `usePreloadOrchestrator.ts` | `frontend/src/hooks/` | - | Asset preloading with ready blob URLs |
|
|
| `usePageSwitch.ts` | `frontend/src/hooks/` | - | Page navigation with preloaded assets |
|
|
| `mediaDuration.ts` | `frontend/src/lib/` | 120 | Duration extraction utility |
|
|
| `assetUrl.ts` | `frontend/src/lib/` | - | URL resolution with presigned cache |
|
|
| `extractPageLinks.ts` | `frontend/src/lib/` | - | Extract navigation links with transition videos |
|
|
| `StorageManager.ts` | `frontend/src/lib/offline/` | - | Cache API storage for video blobs |
|
|
| `VideoPlayerElement.tsx` | `frontend/src/components/UiElements/elements/` | 52 | Video player UI element |
|
|
| `RuntimePresentation.tsx` | `frontend/src/components/` | - | Runtime video rendering |
|
|
| `constructor.tsx` | `frontend/src/pages/` | - | Constructor video preview |
|
|
| `[projectSlug]/index.tsx` | `frontend/src/pages/p/` | - | Production runtime page |
|
|
| `[projectSlug]/stage.tsx` | `frontend/src/pages/p/` | - | Stage runtime page |
|
|
| `preload.config.ts` | `frontend/src/config/` | - | Video priority settings (transition: 150) |
|
|
|
|
---
|
|
|
|
## Troubleshooting
|
|
|
|
### Video Won't Play
|
|
1. Check browser autoplay policy (must be muted)
|
|
2. Verify video URL resolves correctly
|
|
3. Check CORS headers on video source
|
|
4. Ensure video format is supported (MP4/WebM)
|
|
|
|
### Reverse Playback Choppy
|
|
1. Check if video is preloaded (should use 30 FPS)
|
|
2. Verify sufficient buffer is available
|
|
3. Try using cached blob URL for better seeking
|
|
4. Consider reducing video resolution
|
|
|
|
### Transition Doesn't Complete
|
|
1. Check `onEnded` event fires correctly
|
|
2. Verify fallback timer is set
|
|
3. Look for JavaScript errors in console
|
|
4. Ensure `finishTransition` is called
|
|
|
|
### Memory Leaks
|
|
1. Verify blob URLs are revoked on cleanup
|
|
2. Check animation frames are cancelled
|
|
3. Ensure event listeners are removed
|
|
4. Look for orphaned video elements
|