Dmitri 9f111d6226 added documentation

2026-07-03 16:11:24 +02:00

38 KiB

Raw Blame History

Video Playback Feature - E2E Documentation

Overview

The Tour Builder Platform implements a comprehensive video playback system supporting forward and reverse playback modes. The system handles background videos, video player elements, and transition animations between pages in both Constructor (editing) and Runtime (presentation) modes.

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                      Video Types                                 │
│  Background Videos │ Video Player Elements │ Transition Videos  │
└─────────────────────────────────────────────────────────────────┘
                              │
┌─────────────────────────────────────────────────────────────────┐
│                     Playback Hooks                               │
│  useTransitionPlayback │ useReversePlayback │ usePageSwitch     │
│  useBackgroundVideoPlayback (background video time control)      │
└─────────────────────────────────────────────────────────────────┘
                              │
┌─────────────────────────────────────────────────────────────────┐
│                     Preloading Layer                             │
│  usePreloadOrchestrator │ getReadyBlobUrl (O(1)) │ S3 Presign  │
└─────────────────────────────────────────────────────────────────┘
                              │
┌─────────────────────────────────────────────────────────────────┐
│                     Utilities                                    │
│  mediaDuration.ts │ assetUrl.ts │ StorageManager (Cache API)   │
└─────────────────────────────────────────────────────────────────┘
                              │
┌─────────────────────────────────────────────────────────────────┐
│                     Rendering                                    │
│  Constructor Page │ RuntimePresentation │ Transition Overlay    │
└─────────────────────────────────────────────────────────────────┘

Video Types

1. Background Videos

Full-screen videos that play behind page content with configurable playback settings.

// Using useBackgroundVideoPlayback hook for custom time control
const { videoRef, shouldBlockAutoplay } = useBackgroundVideoPlayback({
  videoUrl: page.background_video_url,
  autoplay: page.background_video_autoplay ?? true,
  loop: page.background_video_loop ?? true,
  muted: page.background_video_muted ?? true,
  startTime: page.background_video_start_time ?? null,
  endTime: page.background_video_end_time ?? null,
});

// Block autoplay if video already played this session (when loop=false)
const effectiveAutoplay = (page.background_video_autoplay ?? true) && !shouldBlockAutoplay;

<video
  ref={videoRef}
  src={page.background_video_url}
  autoPlay={effectiveAutoplay}
  loop={page.background_video_end_time == null ? (page.background_video_loop ?? true) : false}
  muted={page.background_video_muted ?? true}
  playsInline
  className="absolute inset-0 w-full h-full object-cover"
/>

Configurable Properties (stored in tour_pages):

Property	Type	Default	Description
`background_video_autoplay`	boolean	true	Start playback automatically
`background_video_loop`	boolean	true	Loop continuously
`background_video_muted`	boolean	true	Mute audio (required for autoplay policy)
`background_video_start_time`	DECIMAL(10,1) \| null	null	Start playback at this time (seconds, 0.1s precision)
`background_video_end_time`	DECIMAL(10,1) \| null	null	Stop/loop at this time (seconds, 0.1s precision)

DECIMAL Parsing (Critical): Sequelize DECIMAL fields return strings from the database (e.g., "2.5" not 2.5). In runtime code, these must be parsed with parseFloat(String(value)) before passing to the hook.

Note: When background_video_end_time is set, native HTML5 loop attribute is disabled and looping is handled via JavaScript (timeupdate event) to properly seek back to startTime.

Session-Scoped Play Once Behavior (when loop=false): When loop is disabled, videos only play once per browser session:

Video plays normally on first navigation to the page
When video ends, the video URL is added to a session-scoped Set (playedVideos)
On subsequent visits within the same session, autoplay is blocked and video shows last frame
Session tracking resets on browser refresh (module-level state, not persisted)

Global Sound Control and Autoplay Compatibility

Browsers block unmuted autoplay until the user interacts with the page. iOS WebKit is especially strict and can show native play overlays for unmuted videos. The platform handles this with a shared global mute state:

Problem: unmuted background audio/video, hover/click effects, and media-player sound cannot reliably autoplay before user interaction.

Solution:

CSS Layer - Hide native iOS video controls via -webkit-media-controls-* selectors in main.css
Always Start Muted - Presentation audio starts muted for autoplay compatibility
Global Sound Control - backgroundAudioController stores shared mute state; useGlobalAudioMute exposes it to React
Custom Sound Button - useVideoSoundControl drives the RuntimeControls sound button and decides when it should be visible
webkit-playsinline - Legacy attribute added to <video> for older iOS versions

// useVideoSoundControl hook usage in RuntimePresentation
const soundControl = useVideoSoundControl({
  pageHasSound: page.background_video_muted === false,
  hasBackgroundVideo: Boolean(backgroundVideoUrl),
  hasBackgroundAudio: Boolean(backgroundAudioUrl),
  hasElementAudio,
});

// Background video follows global presentation mute state
<video
  muted={soundControl.isMuted}
  playsInline
  webkit-playsinline=""
/>

// Sound toggle button in RuntimeControls
<RuntimeControls
  showSoundButton={soundControl.showSoundButton}
  isMuted={soundControl.isMuted}
  onSoundToggle={soundControl.toggleSound}
/>

Key Behaviors:

Behavior	Description
Start muted	Presentation sound starts muted for browser autoplay compatibility
Sound button	Appears when the page has background video sound, background audio, hover/click audio, media players, or gallery/info-panel videos
Global scope	The same mute state controls background audio, background video, hover/click effects, audio/video player elements, and gallery/info-panel videos
Fullscreen gallery	RuntimeControls are hidden while the fullscreen gallery is open, so gallery videos render a small sound button connected to the same global mute state
User control	User can unmute via the sound button in RuntimeControls
Event isolation	Unmuting does not replay already-active hover/click effects; sound effects only play on their own fresh `false -> true` hover/active transition
UI chrome isolation	Runtime/global sound buttons stop event propagation so control clicks do not reach canvas or presentation elements

Files Involved:

frontend/src/css/main.css - CSS to hide native iOS controls
frontend/src/lib/backgroundAudioController.ts - Shared interaction, ducking, and mute state controller
frontend/src/hooks/useGlobalAudioMute.ts - React subscription hook for global mute state
frontend/src/hooks/useVideoSoundControl.ts - RuntimeControls sound-button visibility and actions
frontend/src/components/Runtime/RuntimeControls.tsx - Sound toggle button
frontend/src/components/Constructor/CanvasBackground.tsx - Background video/audio rendering with webkit-playsinline

2. Video Player Elements

Interactive video elements with controls, placed in page UI schema.

// frontend/src/components/UiElements/elements/VideoPlayerElement.tsx
<video
  src={resolve(element.mediaUrl)}
  controls
  autoPlay={Boolean(element.mediaAutoplay)}
  loop={Boolean(element.mediaLoop)}
  muted={Boolean(element.mediaMuted)}
  playsInline
/>

Configurable Properties:

Property	Type	Description
`mediaUrl`	string	Video asset URL
`mediaAutoplay`	boolean	Auto-start playback
`mediaLoop`	boolean	Loop continuously
`mediaMuted`	boolean	Mute audio

Note: Controls are always shown. The component uses resolveAssetPlaybackUrl() for URL resolution.

3. Transition Videos

Animated videos that play during page navigation.

interface TransitionVideo {
  video_url: string;        // Forward transition video
  duration_sec: number;     // Duration in seconds
  supports_reverse: boolean; // Can play in reverse
}

Forward Playback

How It Works

Forward playback uses native HTML5 video capabilities:

1. Set video.src = videoUrl
2. Set video.autoPlay = true (or call video.play())
3. Video plays from start to end
4. 'ended' event fires → trigger next action

Implementation

// Transition overlay forward playback
<video
  ref={videoRef}
  src={transitionVideoUrl}
  autoPlay={!isReverse}
  muted
  playsInline
  preload="auto"
  onEnded={handleTransitionComplete}
/>

Fallback Timer

If video fails to trigger onEnded, a fallback timer ensures navigation continues:

const fallbackTimer = setTimeout(() => {
  finishTransition();
}, transitionDuration * 1000 + 500); // duration + buffer

videoRef.current.onended = () => {
  clearTimeout(fallbackTimer);
  finishTransition();
};

Reverse Playback

Overview

Reverse playback is handled by useReversePlayback hook with two strategies:

┌─────────────────────┐
│  Start Reverse      │
└─────────┬───────────┘
          │
          ▼
┌─────────────────────┐     Supported?     ┌─────────────────────┐
│  Try Native Reverse │ ───────────────────│  Use Native         │
│  (playbackRate=-1)  │        YES         │  playbackRate=-1    │
└─────────┬───────────┘                    └─────────────────────┘
          │ NO
          ▼
┌─────────────────────┐
│  Frame-Stepping     │
│  Fallback           │
└─────────────────────┘

Strategy 1: Native Reverse (Primary)

Uses browser's native reverse playback when supported (Chrome, Edge):

const startNativeReverse = () => {
  video.currentTime = video.duration;
  video.playbackRate = -1;
  video.play();

  video.ontimeupdate = () => {
    if (video.currentTime <= 0.05) {
      video.currentTime = 0;
      onComplete();
    }
  };
};

Browser Support:

Browser	Native Reverse
Chrome	✅ Supported
Edge	✅ Supported
Firefox	❌ Use fallback
Safari	❌ Use fallback

Strategy 2: Frame-Stepping Fallback

Manual frame-by-frame reverse for unsupported browsers:

const startFrameStepping = () => {
  const fps = isPreloaded ? 30 : 15;  // Adaptive FPS
  const frameInterval = 1000 / fps;
  const frameStep = 1 / fps;

  let lastFrameTime = 0;

  const stepFrame = (timestamp: number) => {
    if (timestamp - lastFrameTime >= frameInterval) {
      video.currentTime = Math.max(0, video.currentTime - frameStep);
      lastFrameTime = timestamp;

      if (video.currentTime <= 0) {
        onComplete();
        return;
      }
    }
    requestAnimationFrame(stepFrame);
  };

  requestAnimationFrame(stepFrame);
};

Adaptive FPS:

Condition	FPS	Reason
Preloaded video	30	Smooth playback, data available
Streaming video	15	Conservative, bandwidth-aware

Buffering Handling

The system waits for sufficient buffered data before starting:

const checkBuffered = () => {
  const buffered = video.buffered;
  if (buffered.length > 0) {
    const bufferedEnd = buffered.end(buffered.length - 1);
    const bufferedAmount = bufferedEnd - video.currentTime;

    if (bufferedAmount >= 0.5) {  // 0.5 seconds minimum
      startReverse();
    }
  }
};

Buffering Timeout: 8 seconds before attempting with available buffer.

Hook API

const {
  startReverse,        // Start reverse playback (async)
  stopReverse,         // Cancel reverse playback
  isReversing,         // Currently playing in reverse
  isBuffering,         // Waiting for buffer
  canUseNativeReverse, // Browser supports playbackRate = -1
} = useReversePlayback({
  videoRef,
  onComplete,
  preloadedUrls?,      // Optional: Set of preloaded video URLs
  videoUrl?,           // Optional: Current video URL for cache lookup
  getCachedBlobUrl?,   // Optional: Function to get cached blob URL
});

Constructor Mode

Video Preview System

The Constructor provides comprehensive video testing:

interface TransitionPreview {
  videoUrl: string;           // Forward video URL
  reverseVideoUrl?: string;   // Optional separate reverse video
  reverseMode: 'reverse';     // Playback mode indicator
}

Video Loading Flow

1. User selects transition video
2. Fetch video as blob (better seeking)
3. Create blob URL
4. Monitor readyState events
5. Extract duration from metadata
6. Enable preview controls

Event Monitoring

Constructor monitors comprehensive video events:

// Ready state events
video.addEventListener('loadedmetadata', handleMetadata);
video.addEventListener('canplay', handleCanPlay);
video.addEventListener('canplaythrough', handleReady);

// Playback events
video.addEventListener('playing', handlePlaying);
video.addEventListener('ended', handleEnded);

// Error events
video.addEventListener('error', handleError);
video.addEventListener('abort', handleAbort);
video.addEventListener('stalled', handleStalled);

Duration Extraction

// frontend/src/lib/mediaDuration.ts (120 LOC)

export interface MediaDurationResult {
  duration: number;
  width?: number;   // Only for video
  height?: number;  // Only for video
}

export function probeMediaDuration(
  source: string | File,          // URL string or File object
  mediaType: 'video' | 'audio',   // Media type to probe
  timeoutMs = 10000,              // Timeout in milliseconds
): Promise<MediaDurationResult | null> {
  return new Promise((resolve) => {
    const mediaElement = mediaType === 'video'
      ? document.createElement('video')
      : document.createElement('audio');

    mediaElement.preload = 'metadata';

    mediaElement.onloadedmetadata = () => {
      const result: MediaDurationResult = { duration: mediaElement.duration };
      if (mediaType === 'video' && mediaElement instanceof HTMLVideoElement) {
        result.width = mediaElement.videoWidth;
        result.height = mediaElement.videoHeight;
      }
      cleanup();
      resolve(result);
    };

    mediaElement.onerror = () => {
      cleanup();
      resolve(null);  // Returns null on error, not reject
    };

    // Source can be URL string or File object
    mediaElement.src = typeof source === 'string'
      ? source
      : URL.createObjectURL(source);

    setTimeout(() => {
      cleanup();
      resolve(null);
    }, timeoutMs);
  });
}

// Helper functions for MIME type detection
export function isVideoMimeType(mimeType: string | null | undefined): boolean {
  if (!mimeType) return false;
  return mimeType.startsWith('video/');
}

export function isAudioMimeType(mimeType: string | null | undefined): boolean {
  if (!mimeType) return false;
  return mimeType.startsWith('audio/');
}

Preview Controls

Control	Forward Mode	Reverse Mode
Play	Native play()	startReverse()
Pause	Native pause()	stopReverse()
Progress	timeupdate event	Manual tracking

Runtime Mode (Presentations)

RuntimePresentation Component

Handles video playback during tour presentations:

// frontend/src/components/RuntimePresentation.tsx

// Background video rendering
{page.background_video_url && (
  <video
    src={resolveAssetUrl(page.background_video_url)}
    autoPlay
    loop
    muted
    playsInline
    className="absolute inset-0 w-full h-full object-cover -z-10"
  />
)}

// Video player elements
{element.type === 'video_player' && (
  <video
    src={resolveAssetUrl(element.content_json.mediaUrl)}
    controls={element.content_json.mediaControls}
    autoPlay={element.content_json.autoPlay}
    loop={element.content_json.loop}
    muted={element.content_json.muted}
  />
)}

Transition Overlay Flow

┌──────────────────────────────────────────────────────────────┐
│  1. User clicks navigation element                           │
│     └── Resolve targetPageSlug → targetPageId                │
│     └── startNavigation(targetPageId, element)               │
└──────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌──────────────────────────────────────────────────────────────┐
│  2. Determine transition type                                │
│     └── isReverse = isBack && linkData.supports_reverse      │
└──────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌──────────────────────────────────────────────────────────────┐
│  3. Set overlay transition state                             │
│     └── setOverlayTransition({ videoUrl, isReverse })        │
└──────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌──────────────────────────────────────────────────────────────┐
│  4. Render transition overlay with video                     │
│     └── <TransitionOverlay video={...} />                    │
└──────────────────────────────────────────────────────────────┘
                              │
          ┌───────────────────┴───────────────────┐
          │                                       │
          ▼                                       ▼
┌──────────────────────┐              ┌──────────────────────┐
│  Forward Playback    │              │  Reverse Playback    │
│  - autoPlay={true}   │              │  - useReversePlayback│
│  - Wait for onEnded  │              │  - Wait for complete │
└──────────┬───────────┘              └──────────┬───────────┘
           │                                      │
           └──────────────────┬───────────────────┘
                              │
                              ▼
┌──────────────────────────────────────────────────────────────┐
│  5. Finish transition                                        │
│     └── finishOverlayTransition() → switch pages             │
└──────────────────────────────────────────────────────────────┘

Transition State Management

The useTransitionPlayback hook manages transition video playback with comprehensive state tracking:

// frontend/src/hooks/useTransitionPlayback.ts

// Reverse playback modes
export type ReverseMode = 'none' | 'reverse' | 'separate';

// Transition configuration passed to the hook
export interface TransitionConfig {
  videoUrl: string;           // Forward transition video URL
  storageKey?: string;        // Raw storage path for cache lookup
  reverseMode: ReverseMode;   // How to handle reverse playback
  reverseVideoUrl?: string;   // Separate reverse video (for 'separate' mode)
  durationSec?: number;       // Expected duration in seconds
  targetPageId?: string;      // Target page for navigation
  displayName?: string;       // Human-readable name for logging
}

// Playback phases
export type PlaybackPhase =
  | 'idle'        // No transition active
  | 'preparing'   // Loading video, resolving URLs
  | 'playing'     // Forward playback in progress
  | 'reversing'   // Reverse playback in progress
  | 'finishing'   // Pre-decoding target page images
  | 'completed';  // Transition finished, ready for page switch

// Hook options
export interface UseTransitionPlaybackOptions {
  videoRef: RefObject<HTMLVideoElement | null>;
  transition: TransitionConfig | null;
  onComplete: (targetPageId?: string) => void;
  onError?: (reason: string) => void;

  timeouts?: {
    playbackStartMs?: number;   // Max wait for playback to start (default: 3000)
    durationBufferMs?: number;  // Buffer after expected duration (default: 200)
    hardTimeoutMs?: number;     // Absolute max timeout (default: 45000)
  };
  features?: {
    useBlobUrl?: boolean;             // Force blob URL loading
    preDecodeImages?: boolean;        // Pre-decode target page images
    getTargetPageImages?: () => string[];  // Get images to pre-decode
  };
  preload?: {
    preloadedUrls?: Set<string>;
    getCachedBlobUrl?: (url: string) => Promise<string | null>;
    getReadyBlobUrl?: (url: string) => string | null;
  };
}

// Hook result
export interface UseTransitionPlaybackResult {
  phase: PlaybackPhase;       // Current playback phase
  isBuffering: boolean;       // Waiting for video data
  isReversing: boolean;       // Currently in reverse playback
  cancel: () => void;         // Cancel transition immediately
  forceComplete: () => void;  // Force transition to complete
}

Default Timeouts:

const DEFAULT_TIMEOUTS = {
  playbackStartMs: 3000,   // 3 seconds to start playing
  durationBufferMs: 200,   // 200ms buffer after expected end
  hardTimeoutMs: 45000,    // 45 seconds absolute maximum
};

Usage Example:

const { phase, isBuffering, cancel, forceComplete } = useTransitionPlayback({
  videoRef,
  transition: {
    videoUrl: resolveAssetPlaybackUrl(element.transitionVideoUrl),
    storageKey: element.transitionVideoUrl,
    reverseMode: isBack ? 'reverse' : 'none',
    durationSec: element.transitionDurationSec,
    targetPageId: targetPage.id,
    displayName: element.label,
  },
  onComplete: (targetPageId) => {
    setCurrentPageId(targetPageId);
    setOverlayTransition(null);
  },
  onError: (reason) => {
    console.error('Transition failed:', reason);
    // Navigate anyway to prevent user being stuck
    setCurrentPageId(targetPageId);
    setOverlayTransition(null);
  },
  preload: {
    getReadyBlobUrl: preloadOrchestrator.getReadyBlobUrl,
    getCachedBlobUrl: preloadOrchestrator.getCachedBlobUrl,
  },
});

Video URL Resolution

The resolveAssetPlaybackUrl() utility normalizes video URLs with presigned URL cache lookup:

// frontend/src/lib/assetUrl.ts
export const resolveAssetPlaybackUrl = (value?: string): string => {
  const normalized = String(value || '').trim();
  if (!normalized) return '';

  // Pass through special URLs
  if (normalized.startsWith('data:') || normalized.startsWith('blob:'))
    return normalized;

  // Already an API file download URL
  if (normalized.startsWith('/api/file/download')) return normalized;

  // File download path (prepend API base)
  if (normalized.startsWith('/file/download'))
    return `${baseURLApi}${normalized}`;

  // Full URLs pass through
  if (normalized.startsWith('http://') || normalized.startsWith('https://'))
    return normalized;

  // Relative path - check presigned URL cache first (for fast S3 access)
  const presigned = getPresignedUrl(normalized);
  if (presigned) {
    return presigned;
  }

  // Fallback to backend proxy
  const normalizedPrivateUrl = normalized.replace(/^\/+/, '');
  return `${baseURLApi}/file/download?privateUrl=${encodeURIComponent(normalizedPrivateUrl)}`;
};

Presigned URL Priority: The function first checks if a valid presigned S3 URL is cached for the storage key. If found and not expired, it returns the presigned URL for direct S3 access. Otherwise, it falls back to the backend proxy endpoint.

Cache Integration

S3 Presigned URL Preloading

Videos are preloaded directly from S3 using presigned URLs:

1. POST /api/file/presign { urls: [video1, video2, ...] }  // Max 50 per batch
2. Download from S3 presigned URLs (1-hour expiry)
3. Store in Cache API (dual key: download URL + storage key)
4. Create blob URL: URL.createObjectURL(blob)
5. Store in readyBlobUrlsRef Map (dual key: download URL + storage key) for O(1) instant lookup

Storage Key Mapping: Videos are cached by canonical storage key (e.g., assets/project/transition.mp4) in addition to download URLs. This ensures cache hits even when presigned URL signatures change between sessions.

Ready Blob URL Lookup (Primary)

For instant video playback, use getReadyBlobUrl() which returns pre-created blob URLs. Lookups prioritize storage key over resolved URLs for reliable cache hits:

// 1. Primary: O(1) lookup by storage key (most reliable)
const readyUrl = preloadOrchestrator.getReadyBlobUrl(storageKey);
if (readyUrl) {
  video.src = readyUrl;  // Instant playback - no decode needed for video
  return;
}

// 2. Fallback: O(1) lookup by resolved URL
const readyUrlByResolved = preloadOrchestrator.getReadyBlobUrl(resolvedUrl);
if (readyUrlByResolved) {
  video.src = readyUrlByResolved;
}

Why storage key first: Presigned URL signatures change on each resolution (X-Amz-Signature differs). Storage keys (e.g., assets/project/video.mp4) are canonical and never change.

Cached Blob URL Fallback

If ready URL not available, create blob URL from cache:

const getCachedBlobUrl = async (url: string): Promise<string | null> => {
  const cache = await caches.open('tour-builder-assets-v1');
  const response = await cache.match(url);

  if (response) {
    const blob = await response.blob();
    return URL.createObjectURL(blob);
  }

  return null;
};

Usage in Reverse Playback

// useReversePlayback attempts ready/cached URL for better seeking
// Uses storage key for reliable cache lookup
const setupReverse = async (storageKey: string) => {
  // 1. Try instant ready blob URL by storage key (O(1) lookup, most reliable)
  const readyUrl = preloadOrchestrator.getReadyBlobUrl(storageKey);
  if (readyUrl) {
    video.src = readyUrl;
    isPreloaded = true;
  } else {
    // 2. Fallback: create blob URL from cache by storage key
    const cachedUrl = await getCachedBlobUrl(storageKey);
    if (cachedUrl) {
      video.src = cachedUrl;  // Better seeking with local blob
      isPreloaded = true;
    }
  }

  // Proceed with reverse playback
  startReverse();
};

Presigned URL Fallback Mechanism

The useTransitionPlayback hook includes automatic fallback when presigned S3 URLs fail (typically due to CORS issues):

// Check if URL is a presigned S3 URL
function isPresignedUrl(url: string): boolean {
  return url.includes('X-Amz-Signature=') || url.includes('x-amz-signature=');
}

// Convert presigned URL to proxy URL fallback
function getProxyUrlFallback(
  presignedUrl: string,
  originalStorageKey?: string,
): string | null {
  // If we have the original storage key, use it directly
  if (originalStorageKey && isRelativeStoragePath(originalStorageKey)) {
    const normalizedPath = originalStorageKey.replace(/^\/+/, '');
    return `${baseURLApi}/file/download?privateUrl=${encodeURIComponent(normalizedPath)}`;
  }

  // Try to extract path from presigned URL
  try {
    const url = new URL(presignedUrl);
    const pathParts = url.pathname.split('/').filter(Boolean);
    if (pathParts.length >= 2) {
      const storagePath = pathParts.slice(1).join('/');
      return `${baseURLApi}/file/download?privateUrl=${encodeURIComponent(storagePath)}`;
    }
  } catch {
    // URL parsing failed
  }
  return null;
}

Fallback Flow:

1. Video error event fires on presigned URL
2. Check if current URL is a presigned S3 URL
3. If not already tried fallback:
   a. Mark presigned URL as failed (future resolves use proxy)
   b. Build proxy fallback URL from storage key
   c. Retry video load with proxy URL
4. If fallback also fails → trigger error handler

This mechanism ensures video playback works even when S3 CORS is misconfigured or presigned URLs expire mid-session.

Image Pre-decoding During Transitions

The useTransitionPlayback hook supports pre-decoding target page images during the finishing phase to ensure instant display after transition:

// In useTransitionPlayback.ts
async function waitForImages(urls: string[], timeoutMs = 2000): Promise<void> {
  if (urls.length === 0) return;

  const decodePromises = urls.map(
    (url) =>
      new Promise<void>((resolve) => {
        const img = new Image();
        img.src = url;
        if (typeof img.decode === 'function') {
          img.decode()
            .then(() => resolve())
            .catch(() => resolve());
        } else {
          img.onload = () => resolve();
          img.onerror = () => resolve();
        }
      }),
  );

  // Race against timeout to prevent blocking
  await Promise.race([
    Promise.all(decodePromises),
    new Promise<void>((resolve) => setTimeout(resolve, timeoutMs)),
  ]);
}

Usage in Transition Flow:

// Enable via features option
useTransitionPlayback({
  // ...
  features: {
    preDecodeImages: true,
    getTargetPageImages: () => {
      // Return URLs of images on the target page
      return targetPage.elements
        .filter(el => el.type === 'image')
        .map(el => resolveAssetUrl(el.imageUrl));
    },
  },
});

This ensures the target page appears instantly without visible image loading after the transition video completes.

Comparison: Constructor vs Runtime

Aspect	Constructor	Runtime
Purpose	Testing & preview	Presentation playback
Controls	Full manual control	Automated flow
Logging	Detailed debug logs	Silent operation
Network Info	Displayed in UI	Transparent
Buffering	Status shown	Hidden from user
Error Handling	Detailed messages	Fallback timers
Duration Source	Editable input	From transition data

Error Handling

Reverse Playback Failures

const handleReverseError = (error: Error) => {
  console.error('Reverse playback failed:', error);

  // Fallback chain:
  // 1. Native reverse fails → Try frame-stepping
  // 2. Frame-stepping fails → Use available buffer
  // 3. Buffer timeout → Complete with partial playback
  // 4. All fails → Skip transition, navigate directly

  if (currentStrategy === 'native') {
    switchToFrameStepping();
  } else {
    completeWithAvailableProgress();
  }
};

Forward Playback Failures

const handleVideoError = (event: Event) => {
  console.error('Video playback error:', event);

  // Ensure navigation continues
  clearTimeout(fallbackTimer);
  finishTransition();
};

Network Failures

// Video stalls during loading
video.onstalled = () => {
  console.warn('Video stalled, waiting for data...');
  // Don't block - fallback timer will handle
};

video.onwaiting = () => {
  setIsBuffering(true);
};

video.onplaying = () => {
  setIsBuffering(false);
};

Performance Optimizations

1. Adaptive Frame Rate

// Adjust FPS based on video availability
const fps = isPreloaded ? 30 : 15;

2. Blob URL Caching

// Use cached blob URLs for better seeking
const cachedUrl = await getCachedBlobUrl(videoUrl);
if (cachedUrl) {
  video.src = cachedUrl;
}

3. Preload Priority

Videos receive priority based on context:

const videoPriority = {
  currentPage: 1000,     // Current page assets
  neighborBase: 500,     // Neighbor page assets
  transition: 150,       // Highest type priority - needed immediately on click
  image: 100,            // Backgrounds load during transition
  audio: 50,
  video: 30,             // Background videos can stream progressively
};

Note: maxNeighborDepth defaults to 1 (immediate neighbors only). Transition videos have the highest type priority (+150) because they're needed immediately when navigation is triggered.

4. Memory Management

// Clean up blob URLs
useEffect(() => {
  return () => {
    if (blobUrl) {
      URL.revokeObjectURL(blobUrl);
    }
  };
}, [blobUrl]);

// Cancel animation frames
useEffect(() => {
  return () => {
    if (animationFrameId) {
      cancelAnimationFrame(animationFrameId);
    }
  };
}, []);

5. Event Listener Cleanup

useEffect(() => {
  const video = videoRef.current;

  video.addEventListener('timeupdate', handleTimeUpdate);
  video.addEventListener('ended', handleEnded);

  return () => {
    video.removeEventListener('timeupdate', handleTimeUpdate);
    video.removeEventListener('ended', handleEnded);
  };
}, []);

Key Files Reference

File	Location	LOC	Purpose
`useTransitionPlayback.ts`	`frontend/src/hooks/`	778	Transition video playback coordination
`useReversePlayback.ts`	`frontend/src/hooks/`	399	Reverse playback logic
`useBackgroundVideoPlayback.ts`	`frontend/src/hooks/`	~210	Background video time control, session-scoped play-once when loop=false
`usePreloadOrchestrator.ts`	`frontend/src/hooks/`	-	Asset preloading with ready blob URLs
`usePageSwitch.ts`	`frontend/src/hooks/`	-	Page navigation with preloaded assets
`mediaDuration.ts`	`frontend/src/lib/`	120	Duration extraction utility
`assetUrl.ts`	`frontend/src/lib/`	-	URL resolution with presigned cache
`extractPageLinks.ts`	`frontend/src/lib/`	-	Extract navigation links with transition videos
`StorageManager.ts`	`frontend/src/lib/offline/`	-	Cache API storage for video blobs
`VideoPlayerElement.tsx`	`frontend/src/components/UiElements/elements/`	52	Video player UI element
`RuntimePresentation.tsx`	`frontend/src/components/`	-	Runtime video rendering
`constructor.tsx`	`frontend/src/pages/`	-	Constructor video preview
`[projectSlug]/index.tsx`	`frontend/src/pages/p/`	-	Production runtime page
`[projectSlug]/stage.tsx`	`frontend/src/pages/p/`	-	Stage runtime page
`preload.config.ts`	`frontend/src/config/`	-	Video priority settings (transition: 150)

Troubleshooting

Video Won't Play

Check browser autoplay policy (must be muted)
Verify video URL resolves correctly
Check CORS headers on video source
Ensure video format is supported (MP4/WebM)

Reverse Playback Choppy

Check if video is preloaded (should use 30 FPS)
Verify sufficient buffer is available
Try using cached blob URL for better seeking
Consider reducing video resolution

Transition Doesn't Complete

Check onEnded event fires correctly
Verify fallback timer is set
Look for JavaScript errors in console
Ensure finishTransition is called

Memory Leaks

Verify blob URLs are revoked on cleanup
Check animation frames are cancelled
Ensure event listeners are removed
Look for orphaned video elements

38 KiB Raw Blame History

Video Playback Feature - E2E Documentation

Overview

Architecture

Video Types

1. Background Videos

Global Sound Control and Autoplay Compatibility

2. Video Player Elements

3. Transition Videos

Forward Playback

How It Works

Implementation

Fallback Timer

Reverse Playback

Overview

Strategy 1: Native Reverse (Primary)

Strategy 2: Frame-Stepping Fallback

Buffering Handling

Hook API

Constructor Mode

Video Preview System

Video Loading Flow

Event Monitoring

Duration Extraction

Preview Controls

Runtime Mode (Presentations)

RuntimePresentation Component

Transition Overlay Flow

Transition State Management

Video URL Resolution

Cache Integration

S3 Presigned URL Preloading

Ready Blob URL Lookup (Primary)

Cached Blob URL Fallback

Usage in Reverse Playback

Presigned URL Fallback Mechanism

Image Pre-decoding During Transitions

Comparison: Constructor vs Runtime

Error Handling

Reverse Playback Failures

Forward Playback Failures

Network Failures

Performance Optimizations

1. Adaptive Frame Rate

2. Blob URL Caching

3. Preload Priority

4. Memory Management

5. Event Listener Cleanup

Key Files Reference

Troubleshooting

Video Won't Play

Reverse Playback Choppy

Transition Doesn't Complete

Memory Leaks

38 KiB

Raw Blame History