39948-vm/frontend/docs/ui-element-preloading-analysis.md

UI Element Processing & Neighbor Preloading Analysis

Executive Summary

Deep analysis of how UI elements are processed throughout the Tour Builder Platform - from creation in the Constructor, to rendering in Runtime Presentations, across Online and Offline modes. This document traces each preloading thread step-by-step to verify robustness.

---

1. PRELOAD CONFIGURATION (Single Source of Truth)

File: src/config/preload.config.ts

1.1 Asset URL Fields Configuration

assetFields: {
  // All 18 URL fields for preloading extraction
  all: [
    'iconUrl',                    // Base element icon
    'imageUrl',                   // Generic image reference
    'mediaUrl',                   // Video/audio player source
    'videoUrl',                   // Video element
    'audioUrl',                   // Audio element
    'transitionVideoUrl',         // Navigation transition video
    'backgroundImageUrl',         // Element background
    'reverseVideoUrl',            // Reverse transition video
    'carouselPrevIconUrl',        // Carousel prev button
    'carouselNextIconUrl',        // Carousel next button
    'galleryHeaderImageUrl',      // Gallery header background
    'galleryCarouselPrevIconUrl', // Gallery carousel prev
    'galleryCarouselNextIconUrl', // Gallery carousel next
    'galleryCarouselBackIconUrl', // Gallery carousel back
    'src', 'url', 'poster', 'thumbnail',  // Generic fallbacks
  ],

  // 10 Image-only fields for pre-decode optimization
  images: [
    'iconUrl', 'imageUrl', 'backgroundImageUrl',
    'carouselPrevIconUrl', 'carouselNextIconUrl',
    'galleryHeaderImageUrl', 'galleryCarouselPrevIconUrl',
    'galleryCarouselNextIconUrl', 'galleryCarouselBackIconUrl', 'src',
  ],

  // 3 Nested array fields containing assets
  nested: ['galleryCards', 'carouselSlides', 'galleryInfoSpans'],

  // 3 URL fields within nested items
  nestedUrlFields: ['imageUrl', 'videoUrl', 'iconUrl'],
}

1.2 Configuration Consumers (10 files)

File	Uses	Purpose
extractPageLinks.ts	all, nested, nestedUrlFields	Explicit field extraction
useNeighborGraph.ts	all	Recursive traversal (depth 5)
usePreloadOrchestrator.ts	all	Asset initialization
imagePreDecode.ts	images, nested, nestedUrlFields	Image-only pre-decode
StorageManager.ts	storage config	Storage thresholds
DownloadManager.ts	queue settings	Download concurrency
useStorageQuota.ts	storage config	Quota warnings
usePreloadProgress.ts	autoRemove	Progress cleanup
DownloadContext.tsx	various	Download UI state

---

2. COMPLETE PRELOADING FLOW (Step-by-Step)

2.1 Entry Point: RuntimePresentation Mount

RuntimePresentation.tsx mounts
    ↓
extractPageLinksAndElements(pages)           [lib/extractPageLinks.ts:105-182]
    ├── Parse ui_schema_json for each page
    ├── extractAssetFields(element) → uses PRELOAD_CONFIG.assetFields
    │   ├── Extract top-level fields from assetFields.all
    │   └── Extract nested arrays (galleryCards, carouselSlides, galleryInfoSpans)
    │       └── Within each item, extract nestedUrlFields (imageUrl, videoUrl, iconUrl)
    ├── Build pageLinks[] for navigation graph
    └── Build preloadElements[] with content_json containing asset URLs
    ↓
usePreloadOrchestrator({ pages, pageLinks, elements, currentPageId })

2.2 Thread 1: Neighbor Graph Building

useNeighborGraph({ pages, pageLinks, elements, maxDepth: 1 })
    ↓
Build adjacencyList (Map<pageId, neighborPageIds[]>)     [line 119-141]
    ├── Initialize all pages in map
    └── Add edges from active pageLinks (from_pageId → to_pageId)
    ↓
getNeighbors(currentPageId, depth)                        [line 144-177]
    ├── BFS traversal from current page
    ├── Track visited pages to avoid cycles
    └── Return PreloadNeighborInfo[] sorted by distance
    ↓
getAssetsForPages(pageIds)                                [line 180-224]
    ├── For each page: filter elements by pageId
    ├── extractAssetsFromContent(content_json)            [line 56-106]
    │   ├── Parse JSON content
    │   ├── checkObject() recursive traversal (depth ≤ 5)
    │   │   └── For each field in PRELOAD_CONFIG.assetFields.all:
    │   │       └── If string value found, classify asset type and add to assets[]
    │   └── Return PreloadAssetInfo[] with { url, pageId, assetType, priority }
    └── Also extract transition videos from pageLinks

2.3 Thread 2: Priority Assignment

getPrioritizedAssets(currentPageId, maxDepth)             [line 228-274]
    ↓
Current page assets:
    priority = PRELOAD_CONFIG.priority.currentPage (1000)
             + PRELOAD_CONFIG.priority.assetType[type]

    Asset type priorities:
    ├── transition: +150 (highest - needed on navigation click)
    ├── image: +100 (backgrounds load during transition)
    ├── audio: +50
    └── video: +30
    ↓
Neighbor page assets:
    basePriority = PRELOAD_CONFIG.priority.neighborBase (500) / distance
    priority = basePriority + assetType priority
    ↓
Deduplicate by URL, keep highest priority
Sort descending by priority

2.4 Thread 3: URL Resolution (Presigned URLs)

usePreloadOrchestrator effect on currentPageId change     [line 669-924]
    ↓
Collect storage paths needing presigning:
    ├── Current page: background_image_url, background_video_url, background_audio_url
    ├── Element assets from neighborGraph.getPrioritizedAssets()
    └── Neighbor pages: background URLs
    ↓
queuePresignedUrls(storagePaths)                          [lib/assetUrl.ts]
    ├── POST /api/file/presign { urls: storagePaths[] }
    ├── Response: { [storageKey]: presignedUrl }
    └── Cache presigned URLs (1-hour expiry)
    ↓
resolveUrl(storageKey, presignedUrls)                     [line 761-771]
    ├── If presignedUrls[storageKey] exists → use presigned URL
    └── Fallback → resolveAssetPlaybackUrl (proxy URL)

2.5 Thread 4: Download Queue Processing

addToQueue(item: PreloadQueueItem)                        [line 494-530]
    ├── Skip if already in queue or preloaded
    ├── Insert in priority order (binary search insertion point)
    └── Trigger processQueue()
    ↓
processQueue()                                            [line 355-491]
    ├── Check: isOnline, queue not empty, not already processing
    ├── maxConcurrent = recommendedConcurrency (network-aware)
    ↓
    While queue has items AND activeDownloads < maxConcurrent:
        ├── Shift item from queue
        ├── Skip if already preloaded (preloadedUrls.has(url))
        ├── Skip if already cached (isUrlCached → StorageManager.hasAsset)
        │   └── If cached: createReadyBlobUrl() for instant display
        ↓
        preloadWithProgress(url, jobId, assetId)          [line 140-234]
            ├── Emit downloadEventBus.emitPreloadStart
            ├── fetch(url) with streaming progress
            ├── Collect chunks, emit progress events
            ├── Store in Cache API: caches.open(assets).put(url, response)
            └── Emit downloadEventBus.emitPreloadComplete
        ↓
        On success:
            ├── createReadyBlobUrl(url, storageKey)
            │   ├── StorageManager.getAsset(url) → blob
            │   ├── URL.createObjectURL(blob) → blobUrl
            │   ├── If image: decodeImage(blobUrl) → pre-paint ready
            │   └── readyBlobUrlsRef.set(url, blobUrl)
            └── Also cache under storageKey for post-refresh lookup
        ↓
        On failure (presigned URL CORS):
            ├── markPresignedUrlFailed(storageKey)
            ├── Build proxy URL: /api/file/download?privateUrl=...
            └── Retry with proxy URL

2.6 Thread 5: Ready State & Instant Lookup

getReadyBlobUrl(url): string | null                       [line 582-584]
    └── readyBlobUrlsRef.current.get(url) → O(1) Map lookup

Usage in RuntimePresentation:
    const resolveUrlWithBlob = (url) => {
        const blobUrl = preloadOrchestrator.getReadyBlobUrl(url);
        return blobUrl || resolveAssetPlaybackUrl(url);
    };

    <RuntimeElement resolveUrl={resolveUrlWithBlob} ... />

---

3. ELEMENT TYPE → URL FIELD MAPPING (Complete Audit)

3.1 All Element Types (11 total)

Element Type	URL Fields	Nested Arrays
navigation_next/prev	iconUrl, transitionVideoUrl, reverseVideoUrl	-
gallery	iconUrl, galleryHeaderImageUrl, galleryCarouselPrevIconUrl, galleryCarouselNextIconUrl, galleryCarouselBackIconUrl	galleryCards[].imageUrl, galleryInfoSpans[].iconUrl
carousel	iconUrl, carouselPrevIconUrl, carouselNextIconUrl	carouselSlides[].imageUrl
video_player	iconUrl, mediaUrl	-
audio_player	iconUrl, mediaUrl	-
tooltip	iconUrl	-
description	iconUrl	-
spot	iconUrl	-
logo	iconUrl	-
popup	iconUrl	-

3.2 Cross-Reference Verification

constructor.ts URL Fields vs preload.config.ts assetFields.all:

Field in constructor.ts	In assetFields.all?	Status
iconUrl	Yes	OK
mediaUrl	Yes	OK
backgroundImageUrl	Yes	OK
videoUrl	Yes	OK
audioUrl	Yes	OK
transitionVideoUrl	Yes	OK
reverseVideoUrl	Yes	OK
galleryHeaderImageUrl	Yes	OK
carouselPrevIconUrl	Yes	OK
carouselNextIconUrl	Yes	OK
galleryCarouselPrevIconUrl	Yes	OK
galleryCarouselNextIconUrl	Yes	OK
galleryCarouselBackIconUrl	Yes	OK

Nested Arrays vs nested config:

Nested Array	In nested config?	Status
galleryCards	Yes	OK
carouselSlides	Yes	OK
galleryInfoSpans	Yes	OK

Nested URL Fields vs nestedUrlFields:

Field in Nested Items	In nestedUrlFields?	Status
galleryCards[].imageUrl	Yes	OK
carouselSlides[].imageUrl	Yes	OK
galleryInfoSpans[].iconUrl	Yes	OK

---

4. OFFLINE MODE PROCESSING

4.1 Storage Architecture

StorageManager                                [lib/offline/StorageManager.ts]
    ├── Threshold: OFFLINE_CONFIG.storage.indexedDbMinSize (5MB)
    ├── Small files (< 5MB): Cache API
    │   └── caches.open('vm-assets').put(url, response)
    └── Large files (≥ 5MB): IndexedDB via Dexie
        └── OfflineDbManager.storeAsset(asset)

hasAsset(url): Promise<boolean>
    1. Check IndexedDB: OfflineDbManager.hasAssetByUrl(url)
    2. Check Cache API: caches.open().match(url)

getAsset(url): Promise<Blob | null>
    1. Try IndexedDB: OfflineDbManager.getAssetByUrl(url)
    2. Try Cache API: caches.open().match(url).blob()

4.2 Service Worker Integration

sw.ts (Serwist-generated)
    ├── Precache: static assets (JS, CSS, fonts)
    ├── Runtime caching strategies:
    │   ├── API requests: NetworkFirst
    │   └── Assets: CacheFirst
    └── Offline fallback handling

---

5. IMAGE PRE-DECODE FLOW

5.1 extractPageImageUrls

extractPageImageUrls(page)                    [lib/imagePreDecode.ts:110-175]
    ↓
Extract background_image_url
    ↓
Parse ui_schema_json → elements[]
    ↓
For each element:
    ├── Direct image fields (assetFields.images):
    │   └── iconUrl, imageUrl, backgroundImageUrl, carousel*IconUrl, gallery*IconUrl, src
    └── Nested arrays (assetFields.nested):
        └── Filter nestedUrlFields to images only:
            ├── imageUrl (in images array)
            ├── iconUrl (in images array)
            └── videoUrl (not in images - correctly excluded)

5.2 waitForPageImages

waitForPageImages(page, timeoutMs, cacheProvider)
    ↓
extractPageImageUrls(page) → imageUrls[]
    ↓
decodeImages(imageUrls, timeoutMs, cacheProvider)
    ├── For each URL: decodeImage()
    │   ├── If cacheProvider: try getCachedBlobUrl() → blob URL (local, fast)
    │   └── new Image().decode() or onload fallback
    └── Promise.race([all decoded, timeout])

---

6. ROBUSTNESS VERIFICATION

6.1 All Asset Types Covered

Asset Type	Extracted?	Preloaded?	Pre-decoded?	Cached?
Background images	Yes	Yes	Yes	Yes
Background videos	Yes	Yes	N/A	Yes
Background audio	Yes	Yes	N/A	Yes
Element icons	Yes	Yes	Yes	Yes
Transition videos	Yes	Yes	N/A	Yes
Reverse videos	Yes	Yes	N/A	Yes
Gallery cards images	Yes	Yes	Yes	Yes
Gallery info span icons	Yes	Yes	Yes	Yes
Gallery header image	Yes	Yes	Yes	Yes
Gallery carousel icons	Yes	Yes	Yes	Yes
Carousel slide images	Yes	Yes	Yes	Yes
Carousel nav icons	Yes	Yes	Yes	Yes
Media player URLs	Yes	Yes	N/A	Yes

6.2 Extraction Algorithm Robustness

1. extractPageLinks.ts - Explicit field extraction

   • Uses assetFields.all for top-level fields
   • Uses nested + nestedUrlFields for nested arrays
   • Handles all configured fields

2. useNeighborGraph.ts - Recursive traversal

   • checkObject() recursively traverses to depth 5
   • Matches any field from assetFields.all at any nesting level
   • Handles deeply nested structures

3. imagePreDecode.ts - Image-only extraction

   • Filters nestedUrlFields against assetFields.images
   • Only pre-decodes actual images (not videos/audio)
   • Correctly filters non-image URLs

6.3 Edge Cases Handled

Edge Case	Handling
Empty ui_schema_json	Returns empty arrays safely
Missing nested arrays	Skipped gracefully
Null/undefined values	Filtered out
Already cached assets	Skipped download, creates blob URL
Presigned URL CORS failure	Retries with proxy URL
Network offline	Skips preloading, uses cache
Large files (>=5MB)	Stored in IndexedDB instead of Cache API

---

7. CONCLUSION

The neighbor preloading system is robust and comprehensive:

1. Central Configuration - All asset URL fields defined in preload.config.ts
2. Consistent Extraction - All consumers use the same config
3. Complete Coverage - All 11 element types and their URL fields are covered
4. Nested Arrays - galleryCards, carouselSlides, galleryInfoSpans all handled
5. Dual Storage - Cache API for small files, IndexedDB for large files
6. Fallback Handling - Presigned URLs with proxy fallback
7. Network Awareness - Adaptive concurrency based on connection

No gaps identified - all asset types are properly extracted, preloaded, and cached.

---

Related Documentation

• assets-preloading.md - E2E preloading feature documentation
• runtime-presentation.md - Runtime presentation viewer
• constructor-page-editor.md - Constructor page editor
• hooks-module.md - Custom hooks reference
• lib-module.md - Library utilities reference