411 lines
15 KiB
Markdown
411 lines
15 KiB
Markdown
# UI Element Processing & Neighbor Preloading Analysis
|
|
|
|
## Executive Summary
|
|
|
|
Deep analysis of how UI elements are processed throughout the Tour Builder Platform - from creation in the Constructor, to rendering in Runtime Presentations, across Online and Offline modes. This document traces each preloading thread step-by-step to verify robustness.
|
|
|
|
---
|
|
|
|
## 1. PRELOAD CONFIGURATION (Single Source of Truth)
|
|
|
|
**File:** `src/config/preload.config.ts`
|
|
|
|
### 1.1 Asset URL Fields Configuration
|
|
|
|
```typescript
|
|
assetFields: {
|
|
// All 18 URL fields for preloading extraction
|
|
all: [
|
|
'iconUrl', // Base element icon
|
|
'imageUrl', // Generic image reference
|
|
'mediaUrl', // Video/audio player source
|
|
'videoUrl', // Video element
|
|
'audioUrl', // Audio element
|
|
'transitionVideoUrl', // Navigation transition video
|
|
'backgroundImageUrl', // Element background
|
|
'reverseVideoUrl', // Reverse transition video
|
|
'carouselPrevIconUrl', // Carousel prev button
|
|
'carouselNextIconUrl', // Carousel next button
|
|
'galleryHeaderImageUrl', // Gallery header background
|
|
'galleryCarouselPrevIconUrl', // Gallery carousel prev
|
|
'galleryCarouselNextIconUrl', // Gallery carousel next
|
|
'galleryCarouselBackIconUrl', // Gallery carousel back
|
|
'src', 'url', 'poster', 'thumbnail', // Generic fallbacks
|
|
],
|
|
|
|
// 10 Image-only fields for pre-decode optimization
|
|
images: [
|
|
'iconUrl', 'imageUrl', 'backgroundImageUrl',
|
|
'carouselPrevIconUrl', 'carouselNextIconUrl',
|
|
'galleryHeaderImageUrl', 'galleryCarouselPrevIconUrl',
|
|
'galleryCarouselNextIconUrl', 'galleryCarouselBackIconUrl', 'src',
|
|
],
|
|
|
|
// 3 Nested array fields containing assets
|
|
nested: ['galleryCards', 'carouselSlides', 'galleryInfoSpans'],
|
|
|
|
// 3 URL fields within nested items
|
|
nestedUrlFields: ['imageUrl', 'videoUrl', 'iconUrl'],
|
|
}
|
|
```
|
|
|
|
### 1.2 Configuration Consumers (10 files)
|
|
|
|
| File | Uses | Purpose |
|
|
|------|------|---------|
|
|
| `extractPageLinks.ts` | all, nested, nestedUrlFields | Explicit field extraction |
|
|
| `useNeighborGraph.ts` | all | Recursive traversal (depth 5) |
|
|
| `usePreloadOrchestrator.ts` | all | Asset initialization |
|
|
| `imagePreDecode.ts` | images, nested, nestedUrlFields | Image-only pre-decode |
|
|
| `StorageManager.ts` | storage config | Storage thresholds |
|
|
| `DownloadManager.ts` | queue settings | Download concurrency |
|
|
| `useStorageQuota.ts` | storage config | Quota warnings |
|
|
| `usePreloadProgress.ts` | autoRemove | Progress cleanup |
|
|
| `DownloadContext.tsx` | various | Download UI state |
|
|
|
|
---
|
|
|
|
## 2. COMPLETE PRELOADING FLOW (Step-by-Step)
|
|
|
|
### 2.1 Entry Point: RuntimePresentation Mount
|
|
|
|
```
|
|
RuntimePresentation.tsx mounts
|
|
↓
|
|
extractPageLinksAndElements(pages) [lib/extractPageLinks.ts:105-182]
|
|
├── Parse ui_schema_json for each page
|
|
├── extractAssetFields(element) → uses PRELOAD_CONFIG.assetFields
|
|
│ ├── Extract top-level fields from assetFields.all
|
|
│ └── Extract nested arrays (galleryCards, carouselSlides, galleryInfoSpans)
|
|
│ └── Within each item, extract nestedUrlFields (imageUrl, videoUrl, iconUrl)
|
|
├── Build pageLinks[] for navigation graph
|
|
└── Build preloadElements[] with content_json containing asset URLs
|
|
↓
|
|
usePreloadOrchestrator({ pages, pageLinks, elements, currentPageId })
|
|
```
|
|
|
|
### 2.2 Thread 1: Neighbor Graph Building
|
|
|
|
```
|
|
useNeighborGraph({ pages, pageLinks, elements, maxDepth: 1 })
|
|
↓
|
|
Build adjacencyList (Map<pageId, neighborPageIds[]>) [line 119-141]
|
|
├── Initialize all pages in map
|
|
└── Add edges from active pageLinks (from_pageId → to_pageId)
|
|
↓
|
|
getNeighbors(currentPageId, depth) [line 144-177]
|
|
├── BFS traversal from current page
|
|
├── Track visited pages to avoid cycles
|
|
└── Return PreloadNeighborInfo[] sorted by distance
|
|
↓
|
|
getAssetsForPages(pageIds) [line 180-224]
|
|
├── For each page: filter elements by pageId
|
|
├── extractAssetsFromContent(content_json) [line 56-106]
|
|
│ ├── Parse JSON content
|
|
│ ├── checkObject() recursive traversal (depth ≤ 5)
|
|
│ │ └── For each field in PRELOAD_CONFIG.assetFields.all:
|
|
│ │ └── If string value found, classify asset type and add to assets[]
|
|
│ └── Return PreloadAssetInfo[] with { url, pageId, assetType, priority }
|
|
└── Also extract transition videos from pageLinks
|
|
```
|
|
|
|
### 2.3 Thread 2: Priority Assignment
|
|
|
|
```
|
|
getPrioritizedAssets(currentPageId, maxDepth) [line 228-274]
|
|
↓
|
|
Current page assets:
|
|
priority = PRELOAD_CONFIG.priority.currentPage (1000)
|
|
+ PRELOAD_CONFIG.priority.assetType[type]
|
|
|
|
Asset type priorities:
|
|
├── transition: +150 (highest - needed on navigation click)
|
|
├── image: +100 (backgrounds load during transition)
|
|
├── audio: +50
|
|
└── video: +30
|
|
↓
|
|
Neighbor page assets:
|
|
basePriority = PRELOAD_CONFIG.priority.neighborBase (500) / distance
|
|
priority = basePriority + assetType priority
|
|
↓
|
|
Deduplicate by URL, keep highest priority
|
|
Sort descending by priority
|
|
```
|
|
|
|
### 2.4 Thread 3: URL Resolution (Presigned URLs)
|
|
|
|
```
|
|
usePreloadOrchestrator effect on currentPageId change [line 669-924]
|
|
↓
|
|
Collect storage paths needing presigning:
|
|
├── Current page: background_image_url, background_video_url, background_audio_url
|
|
├── Element assets from neighborGraph.getPrioritizedAssets()
|
|
└── Neighbor pages: background URLs
|
|
↓
|
|
queuePresignedUrls(storagePaths) [lib/assetUrl.ts]
|
|
├── POST /api/file/presign { urls: storagePaths[] }
|
|
├── Response: { [storageKey]: presignedUrl }
|
|
└── Cache presigned URLs (1-hour expiry)
|
|
↓
|
|
resolveUrl(storageKey, presignedUrls) [line 761-771]
|
|
├── If presignedUrls[storageKey] exists → use presigned URL
|
|
└── Fallback → resolveAssetPlaybackUrl (proxy URL)
|
|
```
|
|
|
|
### 2.5 Thread 4: Download Queue Processing
|
|
|
|
```
|
|
addToQueue(item: PreloadQueueItem) [line 494-530]
|
|
├── Skip if already in queue or preloaded
|
|
├── Insert in priority order (binary search insertion point)
|
|
└── Trigger processQueue()
|
|
↓
|
|
processQueue() [line 355-491]
|
|
├── Check: isOnline, queue not empty, not already processing
|
|
├── maxConcurrent = recommendedConcurrency (network-aware)
|
|
↓
|
|
While queue has items AND activeDownloads < maxConcurrent:
|
|
├── Shift item from queue
|
|
├── Skip if already preloaded (preloadedUrls.has(url))
|
|
├── Skip if already cached (isUrlCached → StorageManager.hasAsset)
|
|
│ └── If cached: createReadyBlobUrl() for instant display
|
|
↓
|
|
preloadWithProgress(url, jobId, assetId) [line 140-234]
|
|
├── Emit downloadEventBus.emitPreloadStart
|
|
├── fetch(url) with streaming progress
|
|
├── Collect chunks, emit progress events
|
|
├── Store in Cache API: caches.open(assets).put(url, response)
|
|
└── Emit downloadEventBus.emitPreloadComplete
|
|
↓
|
|
On success:
|
|
├── createReadyBlobUrl(url, storageKey)
|
|
│ ├── StorageManager.getAsset(url) → blob
|
|
│ ├── URL.createObjectURL(blob) → blobUrl
|
|
│ ├── If image: decodeImage(blobUrl) → pre-paint ready
|
|
│ └── readyBlobUrlsRef.set(url, blobUrl)
|
|
└── Also cache under storageKey for post-refresh lookup
|
|
↓
|
|
On failure (presigned URL CORS):
|
|
├── markPresignedUrlFailed(storageKey)
|
|
├── Build proxy URL: /api/file/download?privateUrl=...
|
|
└── Retry with proxy URL
|
|
```
|
|
|
|
### 2.6 Thread 5: Ready State & Instant Lookup
|
|
|
|
```
|
|
getReadyBlobUrl(url): string | null [line 582-584]
|
|
└── readyBlobUrlsRef.current.get(url) → O(1) Map lookup
|
|
|
|
Usage in RuntimePresentation:
|
|
const resolveUrlWithBlob = (url) => {
|
|
const blobUrl = preloadOrchestrator.getReadyBlobUrl(url);
|
|
return blobUrl || resolveAssetPlaybackUrl(url);
|
|
};
|
|
|
|
<RuntimeElement resolveUrl={resolveUrlWithBlob} ... />
|
|
```
|
|
|
|
---
|
|
|
|
## 3. ELEMENT TYPE → URL FIELD MAPPING (Complete Audit)
|
|
|
|
### 3.1 All Element Types (11 total)
|
|
|
|
| Element Type | URL Fields | Nested Arrays |
|
|
|--------------|------------|---------------|
|
|
| `navigation_next/prev` | iconUrl, transitionVideoUrl, reverseVideoUrl | - |
|
|
| `gallery` | iconUrl, galleryHeaderImageUrl, galleryCarouselPrevIconUrl, galleryCarouselNextIconUrl, galleryCarouselBackIconUrl | galleryCards[].imageUrl, galleryInfoSpans[].iconUrl |
|
|
| `carousel` | iconUrl, carouselPrevIconUrl, carouselNextIconUrl | carouselSlides[].imageUrl |
|
|
| `video_player` | iconUrl, mediaUrl | - |
|
|
| `audio_player` | iconUrl, mediaUrl | - |
|
|
| `tooltip` | iconUrl | - |
|
|
| `description` | iconUrl | - |
|
|
| `spot` | iconUrl | - |
|
|
| `logo` | iconUrl | - |
|
|
| `popup` | iconUrl | - |
|
|
|
|
### 3.2 Cross-Reference Verification
|
|
|
|
**constructor.ts URL Fields vs preload.config.ts assetFields.all:**
|
|
|
|
| Field in constructor.ts | In assetFields.all? | Status |
|
|
|-------------------------|---------------------|--------|
|
|
| iconUrl | Yes | OK |
|
|
| mediaUrl | Yes | OK |
|
|
| backgroundImageUrl | Yes | OK |
|
|
| videoUrl | Yes | OK |
|
|
| audioUrl | Yes | OK |
|
|
| transitionVideoUrl | Yes | OK |
|
|
| reverseVideoUrl | Yes | OK |
|
|
| galleryHeaderImageUrl | Yes | OK |
|
|
| carouselPrevIconUrl | Yes | OK |
|
|
| carouselNextIconUrl | Yes | OK |
|
|
| galleryCarouselPrevIconUrl | Yes | OK |
|
|
| galleryCarouselNextIconUrl | Yes | OK |
|
|
| galleryCarouselBackIconUrl | Yes | OK |
|
|
|
|
**Nested Arrays vs nested config:**
|
|
|
|
| Nested Array | In nested config? | Status |
|
|
|--------------|-------------------|--------|
|
|
| galleryCards | Yes | OK |
|
|
| carouselSlides | Yes | OK |
|
|
| galleryInfoSpans | Yes | OK |
|
|
|
|
**Nested URL Fields vs nestedUrlFields:**
|
|
|
|
| Field in Nested Items | In nestedUrlFields? | Status |
|
|
|-----------------------|---------------------|--------|
|
|
| galleryCards[].imageUrl | Yes | OK |
|
|
| carouselSlides[].imageUrl | Yes | OK |
|
|
| galleryInfoSpans[].iconUrl | Yes | OK |
|
|
|
|
---
|
|
|
|
## 4. OFFLINE MODE PROCESSING
|
|
|
|
### 4.1 Storage Architecture
|
|
|
|
```
|
|
StorageManager [lib/offline/StorageManager.ts]
|
|
├── Threshold: OFFLINE_CONFIG.storage.indexedDbMinSize (5MB)
|
|
├── Small files (< 5MB): Cache API
|
|
│ └── caches.open('vm-assets').put(url, response)
|
|
└── Large files (≥ 5MB): IndexedDB via Dexie
|
|
└── OfflineDbManager.storeAsset(asset)
|
|
|
|
hasAsset(url): Promise<boolean>
|
|
1. Check IndexedDB: OfflineDbManager.hasAssetByUrl(url)
|
|
2. Check Cache API: caches.open().match(url)
|
|
|
|
getAsset(url): Promise<Blob | null>
|
|
1. Try IndexedDB: OfflineDbManager.getAssetByUrl(url)
|
|
2. Try Cache API: caches.open().match(url).blob()
|
|
```
|
|
|
|
### 4.2 Service Worker Integration
|
|
|
|
```
|
|
sw.ts (Serwist-generated)
|
|
├── Precache: static assets (JS, CSS, fonts)
|
|
├── Runtime caching strategies:
|
|
│ ├── API requests: NetworkFirst
|
|
│ └── Assets: CacheFirst
|
|
└── Offline fallback handling
|
|
```
|
|
|
|
---
|
|
|
|
## 5. IMAGE PRE-DECODE FLOW
|
|
|
|
### 5.1 extractPageImageUrls
|
|
|
|
```
|
|
extractPageImageUrls(page) [lib/imagePreDecode.ts:110-175]
|
|
↓
|
|
Extract background_image_url
|
|
↓
|
|
Parse ui_schema_json → elements[]
|
|
↓
|
|
For each element:
|
|
├── Direct image fields (assetFields.images):
|
|
│ └── iconUrl, imageUrl, backgroundImageUrl, carousel*IconUrl, gallery*IconUrl, src
|
|
└── Nested arrays (assetFields.nested):
|
|
└── Filter nestedUrlFields to images only:
|
|
├── imageUrl (in images array)
|
|
├── iconUrl (in images array)
|
|
└── videoUrl (not in images - correctly excluded)
|
|
```
|
|
|
|
### 5.2 waitForPageImages
|
|
|
|
```
|
|
waitForPageImages(page, timeoutMs, cacheProvider)
|
|
↓
|
|
extractPageImageUrls(page) → imageUrls[]
|
|
↓
|
|
decodeImages(imageUrls, timeoutMs, cacheProvider)
|
|
├── For each URL: decodeImage()
|
|
│ ├── If cacheProvider: try getCachedBlobUrl() → blob URL (local, fast)
|
|
│ └── new Image().decode() or onload fallback
|
|
└── Promise.race([all decoded, timeout])
|
|
```
|
|
|
|
---
|
|
|
|
## 6. ROBUSTNESS VERIFICATION
|
|
|
|
### 6.1 All Asset Types Covered
|
|
|
|
| Asset Type | Extracted? | Preloaded? | Pre-decoded? | Cached? |
|
|
|------------|------------|------------|--------------|---------|
|
|
| Background images | Yes | Yes | Yes | Yes |
|
|
| Background videos | Yes | Yes | N/A | Yes |
|
|
| Background audio | Yes | Yes | N/A | Yes |
|
|
| Element icons | Yes | Yes | Yes | Yes |
|
|
| Transition videos | Yes | Yes | N/A | Yes |
|
|
| Reverse videos | Yes | Yes | N/A | Yes |
|
|
| Gallery cards images | Yes | Yes | Yes | Yes |
|
|
| Gallery info span icons | Yes | Yes | Yes | Yes |
|
|
| Gallery header image | Yes | Yes | Yes | Yes |
|
|
| Gallery carousel icons | Yes | Yes | Yes | Yes |
|
|
| Carousel slide images | Yes | Yes | Yes | Yes |
|
|
| Carousel nav icons | Yes | Yes | Yes | Yes |
|
|
| Media player URLs | Yes | Yes | N/A | Yes |
|
|
|
|
### 6.2 Extraction Algorithm Robustness
|
|
|
|
1. **extractPageLinks.ts** - Explicit field extraction
|
|
- Uses `assetFields.all` for top-level fields
|
|
- Uses `nested` + `nestedUrlFields` for nested arrays
|
|
- Handles all configured fields
|
|
|
|
2. **useNeighborGraph.ts** - Recursive traversal
|
|
- `checkObject()` recursively traverses to depth 5
|
|
- Matches any field from `assetFields.all` at any nesting level
|
|
- Handles deeply nested structures
|
|
|
|
3. **imagePreDecode.ts** - Image-only extraction
|
|
- Filters `nestedUrlFields` against `assetFields.images`
|
|
- Only pre-decodes actual images (not videos/audio)
|
|
- Correctly filters non-image URLs
|
|
|
|
### 6.3 Edge Cases Handled
|
|
|
|
| Edge Case | Handling |
|
|
|-----------|----------|
|
|
| Empty ui_schema_json | Returns empty arrays safely |
|
|
| Missing nested arrays | Skipped gracefully |
|
|
| Null/undefined values | Filtered out |
|
|
| Already cached assets | Skipped download, creates blob URL |
|
|
| Presigned URL CORS failure | Retries with proxy URL |
|
|
| Network offline | Skips preloading, uses cache |
|
|
| Large files (>=5MB) | Stored in IndexedDB instead of Cache API |
|
|
|
|
---
|
|
|
|
## 7. CONCLUSION
|
|
|
|
The neighbor preloading system is **robust and comprehensive**:
|
|
|
|
1. **Central Configuration** - All asset URL fields defined in `preload.config.ts`
|
|
2. **Consistent Extraction** - All consumers use the same config
|
|
3. **Complete Coverage** - All 11 element types and their URL fields are covered
|
|
4. **Nested Arrays** - `galleryCards`, `carouselSlides`, `galleryInfoSpans` all handled
|
|
5. **Dual Storage** - Cache API for small files, IndexedDB for large files
|
|
6. **Fallback Handling** - Presigned URLs with proxy fallback
|
|
7. **Network Awareness** - Adaptive concurrency based on connection
|
|
|
|
**No gaps identified** - all asset types are properly extracted, preloaded, and cached.
|
|
|
|
---
|
|
|
|
## Related Documentation
|
|
|
|
- [assets-preloading.md](../../documentation/assets-preloading.md) - E2E preloading feature documentation
|
|
- [runtime-presentation.md](./runtime-presentation.md) - Runtime presentation viewer
|
|
- [constructor-page-editor.md](./constructor-page-editor.md) - Constructor page editor
|
|
- [hooks-module.md](./hooks-module.md) - Custom hooks reference
|
|
- [lib-module.md](./lib-module.md) - Library utilities reference
|