1240 lines
47 KiB
Markdown
1240 lines
47 KiB
Markdown
# Asset Upload & Variants
|
||
|
||
Complete documentation for the Tour Builder Platform's asset management system including chunked uploads, storage providers, and variant tracking.
|
||
|
||
For persisted media metadata, the backend is the source of truth for video FPS.
|
||
Frontend upload probing is still used for immediate UX, but the asset
|
||
create/update path now probes the stored file with bundled `ffprobe-static` and
|
||
saves actual `frame_rate` metadata on the asset record.
|
||
|
||
## Overview
|
||
|
||
The platform implements a **robust asset management system** with:
|
||
- **Chunked uploads** - Large files uploaded in 5MB chunks with resumability
|
||
- **Multi-provider storage** - S3, Google Cloud Storage, or local filesystem
|
||
- **Asset variants** - Metadata tracking for different file formats/sizes
|
||
- **Media probing** - Automatic extraction of duration, dimensions
|
||
|
||
## Architecture
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────────────────────┐
|
||
│ Upload Flow │
|
||
│ │
|
||
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
|
||
│ │ Frontend │ │ Backend │ │ Storage │ │
|
||
│ │ Uploader │───▶│ Service │───▶│ Provider │ │
|
||
│ └─────────────┘ └─────────────┘ └─────────────┘ │
|
||
│ │ │ │ │
|
||
│ │ Chunks (5MB) │ Session Mgmt │ S3/GCloud/Local │
|
||
│ ▼ ▼ ▼ │
|
||
│ UploadService.js file.js service AWS SDK / GCloud SDK │
|
||
└─────────────────────────────────────────────────────────────────────────┘
|
||
|
||
┌─────────────────────────────────────────────────────────────────────────┐
|
||
│ Database Structure │
|
||
│ │
|
||
│ assets ─────────────────────────────────────────┐ │
|
||
│ │ │ │
|
||
│ ▼ ▼ │
|
||
│ asset_variants projects │
|
||
│ (variant_type, cdn_url, dimensions) (owner) │
|
||
└─────────────────────────────────────────────────────────────────────────┘
|
||
|
||
┌─────────────────────────────────────────────────────────────────────────┐
|
||
│ Download Flow (S3 with Presigned URLs) │
|
||
│ │
|
||
│ ┌─────────────┐ 1. Get presigned URLs ┌─────────────┐ │
|
||
│ │ Frontend │────────────────────────▶│ Backend │ │
|
||
│ │ Preloader │◀────────────────────────│ /presign │ │
|
||
│ └─────────────┘ 2. Return signed URLs └─────────────┘ │
|
||
│ │ │
|
||
│ │ 3. Direct download (fast) │
|
||
│ ▼ │
|
||
│ ┌─────────────┐ │
|
||
│ │ S3 │ No backend proxy - assets served directly from S3 │
|
||
│ └─────────────┘ │
|
||
└─────────────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
## Chunked Upload System
|
||
|
||
### Overview
|
||
|
||
Large files are uploaded in chunks to handle network interruptions and support resumable uploads.
|
||
|
||
**Key Parameters:**
|
||
| Parameter | Value | Description |
|
||
|-----------|-------|-------------|
|
||
| Chunk Size | 5 MB | `5 * 1024 * 1024` bytes |
|
||
| Max Retries | 3 | Per-chunk retry limit |
|
||
| Retry Backoff | 500ms × retry | Exponential backoff |
|
||
| Session TTL | 24 hours | Expired sessions auto-cleaned |
|
||
|
||
### Upload Session Flow
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────────────────────┐
|
||
│ Phase 1: Initialize Session │
|
||
├─────────────────────────────────────────────────────────────────────────┤
|
||
│ POST /file/upload-sessions/init │
|
||
│ Body: { folder, filename, totalChunks, size, contentType } │
|
||
│ Response: { sessionId, uploadedChunks: [], totalChunks } │
|
||
│ │
|
||
│ - Creates UUID session │
|
||
│ - Stores metadata (local: JSON file, S3: S3 object) │
|
||
│ - Triggers cleanup of expired sessions │
|
||
└─────────────────────────────────────────────────────────────────────────┘
|
||
↓
|
||
┌─────────────────────────────────────────────────────────────────────────┐
|
||
│ Phase 2: Upload Chunks │
|
||
├─────────────────────────────────────────────────────────────────────────┤
|
||
│ PUT /file/upload-sessions/{sessionId}/chunks/{chunkIndex} │
|
||
│ Headers: Content-Type: application/octet-stream │
|
||
│ Body: <binary chunk data> │
|
||
│ Response: { sessionId, chunkIndex, uploadedChunks, totalChunks } │
|
||
│ │
|
||
│ For each chunk (0 to totalChunks-1): │
|
||
│ - Slice file: start = chunkIndex * 5MB, end = min(size, start+5MB) │
|
||
│ - Send raw binary data │
|
||
│ - Retry up to 3 times with exponential backoff │
|
||
│ - Track progress via onProgress callback │
|
||
└─────────────────────────────────────────────────────────────────────────┘
|
||
↓
|
||
┌─────────────────────────────────────────────────────────────────────────┐
|
||
│ Phase 3: Finalize Upload │
|
||
├─────────────────────────────────────────────────────────────────────────┤
|
||
│ POST /file/upload-sessions/{sessionId}/finalize │
|
||
│ Response: { message, privateUrl, url } │
|
||
│ │
|
||
│ - Verify all chunks exist │
|
||
│ - Assemble chunks into final file │
|
||
│ - Validate assembled size matches declared size │
|
||
│ - Upload to final storage location │
|
||
│ - Cleanup session directory/objects │
|
||
│ - Return public and private URLs │
|
||
└─────────────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
### Session Status Check (Resumability)
|
||
|
||
```http
|
||
GET /file/upload-sessions/{sessionId}
|
||
```
|
||
|
||
**Response:**
|
||
```json
|
||
{
|
||
"sessionId": "uuid",
|
||
"totalChunks": 10,
|
||
"uploadedChunks": [0, 1, 2, 3],
|
||
"status": "active"
|
||
}
|
||
```
|
||
|
||
Use this to resume interrupted uploads by checking which chunks are already uploaded.
|
||
|
||
### Session Metadata Structure
|
||
|
||
```javascript
|
||
{
|
||
id: "uuid",
|
||
userId: "user-uuid", // Owner (enforced on all operations)
|
||
folder: "assets/project-id",
|
||
filename: "uuid.mp4",
|
||
totalChunks: 10,
|
||
size: 52428800, // 50MB
|
||
contentType: "video/mp4",
|
||
uploadedChunks: [0, 1, 2], // Completed chunk indices
|
||
status: "active",
|
||
createdAt: "2025-01-01T00:00:00Z",
|
||
updatedAt: "2025-01-01T00:05:00Z"
|
||
}
|
||
```
|
||
|
||
### Session Storage Locations
|
||
|
||
Session management is handled by `UploadSessionManager` class (`backend/src/services/file/UploadSessionManager.ts`).
|
||
|
||
**Local Storage:**
|
||
```
|
||
{uploadDir}/upload_sessions/{sessionId}/
|
||
├── meta.json # Session metadata
|
||
└── chunks/
|
||
├── 0.part # Chunk 0
|
||
├── 1.part # Chunk 1
|
||
└── ...
|
||
```
|
||
|
||
**S3 Storage:**
|
||
```
|
||
{prefix}/_upload_sessions/{sessionId}/
|
||
├── meta.json # Session metadata
|
||
└── chunks/
|
||
├── 0.part # Chunk 0
|
||
├── 1.part # Chunk 1
|
||
└── ...
|
||
```
|
||
|
||
### Chunk Assembly
|
||
|
||
During finalization, chunks are assembled in a temp directory:
|
||
|
||
```javascript
|
||
// Temp assembly location
|
||
const tempDir = path.join(config.uploadDir, '_temp_assembly');
|
||
const assembledPath = path.join(tempDir, `${sessionId}.bin`);
|
||
|
||
// For each chunk, append to assembled file
|
||
for (let chunkIndex = 0; chunkIndex < totalChunks; chunkIndex++) {
|
||
await streamAppendFile(assembledPath, chunkPath);
|
||
}
|
||
|
||
// Validate size
|
||
if (assembledStats.size !== session.size) {
|
||
throw new Error('Assembled file size mismatch');
|
||
}
|
||
```
|
||
|
||
## Storage Providers
|
||
|
||
### Modular File Service Architecture
|
||
|
||
The file service uses a **Strategy Pattern** with modular providers:
|
||
|
||
```
|
||
backend/src/services/
|
||
├── file.js # Unified interface (provider initialization, routing)
|
||
└── file/
|
||
├── index.js # Module exports
|
||
├── BaseStorageProvider.ts # Abstract base class
|
||
├── S3StorageProvider.ts # AWS S3 implementation
|
||
├── LocalStorageProvider.ts # Local filesystem implementation
|
||
└── UploadSessionManager.ts # Chunked upload session management
|
||
```
|
||
|
||
### External Storage Circuit Breaker
|
||
|
||
The unified file service protects S3/GCloud operations used by media processing
|
||
with an in-process circuit breaker. Local filesystem storage is not breaker
|
||
limited, which keeps local development and simple VM filesystem access direct.
|
||
The values are validated in `backend/src/utils/env-validation.ts` and exposed
|
||
through `config.resilience.fileStorage.breaker`; service code does not read
|
||
these environment variables directly.
|
||
|
||
Protected operations:
|
||
|
||
- `downloadToBuffer`
|
||
- `downloadToTempFile`
|
||
- `uploadBuffer`
|
||
- `deleteFile`
|
||
|
||
Configuration overrides:
|
||
|
||
| Variable | Default | Description |
|
||
|----------|---------|-------------|
|
||
| `FILE_STORAGE_BREAKER_FAILURE_THRESHOLD` | `5` | Consecutive failures before the breaker opens |
|
||
| `FILE_STORAGE_BREAKER_COOLDOWN_MS` | `30000` | Cooldown before a half-open probe is allowed |
|
||
| `FILE_STORAGE_BREAKER_SUCCESS_THRESHOLD` | `2` | Half-open successes required to close the breaker |
|
||
|
||
When the breaker is open, protected operations fail fast with a `503`-status
|
||
error instead of piling more requests onto the external storage provider.
|
||
|
||
### Provider Selection Logic
|
||
|
||
**File:** `backend/src/services/file.ts`
|
||
|
||
```javascript
|
||
const getFileStorageProvider = () => {
|
||
// 1. Explicit override from validated backend config
|
||
const provider = config.fileStorage.provider;
|
||
if (provider === 's3' || provider === 'gcloud' || provider === 'local') {
|
||
return provider;
|
||
}
|
||
|
||
// 2. Auto-detect S3 from credentials
|
||
const hasS3 = Boolean(
|
||
config.s3.bucket && config.s3.region &&
|
||
config.s3.accessKeyId && config.s3.secretAccessKey
|
||
);
|
||
if (hasS3) return 's3';
|
||
|
||
// 3. Auto-detect GCloud from credentials
|
||
const hasGCloud = Boolean(
|
||
config.gcloud.projectId &&
|
||
config.gcloud.clientEmail &&
|
||
config.gcloud.privateKey &&
|
||
config.gcloud.bucket &&
|
||
config.gcloud.hash
|
||
);
|
||
if (hasGCloud) return 'gcloud';
|
||
|
||
// 4. Default to local filesystem
|
||
return 'local';
|
||
};
|
||
```
|
||
|
||
`FILE_STORAGE_PROVIDER` is trimmed/lowercased and validated in
|
||
`backend/src/utils/env-validation.ts`; runtime service code reads
|
||
`config.fileStorage.provider` only.
|
||
|
||
### S3 Configuration
|
||
|
||
**Environment Variables:**
|
||
```bash
|
||
FILE_STORAGE_PROVIDER=s3
|
||
AWS_S3_BUCKET=your-bucket-name
|
||
AWS_S3_REGION=us-east-1 # Default: us-east-1
|
||
AWS_ACCESS_KEY_ID=AKIA...
|
||
AWS_SECRET_ACCESS_KEY=secret...
|
||
AWS_S3_PREFIX=optional-prefix # Default: afeefb9d49f5b7977577876b99532ac7
|
||
```
|
||
|
||
**Implementation:**
|
||
```javascript
|
||
const initS3 = () => {
|
||
const client = new S3Client({
|
||
region: config.s3.region,
|
||
credentials: {
|
||
accessKeyId: config.s3.accessKeyId,
|
||
secretAccessKey: config.s3.secretAccessKey,
|
||
},
|
||
});
|
||
|
||
return {
|
||
client,
|
||
bucket: config.s3.bucket,
|
||
region: config.s3.region,
|
||
prefix: config.s3.prefix,
|
||
};
|
||
};
|
||
```
|
||
|
||
**URL Format:**
|
||
```
|
||
https://{bucket}.s3.{region}.amazonaws.com/{prefix}/{folder}/{filename}
|
||
```
|
||
|
||
**SDK:** AWS SDK v3 (`@aws-sdk/client-s3`, `@aws-sdk/s3-request-presigner`). The TypeScript provider uses official AWS SDK command/config/exception types and shared project storage contracts from `backend/src/types/file.ts`.
|
||
|
||
### Google Cloud Storage Configuration
|
||
|
||
**Environment Variables:**
|
||
```bash
|
||
FILE_STORAGE_PROVIDER=gcloud
|
||
GC_PROJECT_ID=your-project-id
|
||
GC_CLIENT_EMAIL=service-account@project.iam.gserviceaccount.com
|
||
GC_PRIVATE_KEY="-----BEGIN PRIVATE KEY-----\n..."
|
||
```
|
||
|
||
**Config (config.ts):** Currently hardcoded values:
|
||
```javascript
|
||
gcloud: {
|
||
bucket: "fldemo-files",
|
||
hash: "afeefb9d49f5b7977577876b99532ac7"
|
||
}
|
||
```
|
||
|
||
**Note:** Unlike S3, GCloud bucket and hash are hardcoded in `config.ts`. To use different values, modify the config file directly.
|
||
|
||
**URL Format:**
|
||
```
|
||
https://storage.googleapis.com/{bucket}/{hash}/{folder}/{filename}
|
||
```
|
||
|
||
**SDK:** `@google-cloud/storage`
|
||
|
||
### Local Storage Configuration
|
||
|
||
**Default behavior** when no cloud credentials are configured.
|
||
|
||
**Upload Directory:** `os.tmpdir()` (OS temp directory)
|
||
|
||
**URL Format:**
|
||
```
|
||
/api/file/download?privateUrl={encodedPath}
|
||
```
|
||
|
||
Files are served via the backend download endpoint. Note: The download endpoint does **not** require authentication.
|
||
|
||
## Asset Database Models
|
||
|
||
### Assets Model
|
||
|
||
**File:** `backend/src/db/models/assets.js`
|
||
|
||
| Field | Type | Required | Description |
|
||
|-------|------|----------|-------------|
|
||
| id | UUID | Yes | Primary key |
|
||
| name | TEXT(255) | No | Display name |
|
||
| asset_type | ENUM | Yes | `image`, `video`, `audio`, `file` |
|
||
| type | ENUM | Yes | `icon`, `background_image`, `audio`, `video`, `transition`, `logo`, `favicon`, `document`, `general` (default) |
|
||
| cdn_url | TEXT | No | Public CDN/storage URL |
|
||
| storage_key | TEXT | No | Private storage path |
|
||
| mime_type | TEXT | No | MIME type (validated format) |
|
||
| size_mb | DECIMAL | No | File size in MB |
|
||
| width_px | INTEGER | No | Image/video width |
|
||
| height_px | INTEGER | No | Image/video height |
|
||
| duration_sec | DECIMAL | No | Video/audio duration |
|
||
| frame_rate | DECIMAL | No | Video FPS from backend `ffprobe` |
|
||
| checksum | TEXT | No | File checksum |
|
||
| is_public | BOOLEAN | Yes | Public visibility (default: false) |
|
||
| projectId | UUID | Yes | FK to projects (CASCADE) |
|
||
|
||
**Indexes:** `projectId`, `asset_type`, `type`, `is_public`, `deletedAt`
|
||
|
||
**Note on Environment Scoping:** Assets are scoped to projects, NOT environments. Unlike `tour_pages` which have `dev`, `stage`, and `production` versions, assets are shared across all environments within a project. When pages are published (dev → stage → production), the same asset URLs are used - only the page content (`ui_schema_json`) is copied between environments.
|
||
|
||
### Asset Variants Model
|
||
|
||
**File:** `backend/src/db/models/asset_variants.js`
|
||
|
||
| Field | Type | Required | Description |
|
||
|-------|------|----------|-------------|
|
||
| id | UUID | Yes | Primary key |
|
||
| variant_type | ENUM | No | `thumbnail`, `preview`, `webp`, `mp4_low`, `mp4_high`, `original` |
|
||
| cdn_url | TEXT(2048) | No | Variant CDN URL (validated URL format) |
|
||
| width_px | INTEGER | No | Variant width |
|
||
| height_px | INTEGER | No | Variant height |
|
||
| size_mb | DECIMAL | No | Variant file size |
|
||
| assetId | UUID | Yes | FK to assets (CASCADE) |
|
||
|
||
**Variant Types:**
|
||
| Type | Description | Use Case |
|
||
|------|-------------|----------|
|
||
| `thumbnail` | Small preview image | List views, galleries |
|
||
| `preview` | Medium quality preview | Quick previews |
|
||
| `webp` | WebP format | Web optimization |
|
||
| `mp4_low` | Low bitrate video | Mobile, slow connections |
|
||
| `mp4_high` | High bitrate video | Desktop, fast connections |
|
||
| `original` | Unmodified original | Full quality access |
|
||
| `reversed` | FFmpeg-reversed video | Back navigation transitions |
|
||
|
||
**Note:** Most variants are **metadata records only** - the system tracks variant information but does not auto-generate them. The exception is `reversed` variants, which are auto-generated by the server when a page with transition videos is saved.
|
||
|
||
**Critical:** The `assetId` field must be properly set when creating variant records. Without it, the variant becomes an orphaned record that cannot be found when looking up an asset's variants via the `asset_variants_asset` association. This is handled in `Asset_variantsDBApi.getFieldMapping()` which must include `assetId` in the field mapping.
|
||
|
||
### Reversed Video Variant
|
||
|
||
Reversed videos are a special variant type generated server-side for back navigation transitions. They use a different storage path pattern than other assets.
|
||
|
||
**Storage Patterns:**
|
||
| Asset Type | Storage Path Pattern | Example |
|
||
|------------|---------------------|---------|
|
||
| Primary assets | `assets/{projectId}/{uuid}.ext` | `assets/abc-123/def-456.mp4` |
|
||
| Reversed videos | `assets/{assetId}/reversed.mp4` | `assets/xyz-789/reversed.mp4` |
|
||
|
||
**Key Difference:** Reversed videos use the **asset ID** (not project ID) in their path. This is because they are tied to a specific asset (the transition video), not just the project.
|
||
|
||
**Generation Flow:**
|
||
1. User adds transition video to navigation element
|
||
2. User saves the tour page
|
||
3. Server detects `transitionVideoUrl` in navigation elements
|
||
4. Server generates reversed video using FFmpeg
|
||
5. Reversed video uploaded to `assets/{assetId}/reversed.mp4`
|
||
6. `reverseVideoUrl` populated in `ui_schema_json`
|
||
|
||
**Note:** Not all assets have reversed videos - only transition videos used in navigation elements.
|
||
|
||
## Frontend Upload Implementation
|
||
|
||
### useAssetUploader Hook
|
||
|
||
**File:** `frontend/src/components/Assets/useAssetUploader.ts`
|
||
|
||
```typescript
|
||
interface UseAssetUploaderOptions {
|
||
selectedProjectId: string;
|
||
onUploadComplete: () => void;
|
||
}
|
||
|
||
interface UseAssetUploaderReturn {
|
||
uploadingSections: string[];
|
||
uploadQueues: Record<string, UploadQueueItem[]>;
|
||
runBatchUpload: (section: AssetSection, files: File[]) => Promise<void>;
|
||
}
|
||
```
|
||
|
||
**Key Features:**
|
||
- **Batch upload** with queue management
|
||
- **2 concurrent uploads** maximum (`maxConcurrent = 2`)
|
||
- **Per-file progress** tracking
|
||
- **Abortable** via AbortController
|
||
- **Media probing** for video/audio duration and dimensions
|
||
|
||
**Status States:**
|
||
| Status | Description |
|
||
|--------|-------------|
|
||
| `queued` | File in queue, waiting to upload |
|
||
| `uploading` | Actively uploading chunks |
|
||
| `saving` | Finalizing upload and creating asset record |
|
||
| `success` | Upload and save completed |
|
||
| `error` | Upload failed (error message available) |
|
||
|
||
### UploadService Class
|
||
|
||
**File:** `frontend/src/components/Uploaders/UploadService.js`
|
||
|
||
**Single File Upload:**
|
||
```javascript
|
||
const result = await FileUploader.upload(path, file, schema);
|
||
// Returns: { id, name, sizeInBytes, privateUrl, publicUrl, new: true }
|
||
```
|
||
|
||
**Chunked Upload:**
|
||
```javascript
|
||
const result = await FileUploader.uploadChunked(
|
||
'assets/project-id', // path
|
||
file, // File object
|
||
{}, // schema (validation)
|
||
{
|
||
chunkSize: 5 * 1024 * 1024, // 5MB
|
||
maxRetries: 3,
|
||
signal: abortController.signal,
|
||
onProgress: (percent, { chunkIndex, totalChunks }) => {},
|
||
onStatus: (status, details) => {},
|
||
}
|
||
);
|
||
```
|
||
|
||
### Media Duration Probing
|
||
|
||
**File:** `frontend/src/lib/mediaDuration.ts`
|
||
|
||
After upload, video/audio files are probed for metadata:
|
||
|
||
```typescript
|
||
interface MediaDurationResult {
|
||
duration: number;
|
||
width?: number; // Video only
|
||
height?: number; // Video only
|
||
}
|
||
|
||
// Probe video
|
||
const result = await probeMediaDuration(file, 'video', 10000);
|
||
// Returns: { duration: 120.5, width: 1920, height: 1080 }
|
||
|
||
// Probe audio
|
||
const result = await probeMediaDuration(file, 'audio', 10000);
|
||
// Returns: { duration: 180.0 }
|
||
```
|
||
|
||
**Implementation:**
|
||
- Creates HTML5 `<video>` or `<audio>` element
|
||
- Sets `preload="metadata"` for efficiency
|
||
- Listens for `loadedmetadata` event
|
||
- Extracts `duration`, `videoWidth`, `videoHeight`
|
||
- 10-second timeout to prevent hanging
|
||
- Cleans up blob URLs after probing
|
||
|
||
### Complete Upload Flow
|
||
|
||
```
|
||
1. User selects files
|
||
└─ useAssetUploader.runBatchUpload(section, files)
|
||
|
||
2. Create queue items with status 'queued'
|
||
└─ Each file gets unique ID
|
||
|
||
3. For each file (max 2 concurrent):
|
||
├─ Status: 'uploading'
|
||
├─ Build validation schema from section.assetFormat
|
||
│ └─ { assetType: 'image' | 'video' | 'audio' }
|
||
├─ Call UploadService.uploadChunked()
|
||
│ ├─ Validate file against schema (MIME type + extension)
|
||
│ │ └─ Throws Error if validation fails
|
||
│ ├─ POST /api/file/upload-sessions/init
|
||
│ ├─ For each chunk:
|
||
│ │ └─ PUT /api/file/upload-sessions/{sessionId}/chunks/{index}
|
||
│ ├─ Status: 'saving' (triggered by onStatus('finalizing'))
|
||
│ └─ POST /api/file/upload-sessions/{sessionId}/finalize
|
||
│
|
||
├─ Probe media duration (video/audio only)
|
||
│ └─ probeMediaDuration(file, 'video'/'audio') // Uses local File object
|
||
│
|
||
├─ POST /api/assets
|
||
│ ├─ Create asset record with metadata
|
||
│ └─ Backend validates asset_type matches mime_type
|
||
│
|
||
└─ Status: 'success'
|
||
|
||
4. onUploadComplete() callback
|
||
└─ Refresh asset list
|
||
```
|
||
|
||
The Assets page refreshes the full selected project's asset list after uploads
|
||
and deletions. It does not apply a frontend `limit` to the `/assets` query;
|
||
assets are filtered into UI sections client-side by `asset_type`, `type`, and
|
||
legacy name tags.
|
||
|
||
**Validation Layers:**
|
||
1. **Frontend (immediate):** UploadService validates file type before upload starts
|
||
2. **Backend (on asset creation):** AssetsService validates MIME type consistency
|
||
|
||
**Note:** The 'saving' status is set when the finalize step begins, before the actual finalize request completes.
|
||
|
||
## API Endpoints
|
||
|
||
**Note:** All file endpoints are mounted at `/api/file/...`. The frontend axios client has a baseURL of `/api`, so frontend code uses `/file/...` paths which resolve to `/api/file/...`.
|
||
|
||
### File Upload Endpoints
|
||
|
||
```http
|
||
# Initialize chunked upload session
|
||
POST /api/file/upload-sessions/init
|
||
Content-Type: application/json
|
||
Authorization: Bearer {token}
|
||
|
||
{
|
||
"folder": "assets/project-uuid",
|
||
"filename": "video.mp4",
|
||
"totalChunks": 10,
|
||
"size": 52428800,
|
||
"contentType": "video/mp4"
|
||
}
|
||
|
||
# Response
|
||
{
|
||
"sessionId": "uuid",
|
||
"uploadedChunks": [],
|
||
"totalChunks": 10
|
||
}
|
||
```
|
||
|
||
```http
|
||
# Upload single chunk
|
||
PUT /api/file/upload-sessions/{sessionId}/chunks/{chunkIndex}
|
||
Content-Type: application/octet-stream
|
||
Authorization: Bearer {token}
|
||
|
||
<binary chunk data>
|
||
|
||
# Response (uploadedChunks is COUNT, not array)
|
||
{
|
||
"sessionId": "uuid",
|
||
"chunkIndex": 0,
|
||
"uploadedChunks": 1,
|
||
"totalChunks": 10
|
||
}
|
||
```
|
||
|
||
```http
|
||
# Check session status
|
||
GET /api/file/upload-sessions/{sessionId}
|
||
Authorization: Bearer {token}
|
||
|
||
# Response (uploadedChunks is ARRAY of chunk indices)
|
||
{
|
||
"sessionId": "uuid",
|
||
"totalChunks": 10,
|
||
"uploadedChunks": [0, 1, 2, 3],
|
||
"status": "active"
|
||
}
|
||
```
|
||
|
||
```http
|
||
# Finalize upload
|
||
POST /api/file/upload-sessions/{sessionId}/finalize
|
||
Authorization: Bearer {token}
|
||
|
||
# Response
|
||
{
|
||
"message": "Uploaded the file successfully: assets/project-uuid/uuid.mp4",
|
||
"privateUrl": "assets/project-uuid/uuid.mp4",
|
||
"url": "https://bucket.s3.region.amazonaws.com/prefix/assets/project-uuid/uuid.mp4"
|
||
}
|
||
```
|
||
|
||
```http
|
||
# Download file (local storage) - NO authentication required
|
||
GET /api/file/download?privateUrl={encodedPath}
|
||
```
|
||
|
||
**Note:** The download endpoint does not require JWT authentication, allowing public access to files. Access control should be implemented at the application level if needed.
|
||
|
||
### Presigned URL Endpoint (S3 Direct Download)
|
||
|
||
For S3 storage, the platform supports presigned URLs for direct browser-to-S3 downloads, bypassing the backend proxy for improved performance.
|
||
|
||
```http
|
||
# Generate presigned URLs for batch of assets
|
||
POST /api/file/presign
|
||
Content-Type: application/json
|
||
|
||
{
|
||
"urls": [
|
||
"assets/project-uuid/video1.mp4",
|
||
"assets/project-uuid/image1.jpg",
|
||
"assets/project-uuid/video2.mp4"
|
||
]
|
||
}
|
||
|
||
# Response
|
||
{
|
||
"presignedUrls": {
|
||
"assets/project-uuid/video1.mp4": "https://bucket.s3.region.amazonaws.com/prefix/assets/project-uuid/video1.mp4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=...&X-Amz-Date=...&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=...",
|
||
"assets/project-uuid/image1.jpg": "https://bucket.s3.region.amazonaws.com/prefix/assets/project-uuid/image1.jpg?...",
|
||
"assets/project-uuid/video2.mp4": "https://bucket.s3.region.amazonaws.com/prefix/assets/project-uuid/video2.mp4?..."
|
||
}
|
||
}
|
||
```
|
||
|
||
**Key Details:**
|
||
- **Max URLs per request:** 50
|
||
- **URL expiry:** 1 hour (3600 seconds)
|
||
- **Fallback:** For non-S3 providers (GCloud, local), returns backend proxy URLs
|
||
- **No authentication required:** Similar to download endpoint
|
||
|
||
**Benefits:**
|
||
| Aspect | Backend Proxy | Presigned URLs |
|
||
|--------|---------------|----------------|
|
||
| Download speed | Slower (proxied) | Direct S3 access |
|
||
| Backend load | Proxies all bytes | Only generates URLs |
|
||
| Scalability | Backend bottleneck | S3 handles traffic |
|
||
|
||
**Frontend Integration:**
|
||
|
||
The frontend implements a multi-layer caching strategy:
|
||
|
||
1. **Presigned URL Cache** (`lib/assetUrl.ts`) - In-memory cache with 1-hour TTL, refreshing 5 minutes before expiry
|
||
2. **Cache API Storage** (`StorageManager`) - Browser Cache API for persistent storage of downloaded assets
|
||
3. **Ready Blob URL Map** (`usePreloadOrchestrator`) - Pre-decoded blob URLs for instant display
|
||
|
||
**Presigned URL Batching:**
|
||
|
||
The `assetUrl.ts` module implements automatic batching for presigned URL requests:
|
||
- `queuePresignedUrl(key)` - Queue single key for batch
|
||
- `queuePresignedUrls(keys)` - Queue multiple keys at once
|
||
- `flushPresignedUrlQueue()` - Flush pending batch immediately
|
||
- `getPresignedUrl(key)` - Synchronous cache lookup
|
||
- `clearPresignedUrlCache()` - Clear all cached URLs
|
||
- Batch delay: 10ms (collects requests before sending)
|
||
- Automatic deduplication of keys
|
||
- Returns cached URLs immediately if valid
|
||
|
||
**Presigned URL Verification:**
|
||
|
||
Before using presigned URLs for playback, the system verifies they work:
|
||
- `presignedUrlsVerified` flag - Tracks if presigned URLs are confirmed working
|
||
- `markPresignedUrlsVerified()` - Called by preloader after successful S3 fetch
|
||
- `resolveAssetPlaybackUrl()` returns proxy URLs until presigned URLs are verified
|
||
- This prevents CORS errors from breaking playback on first load
|
||
|
||
**Utility Functions:**
|
||
|
||
| Function | Purpose |
|
||
|----------|---------|
|
||
| `isRelativeStoragePath(url)` | Check if path is relative (not http/blob/data URL) |
|
||
| `extractStoragePath(url)` | Extract relative path from full S3 URL |
|
||
| `resolveAssetPlaybackUrl(value)` | Resolve any asset path to playable URL |
|
||
| `arePresignedUrlsDisabled()` | Check if presigned URLs are disabled |
|
||
| `markPresignedUrlFailed(key)` | Mark presigned URL failed, disable globally |
|
||
|
||
**CORS Failure Detection:**
|
||
|
||
The module includes automatic CORS failure detection via Axios interceptor:
|
||
- `setupPresignedUrlInterceptor()` - Sets up Axios response interceptor
|
||
- Detects presigned S3 URL failures (likely CORS issues)
|
||
- Automatically disables presigned URLs and falls back to proxy URLs
|
||
- `presignedUrlsDisabled` flag prevents further presigned URL attempts
|
||
|
||
**Complete Preload-to-Display Flow:**
|
||
```
|
||
┌─────────────────────────────────────────────────────────────────────────┐
|
||
│ Asset Preloading Pipeline │
|
||
├─────────────────────────────────────────────────────────────────────────┤
|
||
│ │
|
||
│ 1. Batch fetch presigned URLs │
|
||
│ POST /api/file/presign { urls: [...] } │
|
||
│ ↓ │
|
||
│ 2. Download assets from S3 (parallel, with progress) │
|
||
│ fetch(presignedUrl) → Response blob │
|
||
│ ↓ │
|
||
│ 3. Store in Cache API (persistent) │
|
||
│ caches.open('assets').put(originalUrl, blob) │
|
||
│ ↓ │
|
||
│ 4. Create blob URL (local reference) │
|
||
│ URL.createObjectURL(blob) → blob:... │
|
||
│ ↓ │
|
||
│ 5. Pre-decode if image (paint-ready) │
|
||
│ new Image().decode() on blob URL │
|
||
│ ↓ │
|
||
│ 6. Store in readyBlobUrls Map │
|
||
│ readyBlobUrlsRef.current.set(originalUrl, blobUrl) │
|
||
│ │
|
||
└─────────────────────────────────────────────────────────────────────────┘
|
||
|
||
Page Navigation (instant):
|
||
getReadyBlobUrl(originalUrl) → O(1) Map lookup
|
||
return decodedBlobUrl → Already paint-ready
|
||
set as background → No delay, no flash
|
||
```
|
||
|
||
**Key Methods:**
|
||
| Method | Purpose |
|
||
|--------|---------|
|
||
| `getReadyBlobUrl(url)` | O(1) instant lookup for pre-decoded blob URL (accepts storage key or resolved URL) |
|
||
| `getCachedBlobUrl(url)` | Async fallback - creates blob URL from Cache API (accepts storage key or resolved URL) |
|
||
| `isUrlPreloaded(url)` | Check if asset is ready for instant display |
|
||
|
||
**Storage Key Mapping:**
|
||
|
||
Assets are stored under multiple keys for reliable cache lookup:
|
||
|
||
| Key Type | Example | Purpose |
|
||
|----------|---------|---------|
|
||
| Download URL | `https://s3...?X-Amz-Signature=ABC` | Original presigned URL used for download |
|
||
| Storage Key | `assets/project-123/video.mp4` | Canonical path (most reliable for lookups) |
|
||
| Proxy URL | `/api/file/download?privateUrl=...` | Fallback compatibility |
|
||
|
||
**Why storage keys matter:**
|
||
- Presigned URL signatures change on each resolution (X-Amz-Signature differs)
|
||
- Storage keys are canonical and never change
|
||
- Lookups prioritize storage key for reliable cache hits across URL regeneration
|
||
- After page refresh, Cache API lookup by storage key still works
|
||
|
||
```
|
||
Lookup Priority:
|
||
1. getReadyBlobUrl(storageKey) → instant O(1) Map lookup (same session)
|
||
2. getCachedBlobUrl(storageKey) → Cache API lookup (~5ms, post-refresh)
|
||
3. getReadyBlobUrl(resolvedUrl) → fallback by resolved URL
|
||
4. getCachedBlobUrl(resolvedUrl) → fallback by resolved URL
|
||
5. Network fetch → last resort
|
||
```
|
||
|
||
**Memory Management:** Blob URLs are revoked on unmount via `clearReadyBlobUrls()` to prevent memory leaks.
|
||
|
||
**S3 CORS Configuration Required:**
|
||
|
||
For presigned URLs to work, the S3 bucket must have CORS configured:
|
||
|
||
```json
|
||
[
|
||
{
|
||
"AllowedHeaders": ["*"],
|
||
"AllowedMethods": ["GET", "HEAD"],
|
||
"AllowedOrigins": [
|
||
"https://*.dev.flatlogic.app",
|
||
"https://*.flatlogic.app",
|
||
"http://localhost:3000"
|
||
],
|
||
"ExposeHeaders": ["Content-Length", "Content-Type", "ETag"],
|
||
"MaxAgeSeconds": 3600
|
||
}
|
||
]
|
||
```
|
||
|
||
**Note:** S3 CORS requires exact origin matching or wildcard patterns (`*` only at the beginning). Add your specific subdomain or use wildcards like `https://*.dev.flatlogic.app` for multiple environments.
|
||
|
||
**Apply via AWS CLI:**
|
||
```bash
|
||
aws s3api put-bucket-cors --bucket YOUR_BUCKET_NAME --cors-configuration file://cors.json
|
||
```
|
||
|
||
### Asset CRUD Endpoints
|
||
|
||
```http
|
||
# Create asset record
|
||
POST /api/assets
|
||
Authorization: Bearer {token}
|
||
|
||
{
|
||
"data": {
|
||
"project": "project-uuid",
|
||
"name": "background.mp4",
|
||
"asset_type": "video",
|
||
"type": "background_image",
|
||
"cdn_url": "https://...",
|
||
"storage_key": "assets/project-uuid/uuid.mp4",
|
||
"mime_type": "video/mp4",
|
||
"size_mb": 50.0,
|
||
"duration_sec": 120.5,
|
||
"width_px": 1920,
|
||
"height_px": 1080,
|
||
"frame_rate": 23.976,
|
||
"is_public": false
|
||
}
|
||
}
|
||
```
|
||
|
||
Asset creation accepts the project relation as `data.project` from frontend
|
||
forms. `AssetsDBApi.getFieldMapping()` also maps this to `projectId`, and the
|
||
generic DB API invokes the Sequelize `setProject` association setter with the
|
||
created asset record as its method context. This is required because Sequelize
|
||
belongsTo mixins call `this.set(...)` internally; detached setter calls fail
|
||
before the project foreign key can be persisted.
|
||
|
||
```http
|
||
# List assets
|
||
GET /api/assets?project={projectId}&asset_type=video
|
||
Authorization: Bearer {token}
|
||
```
|
||
|
||
```http
|
||
# Create asset variant
|
||
POST /api/asset_variants
|
||
Authorization: Bearer {token}
|
||
|
||
{
|
||
"data": {
|
||
"asset": "asset-uuid",
|
||
"variant_type": "mp4_low",
|
||
"cdn_url": "https://...",
|
||
"width_px": 854,
|
||
"height_px": 480,
|
||
"size_mb": 10.0
|
||
}
|
||
}
|
||
```
|
||
|
||
## Security & Validation
|
||
|
||
### Backend Validation
|
||
|
||
**Folder Sanitization:**
|
||
```javascript
|
||
const sanitizeFolder = (folder) => {
|
||
const value = String(folder || '').trim().replace(/^\/+|\/+$/g, '');
|
||
if (!value || value.includes('..')) return null; // Prevent path traversal
|
||
return value;
|
||
};
|
||
```
|
||
|
||
**Filename Sanitization:**
|
||
```javascript
|
||
const sanitizeFilename = (filename) => {
|
||
const value = path.basename(String(filename || '').trim());
|
||
if (!value || value === '.' || value === '..') return null;
|
||
return value;
|
||
};
|
||
```
|
||
|
||
**Session Ownership:**
|
||
- Session tied to `req.currentUser.id`
|
||
- All session operations verify `session.userId === req.currentUser.id`
|
||
- Unauthorized access returns HTTP 403
|
||
|
||
**Chunk Validation:**
|
||
- Chunk index bounds checking (`chunkIndex < totalChunks`)
|
||
- File size consistency (`assembledSize === declaredSize`)
|
||
|
||
### Frontend Validation
|
||
|
||
**File:** `frontend/src/components/Uploaders/UploadService.js`
|
||
|
||
The UploadService validates files before upload using both MIME type prefixes and file extensions as fallback.
|
||
|
||
```javascript
|
||
// Valid MIME type prefixes and extensions for each asset type
|
||
const VALID_MIME_TYPES = {
|
||
image: {
|
||
prefixes: ['image/'],
|
||
extensions: ['jpg', 'jpeg', 'png', 'gif', 'webp', 'svg', 'bmp', 'ico'],
|
||
},
|
||
video: {
|
||
prefixes: ['video/'],
|
||
extensions: ['mp4', 'webm', 'mov', 'avi', 'mkv', 'm4v', 'ogv'],
|
||
},
|
||
audio: {
|
||
prefixes: ['audio/'],
|
||
extensions: ['mp3', 'wav', 'ogg', 'aac', 'm4a', 'flac', 'weba'],
|
||
},
|
||
};
|
||
|
||
static validate(file, schema) {
|
||
// Asset type validation (unified approach)
|
||
if (schema.assetType) {
|
||
const result = validateAssetType(file, schema.assetType);
|
||
if (!result.valid) {
|
||
throw new Error(result.error);
|
||
}
|
||
}
|
||
|
||
// Legacy image validation
|
||
if (schema.image) {
|
||
const result = validateAssetType(file, 'image');
|
||
if (!result.valid) {
|
||
throw new Error('You must upload an image');
|
||
}
|
||
}
|
||
|
||
// Legacy video validation
|
||
if (schema.video) {
|
||
const result = validateAssetType(file, 'video');
|
||
if (!result.valid) {
|
||
throw new Error('You must upload a video');
|
||
}
|
||
}
|
||
|
||
// Legacy audio validation
|
||
if (schema.audio) {
|
||
const result = validateAssetType(file, 'audio');
|
||
if (!result.valid) {
|
||
throw new Error('You must upload an audio file');
|
||
}
|
||
}
|
||
|
||
// Size limit
|
||
if (schema.size && file.size > schema.size) {
|
||
const maxSizeMB = (schema.size / (1024 * 1024)).toFixed(1);
|
||
const fileSizeMB = (file.size / (1024 * 1024)).toFixed(1);
|
||
throw new Error(`File is too big. Maximum ${maxSizeMB}MB, got ${fileSizeMB}MB`);
|
||
}
|
||
|
||
// Extension whitelist
|
||
if (schema.formats && !schema.formats.includes(extension)) {
|
||
throw new Error(`Invalid format. Allowed: ${schema.formats.join(', ')}`);
|
||
}
|
||
}
|
||
```
|
||
|
||
**Validation Schemas:**
|
||
|
||
| Schema Property | Description |
|
||
|-----------------|-------------|
|
||
| `assetType` | Asset type: `'image'`, `'video'`, or `'audio'` (unified validation) |
|
||
| `image` | Legacy: Must be an image file |
|
||
| `video` | Legacy: Must be a video file |
|
||
| `audio` | Legacy: Must be an audio file |
|
||
| `size` | Maximum file size in bytes |
|
||
| `formats` | Allowed file extensions array |
|
||
|
||
**Fallback Logic:**
|
||
Validation checks MIME type prefix first, then falls back to file extension. This handles cases where browsers don't report MIME type correctly.
|
||
|
||
### Backend Asset Service Validation
|
||
|
||
**File:** `backend/src/services/assets.ts`
|
||
|
||
The AssetsService validates that `asset_type` and `mime_type` are consistent when creating or updating assets.
|
||
|
||
```typescript
|
||
const VALID_MIME_PATTERNS = {
|
||
image: {
|
||
prefixes: ['image/'],
|
||
description: 'image (jpeg, png, gif, webp, svg, etc.)',
|
||
},
|
||
video: {
|
||
prefixes: ['video/'],
|
||
description: 'video (mp4, webm, mov, etc.)',
|
||
},
|
||
audio: {
|
||
prefixes: ['audio/'],
|
||
description: 'audio (mp3, wav, ogg, etc.)',
|
||
},
|
||
};
|
||
|
||
// Throws ValidationError if mime_type doesn't match asset_type
|
||
const validation = validateAssetMimeType(assetType, mimeType);
|
||
if (!validation.valid) {
|
||
throw new ValidationError(validation.error);
|
||
}
|
||
```
|
||
|
||
**Validation Rules:**
|
||
- On `create()`: Always validates `asset_type` and `mime_type` match
|
||
- On `update()`: Only validates if both `asset_type` AND `mime_type` are provided
|
||
- Skips validation if `asset_type` is not `image`, `video`, or `audio`
|
||
- Skips validation if `mime_type` is missing (browser may not send it)
|
||
|
||
**Error Response:**
|
||
```json
|
||
{
|
||
"code": 400,
|
||
"message": "Invalid file type for video. Expected video (mp4, webm, mov, etc.), got \"image/jpeg\""
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
### MIME Type Validation (DB Level)
|
||
|
||
```javascript
|
||
mime_type: {
|
||
type: DataTypes.TEXT,
|
||
validate: {
|
||
is: {
|
||
args: /^[a-z0-9]+\/[a-z0-9.+-]+$/i,
|
||
msg: 'Invalid MIME type format'
|
||
},
|
||
},
|
||
}
|
||
```
|
||
|
||
## Variant Selection for PWA
|
||
|
||
**File:** `backend/src/services/pwa_manifest.js`
|
||
|
||
Variants are selected based on device type for PWA offline caching:
|
||
|
||
```javascript
|
||
static selectVariants(variants, deviceType) {
|
||
const priority = deviceType === 'mobile'
|
||
? ['mp4_low', 'webp', 'thumbnail', 'preview', 'mp4_high', 'original']
|
||
: ['mp4_high', 'webp', 'preview', 'mp4_low', 'thumbnail', 'original'];
|
||
|
||
// Select first matching variant by priority
|
||
for (const type of priority) {
|
||
const variant = variants.find(v => v.variant_type === type);
|
||
if (variant) return [variant];
|
||
}
|
||
return [];
|
||
}
|
||
```
|
||
|
||
**Device Priorities:**
|
||
| Device | Priority Order |
|
||
|--------|----------------|
|
||
| Mobile | `mp4_low` > `webp` > `thumbnail` > `preview` > `mp4_high` > `original` |
|
||
| Desktop | `mp4_high` > `webp` > `preview` > `mp4_low` > `thumbnail` > `original` |
|
||
|
||
## Session Cleanup
|
||
|
||
### Automatic Cleanup
|
||
|
||
Expired sessions (>24 hours old) are automatically cleaned up:
|
||
|
||
**Local Storage:**
|
||
```javascript
|
||
// Called synchronously on initUploadSession
|
||
cleanupExpiredUploadSessions();
|
||
```
|
||
|
||
**S3 Storage:**
|
||
```javascript
|
||
// Called asynchronously (non-blocking)
|
||
cleanupExpiredS3UploadSessions().catch(err =>
|
||
console.error('S3 session cleanup failed', err)
|
||
);
|
||
```
|
||
|
||
### Manual Cleanup
|
||
|
||
For stuck sessions, directly remove:
|
||
|
||
**Local:**
|
||
```bash
|
||
rm -rf {uploadDir}/upload_sessions/{sessionId}/
|
||
```
|
||
|
||
**S3:**
|
||
```bash
|
||
aws s3 rm --recursive s3://{bucket}/{prefix}/_upload_sessions/{sessionId}/
|
||
```
|
||
|
||
## File Reference
|
||
|
||
### Backend Files
|
||
|
||
| File | Purpose |
|
||
|------|---------|
|
||
| `backend/src/services/file.ts` | Unified file storage service with provider initialization |
|
||
| `backend/src/services/file/index.ts` | Module exports for file service |
|
||
| `backend/src/services/file/BaseStorageProvider.ts` | Abstract base class for storage providers |
|
||
| `backend/src/services/file/S3StorageProvider.ts` | AWS S3 storage implementation |
|
||
| `backend/src/services/file/LocalStorageProvider.ts` | Local filesystem storage implementation |
|
||
| `backend/src/services/file/UploadSessionManager.ts` | Chunked upload session management |
|
||
| `backend/src/routes/file.js` | File API endpoints |
|
||
| `backend/src/db/models/assets.js` | Assets model |
|
||
| `backend/src/db/models/asset_variants.js` | Asset variants model |
|
||
| `backend/src/db/api/assets.ts` | Assets DB operations |
|
||
| `backend/src/services/assets.ts` | Assets service validation and media metadata enrichment |
|
||
| `backend/src/db/api/asset_variants.ts` | Variants DB operations |
|
||
| `backend/src/services/pwa_manifest.js` | PWA manifest with variant selection |
|
||
| `backend/src/config.ts` | Storage provider configuration |
|
||
|
||
### Frontend Files
|
||
|
||
| File | Purpose |
|
||
|------|---------|
|
||
| `frontend/src/components/Assets/useAssetUploader.ts` | Upload hook |
|
||
| `frontend/src/components/Uploaders/UploadService.js` | Upload service class |
|
||
| `frontend/src/lib/mediaDuration.ts` | Media metadata probing |
|
||
| `frontend/src/lib/assetUrl.ts` | Asset URL resolution with presigned URL cache |
|
||
| `frontend/src/lib/offline/StorageManager.ts` | Cache API abstraction for asset storage (small files) and IndexedDB (large files ≥5MB) |
|
||
| `frontend/src/hooks/usePreloadOrchestrator.ts` | Asset preloading with presigned URL batch fetching and blob URL management |
|
||
| `frontend/src/hooks/usePageSwitch.ts` | Page navigation using preloaded blob URLs for instant transitions |
|
||
|
||
## Troubleshooting
|
||
|
||
### Upload Stuck at "Finalizing"
|
||
|
||
1. Check backend logs for assembly errors
|
||
2. Verify all chunks were uploaded: `GET /file/upload-sessions/{sessionId}`
|
||
3. Check disk space for temp assembly directory
|
||
4. Verify storage provider credentials
|
||
|
||
### "Upload Session Not Found" Error
|
||
|
||
1. Session expired (>24 hours) - restart upload
|
||
2. Session cleaned up - check for concurrent cleanup
|
||
3. Wrong sessionId - verify client-side tracking
|
||
|
||
### "Assembled File Size Mismatch" Error
|
||
|
||
1. Network corruption during chunk upload
|
||
2. Client reported wrong file size
|
||
3. Retry full upload from beginning
|
||
|
||
### S3 Upload Errors
|
||
|
||
1. Verify AWS credentials: `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`
|
||
2. Check bucket permissions (PutObject, GetObject, DeleteObject)
|
||
3. Verify region matches bucket location
|
||
4. Check S3 bucket CORS configuration for browser uploads
|
||
|
||
### Variant Not Showing in PWA
|
||
|
||
1. Verify variant record exists in `asset_variants` table
|
||
2. Check `cdn_url` is accessible
|
||
3. Verify `variant_type` matches expected values
|
||
4. Check PWA manifest generation includes asset
|
||
|
||
## Project Cloning and Asset Copying
|
||
|
||
When a project is cloned, all assets (including variants and reversed videos) are copied to give the cloned project independent files.
|
||
|
||
### Optimized S3 Copy
|
||
|
||
The cloning process uses S3's native `CopyObjectCommand` for server-side file copying, which is significantly faster than downloading and re-uploading files.
|
||
|
||
**Performance Comparison:**
|
||
| Metric | Download/Upload | S3 CopyObject |
|
||
|--------|-----------------|---------------|
|
||
| 100 MB file | ~3,000 ms | ~200 ms |
|
||
| Memory usage | File size in RAM | ~5 MB constant |
|
||
| 569 assets (parallel) | Timeouts likely | ~35 seconds |
|
||
|
||
### Clone Process for Assets
|
||
|
||
```
|
||
1. Collect all copy operations:
|
||
- Primary assets: assets/{oldProjectId}/{uuid}.ext → assets/{newProjectId}/{newUuid}.ext
|
||
- Non-reversed variants: Same pattern as primary assets
|
||
|
||
2. Execute parallel S3 copy (10 concurrent)
|
||
|
||
3. Create asset/variant DB records with new storage paths
|
||
|
||
4. Build asset ID mapping: oldAssetId → newAssetId
|
||
|
||
5. Copy reversed videos using asset ID mapping:
|
||
- assets/{oldAssetId}/reversed.mp4 → assets/{newAssetId}/reversed.mp4
|
||
|
||
6. Transform ui_schema_json paths in tour_pages
|
||
```
|
||
|
||
### Why Reversed Videos Need Special Handling
|
||
|
||
Reversed videos use asset-ID-based paths (`assets/{assetId}/reversed.mp4`), not project-ID-based paths. This requires:
|
||
|
||
1. **Asset ID Mapping:** Track which old asset ID maps to which new asset ID
|
||
2. **Separate Copy Phase:** Copy reversed videos after creating asset records (so we have the new asset IDs)
|
||
3. **Path Transformation:** Update `reverseVideoUrl` references in `ui_schema_json`
|
||
|
||
### Error Handling
|
||
|
||
- **Failed copies:** Fall back to original storage path (cloned project shares asset with source)
|
||
- **Missing reversed videos:** Expected for most assets (only navigation transitions have them)
|
||
- **Transaction rollback:** On DB errors; orphaned S3 files are acceptable (can be cleaned later)
|