diff --git a/AGENTS.md b/AGENTS.md index 6f9622b..87b6838 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -66,6 +66,8 @@ These rules are required for new code and for touched code when practical. - Do not create new entity CRUD Redux slices or new Redux thunks for server reads/mutations. - Centralize API access through query hooks or a shared API client. Avoid direct `axios` calls inside feature UI components unless there is a clear existing local pattern. - New feature-specific code should live near the feature/domain it belongs to. Do not put feature logic into generic `components`, `hooks`, or `lib` folders when a feature-local module is clearer. +- When splitting modules by boundaries, update imports to the new owner modules directly. Do not leave legacy compatibility barrels or re-export shims unless the user explicitly asks for a staged migration. +- Do not add new business logic, state machines, parsing, normalization, API orchestration, or reusable UI behavior back into large orchestration files such as `constructor.tsx` or `RuntimePresentation.tsx`. Put new logic in feature-local components, hooks, helpers, action modules, or typed utilities with focused tests. Do not keep splitting files only because they are large; split touched code when it clearly improves architecture boundaries, testability, maintainability, or robustness. ### Frontend TypeScript And React - Do not add new `any` without a specific reason. Prefer existing domain types or add a narrow local type. @@ -130,9 +132,12 @@ The backend defaults to port `3000` in `dev_stage` and `8080` otherwise; set `PO ### Frontend (run from `frontend/` directory) ```bash npm install # Install dependencies -npm run dev # Start dev server with Turbopack (port 3001) +npm run dev # Start stable dev server (port 3001) +npm run dev:turbo # Optional Turbopack dev server; avoid when debugging HMR/runtime issues +npm run test # Unit tests for frontend pure helpers (Node test runner via tsx) +npm run test:e2e # Playwright browser smoke/regression tests npm run typecheck # TypeScript check without production build -npm run verify # Typecheck, lint, and production build +npm run verify # Typecheck, lint, unit tests, Playwright e2e, and production build npm run build # Production build npm run lint # ESLint check (.ts, .tsx files) npm run format # Format code with Prettier diff --git a/README.md b/README.md index c418752..42ed03c 100644 --- a/README.md +++ b/README.md @@ -256,8 +256,12 @@ npm run build # Compile migrated TypeScript files ```bash cd frontend npm run dev # Development server +npm run test # Unit tests for frontend pure helpers +npm run test:e2e # Playwright browser smoke/regression tests +npm run typecheck # TypeScript check npm run build # Production build npm run lint # ESLint +npm run verify # Typecheck, lint, unit tests, Playwright e2e, and production build npm run format # Prettier ``` diff --git a/backend/docs/modules/services.md b/backend/docs/modules/services.md index e6115e8..8e0229a 100644 --- a/backend/docs/modules/services.md +++ b/backend/docs/modules/services.md @@ -46,20 +46,21 @@ The Services module implements the **business logic layer** of the backend appli Generated using `createEntityService()` from `factories/service.factory.ts`. These provide standardized CRUD operations with transaction handling. -| Service | File | Entity | LOC | -|---------|------|--------|-----| -| tour_pages | `tour_pages.ts` | Tour Pages (includes reverse video generation) | ~1,300 | -| permissions | `permissions.ts` | Permissions | 6 | -| asset_variants | `asset_variants.ts` | Asset Variants | 6 | -| presigned_url_requests | `presigned_url_requests.ts` | Presigned URL Requests | 6 | -| publish_events | `publish_events.ts` | Publish Events | 6 | -| pwa_caches | `pwa_caches.ts` | PWA Caches | 6 | -| access_logs | `access_logs.ts` | Access Logs | 6 | -| element_type_defaults | `element_type_defaults.ts` | Element Type Defaults | 6 | -| project_memberships | `project_memberships.ts` | Project Memberships | 6 | -| global_transition_defaults | `global_transition_defaults.ts` | Global transition defaults | 6 | +| Service | File | Entity | LOC | +| -------------------------- | ------------------------------- | ---------------------------------------------- | ------ | +| tour_pages | `tour_pages.ts` | Tour Pages (includes reverse video generation) | ~1,300 | +| permissions | `permissions.ts` | Permissions | 6 | +| asset_variants | `asset_variants.ts` | Asset Variants | 6 | +| presigned_url_requests | `presigned_url_requests.ts` | Presigned URL Requests | 6 | +| publish_events | `publish_events.ts` | Publish Events | 6 | +| pwa_caches | `pwa_caches.ts` | PWA Caches | 6 | +| access_logs | `access_logs.ts` | Access Logs | 6 | +| element_type_defaults | `element_type_defaults.ts` | Element Type Defaults | 6 | +| project_memberships | `project_memberships.ts` | Project Memberships | 6 | +| global_transition_defaults | `global_transition_defaults.ts` | Global transition defaults | 6 | **Example - Factory Service:** + ```javascript // permissions.ts import PermissionsDBApi from '../db/api/permissions.ts'; @@ -74,29 +75,31 @@ export default createEntityService(PermissionsDBApi, { Services with domain-specific business logic beyond simple CRUD. -| Service | File | Purpose | LOC | -|---------|------|---------|-----| -| assets | `assets.ts` | Asset management, MIME validation, embed URL validation, stored media metadata probing | ~300 | -| auth | `auth.ts` | Authentication, password reset, email verification | ~210 | -| users | `users.ts` | User management, invitation emails, Public viewer grants | ~350 | -| projects | `projects.ts` | Project cloning, slug generation, slug uniqueness validation | ~680 | -| roles | `roles.ts` | Role management, permission assignment, CSV import, Public-role hardening | ~170 | -| file | `file.ts` | Multi-provider file storage, downloadToBuffer, uploadBuffer, S3/GCloud circuit breaker for processing paths | ~1,600 | -| publish | `publish.ts` | Dev→Stage→Production publishing | ~400 | -| search | `search.ts` | Global full-text search | 178 | -| pwa_manifest | `pwa_manifest.js` | PWA offline manifest generation | 315 | -| project_audio_tracks | `project_audio_tracks.ts` | Audio track management | 117 | -| project_transition_settings | `project_transition_settings.ts` | Environment-aware transition settings | 209 | -| project_element_defaults | `project_element_defaults.ts` | Element defaults with reset/diff | 34 | -| global_ui_control_defaults | `global_ui_control_defaults.ts` | Global defaults CRUD service for system controls | 6 | -| project_ui_control_settings | `project_ui_control_settings.ts` | Transactional find/upsert/delete for project UI-control overrides | 51 | -| videoProcessing | `videoProcessing.ts` | FFmpeg video reversal for transition videos with single-worker queue, `-threads 1`, hard timeout, metadata logs, and circuit breaker | ~240 | +| Service | File | Purpose | LOC | +| --------------------------- | -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------ | +| assets | `assets.ts` | Asset management, MIME validation, embed URL validation, stored media metadata probing | ~300 | +| auth | `auth.ts` | Authentication, password reset, email verification | ~210 | +| users | `users.ts` | User management, invitation emails, Public viewer grants | ~350 | +| projects | `projects.ts` | Project cloning, slug generation, slug uniqueness validation | ~680 | +| roles | `roles.ts` | Role management, permission assignment, CSV import, Public-role hardening | ~170 | +| file | `file.ts` | Multi-provider file storage, downloadToBuffer, uploadBuffer; provider registry/circuit breaker live in `file/FileStorageRegistry.ts`, pure cache/range/path/MIME helpers live in `file/FileService.helpers.ts` | ~1,200 | +| publish | `publish.ts` | Dev→Stage→Production publishing | ~400 | +| search | `search.ts` | Global full-text search | 178 | +| pwa_manifest | `pwa_manifest.js` | PWA offline manifest generation | 315 | +| project_audio_tracks | `project_audio_tracks.ts` | Audio track management | 117 | +| project_transition_settings | `project_transition_settings.ts` | Environment-aware transition settings | 209 | +| project_element_defaults | `project_element_defaults.ts` | Element defaults with reset/diff | 34 | +| global_ui_control_defaults | `global_ui_control_defaults.ts` | Global defaults CRUD service for system controls | 6 | +| project_ui_control_settings | `project_ui_control_settings.ts` | Transactional find/upsert/delete for project UI-control overrides | 51 | +| videoProcessing | `videoProcessing.ts` | FFmpeg video reversal for transition videos with single-worker queue, `-threads 1`, hard timeout, metadata logs, and circuit breaker | ~240 | ### 3. Specialized Module Directories ``` services/ ├── file/ # Storage providers (Strategy Pattern) +│ ├── FileStorageRegistry.ts # Provider selection, singletons, circuit breaker +│ ├── FileService.helpers.ts # Tested pure cache/range/path/MIME/error helpers │ ├── BaseStorageProvider.ts # Abstract interface │ ├── S3StorageProvider.ts # AWS S3 implementation │ ├── LocalStorageProvider.ts # Local filesystem @@ -161,13 +164,13 @@ function createEntityService(DBApi, options = {}) { **Generated Methods:** -| Method | Signature | Description | -|--------|-----------|-------------| -| `create` | `(data, currentUser) → record` | Create with transaction | -| `bulkImport` | `(req, res) → void` | CSV import with validation | -| `update` | `(data, id, currentUser) → record` | Update with existence check | -| `deleteByIds` | `(ids, currentUser) → void` | Bulk soft delete | -| `remove` | `(id, currentUser) → void` | Single soft delete | +| Method | Signature | Description | +| ------------- | ---------------------------------- | --------------------------- | +| `create` | `(data, currentUser) → record` | Create with transaction | +| `bulkImport` | `(req, res) → void` | CSV import with validation | +| `update` | `(data, id, currentUser) → record` | Update with existence check | +| `deleteByIds` | `(ids, currentUser) → void` | Bulk soft delete | +| `remove` | `(id, currentUser) → void` | Single soft delete | --- @@ -178,6 +181,7 @@ function createEntityService(DBApi, options = {}) { Extends the factory-generated CRUD service with page-specific operations. **Reorder operation:** + - `TourPagesService.reorder(data, currentUser)` accepts `projectId`, `environment`, and `orderedPageIds`. - Reordering is allowed only for `environment='dev'`; this preserves the @@ -220,6 +224,7 @@ class Auth { ``` **Security Features:** + - bcrypt password hashing (`config.bcrypt.saltRounds`) - JWT token generation via `helpers.jwtSign()` - Email verification required for login @@ -232,23 +237,24 @@ Extended factory service with strict reusable TypeScript contracts, MIME type va ```typescript class AssetsService extends BaseService { // Create asset with MIME type validation - static async create({ data, currentUser, transaction, runtimeContext }) + static async create({ data, currentUser, transaction, runtimeContext }); // Update asset with MIME type validation - static async update({ id, data, currentUser, transaction, runtimeContext }) + static async update({ id, data, currentUser, transaction, runtimeContext }); } ``` **MIME Type Validation:** -| Asset Type | Valid MIME Prefixes | Description | -|------------|---------------------|-------------| -| `image` | `image/` | JPEG, PNG, GIF, WebP, SVG, etc. | -| `video` | `video/` | MP4, WebM, MOV, etc. | -| `audio` | `audio/` | MP3, WAV, OGG, etc. | -| `embed` | n/a | MIME validation skipped; HTTPS embed URL domain is validated | +| Asset Type | Valid MIME Prefixes | Description | +| ---------- | ------------------- | ------------------------------------------------------------ | +| `image` | `image/` | JPEG, PNG, GIF, WebP, SVG, etc. | +| `video` | `video/` | MP4, WebM, MOV, etc. | +| `audio` | `audio/` | MP3, WAV, OGG, etc. | +| `embed` | n/a | MIME validation skipped; HTTPS embed URL domain is validated | **Validation Rules:** + - `asset_type` and `mime_type` must be consistent - If `asset_type` is `image`, `mime_type` must start with `image/` - If `asset_type` is `video`, `mime_type` must start with `video/` @@ -257,9 +263,10 @@ class AssetsService extends BaseService { - Missing `mime_type` is allowed (browser may not always send it) **Error Response:** + ```javascript throw new ValidationError( - `Invalid file type for ${assetType}. Expected ${patterns.description}, got "${mimeType}"` + `Invalid file type for ${assetType}. Expected ${patterns.description}, got "${mimeType}"`, ); ``` @@ -269,6 +276,13 @@ throw new ValidationError( Unified file storage using Strategy Pattern for multiple backends. Features comprehensive error handling, AbortController support for client disconnect handling, path validation for security, and structured Pino logging. +Pure cache, range-header, path-validation, MIME, error-mapping, stream-body, +and upload-session input helpers live in `services/file/FileService.helpers.ts` +and are covered by `tests/file-service.test.ts`; `file.ts` remains the owner of +provider orchestration and storage side effects. +Provider selection, S3/GCloud/local singleton initialization, external-storage +circuit breaker wrapping, and upload-session manager singleton live in +`services/file/FileStorageRegistry.ts`. ```javascript // Provider auto-detection @@ -302,11 +316,11 @@ const getS3ErrorStatusCode = (error) → number // HTTP status code mapping The `copyFile()` function uses provider-native copy operations for optimal performance: -| Provider | Implementation | Performance | -|----------|----------------|-------------| -| S3 | `CopyObjectCommand` (server-side) | 15x faster, zero memory | -| Local | `fs.promises.copyFile` | Kernel-level copy | -| GCloud | Download + Upload (fallback) | Legacy behavior | +| Provider | Implementation | Performance | +| -------- | --------------------------------- | ----------------------- | +| S3 | `CopyObjectCommand` (server-side) | 15x faster, zero memory | +| Local | `fs.promises.copyFile` | Kernel-level copy | +| GCloud | Download + Upload (fallback) | Legacy behavior | ```javascript // Single file copy @@ -318,12 +332,14 @@ const copyFilesParallel = async (copies, { concurrency = 10, continueOnError = t ``` **Benefits over download-then-upload:** + - **15x faster**: Server-side copy, no data through backend - **Zero memory**: No file buffering in Node.js - **No timeouts**: Works for large files (>100MB) - **Reduced bandwidth**: No double network transfer **Error Response Format:** + ```javascript // Standardized across all file endpoints { @@ -338,12 +354,12 @@ Downloads automatically abort S3 requests when client disconnects, preventing wa **Provider Selection:** -| Priority | Provider | Detection | -|----------|----------|-----------| -| 1 | `config.fileStorage.provider` | Validated `FILE_STORAGE_PROVIDER` override | -| 2 | S3 | `S3_BUCKET` + `S3_REGION` + credentials | -| 3 | GCloud | Validated `GC_PROJECT_ID` + `GC_CLIENT_EMAIL` + `GC_PRIVATE_KEY` | -| 4 | Local | Default fallback | +| Priority | Provider | Detection | +| -------- | ----------------------------- | ---------------------------------------------------------------- | +| 1 | `config.fileStorage.provider` | Validated `FILE_STORAGE_PROVIDER` override | +| 2 | S3 | `S3_BUCKET` + `S3_REGION` + credentials | +| 3 | GCloud | Validated `GC_PROJECT_ID` + `GC_CLIENT_EMAIL` + `GC_PRIVATE_KEY` | +| 4 | Local | Default fallback | External S3/GCloud operations used by processing paths are protected by the shared file-storage circuit breaker. For S3, only retryable SDK/network errors @@ -382,12 +398,14 @@ When S3 is the storage provider, downloads are cached locally to reduce S3 reque ``` **Cache validation:** + - File exists AND age < `S3_CACHE_MAX_AGE` (default: 1 hour) - No `.downloading` marker file (indicates download in progress) - Size matches `Content-Length` header (verified before rename) **Why atomic writes:** Without atomic writes, concurrent requests could serve truncated cache files: + 1. Request A starts downloading 12MB file, writes to cache 2. After 2MB written, Request B checks cache - file exists, age valid 3. Request B serves 2MB truncated file → **corrupted video/image** @@ -415,6 +433,7 @@ module.exports = class PublishService { ``` **Non-Blocking vs Blocking:** + - `saveToStage()` - Returns immediately, copy runs in background via `setImmediate()` - `publishToProduction()` - Waits for entire copy operation before returning @@ -433,6 +452,7 @@ saveToStage() publishToProduction() │ ``` **Event Status Lifecycle:** + 1. `queued` - Event created 2. `running` - Processing started 3. `success` / `failed` - Completed @@ -444,26 +464,30 @@ Global full-text search with permission filtering. ```typescript export default class SearchService { // Search across all permitted entities - static async search(searchQuery: string, currentUser: CurrentUser | undefined): Promise + static async search( + searchQuery: string, + currentUser: CurrentUser | undefined, + ): Promise; } ``` **Searchable Tables:** -| Table | Text Fields | Numeric Fields | -|-------|-------------|----------------| -| users | firstName, lastName, phoneNumber, email | - | -| projects | name, slug, description, logo_url, favicon_url, og_image_url | - | -| assets | name, cdn_url, storage_key, mime_type, checksum | size_mb, width_px, height_px, duration_sec | -| asset_variants | cdn_url | width_px, height_px, size_mb | -| presigned_url_requests | requested_key, mime_type, status | requested_size_mb | -| tour_pages | source_key, name, slug, background_image_url, background_video_url, background_audio_url, ui_schema_json | sort_order | -| project_audio_tracks | source_key, name, slug, url | volume, sort_order | -| publish_events | error_message | pages_copied, transitions_copied, audios_copied | -| pwa_caches | cache_version, manifest_json, asset_list_json | - | -| access_logs | path, ip_address, user_agent | - | +| Table | Text Fields | Numeric Fields | +| ---------------------- | -------------------------------------------------------------------------------------------------------- | ----------------------------------------------- | +| users | firstName, lastName, phoneNumber, email | - | +| projects | name, slug, description, logo_url, favicon_url, og_image_url | - | +| assets | name, cdn_url, storage_key, mime_type, checksum | size_mb, width_px, height_px, duration_sec | +| asset_variants | cdn_url | width_px, height_px, size_mb | +| presigned_url_requests | requested_key, mime_type, status | requested_size_mb | +| tour_pages | source_key, name, slug, background_image_url, background_video_url, background_audio_url, ui_schema_json | sort_order | +| project_audio_tracks | source_key, name, slug, url | volume, sort_order | +| publish_events | error_message | pages_copied, transitions_copied, audios_copied | +| pwa_caches | cache_version, manifest_json, asset_list_json | - | +| access_logs | path, ip_address, user_agent | - | **Permission Check:** + ```javascript // Only search tables user has READ permission for if (!hasPermission(permissionSet, `READ_${tableName.toUpperCase()}`)) { @@ -498,6 +522,7 @@ export default class ProjectsService { ``` **Slug Validation:** + - `validateSlugUniqueness()` normalizes the slug and checks for duplicates - Uses `excludeId` parameter to skip the current project during updates - Checks soft-deleted projects (`paranoid: false`) to prevent conflicts @@ -528,18 +553,20 @@ Phase G: Clone tour_pages, audio_tracks, element_defaults **Key Implementation Details:** -| Phase | Operation | Notes | -|-------|-----------|-------| -| B-C | Parallel file copy | Uses `FileService.copyFilesParallel()` with S3 `CopyObjectCommand` | -| E | Asset ID mapping | Tracks `oldAssetId → newAssetId` for reversed video copying | -| F | Reversed video copy | Separate phase because reversed videos use asset-ID-based paths, not project-ID-based | -| G | Path transformation | `transformUiSchemaAssetPaths()` updates all asset URLs in `ui_schema_json` | +| Phase | Operation | Notes | +| ----- | ------------------- | ------------------------------------------------------------------------------------- | +| B-C | Parallel file copy | Uses `FileService.copyFilesParallel()` with S3 `CopyObjectCommand` | +| E | Asset ID mapping | Tracks `oldAssetId → newAssetId` for reversed video copying | +| F | Reversed video copy | Separate phase because reversed videos use asset-ID-based paths, not project-ID-based | +| G | Path transformation | `transformUiSchemaAssetPaths()` updates all asset URLs in `ui_schema_json` | **Reversed Video Storage Pattern:** + - Primary assets: `assets/{projectId}/{uuid}.ext` - Reversed videos: `assets/{assetId}/reversed.mp4` (uses asset ID, not project ID) **Error Handling:** + - Failed file copies fall back to original storage path (cloned project still functional, shares assets with source) - Most assets won't have reversed videos - this is expected (only navigation elements with transitions generate them) - Transaction rollback on DB errors; orphaned S3 files acceptable (can be cleaned later) @@ -550,16 +577,17 @@ Role management service for standard CRUD, CSV bulk import, and permission assig ```typescript export default class RolesService { - static assertPublicRoleHasNoPermissions(data, existingRole) - static async create(options) - static async bulkImport(req, res) - static async update(options) - static async deleteByIds(options) - static async remove(options) + static assertPublicRoleHasNoPermissions(data, existingRole); + static async create(options); + static async bulkImport(req, res); + static async update(options); + static async deleteByIds(options); + static async remove(options); } ``` **Public Role Hardening:** + - Creating or updating a role named `Public` rejects non-empty permissions. - Existing role lookup is done before update so renaming a role to `Public` cannot retain assigned permissions through the service boundary. - This keeps customer viewer access separate from admin RBAC permissions. @@ -585,13 +613,14 @@ class Project_element_defaultsService extends BaseService { **Methods:** -| Method | Description | -|--------|-------------| -| `resetToGlobal(id, options)` | Resets a project element default to match the current global element type default | -| `getDiffFromGlobal(id)` | Compares project element default with global default, returns differences | -| `snapshotGlobalDefaults(projectId, options)` | Creates project element defaults by copying all global element type defaults | +| Method | Description | +| -------------------------------------------- | --------------------------------------------------------------------------------- | +| `resetToGlobal(id, options)` | Resets a project element default to match the current global element type default | +| `getDiffFromGlobal(id)` | Compares project element default with global default, returns differences | +| `snapshotGlobalDefaults(projectId, options)` | Creates project element defaults by copying all global element type defaults | **Use Cases:** + - **Project Creation:** `snapshotGlobalDefaults` is called to copy global defaults to new project - **Reset to Global:** User can reset customized project defaults back to global values - **Diff View:** UI can show which settings differ from global defaults @@ -626,14 +655,14 @@ class Project_element_defaultsService extends BaseService { ```typescript export default class BaseStorageProvider { - static get providerName(): string - upload(key, data, options): Promise - download(key): Promise - delete(key): Promise - deleteMany(keys): Promise - exists(key): Promise - list(prefix): Promise - getSignedUrl(key, expiresIn): Promise + static get providerName(): string; + upload(key, data, options): Promise; + download(key): Promise; + delete(key): Promise; + deleteMany(keys): Promise; + exists(key): Promise; + list(prefix): Promise; + getSignedUrl(key, expiresIn): Promise; } ``` @@ -675,15 +704,15 @@ class S3StorageProvider extends BaseStorageProvider { **S3 Error to HTTP Status Code Mapping:** -| S3 Error | HTTP Status | -|----------|-------------| -| NoSuchKey, NotFound, NoSuchBucket | 404 | -| AccessDenied, InvalidAccessKeyId | 403 | -| ExpiredToken | 401 | -| TimeoutError, RequestTimeout | 504 | -| NetworkingError, ServiceUnavailable | 503 | -| ThrottlingException | 429 | -| InternalError | 500 | +| S3 Error | HTTP Status | +| ----------------------------------- | ----------- | +| NoSuchKey, NotFound, NoSuchBucket | 404 | +| AccessDenied, InvalidAccessKeyId | 403 | +| ExpiredToken | 401 | +| TimeoutError, RequestTimeout | 504 | +| NetworkingError, ServiceUnavailable | 503 | +| ThrottlingException | 429 | +| InternalError | 500 | ### LocalStorageProvider @@ -729,6 +758,7 @@ class UploadSessionManager { ``` **Session Directory Structure:** + ``` upload_sessions/ └── {sessionId}/ @@ -749,15 +779,16 @@ Core email sending using Nodemailer. ```typescript export default class EmailSender { - constructor(email: EmailTemplate) - async send(): Promise - static get isConfigured(): boolean - get transportConfig(): SMTPTransport.Options - get from(): string + constructor(email: EmailTemplate); + async send(): Promise; + static get isConfigured(): boolean; + get transportConfig(): SMTPTransport.Options; + get from(): string; } ``` **Configuration (`config.email`):** + ```javascript { host: 'email-smtp.us-east-1.amazonaws.com', @@ -772,19 +803,23 @@ export default class EmailSender { ### Email Templates -| Template | Class | Fields | -|----------|-------|--------| -| Password Reset | `PasswordResetEmail` | to, link | +| Template | Class | Fields | +| ------------------ | ------------------------------- | -------- | +| Password Reset | `PasswordResetEmail` | to, link | | Email Verification | `EmailAddressVerificationEmail` | to, link | -| User Invitation | `InvitationEmail` | to, host | +| User Invitation | `InvitationEmail` | to, host | **Template Pattern:** + ```typescript export default class PasswordResetEmail implements EmailTemplate { constructor({ to, link }: LinkEmailTemplateOptions) {} get subject(): string { - return getNotification('emails.passwordReset.subject', getNotification('app.title')); + return getNotification( + 'emails.passwordReset.subject', + getNotification('app.title'), + ); } async html(): Promise { @@ -794,7 +829,7 @@ export default class PasswordResetEmail implements EmailTemplate { .replace(/{resetUrl}/g, this.link) .replace(/{accountName}/g, this.to); } -}; +} ``` --- @@ -804,6 +839,7 @@ export default class PasswordResetEmail implements EmailTemplate { ### Error Classes **ValidationError (400 Bad Request):** + ```javascript class ValidationError extends Error { constructor(messageCode) { @@ -817,6 +853,7 @@ class ValidationError extends Error { ``` **ForbiddenError (403 Forbidden):** + ```javascript class ForbiddenError extends Error { constructor(messageCode) { @@ -851,15 +888,15 @@ const errors = { userAlreadyExists: 'User with this email already exists', userNotFound: 'User not found', // ... - } + }, }, emails: { invitation: { subject: "You've been invited to {0}", - body: "..." + body: '...', }, // ... - } + }, }; ``` @@ -867,12 +904,12 @@ const errors = { ```javascript // Get notification with parameter substitution -getNotification('emails.invitation.subject', 'Tour Builder') +getNotification('emails.invitation.subject', 'Tour Builder'); // → "You've been invited to Tour Builder" // Check if key exists in catalog -isNotification('auth.userNotFound') // → true -isNotification('custom.message') // → false +isNotification('auth.userNotFound'); // → true +isNotification('custom.message'); // → false ``` --- @@ -931,27 +968,27 @@ static async withProjectPublishLock(projectId, callback) { ### External Packages -| Package | Usage | -|---------|-------| -| `bcrypt` | Password hashing | -| `nodemailer` | Email sending | -| `csv-parser` | CSV import parsing | -| `axios` | External API calls (widgets) | -| `uuid` | Upload session IDs | -| `@aws-sdk/client-s3` | S3 operations | -| `@aws-sdk/s3-request-presigner` | Presigned URLs | -| `@google-cloud/storage` | GCloud storage | -| `lodash/get` | Deep object access | +| Package | Usage | +| ------------------------------- | ---------------------------- | +| `bcrypt` | Password hashing | +| `nodemailer` | Email sending | +| `csv-parser` | CSV import parsing | +| `axios` | External API calls (widgets) | +| `uuid` | Upload session IDs | +| `@aws-sdk/client-s3` | S3 operations | +| `@aws-sdk/s3-request-presigner` | Presigned URLs | +| `@google-cloud/storage` | GCloud storage | +| `lodash/get` | Deep object access | ### Internal Dependencies -| Module | Services Using | -|--------|---------------| -| `db/models` | All services (transaction) | -| `db/api/*` | All entity services | -| `factories/service.factory` | 9 factory services | -| `config` | auth, file, users, roles | -| `helpers` | auth (jwtSign) | +| Module | Services Using | +| --------------------------- | -------------------------- | +| `db/models` | All services (transaction) | +| `db/api/*` | All entity services | +| `factories/service.factory` | 9 factory services | +| `config` | auth, file, users, roles | +| `helpers` | auth (jwtSign) | --- @@ -999,33 +1036,33 @@ static async withProjectPublishLock(projectId, callback) { ### Environment Variables -| Variable | Service | Description | -|----------|---------|-------------| -| `FILE_STORAGE_PROVIDER` | file | Force provider ('s3', 'gcloud', 'local') | -| `AWS_S3_BUCKET` | file | S3 bucket name | -| `AWS_S3_REGION` | file | AWS region (default: us-east-1) | -| `AWS_ACCESS_KEY_ID` | file | AWS access key | -| `AWS_SECRET_ACCESS_KEY` | file | AWS secret key | -| `AWS_S3_PREFIX` | file | Key prefix | -| `AWS_S3_CONNECTION_TIMEOUT` | file | S3 connection timeout in ms (default: 5000) | -| `AWS_S3_REQUEST_TIMEOUT` | file | S3 request timeout in ms (default: 30000) | -| `AWS_S3_MAX_ATTEMPTS` | file | S3 retry attempts (default: 3) | -| `AWS_S3_MAX_SOCKETS` | file | S3 connection pool size (default: 50) | -| `AWS_S3_KEEP_ALIVE` | file | Enable HTTP keep-alive (default: true) | -| `AWS_S3_PRESIGN_EXPIRY` | file | Presigned URL expiry in seconds (default: 3600) | -| `GC_PROJECT_ID` | file | GCloud project | -| `GC_CLIENT_EMAIL` | file | GCloud service account | -| `GC_PRIVATE_KEY` | file | GCloud private key | -| `FFMPEG_REVERSE_TIMEOUT_MS` | videoProcessing | Reverse-video hard timeout in ms (default: 600000) | -| `FFPROBE_TIMEOUT_MS` | videoProcessing | Metadata probe timeout in ms (default: 30000) | -| `FFMPEG_BREAKER_FAILURE_THRESHOLD` | videoProcessing | Failures before FFmpeg breaker opens (default: 3) | -| `FFMPEG_BREAKER_COOLDOWN_MS` | videoProcessing | FFmpeg breaker cooldown in ms (default: 120000) | -| `FFMPEG_BREAKER_SUCCESS_THRESHOLD` | videoProcessing | Half-open successes required to close FFmpeg breaker (default: 1) | -| `FILE_STORAGE_BREAKER_FAILURE_THRESHOLD` | file | Failures before storage breaker opens (default: 5) | -| `FILE_STORAGE_BREAKER_COOLDOWN_MS` | file | Storage breaker cooldown in ms (default: 30000) | -| `FILE_STORAGE_BREAKER_SUCCESS_THRESHOLD` | file | Half-open successes required to close storage breaker (default: 2) | -| `EMAIL_USER` | email | SMTP username | -| `EMAIL_PASS` | email | SMTP password | +| Variable | Service | Description | +| ---------------------------------------- | --------------- | ------------------------------------------------------------------ | +| `FILE_STORAGE_PROVIDER` | file | Force provider ('s3', 'gcloud', 'local') | +| `AWS_S3_BUCKET` | file | S3 bucket name | +| `AWS_S3_REGION` | file | AWS region (default: us-east-1) | +| `AWS_ACCESS_KEY_ID` | file | AWS access key | +| `AWS_SECRET_ACCESS_KEY` | file | AWS secret key | +| `AWS_S3_PREFIX` | file | Key prefix | +| `AWS_S3_CONNECTION_TIMEOUT` | file | S3 connection timeout in ms (default: 5000) | +| `AWS_S3_REQUEST_TIMEOUT` | file | S3 request timeout in ms (default: 30000) | +| `AWS_S3_MAX_ATTEMPTS` | file | S3 retry attempts (default: 3) | +| `AWS_S3_MAX_SOCKETS` | file | S3 connection pool size (default: 50) | +| `AWS_S3_KEEP_ALIVE` | file | Enable HTTP keep-alive (default: true) | +| `AWS_S3_PRESIGN_EXPIRY` | file | Presigned URL expiry in seconds (default: 3600) | +| `GC_PROJECT_ID` | file | GCloud project | +| `GC_CLIENT_EMAIL` | file | GCloud service account | +| `GC_PRIVATE_KEY` | file | GCloud private key | +| `FFMPEG_REVERSE_TIMEOUT_MS` | videoProcessing | Reverse-video hard timeout in ms (default: 600000) | +| `FFPROBE_TIMEOUT_MS` | videoProcessing | Metadata probe timeout in ms (default: 30000) | +| `FFMPEG_BREAKER_FAILURE_THRESHOLD` | videoProcessing | Failures before FFmpeg breaker opens (default: 3) | +| `FFMPEG_BREAKER_COOLDOWN_MS` | videoProcessing | FFmpeg breaker cooldown in ms (default: 120000) | +| `FFMPEG_BREAKER_SUCCESS_THRESHOLD` | videoProcessing | Half-open successes required to close FFmpeg breaker (default: 1) | +| `FILE_STORAGE_BREAKER_FAILURE_THRESHOLD` | file | Failures before storage breaker opens (default: 5) | +| `FILE_STORAGE_BREAKER_COOLDOWN_MS` | file | Storage breaker cooldown in ms (default: 30000) | +| `FILE_STORAGE_BREAKER_SUCCESS_THRESHOLD` | file | Half-open successes required to close storage breaker (default: 2) | +| `EMAIL_USER` | email | SMTP username | +| `EMAIL_PASS` | email | SMTP password | ### Config References @@ -1041,18 +1078,18 @@ module.exports = { secretAccessKey, prefix, // Timeout/retry configuration - connectionTimeout, // 5000ms default - requestTimeout, // 30000ms default - maxAttempts, // 3 retries with adaptive backoff + connectionTimeout, // 5000ms default + requestTimeout, // 30000ms default + maxAttempts, // 3 retries with adaptive backoff // Connection pool - maxSockets, // 50 concurrent connections - keepAlive, // HTTP keep-alive enabled + maxSockets, // 50 concurrent connections + keepAlive, // HTTP keep-alive enabled // Presigned URLs - presignExpirySeconds, // 3600 (1 hour) default + presignExpirySeconds, // 3600 (1 hour) default }, gcloud: { bucket, hash }, uploadDir: './uploads', - flHost: 'https://flatlogic.host', // Widget service + flHost: 'https://flatlogic.host', // Widget service project_uuid: '...', }; ``` @@ -1112,21 +1149,25 @@ describe('PublishService', () => { ## Best Practices ### 1. Transaction Handling + - Always wrap multi-step operations in transactions - Use try/catch/rollback pattern consistently - Pass transaction to all DB API calls ### 2. Error Handling + - Use `ValidationError` for client errors (400) - Use `ForbiddenError` for authorization failures (403) - Use notification catalog for consistent messages ### 3. Service Design + - Keep services focused on business logic - Delegate DB operations to DB API layer - Use factories for simple CRUD services ### 4. File Operations + - Use Strategy Pattern for multi-provider support - Implement chunked uploads for large files - Clean up sessions on completion/failure diff --git a/backend/src/services/file.ts b/backend/src/services/file.ts index 2ff7d02..cce61e5 100644 --- a/backend/src/services/file.ts +++ b/backend/src/services/file.ts @@ -11,23 +11,16 @@ * - Path validation for security */ -import crypto from 'crypto'; import fs from 'fs'; import os from 'os'; import path from 'path'; -import { PassThrough, type Readable } from 'stream'; +import { PassThrough } from 'stream'; import { pipeline } from 'stream/promises'; -import { Storage } from '@google-cloud/storage'; import config from '../config.ts'; import processFile from '../middlewares/upload.ts'; -import { - getCurrentUser, - getRequestLogger, -} from '../utils/request-context.ts'; +import { getCurrentUser, getRequestLogger } from '../utils/request-context.ts'; import type { - CachedFileLookupResult, - FileByteRange, FileCopyFailure, FileCopyOperation, FileCopyOptions, @@ -38,405 +31,50 @@ import type { FileDeleteOptions, FileDeleteResult, FileDownloadToTempFileResult, - FileErrorDetails, - FileErrorResponse, - FileMimeTypeMap, FileServiceFacade, FileServiceRequest, FileServiceResponse, - FileStorageProviderName, FileUploadBufferOptions, FileUploadBufferResult, FileUploadServiceRequest, - GCloudBucketState, UploadChunkRequest, UploadSessionInitRequest, UploadSessionRequest, } from '../types/index.ts'; -import { CircuitBreaker } from '../utils/circuit-breaker.ts'; import { logger } from '../utils/logger.ts'; -import LocalStorageProvider from './file/LocalStorageProvider.ts'; -import S3StorageProvider from './file/S3StorageProvider.ts'; -import UploadSessionManager from './file/UploadSessionManager.ts'; - -// ============================================================================ -// S3 Cache Helpers -// ============================================================================ - -/** - * Get the local cache path for an S3 key - */ -const getCachePath = (privateUrl: string): string => { - // Create a safe filename from the URL - const hash = crypto.createHash('md5').update(privateUrl).digest('hex'); - const ext = path.extname(privateUrl) || ''; - return path.join(config.s3CacheDir, `${hash}${ext}`); -}; - -/** - * Check if a cached file exists and is still valid - * Returns invalid if a download is in progress (.downloading file exists) - */ -const getCachedFile = async ( - cachePath: string, -): Promise => { - try { - // Check if download is in progress - if so, don't use cache - const downloadingPath = cachePath + '.downloading'; - try { - await fs.promises.access(downloadingPath); - // Download in progress, cache is not valid - return { stats: null, valid: false }; - } catch { - // No download in progress, continue checking cache - } - - const stats = await fs.promises.stat(cachePath); - const age = (Date.now() - stats.mtimeMs) / 1000; - if (age < config.s3CacheMaxAge) { - return { stats, valid: true }; - } - return { stats, valid: false }; - } catch { - return { stats: null, valid: false }; - } -}; - -/** - * Ensure cache directory exists - */ -const ensureCacheDir = async (): Promise => { - try { - await fs.promises.mkdir(config.s3CacheDir, { recursive: true }); - } catch (err) { - if (!hasErrorCode(err, 'EEXIST')) throw err; - } -}; - -/** - * Generate ETag from file stats - */ -const generateETag = (stats: fs.Stats): string => { - return `"${stats.size.toString(16)}-${stats.mtimeMs.toString(16)}"`; -}; - -const MIME_TYPES: FileMimeTypeMap = { - '.jpg': 'image/jpeg', - '.jpeg': 'image/jpeg', - '.png': 'image/png', - '.gif': 'image/gif', - '.webp': 'image/webp', - '.svg': 'image/svg+xml', - '.ico': 'image/x-icon', - '.mp4': 'video/mp4', - '.webm': 'video/webm', - '.mp3': 'audio/mpeg', - '.wav': 'audio/wav', - '.ogg': 'audio/ogg', - '.pdf': 'application/pdf', - '.json': 'application/json', -}; - -const hasErrorCode = (error: unknown, code: string): boolean => { - return error instanceof Error && 'code' in error && error.code === code; -}; - -const getErrorStringProperty = ( - error: unknown, - property: 'name' | 'code' | 'message', -): string | undefined => { - if (error instanceof Error && property === 'name') { - return error.name; - } - - if (error instanceof Error && property === 'message') { - return error.message; - } - - if (typeof error !== 'object' || error === null) { - return undefined; - } - - const descriptor = Object.getOwnPropertyDescriptor(error, property); - return typeof descriptor?.value === 'string' ? descriptor.value : undefined; -}; - -const getUnknownErrorMessage = (error: unknown): string => { - return getErrorStringProperty(error, 'message') || String(error); -}; - -const toError = (error: unknown): Error => { - return error instanceof Error - ? error - : new Error(getUnknownErrorMessage(error)); -}; - -const getStorageUploadUrl = (resultUrl: string | undefined): string => { - return resultUrl || ''; -}; - -const toBufferChunk = (chunk: unknown): Buffer => { - if (Buffer.isBuffer(chunk)) { - return chunk; - } - - if (typeof chunk === 'string') { - return Buffer.from(chunk); - } - - if (chunk instanceof Uint8Array) { - return Buffer.from(chunk); - } - - throw new Error('Unsupported stream chunk type'); -}; - -const getSanitizedStringInput = (value: unknown): string => { - if (typeof value === 'string') { - return value; - } - - if (typeof value === 'number' || typeof value === 'boolean') { - return `${value}`; - } - - return ''; -}; - -const isReadableStream = (value: unknown): value is Readable => { - if (typeof value !== 'object' || value === null || !('pipe' in value)) { - return false; - } - - const descriptor = Object.getOwnPropertyDescriptor(value, 'pipe'); - return typeof descriptor?.value === 'function'; -}; - -const hasByteArrayTransformer = ( - value: unknown, -): value is { transformToByteArray(): Promise } => { - return ( - typeof value === 'object' && - value !== null && - 'transformToByteArray' in value && - typeof value.transformToByteArray === 'function' - ); -}; - -// ============================================================================ -// Provider Initialization (Singleton) -// ============================================================================ - -let s3Provider: S3StorageProvider | null = null; -let localProvider: LocalStorageProvider | null = null; -let gcloudBucket: GCloudBucketState['bucket'] | null = null; -let gcloudHash: string | null = null; -let uploadSessionManager: UploadSessionManager | null = null; - -const externalStorageBreaker = new CircuitBreaker({ - name: 'external-file-storage', - failureThreshold: config.resilience.fileStorage.breaker.failureThreshold, - cooldownMs: config.resilience.fileStorage.breaker.cooldownMs, - successThreshold: config.resilience.fileStorage.breaker.successThreshold, - shouldRecordFailure: (error) => - getFileStorageProvider() === 's3' - ? S3StorageProvider.isRetryableError(error) - : true, -}); - -const getFileStorageProvider = (): FileStorageProviderName => { - const provider = config.fileStorage.provider; - if (provider === 's3' || provider === 'gcloud' || provider === 'local') { - return provider; - } - - const hasS3 = Boolean( - config.s3.bucket && - config.s3.region && - config.s3.accessKeyId && - config.s3.secretAccessKey, - ); - if (hasS3) return 's3'; - - const hasGCloud = Boolean( - config.gcloud.projectId && - config.gcloud.clientEmail && - config.gcloud.privateKey && - config.gcloud.bucket && - config.gcloud.hash, - ); - if (hasGCloud) return 'gcloud'; - - return 'local'; -}; - -const executeStorageOperation = async ( - provider: FileStorageProviderName, - operationName: string, - operation: () => Promise, -): Promise => { - if (provider === 'local') { - return operation(); - } - - return externalStorageBreaker.execute( - `${provider}.${operationName}`, - operation, - ); -}; - -const getS3Provider = (): S3StorageProvider => { - if (!s3Provider) { - s3Provider = new S3StorageProvider({ - bucket: config.s3.bucket, - region: config.s3.region, - accessKeyId: config.s3.accessKeyId, - secretAccessKey: config.s3.secretAccessKey, - prefix: config.s3.prefix, - // Timeout and connection pool configuration from config - connectionTimeout: config.s3.connectionTimeout, - requestTimeout: config.s3.requestTimeout, - maxAttempts: config.s3.maxAttempts, - maxSockets: config.s3.maxSockets, - keepAlive: config.s3.keepAlive, - }); - - logger.info( - { - provider: 's3', - bucket: config.s3.bucket, - region: config.s3.region, - connectionTimeout: config.s3.connectionTimeout, - requestTimeout: config.s3.requestTimeout, - maxAttempts: config.s3.maxAttempts, - }, - 'S3 storage provider initialized', - ); - } - return s3Provider; -}; - -const getLocalProvider = (): LocalStorageProvider => { - if (!localProvider) { - localProvider = new LocalStorageProvider({ basePath: config.uploadDir }); - } - return localProvider; -}; - -const getGCloudBucket = (): GCloudBucketState => { - if (!gcloudBucket) { - const privateKey = config.gcloud.privateKey.replace(/\\\n/g, '\n'); - const storage = new Storage({ - projectId: config.gcloud.projectId, - credentials: { - client_email: config.gcloud.clientEmail, - private_key: privateKey, - }, - }); - gcloudBucket = storage.bucket(config.gcloud.bucket); - gcloudHash = config.gcloud.hash; - } - return { bucket: gcloudBucket, hash: gcloudHash || '' }; -}; - -const getUploadSessionManager = (): UploadSessionManager => { - if (!uploadSessionManager) { - uploadSessionManager = new UploadSessionManager({ - sessionDir: path.join(config.uploadDir, 'upload_sessions'), - ttlMs: 24 * 60 * 60 * 1000, - }); - } - return uploadSessionManager; -}; - -// ============================================================================ -// Error Handling Utilities -// ============================================================================ - -/** - * Standardized error response format - * @param {string} message - Error message - * @param {string} [code] - Error code for programmatic handling - * @param {Object} [details] - Additional error details - */ -const createErrorResponse = ( - message: string, - code: string | null = null, - details: FileErrorDetails | null = null, -): FileErrorResponse => { - const response: FileErrorResponse = { message }; - if (code) response.code = code; - if (details) response.details = details; - return response; -}; - -/** - * Get HTTP status code for S3 errors - */ -const getS3ErrorStatusCode = (error: unknown): number => { - return S3StorageProvider.getErrorStatusCode(error); -}; - -/** - * Build user-friendly error message based on error type - */ -const getErrorMessage = ( - error: unknown, - operation = 'process', -): string => { - const errorName = getErrorStringProperty(error, 'name') || ''; - const errorCode = getErrorStringProperty(error, 'code') || ''; - - if ( - errorName === 'NoSuchKey' || - errorName === 'NotFound' || - errorName === 'NoSuchBucket' - ) { - return 'File not found'; - } - if (errorName === 'AccessDenied' || errorName === 'InvalidAccessKeyId') { - return 'Access denied to file'; - } - if (errorName === 'TimeoutError' || errorCode === 'ETIMEDOUT') { - return 'Request timed out while accessing file'; - } - if (errorCode === 'ECONNRESET' || errorCode === 'ECONNREFUSED') { - return 'Connection error while accessing storage'; - } - if (errorName === 'AbortError') { - return 'Request was cancelled'; - } - - return `Could not ${operation} the file`; -}; - -// ============================================================================ -// Path Validation -// ============================================================================ - -/** - * Validate that a path doesn't contain traversal attacks - * @param {string} urlPath - The path to validate - * @returns {boolean} Whether the path is valid - */ -const isValidPath = (urlPath: unknown): urlPath is string => { - if (!urlPath || typeof urlPath !== 'string') return false; - - const trimmed = urlPath.trim(); - if (!trimmed) return false; - - // Check for path traversal attempts - if (trimmed.includes('..')) return false; - if (trimmed.includes('\0')) return false; - - // Check for double slashes (potential injection) - if (trimmed.includes('//')) return false; - - // Check for protocol indicators - if (/^[a-zA-Z]+:/.test(trimmed)) return false; - - return true; -}; +import { + createErrorResponse, + ensureCacheDir, + generateETag, + getCachePath, + getCachedFile, + getErrorMessage, + getErrorStringProperty, + getMimeTypeFromExtension, + getSanitizedStringInput, + getSingleQueryValue, + getStorageUploadUrl, + getUnknownErrorMessage, + hasByteArrayTransformer, + isReadableStream, + isValidPath, + MIME_TYPES, + parseRangeHeader, + sanitizeFilename, + sanitizeFolder, + sendStorageBody, + toBufferChunk, + toError, +} from './file/FileService.helpers.ts'; +import { + executeStorageOperation, + getFileStorageProvider, + getGCloudBucket, + getLocalProvider, + getS3ErrorStatusCode, + getS3Provider, + getUploadSessionManager, +} from './file/FileStorageRegistry.ts'; // ============================================================================ // Unified Upload/Download/Delete Interface @@ -510,63 +148,6 @@ const uploadFile = async ( } }; -/** - * Parse Range header value - * @param {string} rangeHeader - Range header value (e.g., "bytes=0-1000") - * @param {number} totalSize - Total file size - * @returns {{start: number, end: number} | null} - */ -const parseRangeHeader = ( - rangeHeader: string | undefined, - totalSize: number, -): FileByteRange | null => { - if (!rangeHeader || !rangeHeader.startsWith('bytes=')) return null; - - const range = rangeHeader.slice(6); // Remove "bytes=" - const parts = range.split('-'); - const rangeStart = parts[0] || ''; - const rangeEnd = parts[1]; - - let start = parseInt(rangeStart, 10); - let end = rangeEnd ? parseInt(rangeEnd, 10) : totalSize - 1; - - // Handle suffix ranges (e.g., bytes=-500 means last 500 bytes) - if (isNaN(start)) { - start = totalSize - end; - end = totalSize - 1; - } - - // Validate range - if (isNaN(start) || isNaN(end) || start > end || start >= totalSize) { - return null; - } - - // Cap end to file size - end = Math.min(end, totalSize - 1); - - return { start, end }; -}; - -const getSingleQueryValue = (value: unknown): string | undefined => { - return typeof value === 'string' ? value : undefined; -}; - -const sendStorageBody = async ( - body: unknown, - res: FileServiceResponse, -): Promise => { - if (isReadableStream(body)) { - return body.pipe(res); - } - - if (hasByteArrayTransformer(body)) { - const bytes = await body.transformToByteArray(); - return res.send(Buffer.from(bytes)); - } - - return res.send(body); -}; - const downloadFile = async ( req: FileServiceRequest, res: FileServiceResponse, @@ -911,9 +492,7 @@ const downloadFile = async ( if (!res.headersSent) { return res .status(statusCode) - .send( - createErrorResponse(errorMessage, errorName || 'DOWNLOAD_ERROR'), - ); + .send(createErrorResponse(errorMessage, errorName || 'DOWNLOAD_ERROR')); } } @@ -1112,18 +691,6 @@ const uploadBuffer = async ( // Chunked Upload Session Management // ============================================================================ -const sanitizeFolder = (folder: unknown): string | null => { - const value = getSanitizedStringInput(folder) - .trim() - .replace(/^\/+|\/+$/g, ''); - return !value || value.includes('..') ? null : value; -}; - -const sanitizeFilename = (filename: unknown): string | null => { - const value = path.basename(getSanitizedStringInput(filename).trim()); - return !value || value === '.' || value === '..' ? null : value; -}; - const initUploadSession = ( req: UploadSessionInitRequest, res: FileServiceResponse, @@ -1142,9 +709,7 @@ const initUploadSession = ( const filename = sanitizeFilename(req.body?.filename); const totalChunks = Number(req.body?.totalChunks); const size = Number(req.body?.size); - const contentType = getSanitizedStringInput( - req.body?.contentType, - ).trim(); + const contentType = getSanitizedStringInput(req.body?.contentType).trim(); if (!folder || !filename) return res @@ -1205,7 +770,10 @@ const getUploadSession = ( return res .status(404) .send( - createErrorResponse('Upload session not found', 'SESSION_NOT_FOUND'), + createErrorResponse( + 'Upload session not found', + 'SESSION_NOT_FOUND', + ), ); if (session.userId !== currentUser.id) return res.sendStatus(403); @@ -1273,7 +841,9 @@ const uploadChunk = async ( await sessionManager.saveChunk(sessionId, chunkIndex, chunkBuffer); const updatedSession = sessionManager.readMeta(sessionId); if (!updatedSession) { - throw new Error(`Upload session disappeared after chunk save: ${sessionId}`); + throw new Error( + `Upload session disappeared after chunk save: ${sessionId}`, + ); } return res.status(200).send({ @@ -1392,16 +962,6 @@ const finalizeUploadSession = async ( // File Copy Utility // ============================================================================ -/** - * Get MIME type from file extension - * @param {string} filepath - File path or storage key - * @returns {string} MIME type or default 'application/octet-stream' - */ -const getMimeTypeFromExtension = (filepath: string): string => { - const ext = path.extname(filepath).toLowerCase(); - return MIME_TYPES[ext] || 'application/octet-stream'; -}; - /** * Copy a file within storage using provider's native copy * S3: Uses CopyObjectCommand (server-side, no download/upload) diff --git a/backend/src/services/file/FileService.helpers.ts b/backend/src/services/file/FileService.helpers.ts new file mode 100644 index 0000000..185badc --- /dev/null +++ b/backend/src/services/file/FileService.helpers.ts @@ -0,0 +1,277 @@ +import crypto from 'crypto'; +import fs from 'fs'; +import path from 'path'; +import type { Readable } from 'stream'; + +import config from '../../config.ts'; +import type { + CachedFileLookupResult, + FileByteRange, + FileErrorDetails, + FileErrorResponse, + FileMimeTypeMap, + FileServiceResponse, +} from '../../types/index.ts'; + +export const MIME_TYPES: FileMimeTypeMap = { + '.jpg': 'image/jpeg', + '.jpeg': 'image/jpeg', + '.png': 'image/png', + '.gif': 'image/gif', + '.webp': 'image/webp', + '.svg': 'image/svg+xml', + '.ico': 'image/x-icon', + '.mp4': 'video/mp4', + '.webm': 'video/webm', + '.mp3': 'audio/mpeg', + '.wav': 'audio/wav', + '.ogg': 'audio/ogg', + '.pdf': 'application/pdf', + '.json': 'application/json', +}; + +export const getCachePath = (privateUrl: string): string => { + const hash = crypto.createHash('md5').update(privateUrl).digest('hex'); + const ext = path.extname(privateUrl) || ''; + return path.join(config.s3CacheDir, `${hash}${ext}`); +}; + +export const getCachedFile = async ( + cachePath: string, +): Promise => { + try { + const downloadingPath = cachePath + '.downloading'; + try { + await fs.promises.access(downloadingPath); + return { stats: null, valid: false }; + } catch { + // No download marker means the cache file can be checked normally. + } + + const stats = await fs.promises.stat(cachePath); + const age = (Date.now() - stats.mtimeMs) / 1000; + if (age < config.s3CacheMaxAge) { + return { stats, valid: true }; + } + return { stats, valid: false }; + } catch { + return { stats: null, valid: false }; + } +}; + +export const getErrorStringProperty = ( + error: unknown, + property: 'name' | 'code' | 'message', +): string | undefined => { + if (error instanceof Error && property === 'name') { + return error.name; + } + + if (error instanceof Error && property === 'message') { + return error.message; + } + + if (typeof error !== 'object' || error === null) { + return undefined; + } + + const descriptor = Object.getOwnPropertyDescriptor(error, property); + return typeof descriptor?.value === 'string' ? descriptor.value : undefined; +}; + +export const hasErrorCode = (error: unknown, code: string): boolean => { + return getErrorStringProperty(error, 'code') === code; +}; + +export const ensureCacheDir = async (): Promise => { + try { + await fs.promises.mkdir(config.s3CacheDir, { recursive: true }); + } catch (err) { + if (!hasErrorCode(err, 'EEXIST')) throw err; + } +}; + +export const generateETag = (stats: fs.Stats): string => { + return `"${stats.size.toString(16)}-${stats.mtimeMs.toString(16)}"`; +}; + +export const getUnknownErrorMessage = (error: unknown): string => { + return getErrorStringProperty(error, 'message') || String(error); +}; + +export const toError = (error: unknown): Error => { + return error instanceof Error + ? error + : new Error(getUnknownErrorMessage(error)); +}; + +export const getStorageUploadUrl = (resultUrl: string | undefined): string => { + return resultUrl || ''; +}; + +export const toBufferChunk = (chunk: unknown): Buffer => { + if (Buffer.isBuffer(chunk)) { + return chunk; + } + + if (typeof chunk === 'string') { + return Buffer.from(chunk); + } + + if (chunk instanceof Uint8Array) { + return Buffer.from(chunk); + } + + throw new Error('Unsupported stream chunk type'); +}; + +export const getSanitizedStringInput = (value: unknown): string => { + if (typeof value === 'string') { + return value; + } + + if (typeof value === 'number' || typeof value === 'boolean') { + return `${value}`; + } + + return ''; +}; + +export const isReadableStream = (value: unknown): value is Readable => { + if (typeof value !== 'object' || value === null || !('pipe' in value)) { + return false; + } + + const descriptor = Object.getOwnPropertyDescriptor(value, 'pipe'); + return typeof descriptor?.value === 'function'; +}; + +export const hasByteArrayTransformer = ( + value: unknown, +): value is { transformToByteArray(): Promise } => { + return ( + typeof value === 'object' && + value !== null && + 'transformToByteArray' in value && + typeof value.transformToByteArray === 'function' + ); +}; + +export const createErrorResponse = ( + message: string, + code: string | null = null, + details: FileErrorDetails | null = null, +): FileErrorResponse => { + const response: FileErrorResponse = { message }; + if (code) response.code = code; + if (details) response.details = details; + return response; +}; + +export const getErrorMessage = ( + error: unknown, + operation = 'process', +): string => { + const errorName = getErrorStringProperty(error, 'name') || ''; + const errorCode = getErrorStringProperty(error, 'code') || ''; + + if ( + errorName === 'NoSuchKey' || + errorName === 'NotFound' || + errorName === 'NoSuchBucket' + ) { + return 'File not found'; + } + if (errorName === 'AccessDenied' || errorName === 'InvalidAccessKeyId') { + return 'Access denied to file'; + } + if (errorName === 'TimeoutError' || errorCode === 'ETIMEDOUT') { + return 'Request timed out while accessing file'; + } + if (errorCode === 'ECONNRESET' || errorCode === 'ECONNREFUSED') { + return 'Connection error while accessing storage'; + } + if (errorName === 'AbortError') { + return 'Request was cancelled'; + } + + return `Could not ${operation} the file`; +}; + +export const isValidPath = (urlPath: unknown): urlPath is string => { + if (!urlPath || typeof urlPath !== 'string') return false; + + const trimmed = urlPath.trim(); + if (!trimmed) return false; + if (trimmed.includes('..')) return false; + if (trimmed.includes('\0')) return false; + if (trimmed.includes('//')) return false; + if (/^[a-zA-Z]+:/.test(trimmed)) return false; + + return true; +}; + +export const parseRangeHeader = ( + rangeHeader: string | undefined, + totalSize: number, +): FileByteRange | null => { + if (!rangeHeader || !rangeHeader.startsWith('bytes=')) return null; + + const range = rangeHeader.slice(6); + const parts = range.split('-'); + const rangeStart = parts[0] || ''; + const rangeEnd = parts[1]; + + let start = parseInt(rangeStart, 10); + let end = rangeEnd ? parseInt(rangeEnd, 10) : totalSize - 1; + + if (isNaN(start)) { + start = totalSize - end; + end = totalSize - 1; + } + + if (isNaN(start) || isNaN(end) || start > end || start >= totalSize) { + return null; + } + + end = Math.min(end, totalSize - 1); + + return { start, end }; +}; + +export const getSingleQueryValue = (value: unknown): string | undefined => { + return typeof value === 'string' ? value : undefined; +}; + +export const sendStorageBody = async ( + body: unknown, + res: FileServiceResponse, +): Promise => { + if (isReadableStream(body)) { + return body.pipe(res); + } + + if (hasByteArrayTransformer(body)) { + const bytes = await body.transformToByteArray(); + return res.send(Buffer.from(bytes)); + } + + return res.send(body); +}; + +export const sanitizeFolder = (folder: unknown): string | null => { + const value = getSanitizedStringInput(folder) + .trim() + .replace(/^\/+|\/+$/g, ''); + return !value || value.includes('..') ? null : value; +}; + +export const sanitizeFilename = (filename: unknown): string | null => { + const value = path.basename(getSanitizedStringInput(filename).trim()); + return !value || value === '.' || value === '..' ? null : value; +}; + +export const getMimeTypeFromExtension = (filepath: string): string => { + const ext = path.extname(filepath).toLowerCase(); + return MIME_TYPES[ext] || 'application/octet-stream'; +}; diff --git a/backend/src/services/file/FileStorageRegistry.ts b/backend/src/services/file/FileStorageRegistry.ts new file mode 100644 index 0000000..3a7d1d7 --- /dev/null +++ b/backend/src/services/file/FileStorageRegistry.ts @@ -0,0 +1,138 @@ +import path from 'path'; +import { Storage } from '@google-cloud/storage'; + +import config from '../../config.ts'; +import type { + FileStorageProviderName, + GCloudBucketState, +} from '../../types/index.ts'; +import { CircuitBreaker } from '../../utils/circuit-breaker.ts'; +import { logger } from '../../utils/logger.ts'; +import LocalStorageProvider from './LocalStorageProvider.ts'; +import S3StorageProvider from './S3StorageProvider.ts'; +import UploadSessionManager from './UploadSessionManager.ts'; + +let s3Provider: S3StorageProvider | null = null; +let localProvider: LocalStorageProvider | null = null; +let gcloudBucket: GCloudBucketState['bucket'] | null = null; +let gcloudHash: string | null = null; +let uploadSessionManager: UploadSessionManager | null = null; + +export const getFileStorageProvider = (): FileStorageProviderName => { + const provider = config.fileStorage.provider; + if (provider === 's3' || provider === 'gcloud' || provider === 'local') { + return provider; + } + + const hasS3 = Boolean( + config.s3.bucket && + config.s3.region && + config.s3.accessKeyId && + config.s3.secretAccessKey, + ); + if (hasS3) return 's3'; + + const hasGCloud = Boolean( + config.gcloud.projectId && + config.gcloud.clientEmail && + config.gcloud.privateKey && + config.gcloud.bucket && + config.gcloud.hash, + ); + if (hasGCloud) return 'gcloud'; + + return 'local'; +}; + +const externalStorageBreaker = new CircuitBreaker({ + name: 'external-file-storage', + failureThreshold: config.resilience.fileStorage.breaker.failureThreshold, + cooldownMs: config.resilience.fileStorage.breaker.cooldownMs, + successThreshold: config.resilience.fileStorage.breaker.successThreshold, + shouldRecordFailure: (error) => + getFileStorageProvider() === 's3' + ? S3StorageProvider.isRetryableError(error) + : true, +}); + +export const executeStorageOperation = async ( + provider: FileStorageProviderName, + operationName: string, + operation: () => Promise, +): Promise => { + if (provider === 'local') { + return operation(); + } + + return externalStorageBreaker.execute( + `${provider}.${operationName}`, + operation, + ); +}; + +export const getS3Provider = (): S3StorageProvider => { + if (!s3Provider) { + s3Provider = new S3StorageProvider({ + bucket: config.s3.bucket, + region: config.s3.region, + accessKeyId: config.s3.accessKeyId, + secretAccessKey: config.s3.secretAccessKey, + prefix: config.s3.prefix, + connectionTimeout: config.s3.connectionTimeout, + requestTimeout: config.s3.requestTimeout, + maxAttempts: config.s3.maxAttempts, + maxSockets: config.s3.maxSockets, + keepAlive: config.s3.keepAlive, + }); + + logger.info( + { + provider: 's3', + bucket: config.s3.bucket, + region: config.s3.region, + connectionTimeout: config.s3.connectionTimeout, + requestTimeout: config.s3.requestTimeout, + maxAttempts: config.s3.maxAttempts, + }, + 'S3 storage provider initialized', + ); + } + return s3Provider; +}; + +export const getLocalProvider = (): LocalStorageProvider => { + if (!localProvider) { + localProvider = new LocalStorageProvider({ basePath: config.uploadDir }); + } + return localProvider; +}; + +export const getGCloudBucket = (): GCloudBucketState => { + if (!gcloudBucket) { + const privateKey = config.gcloud.privateKey.replace(/\\\n/g, '\n'); + const storage = new Storage({ + projectId: config.gcloud.projectId, + credentials: { + client_email: config.gcloud.clientEmail, + private_key: privateKey, + }, + }); + gcloudBucket = storage.bucket(config.gcloud.bucket); + gcloudHash = config.gcloud.hash; + } + return { bucket: gcloudBucket, hash: gcloudHash || '' }; +}; + +export const getUploadSessionManager = (): UploadSessionManager => { + if (!uploadSessionManager) { + uploadSessionManager = new UploadSessionManager({ + sessionDir: path.join(config.uploadDir, 'upload_sessions'), + ttlMs: 24 * 60 * 60 * 1000, + }); + } + return uploadSessionManager; +}; + +export const getS3ErrorStatusCode = (error: unknown): number => { + return S3StorageProvider.getErrorStatusCode(error); +}; diff --git a/backend/tests/file-service.test.ts b/backend/tests/file-service.test.ts index 3836c68..5ac34d5 100644 --- a/backend/tests/file-service.test.ts +++ b/backend/tests/file-service.test.ts @@ -7,6 +7,14 @@ import { generatePresignedUrls, isValidPath, } from '../src/services/file.ts'; +import { + getMimeTypeFromExtension, + parseRangeHeader, + sanitizeFilename, + sanitizeFolder, + toBufferChunk, +} from '../src/services/file/FileService.helpers.ts'; +import { getFileStorageProvider } from '../src/services/file/FileStorageRegistry.ts'; void test('isValidPath allows relative storage keys and rejects traversal or protocol inputs', () => { assert.equal(isValidPath('projects/demo/image.jpg'), true); @@ -37,6 +45,60 @@ void test('createErrorResponse returns a stable structured error payload', () => ); }); +void test('parseRangeHeader supports bounded, open-ended, and suffix byte ranges', () => { + assert.deepEqual(parseRangeHeader('bytes=0-99', 1000), { + start: 0, + end: 99, + }); + assert.deepEqual(parseRangeHeader('bytes=900-', 1000), { + start: 900, + end: 999, + }); + assert.deepEqual(parseRangeHeader('bytes=-100', 1000), { + start: 900, + end: 999, + }); + assert.deepEqual(parseRangeHeader('bytes=900-1200', 1000), { + start: 900, + end: 999, + }); +}); + +void test('parseRangeHeader rejects invalid ranges', () => { + assert.equal(parseRangeHeader(undefined, 1000), null); + assert.equal(parseRangeHeader('items=0-100', 1000), null); + assert.equal(parseRangeHeader('bytes=100-10', 1000), null); + assert.equal(parseRangeHeader('bytes=1000-1001', 1000), null); + assert.equal(parseRangeHeader('bytes=abc-def', 1000), null); +}); + +void test('upload session input sanitizers keep folders relative and filenames basename-only', () => { + assert.equal(sanitizeFolder('/projects/demo/'), 'projects/demo'); + assert.equal(sanitizeFolder('../secret'), null); + assert.equal(sanitizeFolder(''), null); + + assert.equal(sanitizeFilename('../image.jpg'), 'image.jpg'); + assert.equal(sanitizeFilename(' nested/video.mp4 '), 'video.mp4'); + assert.equal(sanitizeFilename('.'), null); +}); + +void test('file helpers normalize stream chunks and MIME types', () => { + assert.deepEqual(toBufferChunk('hello'), Buffer.from('hello')); + assert.deepEqual( + toBufferChunk(new Uint8Array([1, 2, 3])), + Buffer.from([1, 2, 3]), + ); + assert.equal(getMimeTypeFromExtension('video.MP4'), 'video/mp4'); + assert.equal( + getMimeTypeFromExtension('archive.bin'), + 'application/octet-stream', + ); + assert.throws( + () => toBufferChunk({ invalid: true }), + /Unsupported stream chunk type/, + ); +}); + void test('generatePresignedUrls returns backend download URLs for local storage provider', async () => { const previousProvider = config.fileStorage.provider; config.fileStorage.provider = 'local'; @@ -52,3 +114,13 @@ void test('generatePresignedUrls returns backend download URLs for local storage config.fileStorage.provider = previousProvider; } }); + +void test('getFileStorageProvider honors explicit provider override before credential auto-detection', () => { + const previousProvider = config.fileStorage.provider; + config.fileStorage.provider = 'local'; + try { + assert.equal(getFileStorageProvider(), 'local'); + } finally { + config.fileStorage.provider = previousProvider; + } +}); diff --git a/documentation/asset-upload-variants.md b/documentation/asset-upload-variants.md index d9bebcb..a8033ce 100644 --- a/documentation/asset-upload-variants.md +++ b/documentation/asset-upload-variants.md @@ -10,6 +10,7 @@ saves actual `frame_rate` metadata on the asset record. ## Overview The platform implements a **robust asset management system** with: + - **Chunked uploads** - Large files uploaded in 5MB chunks with resumability - **Multi-provider storage** - S3, Google Cloud Storage, or local filesystem - **Asset variants** - Metadata tracking for different file formats/sizes @@ -64,12 +65,13 @@ The platform implements a **robust asset management system** with: Large files are uploaded in chunks to handle network interruptions and support resumable uploads. **Key Parameters:** -| Parameter | Value | Description | -|-----------|-------|-------------| -| Chunk Size | 5 MB | `5 * 1024 * 1024` bytes | -| Max Retries | 3 | Per-chunk retry limit | -| Retry Backoff | 500ms × retry | Exponential backoff | -| Session TTL | 24 hours | Expired sessions auto-cleaned | + +| Parameter | Value | Description | +| ------------- | ------------- | ----------------------------- | +| Chunk Size | 5 MB | `5 * 1024 * 1024` bytes | +| Max Retries | 3 | Per-chunk retry limit | +| Retry Backoff | 500ms × retry | Exponential backoff | +| Session TTL | 24 hours | Expired sessions auto-cleaned | ### Upload Session Flow @@ -123,6 +125,7 @@ GET /file/upload-sessions/{sessionId} ``` **Response:** + ```json { "sessionId": "uuid", @@ -157,6 +160,7 @@ Use this to resume interrupted uploads by checking which chunks are already uplo Session management is handled by `UploadSessionManager` class (`backend/src/services/file/UploadSessionManager.ts`). **Local Storage:** + ``` {uploadDir}/upload_sessions/{sessionId}/ ├── meta.json # Session metadata @@ -167,6 +171,7 @@ Session management is handled by `UploadSessionManager` class (`backend/src/serv ``` **S3 Storage:** + ``` {prefix}/_upload_sessions/{sessionId}/ ├── meta.json # Session metadata @@ -182,7 +187,7 @@ During finalization, chunks are assembled in a temp directory: ```javascript // Temp assembly location -const tempDir = path.join(config.uploadDir, '_temp_assembly'); +const tempDir = path.join(config.uploadDir, "_temp_assembly"); const assembledPath = path.join(tempDir, `${sessionId}.bin`); // For each chunk, append to assembled file @@ -192,7 +197,7 @@ for (let chunkIndex = 0; chunkIndex < totalChunks; chunkIndex++) { // Validate size if (assembledStats.size !== session.size) { - throw new Error('Assembled file size mismatch'); + throw new Error("Assembled file size mismatch"); } ``` @@ -204,9 +209,11 @@ The file service uses a **Strategy Pattern** with modular providers: ``` backend/src/services/ -├── file.js # Unified interface (provider initialization, routing) +├── file.ts # Unified interface (provider initialization, routing) └── file/ - ├── index.js # Module exports + ├── index.ts # Module exports + ├── FileStorageRegistry.ts # Provider selection, singletons, circuit breaker + ├── FileService.helpers.ts # Pure cache/range/path/MIME/error helpers ├── BaseStorageProvider.ts # Abstract base class ├── S3StorageProvider.ts # AWS S3 implementation ├── LocalStorageProvider.ts # Local filesystem implementation @@ -231,11 +238,11 @@ Protected operations: Configuration overrides: -| Variable | Default | Description | -|----------|---------|-------------| -| `FILE_STORAGE_BREAKER_FAILURE_THRESHOLD` | `5` | Consecutive failures before the breaker opens | -| `FILE_STORAGE_BREAKER_COOLDOWN_MS` | `30000` | Cooldown before a half-open probe is allowed | -| `FILE_STORAGE_BREAKER_SUCCESS_THRESHOLD` | `2` | Half-open successes required to close the breaker | +| Variable | Default | Description | +| ---------------------------------------- | ------- | ------------------------------------------------- | +| `FILE_STORAGE_BREAKER_FAILURE_THRESHOLD` | `5` | Consecutive failures before the breaker opens | +| `FILE_STORAGE_BREAKER_COOLDOWN_MS` | `30000` | Cooldown before a half-open probe is allowed | +| `FILE_STORAGE_BREAKER_SUCCESS_THRESHOLD` | `2` | Half-open successes required to close the breaker | When the breaker is open, protected operations fail fast with a `503`-status error instead of piling more requests onto the external storage provider. @@ -248,29 +255,31 @@ error instead of piling more requests onto the external storage provider. const getFileStorageProvider = () => { // 1. Explicit override from validated backend config const provider = config.fileStorage.provider; - if (provider === 's3' || provider === 'gcloud' || provider === 'local') { + if (provider === "s3" || provider === "gcloud" || provider === "local") { return provider; } // 2. Auto-detect S3 from credentials const hasS3 = Boolean( - config.s3.bucket && config.s3.region && - config.s3.accessKeyId && config.s3.secretAccessKey + config.s3.bucket && + config.s3.region && + config.s3.accessKeyId && + config.s3.secretAccessKey, ); - if (hasS3) return 's3'; + if (hasS3) return "s3"; // 3. Auto-detect GCloud from credentials const hasGCloud = Boolean( config.gcloud.projectId && - config.gcloud.clientEmail && - config.gcloud.privateKey && - config.gcloud.bucket && - config.gcloud.hash + config.gcloud.clientEmail && + config.gcloud.privateKey && + config.gcloud.bucket && + config.gcloud.hash, ); - if (hasGCloud) return 'gcloud'; + if (hasGCloud) return "gcloud"; // 4. Default to local filesystem - return 'local'; + return "local"; }; ``` @@ -281,6 +290,7 @@ const getFileStorageProvider = () => { ### S3 Configuration **Environment Variables:** + ```bash FILE_STORAGE_PROVIDER=s3 AWS_S3_BUCKET=your-bucket-name @@ -291,6 +301,7 @@ AWS_S3_PREFIX=optional-prefix # Default: afeefb9d49f5b7977577876b99532ac7 ``` **Implementation:** + ```javascript const initS3 = () => { const client = new S3Client({ @@ -311,6 +322,7 @@ const initS3 = () => { ``` **URL Format:** + ``` https://{bucket}.s3.{region}.amazonaws.com/{prefix}/{folder}/{filename} ``` @@ -320,6 +332,7 @@ https://{bucket}.s3.{region}.amazonaws.com/{prefix}/{folder}/{filename} ### Google Cloud Storage Configuration **Environment Variables:** + ```bash FILE_STORAGE_PROVIDER=gcloud GC_PROJECT_ID=your-project-id @@ -328,6 +341,7 @@ GC_PRIVATE_KEY="-----BEGIN PRIVATE KEY-----\n..." ``` **Config (config.ts):** Currently hardcoded values: + ```javascript gcloud: { bucket: "fldemo-files", @@ -338,6 +352,7 @@ gcloud: { **Note:** Unlike S3, GCloud bucket and hash are hardcoded in `config.ts`. To use different values, modify the config file directly. **URL Format:** + ``` https://storage.googleapis.com/{bucket}/{hash}/{folder}/{filename} ``` @@ -351,6 +366,7 @@ https://storage.googleapis.com/{bucket}/{hash}/{folder}/{filename} **Upload Directory:** `os.tmpdir()` (OS temp directory) **URL Format:** + ``` /api/file/download?privateUrl={encodedPath} ``` @@ -363,23 +379,23 @@ Files are served via the backend download endpoint. Note: The download endpoint **File:** `backend/src/db/models/assets.js` -| Field | Type | Required | Description | -|-------|------|----------|-------------| -| id | UUID | Yes | Primary key | -| name | TEXT(255) | No | Display name | -| asset_type | ENUM | Yes | `image`, `video`, `audio`, `file` | -| type | ENUM | Yes | `icon`, `background_image`, `audio`, `video`, `transition`, `logo`, `favicon`, `document`, `general` (default) | -| cdn_url | TEXT | No | Public CDN/storage URL | -| storage_key | TEXT | No | Private storage path | -| mime_type | TEXT | No | MIME type (validated format) | -| size_mb | DECIMAL | No | File size in MB | -| width_px | INTEGER | No | Image/video width | -| height_px | INTEGER | No | Image/video height | -| duration_sec | DECIMAL | No | Video/audio duration | -| frame_rate | DECIMAL | No | Video FPS from backend `ffprobe` | -| checksum | TEXT | No | File checksum | -| is_public | BOOLEAN | Yes | Public visibility (default: false) | -| projectId | UUID | Yes | FK to projects (CASCADE) | +| Field | Type | Required | Description | +| ------------ | --------- | -------- | -------------------------------------------------------------------------------------------------------------- | +| id | UUID | Yes | Primary key | +| name | TEXT(255) | No | Display name | +| asset_type | ENUM | Yes | `image`, `video`, `audio`, `file` | +| type | ENUM | Yes | `icon`, `background_image`, `audio`, `video`, `transition`, `logo`, `favicon`, `document`, `general` (default) | +| cdn_url | TEXT | No | Public CDN/storage URL | +| storage_key | TEXT | No | Private storage path | +| mime_type | TEXT | No | MIME type (validated format) | +| size_mb | DECIMAL | No | File size in MB | +| width_px | INTEGER | No | Image/video width | +| height_px | INTEGER | No | Image/video height | +| duration_sec | DECIMAL | No | Video/audio duration | +| frame_rate | DECIMAL | No | Video FPS from backend `ffprobe` | +| checksum | TEXT | No | File checksum | +| is_public | BOOLEAN | Yes | Public visibility (default: false) | +| projectId | UUID | Yes | FK to projects (CASCADE) | **Indexes:** `projectId`, `asset_type`, `type`, `is_public`, `deletedAt` @@ -389,26 +405,27 @@ Files are served via the backend download endpoint. Note: The download endpoint **File:** `backend/src/db/models/asset_variants.js` -| Field | Type | Required | Description | -|-------|------|----------|-------------| -| id | UUID | Yes | Primary key | -| variant_type | ENUM | No | `thumbnail`, `preview`, `webp`, `mp4_low`, `mp4_high`, `original` | -| cdn_url | TEXT(2048) | No | Variant CDN URL (validated URL format) | -| width_px | INTEGER | No | Variant width | -| height_px | INTEGER | No | Variant height | -| size_mb | DECIMAL | No | Variant file size | -| assetId | UUID | Yes | FK to assets (CASCADE) | +| Field | Type | Required | Description | +| ------------ | ---------- | -------- | ----------------------------------------------------------------- | +| id | UUID | Yes | Primary key | +| variant_type | ENUM | No | `thumbnail`, `preview`, `webp`, `mp4_low`, `mp4_high`, `original` | +| cdn_url | TEXT(2048) | No | Variant CDN URL (validated URL format) | +| width_px | INTEGER | No | Variant width | +| height_px | INTEGER | No | Variant height | +| size_mb | DECIMAL | No | Variant file size | +| assetId | UUID | Yes | FK to assets (CASCADE) | **Variant Types:** -| Type | Description | Use Case | -|------|-------------|----------| -| `thumbnail` | Small preview image | List views, galleries | -| `preview` | Medium quality preview | Quick previews | -| `webp` | WebP format | Web optimization | -| `mp4_low` | Low bitrate video | Mobile, slow connections | -| `mp4_high` | High bitrate video | Desktop, fast connections | -| `original` | Unmodified original | Full quality access | -| `reversed` | FFmpeg-reversed video | Back navigation transitions | + +| Type | Description | Use Case | +| ----------- | ---------------------- | --------------------------- | +| `thumbnail` | Small preview image | List views, galleries | +| `preview` | Medium quality preview | Quick previews | +| `webp` | WebP format | Web optimization | +| `mp4_low` | Low bitrate video | Mobile, slow connections | +| `mp4_high` | High bitrate video | Desktop, fast connections | +| `original` | Unmodified original | Full quality access | +| `reversed` | FFmpeg-reversed video | Back navigation transitions | **Note:** Most variants are **metadata records only** - the system tracks variant information but does not auto-generate them. The exception is `reversed` variants, which are auto-generated by the server when a page with transition videos is saved. @@ -419,14 +436,16 @@ Files are served via the backend download endpoint. Note: The download endpoint Reversed videos are a special variant type generated server-side for back navigation transitions. They use a different storage path pattern than other assets. **Storage Patterns:** -| Asset Type | Storage Path Pattern | Example | -|------------|---------------------|---------| -| Primary assets | `assets/{projectId}/{uuid}.ext` | `assets/abc-123/def-456.mp4` | + +| Asset Type | Storage Path Pattern | Example | +| --------------- | ------------------------------- | ----------------------------- | +| Primary assets | `assets/{projectId}/{uuid}.ext` | `assets/abc-123/def-456.mp4` | | Reversed videos | `assets/{assetId}/reversed.mp4` | `assets/xyz-789/reversed.mp4` | **Key Difference:** Reversed videos use the **asset ID** (not project ID) in their path. This is because they are tied to a specific asset (the transition video), not just the project. **Generation Flow:** + 1. User adds transition video to navigation element 2. User saves the tour page 3. Server detects `transitionVideoUrl` in navigation elements @@ -456,6 +475,7 @@ interface UseAssetUploaderReturn { ``` **Key Features:** + - **Batch upload** with queue management - **2 concurrent uploads** maximum (`maxConcurrent = 2`) - **Per-file progress** tracking @@ -463,37 +483,40 @@ interface UseAssetUploaderReturn { - **Media probing** for video/audio duration and dimensions **Status States:** -| Status | Description | -|--------|-------------| -| `queued` | File in queue, waiting to upload | -| `uploading` | Actively uploading chunks | -| `saving` | Finalizing upload and creating asset record | -| `success` | Upload and save completed | -| `error` | Upload failed (error message available) | + +| Status | Description | +| ----------- | ------------------------------------------- | +| `queued` | File in queue, waiting to upload | +| `uploading` | Actively uploading chunks | +| `saving` | Finalizing upload and creating asset record | +| `success` | Upload and save completed | +| `error` | Upload failed (error message available) | ### UploadService Class **File:** `frontend/src/components/Uploaders/UploadService.js` **Single File Upload:** + ```javascript const result = await FileUploader.upload(path, file, schema); // Returns: { id, name, sizeInBytes, privateUrl, publicUrl, new: true } ``` **Chunked Upload:** + ```javascript const result = await FileUploader.uploadChunked( - 'assets/project-id', // path - file, // File object - {}, // schema (validation) + "assets/project-id", // path + file, // File object + {}, // schema (validation) { - chunkSize: 5 * 1024 * 1024, // 5MB + chunkSize: 5 * 1024 * 1024, // 5MB maxRetries: 3, signal: abortController.signal, onProgress: (percent, { chunkIndex, totalChunks }) => {}, onStatus: (status, details) => {}, - } + }, ); ``` @@ -506,20 +529,21 @@ After upload, video/audio files are probed for metadata: ```typescript interface MediaDurationResult { duration: number; - width?: number; // Video only - height?: number; // Video only + width?: number; // Video only + height?: number; // Video only } // Probe video -const result = await probeMediaDuration(file, 'video', 10000); +const result = await probeMediaDuration(file, "video", 10000); // Returns: { duration: 120.5, width: 1920, height: 1080 } // Probe audio -const result = await probeMediaDuration(file, 'audio', 10000); +const result = await probeMediaDuration(file, "audio", 10000); // Returns: { duration: 180.0 } ``` **Implementation:** + - Creates HTML5 `