1143 lines
46 KiB
Markdown
1143 lines
46 KiB
Markdown
# Backend Services Module
|
|
|
|
## Overview
|
|
|
|
The Services module implements the **business logic layer** of the backend application. Services sit between routes (controllers) and the database API layer, encapsulating complex operations, transaction management, and cross-cutting concerns.
|
|
|
|
**Location:** `backend/src/services/`
|
|
|
|
**Total Files:** 37 (24 root services + 13 subdirectory files)
|
|
|
|
---
|
|
|
|
## Architecture Diagram
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────────┐
|
|
│ Routes Layer │
|
|
│ (HTTP Request Handling) │
|
|
└─────────────────────────────────────────────────────────────────────┘
|
|
│
|
|
▼
|
|
┌─────────────────────────────────────────────────────────────────────┐
|
|
│ Services Layer │
|
|
│ ┌───────────────────────────────────────────────────────────────┐ │
|
|
│ │ Service Categories │ │
|
|
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │
|
|
│ │ │ Factory │ │ Custom │ │ Specialized │ │ │
|
|
│ │ │ Services │ │ Services │ │ Modules │ │ │
|
|
│ │ │ (9 files) │ │ (12 files) │ │ (3 dirs) │ │ │
|
|
│ │ └─────────────┘ └─────────────┘ └─────────────┘ │ │
|
|
│ └───────────────────────────────────────────────────────────────┘ │
|
|
└─────────────────────────────────────────────────────────────────────┘
|
|
│
|
|
▼
|
|
┌─────────────────────────────────────────────────────────────────────┐
|
|
│ Database API Layer │
|
|
│ (Sequelize ORM Operations) │
|
|
└─────────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
---
|
|
|
|
## Service Categories
|
|
|
|
### 1. Factory-Generated Services (Simple CRUD)
|
|
|
|
Generated using `createEntityService()` from `factories/service.factory.ts`. These provide standardized CRUD operations with transaction handling.
|
|
|
|
| Service | File | Entity | LOC |
|
|
|---------|------|--------|-----|
|
|
| tour_pages | `tour_pages.ts` | Tour Pages (includes reverse video generation) | ~1,300 |
|
|
| permissions | `permissions.ts` | Permissions | 6 |
|
|
| asset_variants | `asset_variants.ts` | Asset Variants | 6 |
|
|
| presigned_url_requests | `presigned_url_requests.ts` | Presigned URL Requests | 6 |
|
|
| publish_events | `publish_events.ts` | Publish Events | 6 |
|
|
| pwa_caches | `pwa_caches.ts` | PWA Caches | 6 |
|
|
| access_logs | `access_logs.ts` | Access Logs | 6 |
|
|
| element_type_defaults | `element_type_defaults.ts` | Element Type Defaults | 6 |
|
|
| project_memberships | `project_memberships.ts` | Project Memberships | 6 |
|
|
| global_transition_defaults | `global_transition_defaults.ts` | Global transition defaults | 6 |
|
|
|
|
**Example - Factory Service:**
|
|
```javascript
|
|
// permissions.ts
|
|
import PermissionsDBApi from '../db/api/permissions.ts';
|
|
import { createEntityService } from '../factories/service.factory.ts';
|
|
|
|
export default createEntityService(PermissionsDBApi, {
|
|
entityName: 'permissions',
|
|
});
|
|
```
|
|
|
|
### 2. Custom Services (Business Logic)
|
|
|
|
Services with domain-specific business logic beyond simple CRUD.
|
|
|
|
| Service | File | Purpose | LOC |
|
|
|---------|------|---------|-----|
|
|
| assets | `assets.ts` | Asset management, MIME validation, embed URL validation, stored media metadata probing | ~300 |
|
|
| auth | `auth.ts` | Authentication, password reset, email verification | ~210 |
|
|
| users | `users.ts` | User management, invitation emails, Public viewer grants | ~350 |
|
|
| projects | `projects.ts` | Project cloning, slug generation, slug uniqueness validation | ~680 |
|
|
| roles | `roles.ts` | Role management, permission assignment, CSV import, Public-role hardening | ~170 |
|
|
| file | `file.ts` | Multi-provider file storage, downloadToBuffer, uploadBuffer, S3/GCloud circuit breaker for processing paths | ~1,600 |
|
|
| publish | `publish.ts` | Dev→Stage→Production publishing | ~400 |
|
|
| search | `search.ts` | Global full-text search | 178 |
|
|
| pwa_manifest | `pwa_manifest.js` | PWA offline manifest generation | 315 |
|
|
| project_audio_tracks | `project_audio_tracks.ts` | Audio track management | 117 |
|
|
| project_transition_settings | `project_transition_settings.ts` | Environment-aware transition settings | 209 |
|
|
| project_element_defaults | `project_element_defaults.ts` | Element defaults with reset/diff | 34 |
|
|
| global_ui_control_defaults | `global_ui_control_defaults.ts` | Global defaults CRUD service for system controls | 6 |
|
|
| project_ui_control_settings | `project_ui_control_settings.ts` | Transactional find/upsert/delete for project UI-control overrides | 51 |
|
|
| videoProcessing | `videoProcessing.ts` | FFmpeg video reversal for transition videos with single-worker queue, `-threads 1`, hard timeout, metadata logs, and circuit breaker | ~240 |
|
|
|
|
### 3. Specialized Module Directories
|
|
|
|
```
|
|
services/
|
|
├── file/ # Storage providers (Strategy Pattern)
|
|
│ ├── BaseStorageProvider.ts # Abstract interface
|
|
│ ├── S3StorageProvider.ts # AWS S3 implementation
|
|
│ ├── LocalStorageProvider.ts # Local filesystem
|
|
│ ├── UploadSessionManager.ts # Chunked uploads
|
|
│ └── index.js # Provider factory (27 LOC)
|
|
│
|
|
├── email/ # Email sending
|
|
│ ├── index.js # EmailSender class (44 LOC)
|
|
│ └── list/
|
|
│ ├── passwordReset.ts # Password reset email
|
|
│ ├── addressVerification.ts # Email verification
|
|
│ └── invitation.ts # User invitation email
|
|
│
|
|
└── notifications/ # Error handling & i18n
|
|
├── helpers.js # getNotification helper (30 LOC)
|
|
├── list.js # Error message catalog (100 LOC)
|
|
└── errors/
|
|
├── validation.js # ValidationError class (16 LOC)
|
|
└── forbidden.js # ForbiddenError class (16 LOC)
|
|
```
|
|
|
|
---
|
|
|
|
## Service Factory Pattern
|
|
|
|
### `factories/service.factory.ts`
|
|
|
|
Generates standardized service classes with transaction handling.
|
|
|
|
```javascript
|
|
function createEntityService(DBApi, options = {}) {
|
|
const entityName = options.entityName || 'Entity';
|
|
|
|
return class GenericService {
|
|
// Create with transaction
|
|
static async create({ data, currentUser, transaction, runtimeContext }) {
|
|
const transaction = await db.sequelize.transaction();
|
|
try {
|
|
const record = await DBApi.create({ data, currentUser, transaction, runtimeContext });
|
|
await transaction.commit();
|
|
return record;
|
|
} catch (error) {
|
|
await transaction.rollback();
|
|
throw error;
|
|
}
|
|
}
|
|
|
|
// Bulk import from CSV
|
|
static async bulkImport(req, res) { ... }
|
|
|
|
// Update with existence check
|
|
static async update({ id, data, currentUser, transaction, runtimeContext }) { ... }
|
|
|
|
// Delete multiple by IDs
|
|
static async deleteByIds({ ids, currentUser, transaction, runtimeContext }) { ... }
|
|
|
|
// Soft delete single
|
|
static async remove({ id, currentUser, transaction, runtimeContext }) { ... }
|
|
};
|
|
}
|
|
```
|
|
|
|
**Generated Methods:**
|
|
|
|
| Method | Signature | Description |
|
|
|--------|-----------|-------------|
|
|
| `create` | `(data, currentUser) → record` | Create with transaction |
|
|
| `bulkImport` | `(req, res) → void` | CSV import with validation |
|
|
| `update` | `(data, id, currentUser) → record` | Update with existence check |
|
|
| `deleteByIds` | `(ids, currentUser) → void` | Bulk soft delete |
|
|
| `remove` | `(id, currentUser) → void` | Single soft delete |
|
|
|
|
---
|
|
|
|
## Core Services Detail
|
|
|
|
### Tour Pages Service (`tour_pages.ts`)
|
|
|
|
Extends the factory-generated CRUD service with page-specific operations.
|
|
|
|
**Reorder operation:**
|
|
- `TourPagesService.reorder(data, currentUser)` accepts `projectId`,
|
|
`environment`, and `orderedPageIds`.
|
|
- Reordering is allowed only for `environment='dev'`; this preserves the
|
|
publishing model where stage and production are derived environments.
|
|
- The service loads all pages for the project/dev environment inside a
|
|
transaction, verifies that the ordered list contains every page exactly once,
|
|
and then updates `sort_order` sequentially.
|
|
- The operation updates only `sort_order`; it does not change page content,
|
|
slugs, backgrounds, `ui_schema_json`, navigation, transitions, or media.
|
|
- Stage receives the new order after Save to Stage; production receives it after
|
|
Publish.
|
|
|
|
### Auth Service (`auth.ts`)
|
|
|
|
Handles authentication, password management, and email verification.
|
|
|
|
```typescript
|
|
class Auth {
|
|
// User login with bcrypt verification
|
|
static async signin(email, password) → JWT
|
|
|
|
// Send email verification link
|
|
static async sendEmailAddressVerificationEmail(email, host)
|
|
|
|
// Send password reset or invitation email
|
|
static async sendPasswordResetEmail(email, type, host)
|
|
|
|
// Verify email from token
|
|
static async verifyEmail(token, options) → boolean
|
|
|
|
// Update password (requires current password)
|
|
static async passwordUpdate(currentPassword, newPassword, options)
|
|
|
|
// Reset password from token
|
|
static async passwordReset(token, password, options)
|
|
|
|
// Update user profile
|
|
static async updateProfile(data, currentUser)
|
|
}
|
|
```
|
|
|
|
**Security Features:**
|
|
- bcrypt password hashing (`config.bcrypt.saltRounds`)
|
|
- JWT token generation via `helpers.jwtSign()`
|
|
- Email verification required for login
|
|
- Password reset tokens with expiration
|
|
|
|
### Assets Service (`assets.ts`)
|
|
|
|
Extended factory service with strict reusable TypeScript contracts, MIME type validation for asset uploads, embed URL validation, and stored audio/video metadata probing.
|
|
|
|
```typescript
|
|
class AssetsService extends BaseService {
|
|
// Create asset with MIME type validation
|
|
static async create({ data, currentUser, transaction, runtimeContext })
|
|
|
|
// Update asset with MIME type validation
|
|
static async update({ id, data, currentUser, transaction, runtimeContext })
|
|
}
|
|
```
|
|
|
|
**MIME Type Validation:**
|
|
|
|
| Asset Type | Valid MIME Prefixes | Description |
|
|
|------------|---------------------|-------------|
|
|
| `image` | `image/` | JPEG, PNG, GIF, WebP, SVG, etc. |
|
|
| `video` | `video/` | MP4, WebM, MOV, etc. |
|
|
| `audio` | `audio/` | MP3, WAV, OGG, etc. |
|
|
| `embed` | n/a | MIME validation skipped; HTTPS embed URL domain is validated |
|
|
|
|
**Validation Rules:**
|
|
- `asset_type` and `mime_type` must be consistent
|
|
- If `asset_type` is `image`, `mime_type` must start with `image/`
|
|
- If `asset_type` is `video`, `mime_type` must start with `video/`
|
|
- If `asset_type` is `audio`, `mime_type` must start with `audio/`
|
|
- Asset types not in the validation list (e.g., `file`) skip validation
|
|
- Missing `mime_type` is allowed (browser may not always send it)
|
|
|
|
**Error Response:**
|
|
```javascript
|
|
throw new ValidationError(
|
|
`Invalid file type for ${assetType}. Expected ${patterns.description}, got "${mimeType}"`
|
|
);
|
|
```
|
|
|
|
---
|
|
|
|
### File Service (`file.js`)
|
|
|
|
Unified file storage using Strategy Pattern for multiple backends. Features comprehensive error handling,
|
|
AbortController support for client disconnect handling, path validation for security, and structured Pino logging.
|
|
|
|
```javascript
|
|
// Provider auto-detection
|
|
const getFileStorageProvider = () → 's3' | 'gcloud' | 'local'
|
|
|
|
// Core operations (with AbortController support for S3)
|
|
const uploadFile = async (folder, req, res) → { url }
|
|
const downloadFile = async (req, res) → stream // Aborts on client disconnect
|
|
const deleteFile = async (privateUrl, { throwOnError }) → { success, error? }
|
|
|
|
// Server-side file copy (S3 uses CopyObjectCommand, Local uses fs.copyFile)
|
|
const copyFile = async (sourceKey, destKey, options) → { url } | { key }
|
|
const copyFilesParallel = async (copies, options) → { succeeded, failed }
|
|
|
|
// Chunked upload session management
|
|
const initUploadSession = async (req, res) → { sessionId, totalChunks }
|
|
const getUploadSession = async (req, res) → { status, uploadedChunks }
|
|
const uploadChunk = async (req, res) → { chunkIndex, uploadedChunks }
|
|
const finalizeUploadSession = async (req, res) → { url, privateUrl }
|
|
|
|
// Presigned URL generation (S3 only, with path validation)
|
|
const generatePresignedUrls = async (urls) → { [url]: presignedUrl }
|
|
|
|
// Utilities (exported for route layer)
|
|
const isValidPath = (urlPath) → boolean // Path traversal protection
|
|
const createErrorResponse = (message, code, details) → { message, code?, details? }
|
|
const getS3ErrorStatusCode = (error) → number // HTTP status code mapping
|
|
```
|
|
|
|
**Server-Side File Copy (S3 Native):**
|
|
|
|
The `copyFile()` function uses provider-native copy operations for optimal performance:
|
|
|
|
| Provider | Implementation | Performance |
|
|
|----------|----------------|-------------|
|
|
| S3 | `CopyObjectCommand` (server-side) | 15x faster, zero memory |
|
|
| Local | `fs.promises.copyFile` | Kernel-level copy |
|
|
| GCloud | Download + Upload (fallback) | Legacy behavior |
|
|
|
|
```javascript
|
|
// Single file copy
|
|
const copyFile = async (sourceKey, destKey, { contentType }) → { url }
|
|
|
|
// Parallel batch copy with concurrency control
|
|
const copyFilesParallel = async (copies, { concurrency = 10, continueOnError = true })
|
|
→ { succeeded: [{ sourceKey, destKey }], failed: [{ sourceKey, error }] }
|
|
```
|
|
|
|
**Benefits over download-then-upload:**
|
|
- **15x faster**: Server-side copy, no data through backend
|
|
- **Zero memory**: No file buffering in Node.js
|
|
- **No timeouts**: Works for large files (>100MB)
|
|
- **Reduced bandwidth**: No double network transfer
|
|
|
|
**Error Response Format:**
|
|
```javascript
|
|
// Standardized across all file endpoints
|
|
{
|
|
message: 'Human-readable error message',
|
|
code: 'ERROR_CODE', // For programmatic handling
|
|
details: { ... } // Optional additional context
|
|
}
|
|
```
|
|
|
|
**Request Cancellation:**
|
|
Downloads automatically abort S3 requests when client disconnects, preventing wasted bandwidth and server resources.
|
|
|
|
**Provider Selection:**
|
|
|
|
| Priority | Provider | Detection |
|
|
|----------|----------|-----------|
|
|
| 1 | `config.fileStorage.provider` | Validated `FILE_STORAGE_PROVIDER` override |
|
|
| 2 | S3 | `S3_BUCKET` + `S3_REGION` + credentials |
|
|
| 3 | GCloud | Validated `GC_PROJECT_ID` + `GC_CLIENT_EMAIL` + `GC_PRIVATE_KEY` |
|
|
| 4 | Local | Default fallback |
|
|
|
|
External S3/GCloud operations used by processing paths are protected by the
|
|
shared file-storage circuit breaker. For S3, only retryable SDK/network errors
|
|
count toward opening the breaker; expected 4xx errors do not.
|
|
|
|
**Chunked Upload Flow:**
|
|
|
|
```
|
|
1. POST /api/file/upload-sessions/init
|
|
← { sessionId, totalChunks }
|
|
|
|
2. PUT /api/file/upload-sessions/:sessionId/chunks/:chunkIndex
|
|
← { uploadedChunks: N }
|
|
|
|
3. POST /api/file/upload-sessions/:sessionId/finalize
|
|
← { url, privateUrl }
|
|
```
|
|
|
|
**S3 Local Cache (Atomic Writes):**
|
|
|
|
When S3 is the storage provider, downloads are cached locally to reduce S3 requests and improve latency. The cache uses **atomic writes** to prevent race conditions:
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ Request A Request B (concurrent) │
|
|
├─────────────────────────────────────────────────────────────────┤
|
|
│ 1. Check cache → miss 1. Check cache → miss │
|
|
│ 2. Create .downloading 2. See .downloading → skip cache │
|
|
│ 3. Stream to .tmp file 3. Stream directly from S3 │
|
|
│ 4. Verify size matches 4. Complete │
|
|
│ 5. Rename .tmp → final ↓ │
|
|
│ 6. Delete .downloading Request C (later) │
|
|
│ ↓ 1. Check cache → hit │
|
|
│ 2. Serve from cache │
|
|
└─────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
**Cache validation:**
|
|
- File exists AND age < `S3_CACHE_MAX_AGE` (default: 1 hour)
|
|
- No `.downloading` marker file (indicates download in progress)
|
|
- Size matches `Content-Length` header (verified before rename)
|
|
|
|
**Why atomic writes:**
|
|
Without atomic writes, concurrent requests could serve truncated cache files:
|
|
1. Request A starts downloading 12MB file, writes to cache
|
|
2. After 2MB written, Request B checks cache - file exists, age valid
|
|
3. Request B serves 2MB truncated file → **corrupted video/image**
|
|
|
|
The atomic write pattern (write to `.tmp`, verify size, rename) prevents this.
|
|
|
|
### Publish Service (`publish.ts`)
|
|
|
|
Three-tier content environment publishing workflow.
|
|
|
|
```javascript
|
|
module.exports = class PublishService {
|
|
// Acquire project lock and run callback
|
|
static async withProjectPublishLock(projectId, callback)
|
|
|
|
// Copy stage content to production (blocking)
|
|
static async publishToProduction(projectId, currentUser, title, description)
|
|
|
|
// Copy dev content to stage (non-blocking, returns immediately)
|
|
static async saveToStage(projectId, currentUser)
|
|
|
|
// Generic environment copy
|
|
static async copyEnvironment(projectId, fromEnv, toEnv, currentUser, transaction)
|
|
}
|
|
```
|
|
|
|
**Non-Blocking vs Blocking:**
|
|
- `saveToStage()` - Returns immediately, copy runs in background via `setImmediate()`
|
|
- `publishToProduction()` - Waits for entire copy operation before returning
|
|
|
|
**Publishing Flow:**
|
|
|
|
```
|
|
┌─────────┐ ┌─────────┐ ┌────────────┐
|
|
│ DEV │──────▶│ STAGE │──────▶│ PRODUCTION │
|
|
│(editing)│ │(preview)│ │ (public) │
|
|
└─────────┘ └─────────┘ └────────────┘
|
|
│ │ │
|
|
saveToStage() publishToProduction() │
|
|
│ │ │
|
|
└──────────────────┴────────────────────┘
|
|
Creates publish_event record
|
|
```
|
|
|
|
**Event Status Lifecycle:**
|
|
1. `queued` - Event created
|
|
2. `running` - Processing started
|
|
3. `success` / `failed` - Completed
|
|
|
|
### Search Service (`search.ts`)
|
|
|
|
Global full-text search with permission filtering.
|
|
|
|
```typescript
|
|
export default class SearchService {
|
|
// Search across all permitted entities
|
|
static async search(searchQuery: string, currentUser: CurrentUser | undefined): Promise<SearchResult>
|
|
}
|
|
```
|
|
|
|
**Searchable Tables:**
|
|
|
|
| Table | Text Fields | Numeric Fields |
|
|
|-------|-------------|----------------|
|
|
| users | firstName, lastName, phoneNumber, email | - |
|
|
| projects | name, slug, description, logo_url, favicon_url, og_image_url | - |
|
|
| assets | name, cdn_url, storage_key, mime_type, checksum | size_mb, width_px, height_px, duration_sec |
|
|
| asset_variants | cdn_url | width_px, height_px, size_mb |
|
|
| presigned_url_requests | requested_key, mime_type, status | requested_size_mb |
|
|
| tour_pages | source_key, name, slug, background_image_url, background_video_url, background_audio_url, ui_schema_json | sort_order |
|
|
| project_audio_tracks | source_key, name, slug, url | volume, sort_order |
|
|
| publish_events | error_message | pages_copied, transitions_copied, audios_copied |
|
|
| pwa_caches | cache_version, manifest_json, asset_list_json | - |
|
|
| access_logs | path, ip_address, user_agent | - |
|
|
|
|
**Permission Check:**
|
|
```javascript
|
|
// Only search tables user has READ permission for
|
|
if (!hasPermission(permissionSet, `READ_${tableName.toUpperCase()}`)) {
|
|
return [];
|
|
}
|
|
```
|
|
|
|
### Projects Service (`projects.ts`)
|
|
|
|
Project management with cloning capabilities and slug uniqueness validation.
|
|
|
|
```javascript
|
|
export default class ProjectsService {
|
|
// Normalize slug for URL safety
|
|
static normalizeSlug(value) → slug
|
|
|
|
// Generate unique slug with -copy suffix
|
|
static async generateUniqueSlug(baseSlug, transaction) → uniqueSlug
|
|
|
|
// Validate slug uniqueness before create/update (throws ValidationError if duplicate)
|
|
static async validateSlugUniqueness(slug, excludeId, transaction) → normalizedSlug
|
|
|
|
// Create new project (validates slug uniqueness)
|
|
static async create({ data, currentUser, transaction, runtimeContext }) → project
|
|
|
|
// Clone project with all assets and variants
|
|
static async cloneFromProject(sourceProjectId, currentUser) → clonedProject
|
|
|
|
// Update project (validates slug uniqueness if changed)
|
|
static async update({ id, data, currentUser, transaction, runtimeContext }) → project
|
|
}
|
|
```
|
|
|
|
**Slug Validation:**
|
|
- `validateSlugUniqueness()` normalizes the slug and checks for duplicates
|
|
- Uses `excludeId` parameter to skip the current project during updates
|
|
- Checks soft-deleted projects (`paranoid: false`) to prevent conflicts
|
|
- Throws `ValidationError('iam.errors.slugAlreadyExists')` if duplicate found
|
|
|
|
**Clone Process (Optimized with S3 Native Copy):**
|
|
|
|
The clone process uses S3's native `CopyObjectCommand` for server-side file copying (15x faster than download-then-upload). Files are copied in parallel with configurable concurrency.
|
|
|
|
```
|
|
Phase A: Create cloned project record
|
|
↓
|
|
Phase B: Collect all copy operations (assets + non-reversed variants)
|
|
↓
|
|
Phase C: Execute parallel S3 copy (10 concurrent, continueOnError=true)
|
|
↓
|
|
Phase D: Build assetPathMap from copy results (failed → use original path)
|
|
↓
|
|
Phase E: Create asset/variant records, build assetIdMap (old → new asset IDs)
|
|
↓
|
|
Phase F: Copy reversed videos using asset ID mapping
|
|
└── Reversed videos use pattern: assets/{assetId}/reversed.mp4
|
|
└── Copy from old asset ID path to new asset ID path
|
|
↓
|
|
Phase G: Clone tour_pages, audio_tracks, element_defaults
|
|
└── Transform ui_schema_json asset paths using assetPathMap
|
|
```
|
|
|
|
**Key Implementation Details:**
|
|
|
|
| Phase | Operation | Notes |
|
|
|-------|-----------|-------|
|
|
| B-C | Parallel file copy | Uses `FileService.copyFilesParallel()` with S3 `CopyObjectCommand` |
|
|
| E | Asset ID mapping | Tracks `oldAssetId → newAssetId` for reversed video copying |
|
|
| F | Reversed video copy | Separate phase because reversed videos use asset-ID-based paths, not project-ID-based |
|
|
| G | Path transformation | `transformUiSchemaAssetPaths()` updates all asset URLs in `ui_schema_json` |
|
|
|
|
**Reversed Video Storage Pattern:**
|
|
- Primary assets: `assets/{projectId}/{uuid}.ext`
|
|
- Reversed videos: `assets/{assetId}/reversed.mp4` (uses asset ID, not project ID)
|
|
|
|
**Error Handling:**
|
|
- Failed file copies fall back to original storage path (cloned project still functional, shares assets with source)
|
|
- Most assets won't have reversed videos - this is expected (only navigation elements with transitions generate them)
|
|
- Transaction rollback on DB errors; orphaned S3 files acceptable (can be cleaned later)
|
|
|
|
### Roles Service (`roles.ts`)
|
|
|
|
Role management service for standard CRUD, CSV bulk import, and permission assignment. The service wraps DB writes in transactions when a caller does not provide one.
|
|
|
|
```typescript
|
|
export default class RolesService {
|
|
static assertPublicRoleHasNoPermissions(data, existingRole)
|
|
static async create(options)
|
|
static async bulkImport(req, res)
|
|
static async update(options)
|
|
static async deleteByIds(options)
|
|
static async remove(options)
|
|
}
|
|
```
|
|
|
|
**Public Role Hardening:**
|
|
- Creating or updating a role named `Public` rejects non-empty permissions.
|
|
- Existing role lookup is done before update so renaming a role to `Public` cannot retain assigned permissions through the service boundary.
|
|
- This keeps customer viewer access separate from admin RBAC permissions.
|
|
|
|
~~Generates offline manifests for PWA asset downloads.~~
|
|
|
|
### Project Element Defaults Service (`project_element_defaults.ts`)
|
|
|
|
Extended factory service with custom methods for managing project-level element defaults.
|
|
|
|
```javascript
|
|
class Project_element_defaultsService extends BaseService {
|
|
// Reset project default to current global default
|
|
static async resetToGlobal(id, options) → updated record
|
|
|
|
// Get diff between project default and global default
|
|
static async getDiffFromGlobal(id) → { hasChanges, diff }
|
|
|
|
// Snapshot all global defaults to a project (called on project creation)
|
|
static async snapshotGlobalDefaults(projectId, options) → created records
|
|
}
|
|
```
|
|
|
|
**Methods:**
|
|
|
|
| Method | Description |
|
|
|--------|-------------|
|
|
| `resetToGlobal(id, options)` | Resets a project element default to match the current global element type default |
|
|
| `getDiffFromGlobal(id)` | Compares project element default with global default, returns differences |
|
|
| `snapshotGlobalDefaults(projectId, options)` | Creates project element defaults by copying all global element type defaults |
|
|
|
|
**Use Cases:**
|
|
- **Project Creation:** `snapshotGlobalDefaults` is called to copy global defaults to new project
|
|
- **Reset to Global:** User can reset customized project defaults back to global values
|
|
- **Diff View:** UI can show which settings differ from global defaults
|
|
|
|
---
|
|
|
|
## File Storage Module (`services/file/`)
|
|
|
|
### Strategy Pattern Implementation
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────────┐
|
|
│ BaseStorageProvider │
|
|
│ (Abstract Interface) │
|
|
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌───────────────┐ │
|
|
│ │ upload │ │ download │ │ delete │ │ getSignedUrl │ │
|
|
│ │ (abstract) │ │ (abstract) │ │ (abstract) │ │ (optional) │ │
|
|
│ └─────────────┘ └─────────────┘ └─────────────┘ └───────────────┘ │
|
|
└─────────────────────────────────────────────────────────────────────┘
|
|
△ △ △
|
|
│ │ │
|
|
┌──────────┴─────────┐ ┌───────┴────────┐ ┌────────┴───────┐
|
|
│ S3StorageProvider │ │LocalStorage │ │(GCloud inline) │
|
|
│ │ │Provider │ │ │
|
|
│ • AWS SDK v3 │ │ • fs module │ │ • @google-cloud│
|
|
│ • Presigned URLs │ │ • MIME types │ │ /storage │
|
|
│ • Batch delete │ │ • Recursive │ │ • Resumable │
|
|
└────────────────────┘ └────────────────┘ └────────────────┘
|
|
```
|
|
|
|
### BaseStorageProvider Interface
|
|
|
|
```typescript
|
|
export default class BaseStorageProvider {
|
|
static get providerName(): string
|
|
upload(key, data, options): Promise<StorageUploadResult>
|
|
download(key): Promise<StorageDownloadResult>
|
|
delete(key): Promise<void>
|
|
deleteMany(keys): Promise<void>
|
|
exists(key): Promise<boolean>
|
|
list(prefix): Promise<string[]>
|
|
getSignedUrl(key, expiresIn): Promise<string | null>
|
|
}
|
|
```
|
|
|
|
### S3StorageProvider
|
|
|
|
AWS S3 implementation using SDK v3 with robust timeout, retry, and error handling.
|
|
|
|
`S3StorageProvider.ts` uses official AWS SDK v3 types for client config, command inputs, signed URL generation, streaming download bodies, and `S3ServiceException` metadata. Project-specific storage contracts live in `backend/src/types/file.ts`.
|
|
|
|
```javascript
|
|
class S3StorageProvider extends BaseStorageProvider {
|
|
constructor({
|
|
bucket, region, accessKeyId, secretAccessKey, prefix,
|
|
connectionTimeout, // Default: 5000ms
|
|
requestTimeout, // Default: 30000ms
|
|
maxAttempts, // Default: 3 (adaptive retry)
|
|
maxSockets, // Default: 50 (connection pool)
|
|
keepAlive // Default: true
|
|
})
|
|
|
|
// Static methods for error handling
|
|
static getErrorStatusCode(error) // Map S3 errors to HTTP status codes
|
|
static isRetryableError(error) // Check if error should be retried
|
|
|
|
// Instance methods (all support AbortSignal for cancellation)
|
|
buildKey(key) // Add prefix to key
|
|
async upload(key, data, { signal }) // PutObjectCommand
|
|
async download(key, { signal }) // GetObjectCommand
|
|
async copy(sourceKey, destKey, { signal, contentType }) // CopyObjectCommand (server-side)
|
|
async delete(key, { signal }) // DeleteObjectCommand
|
|
async deleteMany(keys, { signal }) // DeleteObjectsCommand (batched 1000)
|
|
async exists(key, { signal }) // HeadObjectCommand
|
|
async list(prefix, { signal }) // ListObjectsV2Command (paginated)
|
|
async getSignedUrl(key, expiry) // @aws-sdk/s3-request-presigner
|
|
getConfig() // Get provider config for debugging
|
|
destroy() // Cleanup connection pool
|
|
}
|
|
```
|
|
|
|
**S3 Error to HTTP Status Code Mapping:**
|
|
|
|
| S3 Error | HTTP Status |
|
|
|----------|-------------|
|
|
| NoSuchKey, NotFound, NoSuchBucket | 404 |
|
|
| AccessDenied, InvalidAccessKeyId | 403 |
|
|
| ExpiredToken | 401 |
|
|
| TimeoutError, RequestTimeout | 504 |
|
|
| NetworkingError, ServiceUnavailable | 503 |
|
|
| ThrottlingException | 429 |
|
|
| InternalError | 500 |
|
|
|
|
### LocalStorageProvider
|
|
|
|
Local filesystem implementation.
|
|
|
|
`LocalStorageProvider.ts` shares the same typed storage contracts as S3 while using Node filesystem and stream APIs.
|
|
|
|
```javascript
|
|
class LocalStorageProvider extends BaseStorageProvider {
|
|
constructor({ basePath = './uploads' })
|
|
|
|
buildPath(key) // path.join(basePath, key)
|
|
async upload(key, data) // fs.writeFileSync / stream.pipeline
|
|
async download(key) // fs.createReadStream
|
|
async copy(sourceKey, destKey) // fs.promises.copyFile (kernel-level)
|
|
async delete(key) // fs.unlinkSync
|
|
async exists(key) // fs.existsSync
|
|
async list(prefix) // fs.readdirSync (recursive)
|
|
getContentType(ext) // MIME type mapping
|
|
}
|
|
```
|
|
|
|
### UploadSessionManager
|
|
|
|
Chunked upload session management for large files.
|
|
|
|
`UploadSessionManager.ts` uses reusable upload-session metadata contracts from `backend/src/types/file.ts` and validates `meta.json` after parsing instead of relying on type assertions.
|
|
|
|
```javascript
|
|
class UploadSessionManager {
|
|
constructor({ sessionDir, ttlMs = 24h })
|
|
|
|
createSession(options) // → sessionId (UUID)
|
|
readMeta(sessionId) // → session metadata
|
|
writeMeta(sessionId, payload) // Save session state
|
|
saveChunk(sessionId, index, data) // Save chunk file
|
|
chunkExists(sessionId, index) // Check chunk exists
|
|
isComplete(sessionId) // All chunks uploaded?
|
|
assembleChunks(sessionId, path) // Combine chunks
|
|
removeSession(sessionId) // Cleanup session
|
|
cleanupExpiredSessions() // Remove stale sessions
|
|
}
|
|
```
|
|
|
|
**Session Directory Structure:**
|
|
```
|
|
upload_sessions/
|
|
└── {sessionId}/
|
|
├── meta.json # Session metadata
|
|
└── chunks/
|
|
├── 0.part
|
|
├── 1.part
|
|
└── N.part
|
|
```
|
|
|
|
---
|
|
|
|
## Email Module (`services/email/`)
|
|
|
|
### EmailSender Class
|
|
|
|
Core email sending using Nodemailer.
|
|
|
|
```typescript
|
|
export default class EmailSender {
|
|
constructor(email: EmailTemplate)
|
|
async send(): Promise<EmailSendResult>
|
|
static get isConfigured(): boolean
|
|
get transportConfig(): SMTPTransport.Options
|
|
get from(): string
|
|
}
|
|
```
|
|
|
|
**Configuration (`config.email`):**
|
|
```javascript
|
|
{
|
|
host: 'email-smtp.us-east-1.amazonaws.com',
|
|
port: 587,
|
|
auth: {
|
|
user: process.env.EMAIL_USER || '',
|
|
pass: process.env.EMAIL_PASS || ''
|
|
},
|
|
from: 'Tour Builder Platform <app@flatlogic.app>'
|
|
}
|
|
```
|
|
|
|
### Email Templates
|
|
|
|
| Template | Class | Fields |
|
|
|----------|-------|--------|
|
|
| Password Reset | `PasswordResetEmail` | to, link |
|
|
| Email Verification | `EmailAddressVerificationEmail` | to, link |
|
|
| User Invitation | `InvitationEmail` | to, host |
|
|
|
|
**Template Pattern:**
|
|
```typescript
|
|
export default class PasswordResetEmail implements EmailTemplate {
|
|
constructor({ to, link }: LinkEmailTemplateOptions) {}
|
|
|
|
get subject(): string {
|
|
return getNotification('emails.passwordReset.subject', getNotification('app.title'));
|
|
}
|
|
|
|
async html(): Promise<string> {
|
|
const template = await fs.readFile(templatePath, 'utf8');
|
|
return template
|
|
.replace(/{appTitle}/g, appTitle)
|
|
.replace(/{resetUrl}/g, this.link)
|
|
.replace(/{accountName}/g, this.to);
|
|
}
|
|
};
|
|
```
|
|
|
|
---
|
|
|
|
## Notifications Module (`services/notifications/`)
|
|
|
|
### Error Classes
|
|
|
|
**ValidationError (400 Bad Request):**
|
|
```javascript
|
|
class ValidationError extends Error {
|
|
constructor(messageCode) {
|
|
const message = isNotification(messageCode)
|
|
? getNotification(messageCode)
|
|
: getNotification('errors.validation.message');
|
|
super(message);
|
|
this.code = 400;
|
|
}
|
|
}
|
|
```
|
|
|
|
**ForbiddenError (403 Forbidden):**
|
|
```javascript
|
|
class ForbiddenError extends Error {
|
|
constructor(messageCode) {
|
|
const message = isNotification(messageCode)
|
|
? getNotification(messageCode)
|
|
: getNotification('errors.forbidden.message');
|
|
super(message);
|
|
this.code = 403;
|
|
}
|
|
}
|
|
```
|
|
|
|
### Notification Catalog (`list.js`)
|
|
|
|
Centralized error messages and i18n strings.
|
|
|
|
```javascript
|
|
const errors = {
|
|
app: {
|
|
title: 'Tour Builder Platform',
|
|
},
|
|
auth: {
|
|
userDisabled: 'Your account is disabled',
|
|
forbidden: 'Forbidden',
|
|
unauthorized: 'Unauthorized',
|
|
userNotFound: "Sorry, we don't recognize your credentials",
|
|
wrongPassword: "Sorry, we don't recognize your credentials",
|
|
// ...
|
|
},
|
|
iam: {
|
|
errors: {
|
|
userAlreadyExists: 'User with this email already exists',
|
|
userNotFound: 'User not found',
|
|
// ...
|
|
}
|
|
},
|
|
emails: {
|
|
invitation: {
|
|
subject: "You've been invited to {0}",
|
|
body: "..."
|
|
},
|
|
// ...
|
|
}
|
|
};
|
|
```
|
|
|
|
### Helper Functions
|
|
|
|
```javascript
|
|
// Get notification with parameter substitution
|
|
getNotification('emails.invitation.subject', 'Tour Builder')
|
|
// → "You've been invited to Tour Builder"
|
|
|
|
// Check if key exists in catalog
|
|
isNotification('auth.userNotFound') // → true
|
|
isNotification('custom.message') // → false
|
|
```
|
|
|
|
---
|
|
|
|
## Transaction Patterns
|
|
|
|
### Standard Transaction Pattern
|
|
|
|
All services use consistent transaction handling:
|
|
|
|
```javascript
|
|
static async create({ data, currentUser, transaction: externalTransaction, runtimeContext }) {
|
|
const transaction = externalTransaction || await db.sequelize.transaction();
|
|
const ownsTransaction = !externalTransaction;
|
|
try {
|
|
const record = await DBApi.create({ data, currentUser, transaction, runtimeContext });
|
|
if (ownsTransaction) await transaction.commit();
|
|
return record;
|
|
} catch (error) {
|
|
if (ownsTransaction) await transaction.rollback();
|
|
throw error;
|
|
}
|
|
}
|
|
```
|
|
|
|
### Lock Pattern (Publish Service)
|
|
|
|
```javascript
|
|
static async withProjectPublishLock(projectId, callback) {
|
|
return db.sequelize.transaction(async (transaction) => {
|
|
// Acquire row lock
|
|
const project = await db.projects.findByPk(projectId, {
|
|
transaction,
|
|
lock: transaction.LOCK.UPDATE,
|
|
});
|
|
|
|
// Check for concurrent operations
|
|
const runningEvent = await db.publish_events.findOne({
|
|
where: { projectId, status: EVENT_STATUS.RUNNING },
|
|
transaction,
|
|
lock: transaction.LOCK.UPDATE,
|
|
});
|
|
|
|
if (runningEvent) {
|
|
throw new Error('Publish is already running for this project');
|
|
}
|
|
|
|
return callback(transaction);
|
|
});
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## Dependencies
|
|
|
|
### External Packages
|
|
|
|
| Package | Usage |
|
|
|---------|-------|
|
|
| `bcrypt` | Password hashing |
|
|
| `nodemailer` | Email sending |
|
|
| `csv-parser` | CSV import parsing |
|
|
| `axios` | External API calls (widgets) |
|
|
| `uuid` | Upload session IDs |
|
|
| `@aws-sdk/client-s3` | S3 operations |
|
|
| `@aws-sdk/s3-request-presigner` | Presigned URLs |
|
|
| `@google-cloud/storage` | GCloud storage |
|
|
| `lodash/get` | Deep object access |
|
|
|
|
### Internal Dependencies
|
|
|
|
| Module | Services Using |
|
|
|--------|---------------|
|
|
| `db/models` | All services (transaction) |
|
|
| `db/api/*` | All entity services |
|
|
| `factories/service.factory` | 9 factory services |
|
|
| `config` | auth, file, users, roles |
|
|
| `helpers` | auth (jwtSign) |
|
|
|
|
---
|
|
|
|
## Service Relationships
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────────┐
|
|
│ Service Graph │
|
|
│ │
|
|
│ ┌──────────┐ │
|
|
│ │ auth │◄────────────────────┐ │
|
|
│ └────┬─────┘ │ │
|
|
│ │ uses │ sends invitations │
|
|
│ ▼ │ │
|
|
│ ┌──────────┐ ┌───────────┐ │ │
|
|
│ │ users │────▶│ email │───┘ │
|
|
│ │ (DBApi) │ └───────────┘ │
|
|
│ └──────────┘ │ │
|
|
│ │ uses templates │
|
|
│ ▼ │
|
|
│ ┌─────────────────┐ │
|
|
│ │ notifications │ │
|
|
│ │ (error catalog) │ │
|
|
│ └─────────────────┘ │
|
|
│ ▲ │
|
|
│ │ throws errors │
|
|
│ │ │
|
|
│ ┌──────────┐ ┌─────┴─────┐ ┌──────────┐ │
|
|
│ │ projects │ │ publish │ │ search │ │
|
|
│ │ (clone) │ │ (env copy)│ │(fulltext)│ │
|
|
│ └──────────┘ └───────────┘ └──────────┘ │
|
|
│ │
|
|
│ ┌──────────────────────────────────────────────────────────────┐ │
|
|
│ │ File Service │ │
|
|
│ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────────────┐ │ │
|
|
│ │ │ S3Provider │ │LocalProvider │ │UploadSessionManager │ │ │
|
|
│ │ └──────────────┘ └──────────────┘ └──────────────────────┘ │ │
|
|
│ └──────────────────────────────────────────────────────────────┘ │
|
|
└─────────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
---
|
|
|
|
## Configuration
|
|
|
|
### Environment Variables
|
|
|
|
| Variable | Service | Description |
|
|
|----------|---------|-------------|
|
|
| `FILE_STORAGE_PROVIDER` | file | Force provider ('s3', 'gcloud', 'local') |
|
|
| `AWS_S3_BUCKET` | file | S3 bucket name |
|
|
| `AWS_S3_REGION` | file | AWS region (default: us-east-1) |
|
|
| `AWS_ACCESS_KEY_ID` | file | AWS access key |
|
|
| `AWS_SECRET_ACCESS_KEY` | file | AWS secret key |
|
|
| `AWS_S3_PREFIX` | file | Key prefix |
|
|
| `AWS_S3_CONNECTION_TIMEOUT` | file | S3 connection timeout in ms (default: 5000) |
|
|
| `AWS_S3_REQUEST_TIMEOUT` | file | S3 request timeout in ms (default: 30000) |
|
|
| `AWS_S3_MAX_ATTEMPTS` | file | S3 retry attempts (default: 3) |
|
|
| `AWS_S3_MAX_SOCKETS` | file | S3 connection pool size (default: 50) |
|
|
| `AWS_S3_KEEP_ALIVE` | file | Enable HTTP keep-alive (default: true) |
|
|
| `AWS_S3_PRESIGN_EXPIRY` | file | Presigned URL expiry in seconds (default: 3600) |
|
|
| `GC_PROJECT_ID` | file | GCloud project |
|
|
| `GC_CLIENT_EMAIL` | file | GCloud service account |
|
|
| `GC_PRIVATE_KEY` | file | GCloud private key |
|
|
| `FFMPEG_REVERSE_TIMEOUT_MS` | videoProcessing | Reverse-video hard timeout in ms (default: 600000) |
|
|
| `FFPROBE_TIMEOUT_MS` | videoProcessing | Metadata probe timeout in ms (default: 30000) |
|
|
| `FFMPEG_BREAKER_FAILURE_THRESHOLD` | videoProcessing | Failures before FFmpeg breaker opens (default: 3) |
|
|
| `FFMPEG_BREAKER_COOLDOWN_MS` | videoProcessing | FFmpeg breaker cooldown in ms (default: 120000) |
|
|
| `FFMPEG_BREAKER_SUCCESS_THRESHOLD` | videoProcessing | Half-open successes required to close FFmpeg breaker (default: 1) |
|
|
| `FILE_STORAGE_BREAKER_FAILURE_THRESHOLD` | file | Failures before storage breaker opens (default: 5) |
|
|
| `FILE_STORAGE_BREAKER_COOLDOWN_MS` | file | Storage breaker cooldown in ms (default: 30000) |
|
|
| `FILE_STORAGE_BREAKER_SUCCESS_THRESHOLD` | file | Half-open successes required to close storage breaker (default: 2) |
|
|
| `EMAIL_USER` | email | SMTP username |
|
|
| `EMAIL_PASS` | email | SMTP password |
|
|
|
|
### Config References
|
|
|
|
```javascript
|
|
// config.ts
|
|
module.exports = {
|
|
bcrypt: { saltRounds: 12 },
|
|
email: { host, port, auth, from },
|
|
s3: {
|
|
bucket,
|
|
region,
|
|
accessKeyId,
|
|
secretAccessKey,
|
|
prefix,
|
|
// Timeout/retry configuration
|
|
connectionTimeout, // 5000ms default
|
|
requestTimeout, // 30000ms default
|
|
maxAttempts, // 3 retries with adaptive backoff
|
|
// Connection pool
|
|
maxSockets, // 50 concurrent connections
|
|
keepAlive, // HTTP keep-alive enabled
|
|
// Presigned URLs
|
|
presignExpirySeconds, // 3600 (1 hour) default
|
|
},
|
|
gcloud: { bucket, hash },
|
|
uploadDir: './uploads',
|
|
flHost: 'https://flatlogic.host', // Widget service
|
|
project_uuid: '...',
|
|
};
|
|
```
|
|
|
|
---
|
|
|
|
## Testing Guidelines
|
|
|
|
### Unit Testing Services
|
|
|
|
```javascript
|
|
// Mock transaction
|
|
jest.mock('../db/models', () => ({
|
|
sequelize: {
|
|
transaction: jest.fn(() => ({
|
|
commit: jest.fn(),
|
|
rollback: jest.fn(),
|
|
})),
|
|
},
|
|
}));
|
|
|
|
// Mock DB API
|
|
jest.mock('../db/api/assets');
|
|
|
|
describe('AssetsService', () => {
|
|
it('should create asset with transaction', async () => {
|
|
const result = await AssetsService.create({
|
|
data: mockData,
|
|
currentUser: mockUser,
|
|
});
|
|
expect(transaction.commit).toHaveBeenCalled();
|
|
});
|
|
});
|
|
```
|
|
|
|
### Integration Testing
|
|
|
|
```javascript
|
|
describe('PublishService', () => {
|
|
it('should copy dev to stage', async () => {
|
|
// Create test project with pages
|
|
const project = await createTestProject();
|
|
await createTestPages(project.id, 'dev');
|
|
|
|
// Publish to stage
|
|
const result = await PublishService.saveToStage(project.id, mockUser);
|
|
|
|
// Verify stage pages created
|
|
const stagePages = await findPages(project.id, 'stage');
|
|
expect(stagePages.length).toBe(result.summary.pages_copied);
|
|
});
|
|
});
|
|
```
|
|
|
|
---
|
|
|
|
## Best Practices
|
|
|
|
### 1. Transaction Handling
|
|
- Always wrap multi-step operations in transactions
|
|
- Use try/catch/rollback pattern consistently
|
|
- Pass transaction to all DB API calls
|
|
|
|
### 2. Error Handling
|
|
- Use `ValidationError` for client errors (400)
|
|
- Use `ForbiddenError` for authorization failures (403)
|
|
- Use notification catalog for consistent messages
|
|
|
|
### 3. Service Design
|
|
- Keep services focused on business logic
|
|
- Delegate DB operations to DB API layer
|
|
- Use factories for simple CRUD services
|
|
|
|
### 4. File Operations
|
|
- Use Strategy Pattern for multi-provider support
|
|
- Implement chunked uploads for large files
|
|
- Clean up sessions on completion/failure
|
|
|
|
---
|
|
|
|
## Related Documentation
|
|
|
|
- [Backend Architecture](../backend-architecture.md) - Overall backend design
|
|
- [Database Schema](../database-schema.md) - Data models
|
|
- [API Endpoints](../api-endpoints.md) - REST API reference
|
|
- [Auth Module](./auth.md) - Authentication details
|
|
- [Routes Module](./routes.md) - Route implementations
|