diff --git a/.gitignore b/.gitignore index 0ff3a2c..406f4a4 100644 --- a/.gitignore +++ b/.gitignore @@ -11,7 +11,6 @@ frontend/public/sw.js frontend/next-env.d.ts package-lock.json !backend/package-lock.json -AGENTS.md .codex/ !frontend/package-lock.json diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..6f9622b --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,447 @@ +# AGENTS.md + +This file provides guidance to CODEX when working with code in this repository. + +**MAIN RULE:** DON'T MADE UP ANYTHING!!! IF YOU NOT SURE - DOUBLECHECK IT VIA PROJECT DOCUMENTATION, TOOL DOCUMENTATION, APPROPRIATE MCP, WEB SEARCH OR JUST ASK FOR ME TO CLARIFY. + +## Project Overview + +Tour Builder Platform - a web application for building and managing interactive virtual tours. Built with: +- **Frontend**: Next.js 15 with React 19, TypeScript, Redux Toolkit, Tailwind CSS +- **Backend**: Node.js/Express with Sequelize ORM +- **Database**: PostgreSQL + +## System Requirements + +### Required Software +- **Node.js** 24.x for backend runtime and TypeScript migration work +- **PostgreSQL** 14+ + +### FFmpeg + +FFmpeg is bundled with the backend via `ffmpeg-static` and `ffprobe-static` npm packages. No manual installation is required. + +**How it works:** +- Pre-compiled binaries are downloaded during `npm install` +- `fluent-ffmpeg` is configured to use bundled binaries in `backend/src/services/videoProcessing.js` +- Works across all platforms (Linux, macOS, Windows) and Docker environments + +**Supported use case:** +- Reversed video generation for back navigation transitions + +## Important Rules + +- **Never change global app configs casually** - Do not modify `.env` or database configuration without verifying the target environment. These configs are shared across environments and incorrect changes can break production. +- **Centralize environment variables** - Do not read new app/runtime environment variables directly from services, routes, components, hooks, or feature modules. Backend app env vars must be added to `backend/src/utils/env-validation.ts`, typed in `backend/src/types/env.ts` / `backend/src/types/config.ts`, exposed through `backend/src/config.ts`, and consumed via `config`. Frontend public env vars must be centralized in `frontend/src/config.ts`. Direct `process.env` access is acceptable only in bootstrap/config entrypoints (`load-env`, logger bootstrap, DB/Umzug config, Next config), scripts, migrations, seeders, and tests. +- **Backend async handlers** must be wrapped with `wrapAsync` helper for error propagation. +- **Use Passport JWT** for protected routes: `passport.authenticate('jwt', { session: false })`. Secure routes must read the authenticated user through `getCurrentUser(req)` from `backend/src/utils/request-context.ts`; if absent, return `ForbiddenError`. +- **Access model** - Keep the current permission model: effective permissions are `app_role.permissions + user.custom_permissions`. `Administrator`, `Platform Owner`, and `Account Manager` are platform-wide internal roles. Other internal users currently have all-project scope by default unless a future explicit override system is implemented. `Public` users must not have admin API permissions and can only access public production pages plus explicitly granted private production presentations. +- **Public role hardening** - Do not grant RBAC permissions or custom permissions to `Public` users. Do not represent private production presentation grants as `READ_PROJECTS`, `READ_TOUR_PAGES`, or other admin permissions; use `production_presentation_access`. +- **Centralize access decisions** - New authorization logic must go through an `AccessPolicy`-style helper/service rather than ad hoc checks in routes/components. Keep admin API permissions separate from runtime presentation access. + +## Mandatory Rules For New Code + +These rules are required for new code and for touched code when practical. + +### Backend Boundaries +- **Routes/controllers**: only authenticate, read runtime context, validate request input, call a service, and map the response. Do not put business logic in routes. +- **Services/domain**: own business logic, transactions, permission decisions, and orchestration. +- **DB API/repository**: only data access, filters, includes, pagination, and persistence mapping. +- **Policy**: all role/permission/runtime access decisions belong in policy/helper services, not scattered across routes. +- **Validation**: all new external `body`, `query`, and `params` inputs must be validated before service calls. `Joi` is already available in backend and should be preferred unless there is a clear reason to use another validator. +- **ID handling**: route params are canonical. For `PUT/PATCH/DELETE /:id`, use `req.params.id`; reject mismatched body ids. +- **Query safety**: new list/autocomplete endpoints must define max `limit`, default pagination, allowed sort fields, and allowed sort directions. +- **Service contracts**: new service/DB API methods should use object/options signatures, e.g. `Service.update({ id, data, currentUser, transaction, runtimeContext })` and `DBApi.findAll(filter, { currentUser, transaction, runtimeContext })`. +- **Logging**: use the project logger in runtime backend code (`services`, `routes`, `middlewares`, `db/api`, `utils`, app/bootstrap/config). Do not add new `console.*` calls outside migrations, seeders, scripts, or explicit debug-only CLI tooling. + +### Backend Typing +- New backend pure helpers, validators, and policy modules should use TypeScript or JSDoc/checkJs-compatible types. +- Define shared shapes for `currentUser`, `runtimeContext`, service options, and validation results when touching related code. +- Do not add global `Express.Request` augmentation for project request fields. Use `backend/src/utils/request-context.ts` helpers for request-scoped `currentUser`, `runtimeContext`, logging, runtime-public flags, and permission overrides. +- Keep active backend source in strict TypeScript/ESM. + +### Frontend State And API +- **Redux is for client/app state only**: auth/session UI, theme/style, layout/sidebar, constructor UI state, and app preferences. +- **TanStack Query is for server state**: API reads, entity lists/details, mutations, invalidation, and background refetch. +- Do not create new entity CRUD Redux slices or new Redux thunks for server reads/mutations. +- Centralize API access through query hooks or a shared API client. Avoid direct `axios` calls inside feature UI components unless there is a clear existing local pattern. +- New feature-specific code should live near the feature/domain it belongs to. Do not put feature logic into generic `components`, `hooks`, or `lib` folders when a feature-local module is clearer. + +### Frontend TypeScript And React +- Do not add new `any` without a specific reason. Prefer existing domain types or add a narrow local type. +- Avoid new `eslint-disable`, `@ts-ignore`, and hook dependency suppressions. If unavoidable, add a short reason. +- New hooks must follow React hook rules and should be structured so `react-hooks/exhaustive-deps` can be enabled. +- Do not add new client usage of `jsonwebtoken`; use `jwt-decode` or `/auth/me` for client-side identity. + +### Database And Migrations +- Do not add indexes speculatively. Add indexes after checking query patterns or explaining the expected query. +- Migrations must be reversible or include an explicit rollback/backup plan. +- Do not drop columns/tables in the same change that stops using them unless explicitly requested and production data safety is covered. +- Do not normalize `ui_schema_json` into new tables unless there is a concrete current need; prefer validation/extraction helpers first. +- Do not rewrite, rename, or reformat already applied backend migration files. Add new migrations only for new schema changes. + +## Disabled Features + +The following features are implemented but currently disabled with `false &&` conditions: + +- **Navigation blocking while preloading** - Navigation buttons can be disabled until neighbor page backgrounds are preloaded. Currently disabled to allow instant navigation response. To re-enable, remove `false &&` in: + - `frontend/src/components/RuntimePresentation.tsx` (isForwardNavDisabled, handleElementClick) + - `frontend/src/pages/constructor.tsx` (handleElementClick, isNavDisabled calculation) + +## Documentation Workflow + +- **Before each task**: Research relevant documentation in `documentation/` to understand existing implementations, patterns, and conventions. +- **After each task**: Update affected documentation to reflect changes (API changes, new fields, modified workflows). +- **After implementing new features/modules**: Create new documentation file in `documentation/` following the existing format, and add reference to the Feature Documentation table in this file. + +## Quick Start +```bash +# Terminal 1 - Backend (port 3000) +cd backend && npm run start-dev + +# Terminal 2 - Frontend (port 3001) +cd frontend && npm run dev +``` + +## Common Commands + +### Backend (run from `backend/` directory, runs on port 3000 in the default dev_stage flow) +```bash +npm install # Install dependencies +npm run start-dev # Start server (loads .env, migrates, seeds, watches) +npm run lint # ESLint check +npm run typecheck # Strict TypeScript check for migrated backend scope +npm run test # Unit tests (Node test runner) +npm run test:integration # Integration tests; DB tests skip when Postgres is unavailable +npm run test:e2e # E2E HTTP tests with a local listener +npm run test:all # Unit, integration, and e2e suites +npm run build # Compile migrated TypeScript files +npm run db:migrate # Run migrations only +npm run db:migrate:undo # Undo last migration +npm run db:seed # Seed database only +npm run db:reset # Drop, create, migrate, and seed +``` + +**Creating new migrations:** Add new migration files deliberately under `backend/src/db/migrations/` only when a schema change is required. Keep them reversible and preserve already applied migration files unchanged. + +**Note:** Backend `.env` is loaded centrally by `backend/src/load-env.ts` for app and DB entrypoints. If `NODE_ENV` is absent, it defaults to `dev_stage`, matching the standard VM backend flow and using the `.env` DB settings. Do not add `NODE_ENV=production` to local startup unless a task explicitly requires the production config. +The backend defaults to port `3000` in `dev_stage` and `8080` otherwise; set `PORT` explicitly when a task needs a different port. + +### Frontend (run from `frontend/` directory) +```bash +npm install # Install dependencies +npm run dev # Start dev server with Turbopack (port 3001) +npm run typecheck # TypeScript check without production build +npm run verify # Typecheck, lint, and production build +npm run build # Production build +npm run lint # ESLint check (.ts, .tsx files) +npm run format # Format code with Prettier +``` + +### Standard VM Environment + +The standard VM port split is: +- **Frontend PM2 app `frontend-dev`**: Next.js production server on port `3001` +- **Backend PM2 app `backend-dev`**: `npm run start` with `NODE_ENV=dev_stage` on port `3000` +- **Apache**: public entrypoint and reverse proxy on port `80` + +Direct VM backend health checks should target `http://127.0.0.1:3000/api/...`. A protected endpoint returning `401 Unauthorized` without JWT means the backend is reachable. + +### Docker (run from `docker/` directory) +```bash +chmod +x start-backend.sh && chmod +x wait-for-it.sh # First time setup +docker-compose up # Start all services +rm -rf data && docker-compose up # Start with fresh database +``` + +## Architecture + +### Backend Structure (`backend/src/`) +- **routes/**: Express route handlers (RESTful endpoints) +- **db/api/**: Database access layer (CRUD operations per model) +- **db/models/**: Sequelize model definitions +- **db/migrations/**: Database migrations +- **db/seeders/**: Seed data +- **auth/**: Passport.js authentication (JWT, Google OAuth, Microsoft OAuth) +- **services/**: Business logic services (emails, notifications, publishing, file storage) +- **factories/**: Router and service generation (`router.factory.js`, `service.factory.js`) +- **middlewares/**: Permission checking, runtime context handling +- **helpers/**: Utility functions (e.g., `wrapAsync` for async error handling) + +### Frontend Structure (`frontend/src/`) +- **pages/**: Next.js pages (each entity has `-list`, `-edit`, `-new`, `-view`, `-table` variants) +- **components/**: React components (PascalCase naming) +- **stores/**: Redux Toolkit slices (per entity + auth, main, style slices) +- **layouts/**: Page layouts (Authenticated, Guest) +- **hooks/**: Global custom hooks (`usePreloadOrchestrator`, `useNeighborGraph`, etc.) +- **lib/**: Utility libraries (`elementStyles`, `imagePreDecode`, `mediaDuration`, `parseJson`) +- **types/**: TypeScript type definitions (`constructor.ts`, `runtime.ts`, `preload.ts`) +- **helpers/**: Utility functions +- **styles/**: Global CSS with Tailwind (`_theme.css` uses `@apply`) + +### Key Domain Models +Projects, Tour Pages, Assets, Asset Variants, Permissions, Roles, Users, Publish Events, PWA Caches, Element Type Defaults, Project Element Defaults + +**Note:** Page elements, navigation links, and transitions are stored directly in `tour_pages.ui_schema_json`. + +### Element Defaults Hierarchy +The system has a two-level defaults cascade for UI elements: +1. **element_type_defaults** - Global platform-wide defaults (11 predefined types) +2. **project_element_defaults** - Project-specific overrides (auto-snapshotted from global on project creation) + +When creating new elements, defaults cascade: global → project. Instance-specific settings are stored in `tour_pages.ui_schema_json`. + +### Special Pages +- **`constructor.tsx`**: Tour builder/editor with drag-drop, element positioning, page management +- **`runtime.tsx`**: Tour playback with transitions, preloading, navigation +- **`p/[slug].tsx`**: Public tour pages with PWA offline support + +### State Management +- **Redux Toolkit is the default only for client/app state** (auth/session UI, style/theme, layout/sidebar, constructor UI state, app preferences) +- **TanStack Query is the default for server state** (API reads, entity lists/details, mutations, invalidation, background refetch) +- Do not create new entity CRUD Redux slices in `stores/[entity]/[entity]Slice.ts`. +- Core slices: `authSlice.ts` (auth), `mainSlice.ts` (UI), `styleSlice.ts` (theming), `constructorSlice.ts` (editor) +- Use `useAppSelector` and `useAppDispatch` hooks from `stores/hooks.ts` +- **Local hooks acceptable** for ephemeral, component-scoped state (e.g., `usePageNavigation` for session-scoped page history) +- See [stores-module.md](frontend/docs/stores-module.md#state-management-guidelines) for decision tree + +## Code Conventions + +### Backend +- ES6+ with arrow functions, const/let +- Document endpoints with Swagger JSDoc comments +- Lowercase filenames (e.g., `auth.js`, `projects.js`) + +### Frontend +- Functional components with TypeScript +- PascalCase for components and types, camelCase for variables/functions +- Custom hooks prefixed with `use` (e.g., `useAuth`) +- Tailwind CSS with theme customization in `_theme.css` + +### Styling +- Theme customization uses `@apply` directive in `_theme.css` +- **Sidebar styles**: Target `#asideMenu` (defined in `AsideMenuLayer.tsx`) for sidebar overrides +- Use highly specific selectors when overriding Tailwind utilities to avoid conflicts +- Themed blocks (`.theme-pink`, `.theme-green`) standardize UI appearance - ensure custom overrides integrate cleanly + +### Error Handling +- Backend: Use centralized `commonErrorHandler` middleware for uniform error responses +- Frontend: Use React error boundaries for runtime errors + +## API Patterns + +Routes follow pattern: `/api/[entity]/` with standard CRUD operations +- `POST /api/auth/signin/local` - Login +- `GET /api/auth/me` - Current user (requires JWT) +- Self-registration is disabled. New users are created by Administrator, Platform Owner, or Account Manager through the Users flow and receive an invitation/setup link. + +All entity routes support: list, findOne, create, update, destroy operations with pagination and filtering. + +## Backend Factory Patterns + +### Router Factory (`factories/router.factory.js`) +Generates standard CRUD routes for entities: +```javascript +module.exports = createEntityRouter('assets', AssetsService, AssetsDBApi, { + permissionEntity: 'assets', +}); +``` + +### Service Factory (`factories/service.factory.js`) +Generates service classes with transaction handling: +```javascript +module.exports = createEntityService('assets', AssetsDBApi); +``` + +### Base DB API (`db/api/base.api.js`) +All entity APIs extend `GenericDBApi` and override: +- `MODEL` - Sequelize model reference +- `SEARCHABLE_FIELDS` - Fields for text search +- `RANGE_FIELDS` - Fields for range filtering +- `ENUM_FIELDS` - Fields for exact match filtering +- `getFieldMapping(data)` - Transform input data before save + +## File Storage + +Controlled by `FILE_STORAGE_PROVIDER` env var or auto-detected from credentials: +- `s3` - AWS S3 (requires `S3_BUCKET`, `S3_REGION`, `S3_ACCESS_KEY_ID`, `S3_SECRET_ACCESS_KEY`) +- `gcloud` - Google Cloud Storage +- `local` - Local filesystem (default) + +## Publishing Workflow + +Three-tier **content environment** model (distinct from `NODE_ENV` server environment): +- **Dev** (constructor editing) → **Stage** (preview) → **Production** (public) + +**Route-based environment access:** +- `/p/[slug]` - Production presentation +- `/p/[slug]/stage` - Stage presentation +- `/constructor?projectId=` - Dev editing (always dev environment) + +Key endpoints: +- `POST /api/publish/save-to-stage` - Dev → Stage (body: `{ projectId }`) +- `POST /api/publish` - Stage → Production (body: `{ projectId }`) + +**Environment isolation:** Pages have `environment` field (`dev`, `stage`, `production`). The `X-Runtime-Environment` header (set by frontend) determines which content environment to query. Both frontend (`RuntimePresentation.tsx`) and backend (`runtime-context.js`) filter by this field. + +**Server Environment vs Content Environment:** +- `NODE_ENV` controls database config (production/dev_stage/development) +- `tour_pages.environment` controls content visibility (dev/stage/production) + +### Private Production Presentations + +Production presentations are public by default at `/p/[slug]`. Each project can switch production runtime visibility through `projects.production_presentation_visibility` (`public` default, `private` optional). + +- Platform staff users with any RBAC permission can visit every private production presentation +- Customer users with `Public` role can visit only private production presentations granted in `production_presentation_access` +- `Administrator`, `Platform Owner`, and `Account Manager` can create users +- When creating or editing a `Public` user, the form shows a selector for private production presentations and saves DB access grants +- Do not use config files or env vars for customer allowlists +- Do not grant customer viewer users broad permissions such as `READ_PROJECTS` or `READ_TOUR_PAGES`; any RBAC permission makes the user platform staff for private presentation access + +See [private-production-presentations.md](documentation/private-production-presentations.md). + +## PWA & Offline Support + +- **Service Worker**: Generated by Serwist from `frontend/src/sw.ts` → `public/sw.js` +- **Offline Caching**: PWA_Caches model tracks cached assets per project +- **Runtime Mode**: Middleware distinguishes public vs authenticated access for offline tours + +## Asset Preloading Architecture + +Runtime presentations use direct S3 downloads via presigned URLs for instant page navigation: + +``` +1. Request presigned URLs (max 50 per batch, 1-hour expiry) + POST /api/file/presign { urls: ["assets/img.jpg", ...] } + +2. Download directly from S3 → Store in Cache API (< 5MB) or IndexedDB (≥ 5MB) + +3. Create blob URL → Decode image → Store in readyBlobUrlsRef + +4. Instant lookup during navigation (O(1)) + const blobUrl = preloadOrchestrator.getReadyBlobUrl(originalUrl); +``` + +**Preload Priority:** Transition videos (+150) > Images (+100) > Audio (+50) > Video (+30) + +**Storage Layers:** +| Layer | Size Limit | Purpose | +|-------|------------|---------| +| Cache API | < 5MB | Fast asset storage | +| IndexedDB (Dexie) | ≥ 5MB | Large assets, offline data | +| Blob URLs | Memory | Pre-decoded for instant display | + +## Frontend Patterns + +### MUI X Data Grid v7 +Use new `valueGetter` signature: +```typescript +// For value transformation +valueGetter: (value) => value?.id ?? value + +// For row access +valueGetter: (_value, row) => new Date(row.created_at) +``` + +### Custom Hooks +Hooks are located in three places: +- **`src/hooks/`**: Global hooks (`usePreloadOrchestrator`, `usePageNavigationState`, `useTransitionPlayback`, `usePageNavigation`) +- **`src/hooks/video/`**: Video playback primitives (8 composable hooks for video playback scenarios) +- **Component directories**: Domain-specific hooks (`components/Assets/useAssetUploader.ts`) + +Key runtime hooks: +| Hook | Purpose | +|------|---------| +| `usePreloadOrchestrator` | Stream-first asset preloading (current page + transitions only) | +| `usePageNavigationState` | Unified navigation state machine (replaces 6+ hooks) | +| `useTransitionPlayback` | Video transition playback coordination | +| `usePageNavigation` | Page history tracking with browser-like back behavior | +| `useCanvasScale` | Responsive canvas scaling with letterbox mode | +| `useNetworkAware` | Network condition monitoring for adaptive transitions | +| `useBackgroundDimensionSuggestion` | Detect background media dimensions for canvas size suggestions | + +**Video Hooks (`src/hooks/video/`):** +| Hook | Purpose | +|------|---------| +| `useVideoBlobUrl` | Resolve video URLs to blob URLs from preload cache | +| `useVideoPlaybackCore` | Core playback logic combining multiple primitives | +| `useVideoPlayer` | Complete UI video player hook | + +### Redux Entity Pattern +Use this pattern only when maintaining an existing Redux entity flow; new server-state code should use TanStack Query hooks instead. +```typescript +import { fetch, deleteItem } from '../../stores/assets/assetsSlice'; +const dispatch = useAppDispatch(); +dispatch(fetch({ query: '?limit=100' })); +``` + +## Feature Documentation + +Detailed feature documentation is available in `documentation/`: + +| Document | Description | +|----------|-------------| +| [project-architecture.md](documentation/project-architecture.md) | Overall project structure and architecture | +| [api-reference.md](documentation/api-reference.md) | API endpoints reference | +| [authentication-system.md](documentation/authentication-system.md) | JWT, OAuth (Google, Microsoft) authentication | +| [rbac-system.md](documentation/rbac-system.md) | Role-based access control system | +| [user-management.md](documentation/user-management.md) | User CRUD, roles assignment, account management | +| [project-memberships.md](documentation/project-memberships.md) | Team collaboration, per-project access control | +| [asset-upload-variants.md](documentation/asset-upload-variants.md) | Asset upload pipeline and variant generation | +| [ui-elements.md](documentation/ui-elements.md) | UI Elements stored in `ui_schema_json` (buttons, hotspots, galleries, tooltips, media players) | +| [page-transitions.md](documentation/page-transitions.md) | Video transitions stored on navigation elements | +| [project-transition-settings.md](documentation/project-transition-settings.md) | Environment-aware CSS transition settings (fade, duration, easing) | +| [video-playback.md](documentation/video-playback.md) | Video player implementation, iOS autoplay compatibility | +| [assets-preloading.md](documentation/assets-preloading.md) | Asset preloading and caching strategy | +| [publishing-workflow.md](documentation/publishing-workflow.md) | Dev → Stage → Production publishing | +| [private-production-presentations.md](documentation/private-production-presentations.md) | Private production presentation allowlist, viewer users, and runtime access flow | +| [offline-pwa-mode.md](documentation/offline-pwa-mode.md) | PWA offline capabilities and caching | +| [email-notification-service.md](documentation/email-notification-service.md) | Nodemailer/SES integration, verification emails, invitations | +| [search-system.md](documentation/search-system.md) | Global full-text search, permission-based filtering | +| [deployment-vm.md](documentation/deployment-vm.md) | Standard VM deployment topology, PM2 recovery, ports, OOM/ffmpeg diagnostics | +| [custom-domains-apache.md](documentation/custom-domains-apache.md) | Customer-owned presentation domains via Apache, DNS A records, Certbot, and host/path routing | +| [page-links-navigation.md](documentation/page-links-navigation.md) | Page navigation using `targetPageSlug` in elements | +| [access-logs-audit-trail.md](documentation/access-logs-audit-trail.md) | Access logging, audit trail, activity tracking | +| [db-cleanup-audit.md](documentation/db-cleanup-audit.md) | Non-destructive DB cleanup audit, orphan checks, legacy schema checks, and soft-delete retention policy | +| [global-ui-controls.md](documentation/global-ui-controls.md) | Configurable fullscreen, sound, and offline system controls with global/project/page cascade | +| [backend/database-schema.md](backend/docs/database-schema.md) | Complete database schema - all models, fields, relationships, indexes, constraints | +| [backend/backend-architecture.md](backend/docs/backend-architecture.md) | Backend architecture - layers, design patterns, middleware, factories, file storage | +| [backend/api-endpoints.md](backend/docs/api-endpoints.md) | Complete API reference - all endpoints, request/response formats, authentication, rate limits | +| [backend/modules/core.md](backend/docs/modules/core.md) | Core module - index.js (entry), config.ts (configuration), helpers.js/utilities | +| [backend/modules/auth.md](backend/docs/modules/auth.md) | Auth module - Passport.js strategies (JWT, Google, Microsoft), login, invitation setup, password reset, email verification | +| [backend/modules/middleware.md](backend/docs/modules/middleware.md) | Middleware module - rate limiting, permissions, runtime context, public access control, file uploads | +| [backend/modules/routes.md](backend/docs/modules/routes.md) | Routes module - 22 route files, factory pattern, CRUD endpoints, Swagger docs, auth/file/search/sql routes | +| [backend/modules/services.md](backend/docs/modules/services.md) | Services module - business logic layer, 34 service files, factory pattern, file storage (S3/GCloud/Local), email, notifications, publishing | +| [backend/modules/email.md](backend/docs/modules/email.md) | Email module - Nodemailer/AWS SES, transactional emails, HTML templates, token management, email verification, password reset, invitations | +| [backend/modules/notifications.md](backend/docs/modules/notifications.md) | Notifications module - error classes (ValidationError, ForbiddenError), i18n message catalog, helper functions | +| [backend/modules/factories.md](backend/docs/modules/factories.md) | Factories module - createEntityRouter and createEntityService functions, code generation patterns for CRUD operations, boilerplate elimination | +| [backend/modules/db-models.md](backend/docs/modules/db-models.md) | DB Models module - 16 Sequelize models, entity definitions, associations, validations, lifecycle hooks, soft delete patterns | +| [backend/modules/db-api.md](backend/docs/modules/db-api.md) | DB API module - GenericDBApi base class, 18 entity APIs, declarative configuration, query filtering, runtime context helpers | +| [backend/modules/db-migrations.md](backend/docs/modules/db-migrations.md) | DB Migrations module - Umzug runner, migration safety rules, schema evolution, data backfill | +| [backend/modules/db-seeders.md](backend/docs/modules/db-seeders.md) | DB Seeders module - Umzug seeders, RBAC setup (7 roles, 54 permissions), sample data opt-in | +| [backend/modules/db-config.md](backend/docs/modules/db-config.md) | DB Config module - typed DB config, environment validation (Joi), Umzug commands, sync/reset scripts | +| [backend/modules/utilities.md](backend/docs/modules/utilities.md) | Utilities module - error classes, Pino logging, env validation, request helpers (wrapAsync, commonErrorHandler), DB utils, i18n messages | +| [backend/testing.md](backend/docs/testing.md) | Backend test strategy - unit, integration, and e2e coverage, helpers, commands, and environment notes | +| [frontend/hooks-reference.md](frontend/docs/hooks-reference.md) | React hooks reference (useFormSync, useEntityTable, useOfflineMode, etc.) | +| [frontend/video-hooks-module.md](frontend/docs/video-hooks-module.md) | Video playback primitive hooks (8 composable hooks for video scenarios) | +| [frontend/constructor-page-editor.md](frontend/docs/constructor-page-editor.md) | Constructor page editor - visual tour builder with drag-drop elements | +| [frontend/runtime-presentation.md](frontend/docs/runtime-presentation.md) | Runtime presentation viewer - full-screen tour playback with transitions | +| [frontend/navigation-smooth-transitions.md](frontend/docs/navigation-smooth-transitions.md) | Navigation & smooth transitions - page switching, transition video playback, last frame preservation, online/offline modes | +| [frontend/ui-adaptivity-system.md](frontend/docs/ui-adaptivity-system.md) | UI Adaptivity System - canvas units (--cu), responsive scaling, letterbox mode, element styling | +| [frontend/ui-element-preloading-analysis.md](frontend/docs/ui-element-preloading-analysis.md) | UI element processing & neighbor preloading - deep analysis of asset extraction, preload flow, offline caching | +| [frontend/asset-upload-preloading-pwa-analysis.md](frontend/docs/asset-upload-preloading-pwa-analysis.md) | Deep analysis of asset upload, preload orchestrator, PWA/offline mode, storage layers, network awareness | +| [frontend/frontend-architecture.md](frontend/docs/frontend-architecture.md) | Frontend architecture - 386 files, 14 modules, factories, hooks, Redux, PWA/offline, design patterns | +| [frontend/pages-module.md](frontend/docs/pages-module.md) | Pages module - 99 pages, _app.tsx entry, entity CRUD pattern, constructor, runtime presentations, layouts | +| [frontend/components-module.md](frontend/docs/components-module.md) | Components module - 183 files, entity tables, factories, constructor, element settings, runtime, PWA/offline | +| [frontend/hooks-module.md](frontend/docs/hooks-module.md) | Hooks module - 52 custom hooks, runtime/preloading, PWA/offline, constructor, tables/forms, UI utilities | +| [frontend/stores-module.md](frontend/docs/stores-module.md) | Stores module - Redux Toolkit, 17 slices (4 core + 13 entity), createEntitySlice factory, typed hooks | +| [frontend/lib-module.md](frontend/docs/lib-module.md) | Lib module - 19 utility files, asset URL management, element defaults/styles/effects, offline storage (Cache API + IndexedDB), download queue | +| [frontend/types-module.md](frontend/docs/types-module.md) | Types module - 18 TypeScript definition files, entity types, constructor/runtime types, API/Redux types, permissions enum, offline/preload types | +| [frontend/factories-module.md](frontend/docs/factories-module.md) | Factories module - 5 factory files (~1,285 LOC), createListPage, createFormPage, createTableComponent, configBuilderFactory, createEntitySlice for 91% boilerplate reduction | +| [frontend/schemas-module.md](frontend/docs/schemas-module.md) | Schemas module - 6 Zod validation schema files (~257 LOC), form validation for User, Asset, Project, TourPage, Role entities with type inference | +| [frontend/helpers-module.md](frontend/docs/helpers-module.md) | Helpers module - 6 utility files (~304 LOC), dataFormatter (entity display), hasPermission (RBAC), notifyStateHandler (Redux), text formatters, file saver | +| [frontend/layouts-module.md](frontend/docs/layouts-module.md) | Layouts module - 2 layout files (~262 LOC), LayoutAuthenticated (JWT auth, permissions, UI chrome), LayoutGuest (public pages), getLayout pattern | +| [frontend/config-module.md](frontend/docs/config-module.md) | Config module - 12 config files (~523 LOC), Next.js/Tailwind/TypeScript build config, runtime settings (config.ts), menu configs, offline/preload configs, ESLint rules | +| [frontend/context-module.md](frontend/docs/context-module.md) | Context module - DownloadContext (~256 LOC) for PWA download progress state, DownloadEventBus integration, useDownloadContext/useDownloadContextOptional hooks | +| [frontend/interfaces-module.md](frontend/docs/interfaces-module.md) | Interfaces module - ~15 type definition files (~2,200+ LOC), entity types, API contracts, Redux state shapes, permissions enum, constructor/runtime/offline/preload interfaces | diff --git a/README.md b/README.md index d8dbe87..c418752 100644 --- a/README.md +++ b/README.md @@ -100,6 +100,15 @@ After seeding, use the credentials configured in backend environment/config valu └── README.md # Docker documentation ``` +## Documentation + +- [General feature documentation](documentation/project-architecture.md) - platform architecture and cross-cutting features +- [API reference](documentation/api-reference.md) - public API overview and workflow-level endpoint notes +- [Backend documentation](backend/docs/backend-architecture.md) - backend architecture, modules, database, and tests +- [Backend API endpoints](backend/docs/api-endpoints.md) - backend endpoint reference +- [Frontend documentation](frontend/docs/frontend-architecture.md) - frontend architecture, pages, components, hooks, and runtime flows +- [VM deployment](documentation/deployment-vm.md) - standard VM topology, ports, PM2, Apache, and diagnostics + ## Key Workflows ### Tour Creation diff --git a/backend/README.md b/backend/README.md index 0227a86..8bdc4f5 100644 --- a/backend/README.md +++ b/backend/README.md @@ -53,6 +53,15 @@ npm run check:public-access non-Public private production presentation grants. Review its output before running `npm run fix:public-access`. +## Documentation + +- [Backend architecture](docs/backend-architecture.md) - service boundaries, app bootstrap, middleware, and factories +- [API endpoints](docs/api-endpoints.md) - backend REST API reference +- [Database schema](docs/database-schema.md) - models, relationships, indexes, and constraints +- [Testing strategy](docs/testing.md) - unit, integration, and e2e test notes +- [Backend modules](docs/modules/core.md) - module-level documentation entry point +- [VM deployment](../documentation/deployment-vm.md) - standard VM ports, PM2, Apache, and backend health checks + ## Environment Variables Create a `.env` file in the backend directory: diff --git a/backend/docs/api-endpoints.md b/backend/docs/api-endpoints.md new file mode 100644 index 0000000..f3357b9 --- /dev/null +++ b/backend/docs/api-endpoints.md @@ -0,0 +1,1671 @@ +# Backend API Endpoints Documentation + +Complete reference for all API endpoints in the Tour Builder Platform backend. + +## Overview + +- **Base URL**: `http://localhost:3000/api` for local development +- **Content-Type**: `application/json` +- **Authentication**: JWT Bearer Token (except public endpoints) +- **API Documentation UI**: `http://localhost:3000/api-docs` (Swagger, local development) + +**Standard VM note:** the VM `dev_stage` backend listens on port `3000`, while +the frontend listens on `3001` and Apache serves the public domain on port `80`. +Use `http://127.0.0.1:3000/api/...` for direct VM backend checks. Protected +routes returning `401 Unauthorized` without JWT are healthy. See +[`deployment-vm.md`](../../documentation/deployment-vm.md). + +--- + +## Table of Contents + +1. [Authentication](#1-authentication) +2. [Health & System](#2-health--system) +3. [Users](#3-users) +4. [Roles](#4-roles) +5. [Permissions](#5-permissions) +6. [Projects](#6-projects) +7. [Project Memberships](#7-project-memberships) +8. [Tour Pages](#8-tour-pages) +9. [Assets](#9-assets) +10. [Asset Variants](#10-asset-variants) +11. [Project Audio Tracks](#11-project-audio-tracks) +12. [Project Transition Settings](#12-project-transition-settings) +13. [Global UI Controls](#13-global-ui-controls) +14. [Element Defaults](#14-element-defaults) +15. [Publishing](#15-publishing) +16. [File Management](#16-file-management) +17. [Search](#17-search) +20. [Access Logs](#20-access-logs) +21. [Publish Events](#21-publish-events) +22. [PWA Caches](#22-pwa-caches) +23. [Presigned URL Requests](#23-presigned-url-requests) + +--- + +## Common Patterns + +### Authentication Header + +```http +Authorization: Bearer +``` + +### Query Parameters (List Endpoints) + +| Parameter | Type | Description | +|-----------|------|-------------| +| `page` | number | Page number (0-indexed) | +| `limit` | number | Items per page (default: 10) | +| `field` | string | Field to sort by | +| `sort` | string | Sort direction: `asc` or `desc` | +| `filetype` | string | Set to `csv` for CSV export | +| `` | string | Text search filter (ILIKE) | +| `Range` | string | Range filter: `[min,max]` | + +### Standard List Response + +```json +{ + "rows": [...], + "count": 100 +} +``` + +### Rate Limits + +| Endpoint Type | Limit | Window | +|---------------|-------|--------| +| Auth | 10 requests | 15 minutes | +| Signup | 5 requests | 1 hour | +| Password Reset | 5 requests | 1 hour | +| API (general) | 100 requests | 1 minute | +| Upload | 10 requests | 1 minute | +| Download | 200 requests | 1 minute | +| Search | 30 requests | 1 minute | +| AI | 20 requests | 1 minute | + +### Rate Limit Headers + +```http +X-RateLimit-Limit: 100 +X-RateLimit-Remaining: 99 +X-RateLimit-Reset: 2026-03-30T12:00:00.000Z +Retry-After: 60 # (only on 429) +``` + +--- + +## 1. Authentication + +### POST /api/auth/signin/local + +Sign in with email and password. + +**Authentication**: None +**Rate Limit**: 10/15min + +**Request:** +```json +{ + "email": "user@example.com", + "password": "password123" +} +``` + +**Response (200):** +```json +"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9..." +``` + +**Errors:** +- `400`: Invalid credentials + +--- + +### Self-Registration + +Self-registration is disabled. `POST /api/auth/signup` is not registered. +New users are created through the authenticated Users API/UI and receive an +invitation/setup link. + +--- + +### GET /api/auth/me + +Get current authenticated user. + +**Authentication**: Required + +**Response (200):** +```json +{ + "id": "uuid", + "email": "user@example.com", + "firstName": "John", + "lastName": "Doe", + "phoneNumber": "+1234567890", + "emailVerified": true, + "disabled": false, + "app_role": { + "id": "uuid", + "name": "User", + "globalAccess": false + }, + "custom_permissions": [] +} +``` + +--- + +### PUT /api/auth/password-reset + +Reset password using token. + +**Authentication**: None + +**Request:** +```json +{ + "token": "reset-token-from-email", + "password": "newPassword123" +} +``` + +**Response (200):** `true` + +--- + +### POST /api/auth/send-password-reset-email + +Send password reset email. + +**Authentication**: None +**Rate Limit**: 5/hour + +**Request:** +```json +{ + "email": "user@example.com" +} +``` + +**Response (200):** `true` + +--- + +### PUT /api/auth/password-update + +Update password for authenticated user. + +**Authentication**: Required + +**Request:** +```json +{ + "currentPassword": "oldPassword", + "newPassword": "newPassword123" +} +``` + +**Response (200):** `true` + +--- + +### PUT /api/auth/profile + +Update user profile. + +**Authentication**: Required + +**Request:** +```json +{ + "profile": { + "firstName": "John", + "lastName": "Doe", + "phoneNumber": "+1234567890" + } +} +``` + +**Response (200):** `true` + +--- + +### PUT /api/auth/verify-email + +Verify email address. + +**Authentication**: None + +**Request:** +```json +{ + "token": "verification-token-from-email" +} +``` + +**Response (200):** JWT token + +--- + +### GET /api/auth/email-configured + +Check if email service is configured. + +**Authentication**: None + +**Response (200):** `true` or `false` + +--- + +### GET /api/auth/signin/google + +Initiate Google OAuth flow. + +**Authentication**: None + +**Query Parameters:** +- `app`: Optional state parameter + +**Response**: Redirect to Google + +--- + +### GET /api/auth/signin/google/callback + +Google OAuth callback. + +**Response**: Redirect to frontend with token + +--- + +### GET /api/auth/signin/microsoft + +Initiate Microsoft OAuth flow. + +**Authentication**: None + +**Response**: Redirect to Microsoft + +--- + +### GET /api/auth/signin/microsoft/callback + +Microsoft OAuth callback. + +**Response**: Redirect to frontend with token + +--- + +## 2. Health & System + +### GET /api/health + +Health check endpoint. + +**Authentication**: None + +**Response (200):** +```json +{ + "status": "ok", + "timestamp": "2026-03-30T12:00:00.000Z", + "uptime": 12345.678, + "environment": "production", + "database": "connected" +} +``` + +**Response (503):** (when database is down) +```json +{ + "status": "degraded", + "timestamp": "2026-03-30T12:00:00.000Z", + "uptime": 12345.678, + "environment": "production", + "database": "disconnected", + "databaseError": "Connection refused" +} +``` + +--- + +### GET /api/runtime-context + +Get current runtime context (environment detection). + +**Authentication**: None + +**Response (200):** +```json +{ + "mode": "workspace", + "projectSlug": "my-project", + "headerEnvironment": "production" +} +``` + +--- + +## 3. Users + +### Standard CRUD Endpoints + +| Method | Endpoint | Description | +|--------|----------|-------------| +| POST | `/api/users` | Create user | +| GET | `/api/users` | List users | +| GET | `/api/users/count` | Count users | +| GET | `/api/users/autocomplete` | Autocomplete search | +| GET | `/api/users/:id` | Get user by ID | +| PUT | `/api/users/:id` | Update user | +| DELETE | `/api/users/:id` | Delete user | +| POST | `/api/users/deleteByIds` | Bulk delete | +| POST | `/api/users/bulk-import` | Bulk import | + +**Authentication**: Required +**Permissions**: `CREATE_USERS`, `READ_USERS`, `UPDATE_USERS`, `DELETE_USERS` + +### POST /api/users + +**Request:** +```json +{ + "data": { + "firstName": "John", + "lastName": "Doe", + "email": "john@example.com", + "phoneNumber": "+1234567890", + "app_role": "uuid-of-role", + "custom_permissions": ["uuid-1", "uuid-2"], + "password": "password123", + "disabled": false + } +} +``` + +--- + +### GET /api/users + +**Query Parameters:** +- `firstName`: Filter by first name +- `lastName`: Filter by last name +- `email`: Filter by email +- `app_role`: Filter by role ID +- Standard pagination params + +**Response:** +```json +{ + "rows": [ + { + "id": "uuid", + "firstName": "John", + "lastName": "Doe", + "email": "john@example.com", + "phoneNumber": "+1234567890", + "disabled": false, + "emailVerified": true, + "app_role": { "id": "uuid", "name": "User" } + } + ], + "count": 1 +} +``` + +--- + +## 4. Roles + +### Standard CRUD Endpoints + +| Method | Endpoint | Description | +|--------|----------|-------------| +| POST | `/api/roles` | Create role | +| GET | `/api/roles` | List roles | +| GET | `/api/roles/count` | Count roles | +| GET | `/api/roles/autocomplete` | Autocomplete search | +| GET | `/api/roles/:id` | Get role by ID | +| PUT | `/api/roles/:id` | Update role | +| DELETE | `/api/roles/:id` | Delete role | +| POST | `/api/roles/deleteByIds` | Bulk delete | + +**Permissions**: `CREATE_ROLES`, `READ_ROLES`, `UPDATE_ROLES`, `DELETE_ROLES` + +### POST /api/roles + +**Request:** +```json +{ + "data": { + "name": "Editor", + "globalAccess": false, + "permissions": ["uuid-1", "uuid-2"] + } +} +``` + +--- + +## 5. Permissions + +### Standard CRUD Endpoints + +| Method | Endpoint | Description | +|--------|----------|-------------| +| POST | `/api/permissions` | Create permission | +| GET | `/api/permissions` | List permissions | +| GET | `/api/permissions/count` | Count permissions | +| GET | `/api/permissions/autocomplete` | Autocomplete search | +| GET | `/api/permissions/:id` | Get permission by ID | +| PUT | `/api/permissions/:id` | Update permission | +| DELETE | `/api/permissions/:id` | Delete permission | +| POST | `/api/permissions/deleteByIds` | Bulk delete | + +**Permissions**: `CREATE_PERMISSIONS`, `READ_PERMISSIONS`, `UPDATE_PERMISSIONS`, `DELETE_PERMISSIONS` + +### POST /api/permissions + +**Request:** +```json +{ + "data": { + "name": "READ_CUSTOM_FEATURE" + } +} +``` + +--- + +## 6. Projects + +### Standard CRUD Endpoints + +| Method | Endpoint | Description | +|--------|----------|-------------| +| POST | `/api/projects` | Create project | +| GET | `/api/projects` | List projects | +| GET | `/api/projects/count` | Count projects | +| GET | `/api/projects/autocomplete` | Autocomplete search | +| GET | `/api/projects/:id` | Get project by ID | +| PUT | `/api/projects/:id` | Update project | +| DELETE | `/api/projects/:id` | Delete project | +| POST | `/api/projects/deleteByIds` | Bulk delete | + +**Permissions**: `CREATE_PROJECTS`, `READ_PROJECTS`, `UPDATE_PROJECTS`, `DELETE_PROJECTS` + +**Runtime Public Access**: GET endpoints accessible without auth in production mode. + +### POST /api/projects + +**Request:** +```json +{ + "data": { + "name": "My Tour", + "slug": "my-tour", + "description": "A virtual tour of the museum", + "logo_url": "assets/logo.png", + "favicon_url": "assets/favicon.ico", + "og_image_url": "assets/og-image.jpg" + } +} +``` + +**Fields:** + +| Field | Type | Description | +|-------|------|-------------| +| `name` | string | Project display name | +| `slug` | string | URL-safe identifier (auto-generated if not provided) | +| `description` | string | Project description | +| `logo_url` | string | Path to project logo | +| `favicon_url` | string | Path to favicon | +| `og_image_url` | string | Path to Open Graph image | + +--- + +### POST /api/projects/:id/clone + +Clone an existing project with all pages and settings. + +**Request:** None (uses URL parameter) + +**Response:** +```json +{ + "id": "new-project-uuid", + "name": "My Tour (Copy)", + "slug": "my-tour-copy" +} +``` + +--- + +### GET /api/projects/:id/offline-manifest + +Get PWA offline manifest for a project. + +**Query Parameters:** +- `variant`: `mobile` or `desktop` (default: `desktop`) + +**Response:** +```json +{ + "projectId": "uuid", + "version": "2026-03-30T12:00:00.000Z", + "assets": [ + { + "url": "assets/image.jpg", + "type": "image", + "size": 1024000 + } + ], + "pages": [ + { + "slug": "home", + "name": "Home Page" + } + ] +} +``` + +--- + +## 7. Project Memberships + +### Standard CRUD Endpoints + +| Method | Endpoint | Description | +|--------|----------|-------------| +| POST | `/api/project_memberships` | Create membership | +| GET | `/api/project_memberships` | List memberships | +| GET | `/api/project_memberships/count` | Count memberships | +| GET | `/api/project_memberships/:id` | Get membership by ID | +| PUT | `/api/project_memberships/:id` | Update membership | +| DELETE | `/api/project_memberships/:id` | Delete membership | + +**Permissions**: `CREATE_PROJECT_MEMBERSHIPS`, `READ_PROJECT_MEMBERSHIPS`, etc. + +### POST /api/project_memberships + +**Request:** +```json +{ + "data": { + "user": "user-uuid", + "project": "project-uuid", + "membershipRole": "editor" + } +} +``` + +**Membership Roles:** +- `owner` - Full access including delete +- `editor` - Can edit content +- `viewer` - Read-only access + +--- + +## 8. Tour Pages + +### Standard CRUD Endpoints + +| Method | Endpoint | Description | +|--------|----------|-------------| +| POST | `/api/tour_pages` | Create page | +| GET | `/api/tour_pages` | List pages | +| GET | `/api/tour_pages/count` | Count pages | +| GET | `/api/tour_pages/:id` | Get page by ID | +| PUT | `/api/tour_pages/:id` | Update page | +| DELETE | `/api/tour_pages/:id` | Delete page | +| POST | `/api/tour_pages/deleteByIds` | Bulk delete | +| POST | `/api/tour_pages/reorder` | Reorder pages by updating `sort_order` only | +| POST | `/api/tour_pages/:id/duplicate` | Duplicate a dev page as a new independent dev page | + +**Permissions**: `CREATE_TOUR_PAGES`, `READ_TOUR_PAGES`, etc. + +**Runtime Public Access**: GET endpoints accessible without auth in production mode. + +### POST /api/tour_pages/reorder + +Reorders pages for a project in the constructor environment. + +**Auth:** Required + +**Request:** + +```json +{ + "data": { + "projectId": "project-uuid", + "environment": "dev", + "orderedPageIds": ["page-uuid-1", "page-uuid-2", "page-uuid-3"] + } +} +``` + +**Response:** + +```json +[ + { "id": "page-uuid-1", "sort_order": 1 }, + { "id": "page-uuid-2", "sort_order": 2 }, + { "id": "page-uuid-3", "sort_order": 3 } +] +``` + +**Rules:** +- Only `environment: "dev"` is accepted. Stage and production are updated by + publishing, not by direct reorder writes. +- `orderedPageIds` must include every page in the project/dev environment + exactly once. +- The endpoint updates only `tour_pages.sort_order`; it does not alter + `ui_schema_json`, slugs, backgrounds, media, navigation, or transitions. +- The visible stage order changes after Save to Stage. The visible production + order changes after Publish. + +**Validation failures:** +- Missing `projectId` +- Empty or invalid `orderedPageIds` +- Duplicate page IDs +- Missing pages from the ordered list +- Unknown page IDs or page IDs outside the project/environment +- Any environment other than `dev` + +### POST /api/tour_pages/:id/duplicate + +Duplicates a constructor/dev page into a new independent dev page. + +**Auth:** Required + +**Request:** + +```json +{ + "data": { + "projectId": "project-uuid", + "environment": "dev", + "name": "Lobby Copy", + "slug": "lobby-copy" + } +} +``` + +**Rules:** +- Only source pages in `environment: "dev"` can be duplicated. +- The requested target environment must also be `dev`. +- The source page must belong to the requested project. +- The new page is appended to constructor order with + `sort_order = max(project dev sort_order) + 1`. +- The duplicate receives a new page ID, unique slug, and empty `source_key`. +- Page settings, background media fields, design dimensions, `requires_auth`, + and `ui_schema_json` are copied. +- Inline `ui_schema_json.elements[]` IDs and nested gallery/carousel/info-panel + item IDs are regenerated for independence. +- Asset URLs, transition URLs, and navigation `targetPageSlug` references are + preserved. +- Reverse-video processing uses the existing `TourPagesService` path. + +**Validation failures:** +- Invalid source page ID +- Source page not found +- Source page outside requested project +- Source or target environment other than `dev` + +### POST /api/tour_pages + +**Request:** +```json +{ + "data": { + "projectId": "project-uuid", + "environment": "dev", + "name": "Home Page", + "slug": "home", + "sort_order": 1, + "background_image_url": "assets/bg.jpg", + "background_video_url": null, + "background_audio_url": "assets/ambient.mp3", + "background_loop": true, + "requires_auth": false, + "ui_schema_json": { + "elements": [], + "canvasSettings": {} + } + } +} +``` + +**ui_schema_json Structure:** +```json +{ + "elements": [ + { + "id": "element-uuid", + "type": "button", + "props": { + "x": 100, + "y": 200, + "width": 150, + "height": 50, + "label": "Click Me", + "targetPageSlug": "next-page", + "transitionVideo": "assets/transition.mp4" + } + } + ], + "canvasSettings": { + "width": 1920, + "height": 1080 + } +} +``` + +--- + +## 9. Assets + +### Standard CRUD Endpoints + +| Method | Endpoint | Description | +|--------|----------|-------------| +| POST | `/api/assets` | Create asset record | +| GET | `/api/assets` | List assets | +| GET | `/api/assets/count` | Count assets | +| GET | `/api/assets/autocomplete` | Autocomplete search | +| GET | `/api/assets/:id` | Get asset by ID | +| PUT | `/api/assets/:id` | Update asset | +| DELETE | `/api/assets/:id` | Delete asset | +| POST | `/api/assets/deleteByIds` | Bulk delete | + +**Permissions**: `CREATE_ASSETS`, `READ_ASSETS`, etc. + +### POST /api/assets + +Create an asset record with MIME type validation. + +**Request:** +```json +{ + "data": { + "name": "Hero Image", + "asset_type": "image", + "type": "background", + "cdn_url": "https://cdn.example.com/assets/hero.jpg", + "storage_key": "assets/hero.jpg", + "mime_type": "image/jpeg", + "size_mb": 2.5, + "width_px": 1920, + "height_px": 1080, + "duration_sec": null, + "checksum": "abc123...", + "is_public": true, + "project": "project-uuid" + } +} +``` + +**Asset Types:** `image`, `video`, `audio`, `document`, `other` + +**Type (Usage):** `background`, `transition`, `element`, `general` + +**MIME Type Validation:** + +The `mime_type` must match the `asset_type`: + +| asset_type | Valid mime_type prefixes | +|------------|-------------------------| +| `image` | `image/` (jpeg, png, gif, webp, svg, etc.) | +| `video` | `video/` (mp4, webm, mov, etc.) | +| `audio` | `audio/` (mp3, wav, ogg, etc.) | + +Other asset types (`document`, `other`, `file`) skip MIME validation. + +**Error Response (400):** +```json +{ + "message": "Invalid file type for image. Expected image (jpeg, png, gif, webp, svg, etc.), got \"video/mp4\"" +} +``` + +--- + +## 10. Asset Variants + +### Standard CRUD Endpoints + +| Method | Endpoint | Description | +|--------|----------|-------------| +| POST | `/api/asset_variants` | Create variant | +| GET | `/api/asset_variants` | List variants | +| GET | `/api/asset_variants/:id` | Get variant by ID | +| PUT | `/api/asset_variants/:id` | Update variant | +| DELETE | `/api/asset_variants/:id` | Delete variant | + +**Permissions**: `CREATE_ASSET_VARIANTS`, `READ_ASSET_VARIANTS`, etc. + +### POST /api/asset_variants + +**Request:** +```json +{ + "data": { + "asset": "asset-uuid", + "variant_type": "mobile", + "cdn_url": "https://cdn.example.com/assets/hero-mobile.jpg", + "width_px": 768, + "height_px": 432, + "size_mb": 0.8 + } +} +``` + +**Variant Types:** `desktop`, `mobile`, `thumbnail` + +--- + +## 11. Project Audio Tracks + +### Standard CRUD Endpoints + +| Method | Endpoint | Description | +|--------|----------|-------------| +| POST | `/api/project_audio_tracks` | Create audio track | +| GET | `/api/project_audio_tracks` | List audio tracks | +| GET | `/api/project_audio_tracks/:id` | Get audio track | +| PUT | `/api/project_audio_tracks/:id` | Update audio track | +| DELETE | `/api/project_audio_tracks/:id` | Delete audio track | + +**Runtime Public Access**: GET endpoints accessible without auth in production mode. + +### POST /api/project_audio_tracks + +**Request:** +```json +{ + "data": { + "projectId": "project-uuid", + "environment": "dev", + "name": "Background Music", + "slug": "background-music", + "url": "assets/music.mp3", + "loop": true, + "volume": 0.5, + "sort_order": 1, + "is_enabled": true + } +} +``` + +--- + +## 12. Project Transition Settings + +Environment-aware CSS transition settings for page navigation. + +### Authentication Model + +Uses **URL-path-based public access** - no headers required: + +| Endpoint | Method | Environment | Auth Required | +|----------|--------|-------------|---------------| +| `/project/:id/env/production` | GET | production | **No** (public) | +| `/project/:id/env/dev` | GET | dev | JWT + READ_PAGE_ELEMENTS | +| `/project/:id/env/stage` | GET | stage | JWT + READ_PAGE_ELEMENTS | +| `/project/:id/env/*` | PUT/DELETE | any | JWT + UPDATE_PAGE_ELEMENTS | +| Standard CRUD | all | n/a | JWT + PAGE_ELEMENTS CRUD permission | + +This allows public presentations (`/p/[slug]`) to fetch production transition settings without authentication in incognito mode. + +### Standard CRUD Endpoints (All Require Auth) + +| Method | Endpoint | Description | +|--------|----------|-------------| +| POST | `/api/project-transition-settings` | Create settings | +| GET | `/api/project-transition-settings` | List all settings | +| GET | `/api/project-transition-settings/:id` | Get by ID | +| PUT | `/api/project-transition-settings/:id` | Update by ID | +| DELETE | `/api/project-transition-settings/:id` | Delete by ID | + +### Environment-Specific Endpoints + +| Method | Endpoint | Auth | Description | +|--------|----------|------|-------------| +| GET | `/project/:projectId/env/production` | None | Get production settings (public) | +| GET | `/project/:projectId/env/dev` | JWT | Get dev settings | +| GET | `/project/:projectId/env/stage` | JWT | Get stage settings | +| PUT | `/project/:projectId/env/:environment` | JWT + UPDATE_PAGE_ELEMENTS | Create or update (upsert) | +| DELETE | `/project/:projectId/env/:environment` | JWT + UPDATE_PAGE_ELEMENTS | Reset settings to global defaults | + +### GET /api/project-transition-settings/project/:projectId/env/:environment + +**Parameters:** +- `projectId` (path): Project UUID +- `environment` (path): `dev`, `stage`, or `production` + +**Authentication:** +- `production`: None required (public access) +- `dev`/`stage`: `Authorization: Bearer {token}` + +**Response (200):** +```json +{ + "id": "uuid", + "projectId": "project-uuid", + "environment": "production", + "transition_type": "fade", + "duration_ms": 700, + "easing": "ease-in-out", + "overlay_color": "#000000", + "createdAt": "2026-05-01T12:00:00Z", + "updatedAt": "2026-05-01T12:00:00Z" +} +``` + +**Response (200):** `null` when no settings exist (use global/fallback defaults) + +**Response (401):** Authentication required (for dev/stage without JWT) + +### PUT /api/project-transition-settings/project/:projectId/env/:environment + +Creates or updates settings for a project/environment combination. + +**Authentication**: Required (JWT + UPDATE_PAGE_ELEMENTS) + +**Request:** +```json +{ + "transition_type": "fade", + "duration_ms": 1000, + "easing": "ease-out", + "overlay_color": "#1a1a1a" +} +``` + +**Response (200):** Created or updated settings object + +--- + +## 13. Global UI Controls + +Configurable fullscreen, global sound, and offline buttons. Settings cascade +from global defaults to project/environment overrides and then page overrides. + +### Global Defaults + +| Method | Endpoint | Auth | Description | +|--------|----------|------|-------------| +| GET | `/api/global-ui-control-defaults` | None | Get singleton global UI-control defaults | +| GET | `/api/global-ui-control-defaults/:id` | None | Get defaults by ID | +| PUT | `/api/global-ui-control-defaults/:id` | JWT + UPDATE_PAGE_ELEMENTS | Update singleton defaults | + +### Project/Environment Overrides + +| Method | Endpoint | Auth | Description | +|--------|----------|------|-------------| +| GET | `/api/project-ui-control-settings/project/:projectId/env/production` | None or JWT for private production | Get production project overrides | +| GET | `/api/project-ui-control-settings/project/:projectId/env/dev` | JWT | Get dev project overrides | +| GET | `/api/project-ui-control-settings/project/:projectId/env/stage` | JWT | Get stage project overrides | +| PUT | `/api/project-ui-control-settings/project/:projectId/env/:environment` | JWT + UPDATE_PAGE_ELEMENTS | Upsert project overrides | +| DELETE | `/api/project-ui-control-settings/project/:projectId/env/:environment` | JWT + UPDATE_PAGE_ELEMENTS | Reset project overrides to global defaults | + +`production` reads follow private-production presentation rules: public +presentations are public-readable, private presentations require JWT and an +allowlist/staff access check. + +### Settings JSON + +```json +{ + "offline": { + "enabled": true, + "hidden": false, + "xPercent": 89.5, + "yPercent": 6, + "anchor": "center", + "buttonSizePercent": 2.6, + "iconSizePercent": 1.35, + "defaultIconUrl": "", + "activeIconUrl": "", + "defaultBackgroundColor": "#2563EB", + "activeBackgroundColor": "#059669", + "hoverBackgroundColor": "#1D4ED8", + "color": "#FFFFFF", + "defaultBorderColor": "#2563EB", + "activeBorderColor": "#059669", + "borderRadiusPercent": 0.42, + "opacity": 1, + "boxShadow": "", + "zIndex": 900, + "order": 1 + } +} +``` + +--- + +## 14. Element Defaults + +### Global Transition Defaults + +Platform-wide default transition settings (singleton). + +| Method | Endpoint | Auth | Description | +|--------|----------|------|-------------| +| GET | `/api/global-transition-defaults` | **None** | Get defaults (public) | +| GET | `/api/global-transition-defaults/:id` | **None** | Get by ID (public) | +| PUT | `/api/global-transition-defaults/:id` | JWT + UPDATE_PAGE_ELEMENTS | Update defaults | + +**Authentication Model**: GET is always public (for runtime presentations), PUT requires JWT. + +**GET /api/global-transition-defaults** + +```http +GET /api/global-transition-defaults +# No authentication required +``` + +**Response (200):** +```json +{ + "id": "uuid", + "transition_type": "fade", + "duration_ms": 700, + "easing": "ease-in-out", + "overlay_color": "#000000" +} +``` + +**PUT /api/global-transition-defaults/:id** + +```http +PUT /api/global-transition-defaults/:id +Authorization: Bearer {token} +Content-Type: application/json + +{ + "data": { + "transition_type": "fade", + "duration_ms": 1000, + "easing": "ease-out" + } +} +``` + +--- + +### Element Type Defaults (Global) + +| Method | Endpoint | Description | +|--------|----------|-------------| +| GET | `/api/element-type-defaults` | List global defaults | +| GET | `/api/element-type-defaults/:id` | Get default by ID | +| PUT | `/api/element-type-defaults/:id` | Update default | + +**Alternative Path:** `/api/ui-elements` (backwards compatibility) + +**Element Types (11 predefined):** +- `button`, `hotspot`, `tooltip`, `gallery` +- `media_player`, `text_block`, `popup` +- `logo`, `spot`, `hamburger_menu`, `image` + +--- + +### Project Element Defaults + +| Method | Endpoint | Description | +|--------|----------|-------------| +| GET | `/api/project-element-defaults` | List project defaults | +| GET | `/api/project-element-defaults/:id` | Get default by ID | +| PUT | `/api/project-element-defaults/:id` | Update default | +| POST | `/api/project-element-defaults/:id/reset` | Reset to global | +| GET | `/api/project-element-defaults/:id/diff` | Compare with global | + +### POST /api/project-element-defaults/:id/reset + +Reset project element default to current global settings. + +**Response:** +```json +{ + "id": "uuid", + "element_type": "button", + "projectId": "project-uuid", + "default_props_json": {...} +} +``` + +--- + +### GET /api/project-element-defaults/:id/diff + +Compare project default with global default. + +**Response:** +```json +{ + "hasDifferences": true, + "projectDefault": {...}, + "globalDefault": {...}, + "differences": { + "backgroundColor": { + "project": "#FF0000", + "global": "#0000FF" + } + } +} +``` + +--- + +## 15. Publishing + +### POST /api/publish + +Publish from stage to production. + +**Authentication**: Required +**Permissions**: `CREATE_PUBLISH_EVENTS` + +**Request:** +```json +{ + "projectId": "project-uuid", + "title": "Version 1.0 Release", + "description": "Initial public release with all pages" +} +``` + +**Response:** +```json +{ + "success": true, + "publishEventId": "event-uuid", + "summary": { + "pages_copied": 10, + "audios_copied": 3 + } +} +``` + +--- + +### POST /api/publish/save-to-stage + +Copy dev content to stage environment. + +**Authentication**: Required +**Permissions**: `CREATE_PUBLISH_EVENTS` + +**Request:** +```json +{ + "projectId": "project-uuid" +} +``` + +**Response:** +```json +{ + "success": true, + "publishEventId": "event-uuid", + "summary": { + "pages_copied": 10, + "audios_copied": 3 + } +} +``` + +**Errors:** +- `400`: Publish already in progress +- `404`: Project not found + +--- + +## 16. File Management + +### GET /api/file/download + +Download a file from storage. Supports automatic client disconnect handling via AbortController. + +**Authentication**: None (uses private URL) +**Rate Limit**: 200/min + +**Query Parameters:** +- `privateUrl`: Storage key (e.g., `assets/image.jpg`) + +**Response:** File stream with appropriate Content-Type + +**Error Responses:** + +| HTTP Status | Condition | S3 Error Types | +|-------------|-----------|----------------| +| 400 | Missing privateUrl parameter | - | +| 401 | Expired credentials | ExpiredToken | +| 403 | Access denied | AccessDenied, InvalidAccessKeyId | +| 404 | File not found | NoSuchKey, NotFound, NoSuchBucket | +| 429 | Rate limited | ThrottlingException | +| 500 | Internal server error | InternalError | +| 503 | Service unavailable | NetworkingError, ServiceUnavailable | +| 504 | Gateway timeout | TimeoutError, RequestTimeout | + +**Error Response Format:** +```json +{ + "message": "Could not download the file. NoSuchKey: The specified key does not exist." +} +``` + +**Client Disconnect:** When a client disconnects during download, the backend automatically aborts the S3 request to prevent wasted bandwidth. + +--- + +### POST /api/file/presign + +Generate presigned URLs for direct S3 access. Includes path validation to prevent directory traversal attacks. + +**Authentication**: None +**Rate Limit**: 200/min + +**Request:** +```json +{ + "urls": [ + "assets/image1.jpg", + "assets/image2.jpg", + "assets/video.mp4" + ] +} +``` + +**Response:** +```json +{ + "presignedUrls": { + "assets/image1.jpg": "https://bucket.s3.amazonaws.com/assets/image1.jpg?...", + "assets/image2.jpg": "https://bucket.s3.amazonaws.com/assets/image2.jpg?...", + "assets/video.mp4": "https://bucket.s3.amazonaws.com/assets/video.mp4?..." + } +} +``` + +**Limits:** +- Maximum 50 URLs per request +- Presigned URLs expire in 1 hour (configurable via `AWS_S3_PRESIGN_EXPIRY`) + +**Path Validation:** +URLs are validated to prevent path traversal and ensure security: +- Must be non-empty strings +- Must not contain `..` (parent directory traversal) +- Must not start with `/` (absolute paths) +- Must not contain null bytes + +**Error Responses:** + +| HTTP Status | Condition | +|-------------|-----------| +| 400 | Missing `urls` array | +| 400 | `urls` array is empty | +| 400 | `urls` exceeds maximum of 50 | +| 400 | Invalid URL format (contains `..`, starts with `/`, etc.) | +| 500 | S3 presigning failed | +| 503 | S3 service unavailable | + +**Error Response Format:** +```json +{ + "message": "Invalid URL format", + "code": "INVALID_PATH", + "details": { "invalidUrls": ["../etc/passwd", "/root/secret"] } +} +``` + +--- + +### POST /api/file/upload/:table/:field + +Upload a file (single request). + +**Authentication**: Required +**Rate Limit**: 10/min +**Content-Type**: `multipart/form-data` + +**Form Fields:** +- `file`: Binary file data +- `filename`: Target filename + +**Response:** +```json +{ + "message": "Uploaded the file successfully: assets/hero.jpg", + "url": "https://bucket.s3.amazonaws.com/assets/hero.jpg" +} +``` + +--- + +### Chunked Upload (Large Files) + +For files larger than a few MB, use chunked upload: + +#### 1. Initialize Session + +**POST /api/file/upload-sessions/init** + +**Request:** +```json +{ + "folder": "assets", + "filename": "large-video.mp4", + "totalChunks": 10, + "size": 104857600, + "contentType": "video/mp4" +} +``` + +**Response:** +```json +{ + "sessionId": "session-uuid", + "uploadedChunks": [], + "totalChunks": 10 +} +``` + +--- + +#### 2. Upload Chunks + +**PUT /api/file/upload-sessions/:sessionId/chunks/:chunkIndex** + +**Content-Type**: `application/octet-stream` +**Body**: Raw binary chunk data + +**Response:** +```json +{ + "sessionId": "session-uuid", + "chunkIndex": 0, + "uploadedChunks": 1, + "totalChunks": 10 +} +``` + +--- + +#### 3. Check Session Status + +**GET /api/file/upload-sessions/:sessionId** + +**Response:** +```json +{ + "sessionId": "session-uuid", + "totalChunks": 10, + "uploadedChunks": [0, 1, 2, 3, 4], + "status": "uploading" +} +``` + +--- + +#### 4. Finalize Upload + +**POST /api/file/upload-sessions/:sessionId/finalize** + +**Response:** +```json +{ + "message": "Uploaded the file successfully: assets/large-video.mp4", + "privateUrl": "assets/large-video.mp4", + "url": "https://bucket.s3.amazonaws.com/assets/large-video.mp4" +} +``` + +--- + +## 17. Search + +### POST /api/search + +Global full-text search across entities. + +**Authentication**: Required +**Rate Limit**: 30/min +**Permissions**: `READ_SEARCH` (implicit via entity permissions) + +**Request:** +```json +{ + "searchQuery": "museum" +} +``` + +**Response:** +```json +[ + { + "id": "uuid", + "name": "Museum Tour", + "tableName": "projects", + "matchAttribute": ["name", "description"] + }, + { + "id": "uuid", + "slug": "museum-entrance", + "tableName": "tour_pages", + "matchAttribute": ["slug"] + } +] +``` + +**Searchable Tables:** + +| Table | Text Fields | Numeric Fields | +|-------|-------------|----------------| +| `users` | firstName, lastName, phoneNumber, email | - | +| `projects` | name, slug, description, logo_url, favicon_url, og_image_url | - | +| `assets` | name, cdn_url, storage_key, mime_type, checksum | size_mb, width_px, height_px, duration_sec | +| `asset_variants` | cdn_url | width_px, height_px, size_mb | +| `presigned_url_requests` | requested_key, mime_type, status | requested_size_mb | +| `tour_pages` | source_key, name, slug, background_image_url, background_video_url, background_audio_url, ui_schema_json | sort_order | +| `project_audio_tracks` | source_key, name, slug, url | volume, sort_order | +| `publish_events` | error_message | pages_copied, transitions_copied, audios_copied | +| `pwa_caches` | cache_version, manifest_json, asset_list_json | - | +| `access_logs` | path, ip_address, user_agent | - | + +--- + +## 20. Access Logs + +### Standard CRUD Endpoints + +| Method | Endpoint | Description | +|--------|----------|-------------| +| GET | `/api/access_logs` | List access logs | +| GET | `/api/access_logs/count` | Count access logs | +| GET | `/api/access_logs/:id` | Get access log by ID | + +**Permissions**: `READ_ACCESS_LOGS` + +### GET /api/access_logs + +**Query Parameters:** +- `path`: Filter by path +- `ip_address`: Filter by IP +- `user_agent`: Filter by user agent +- Standard pagination params + +**Response:** +```json +{ + "rows": [ + { + "id": "uuid", + "path": "/p/my-tour/", + "ip_address": "192.168.1.1", + "user_agent": "Mozilla/5.0...", + "project": { "id": "uuid", "name": "My Tour" }, + "createdAt": "2026-03-30T12:00:00.000Z" + } + ], + "count": 100 +} +``` + +--- + +## 21. Publish Events + +### Standard CRUD Endpoints + +| Method | Endpoint | Description | +|--------|----------|-------------| +| GET | `/api/publish_events` | List publish events | +| GET | `/api/publish_events/count` | Count publish events | +| GET | `/api/publish_events/:id` | Get publish event by ID | + +**Permissions**: `READ_PUBLISH_EVENTS` + +### GET /api/publish_events + +**Query Parameters:** +- `project`: Filter by project ID +- `status`: Filter by status (queued, running, success, failed) +- `from_environment`: Filter by source (dev, stage) +- `to_environment`: Filter by target (stage, production) +- Standard pagination params + +**Response:** +```json +{ + "rows": [ + { + "id": "uuid", + "title": "Version 1.0", + "description": "Initial release", + "from_environment": "stage", + "to_environment": "production", + "status": "success", + "pages_copied": 10, + "audios_copied": 3, + "started_at": "2026-03-30T12:00:00.000Z", + "finished_at": "2026-03-30T12:00:05.000Z", + "project": { "id": "uuid", "name": "My Tour" }, + "user": { "id": "uuid", "firstName": "John" } + } + ], + "count": 5 +} +``` + +--- + +## 22. PWA Caches + +### Standard CRUD Endpoints + +| Method | Endpoint | Description | +|--------|----------|-------------| +| POST | `/api/pwa_caches` | Create PWA cache record | +| GET | `/api/pwa_caches` | List PWA caches | +| GET | `/api/pwa_caches/:id` | Get PWA cache by ID | +| PUT | `/api/pwa_caches/:id` | Update PWA cache | +| DELETE | `/api/pwa_caches/:id` | Delete PWA cache | + +**Permissions**: `CREATE_PWA_CACHES`, `READ_PWA_CACHES`, etc. + +### POST /api/pwa_caches + +**Request:** +```json +{ + "data": { + "project": "project-uuid", + "cache_version": "v1.0.0", + "manifest_json": { + "name": "My Tour", + "short_name": "Tour" + }, + "asset_list_json": [ + "assets/image1.jpg", + "assets/image2.jpg" + ] + } +} +``` + +--- + +## 23. Presigned URL Requests + +### Standard CRUD Endpoints + +| Method | Endpoint | Description | +|--------|----------|-------------| +| POST | `/api/presigned_url_requests` | Create presign request | +| GET | `/api/presigned_url_requests` | List presign requests | +| GET | `/api/presigned_url_requests/:id` | Get presign request | +| PUT | `/api/presigned_url_requests/:id` | Update presign request | +| DELETE | `/api/presigned_url_requests/:id` | Delete presign request | + +**Permissions**: `CREATE_PRESIGNED_URL_REQUESTS`, `READ_PRESIGNED_URL_REQUESTS`, etc. + +### POST /api/presigned_url_requests + +**Request:** +```json +{ + "data": { + "requested_key": "assets/new-upload.jpg", + "mime_type": "image/jpeg", + "requested_size_mb": 2.5, + "status": "pending", + "project": "project-uuid", + "user": "user-uuid" + } +} +``` + +--- + +## Error Codes Reference + +| Code | Message | Description | +|------|---------|-------------| +| 400 | Bad Request | Invalid input data or validation error | +| 401 | Unauthorized | Missing or invalid JWT token | +| 403 | Forbidden | Insufficient permissions | +| 404 | Not Found | Resource does not exist | +| 409 | Conflict | Resource conflict (e.g., duplicate) | +| 422 | Unprocessable Entity | Validation failed | +| 429 | Too Many Requests | Rate limit exceeded | +| 500 | Internal Server Error | Unexpected server error | + +--- + +## Headers Reference + +### Request Headers + +| Header | Description | Required | +|--------|-------------|----------| +| `Authorization` | `Bearer ` | For protected endpoints | +| `Content-Type` | `application/json` | For JSON bodies | +| `X-Runtime-Environment` | `dev`, `stage`, or `production` | For runtime context | +| `X-Request-Id` | UUID for request tracing | Optional | + +### Response Headers + +| Header | Description | +|--------|-------------| +| `X-Request-Id` | Request ID for tracing | +| `X-RateLimit-Limit` | Rate limit maximum | +| `X-RateLimit-Remaining` | Remaining requests | +| `X-RateLimit-Reset` | Reset timestamp | +| `Retry-After` | Seconds until rate limit resets (on 429) | +| `Cross-Origin-Resource-Policy` | `cross-origin` (on file downloads) | diff --git a/backend/docs/backend-architecture.md b/backend/docs/backend-architecture.md new file mode 100644 index 0000000..f841882 --- /dev/null +++ b/backend/docs/backend-architecture.md @@ -0,0 +1,681 @@ +# Backend Architecture Documentation + +This document provides a comprehensive analysis of the Tour Builder Platform backend architecture, including design patterns, layers, and implementation details. + +## Overview + +- **Runtime**: Node.js 24 LTS +- **Framework**: Express.js 4.x +- **ORM**: Sequelize 6.x with PostgreSQL +- **Authentication**: Passport.js (JWT, Google OAuth, Microsoft OAuth) +- **Documentation**: Swagger/OpenAPI 3.0 +- **Logging**: Pino (structured JSON logging) +- **TypeScript/ESM**: the backend package is `"type": "module"` and active + backend source is strict TypeScript ESM. +- **File Storage**: AWS S3 / Local filesystem (Strategy Pattern, provider-based) + +--- + +## Architecture Diagram + +``` +┌──────────────────────────────────────────────────────────────────────────┐ +│ Express Application │ +│ (src/index.ts) │ +└──────────────────────────────────────────────────────────────────────────┘ + │ + ┌─────────────────┼─────────────────┐ + ▼ ▼ ▼ +┌────────────────────────┐ ┌────────────────┐ ┌────────────────────────┐ +│ Middleware │ │ Rate Limiter │ │ Request Logger │ +│ • runtimeContext │ │ • auth (10/15m)│ │ • Pino structured │ +│ • runtimePublic │ │ • api (100/1m) │ │ • Request ID tracking │ +│ • checkPermissions │ │ • upload (10/m)│ │ • Duration metrics │ +│ • passport JWT │ │ • download(200)│ └────────────────────────┘ +└────────────────────────┘ │ • search (30/m)│ + │ • AI (20/1m) │ + └────────────────┘ + │ +┌──────────────────────────────────────────────────────────────────────────┐ +│ Routes Layer │ +│ (src/routes/*.ts) │ +│ ┌────────────────────────────────────────────────────────────────────┐ │ +│ │ Factory-generated routes: router.factory.js │ │ +│ │ • Standard CRUD: POST /, PUT /:id, DELETE /:id, GET /, GET /:id │ │ +│ │ • Extra: GET /count, GET /autocomplete, POST /deleteByIds │ │ +│ │ • CSV export: GET /?filetype=csv │ │ +│ └────────────────────────────────────────────────────────────────────┘ │ +│ ┌────────────────────────────────────────────────────────────────────┐ │ +│ │ Custom routes: auth, file, publish, search, sql, runtime │ │ +│ └────────────────────────────────────────────────────────────────────┘ │ +└──────────────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌──────────────────────────────────────────────────────────────────────────┐ +│ Service Layer │ +│ (src/services/*.js) │ +│ ┌────────────────────────────────────────────────────────────────────┐ │ +│ │ Factory-generated services: service.factory.js │ │ +│ │ • Transaction-wrapped CRUD operations │ │ +│ │ • Delegates to DB API layer │ │ +│ └────────────────────────────────────────────────────────────────────┘ │ +│ ┌────────────────────────────────────────────────────────────────────┐ │ +│ │ Custom services: │ │ +│ │ • auth.js - Authentication logic (signin, signup, password) │ │ +│ │ • publish.ts - Publishing workflow service (dev→stage→production)│ │ +│ │ • file.ts - File storage operations │ │ +│ │ • email/index.js - Email sending via Nodemailer/SES │ │ +│ │ • search.ts - Full-text search route │ │ +│ │ • pwa_manifest.js - PWA offline manifest generation │ │ +│ └────────────────────────────────────────────────────────────────────┘ │ +└──────────────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌──────────────────────────────────────────────────────────────────────────┐ +│ Database API Layer │ +│ (src/db/api/*.js) │ +│ ┌────────────────────────────────────────────────────────────────────┐ │ +│ │ GenericDBApi (base.api.js) - Template Method Pattern │ │ +│ │ • Configurable: MODEL, SEARCHABLE_FIELDS, RANGE_FIELDS, etc. │ │ +│ │ • CRUD: create, update, remove, deleteByIds │ │ +│ │ • Query: findBy, findAll, findAllAutocomplete │ │ +│ │ • Data Transform: getFieldMapping(), JSON_FIELDS, FIELD_DEFAULTS │ │ +│ └────────────────────────────────────────────────────────────────────┘ │ +│ ┌────────────────────────────────────────────────────────────────────┐ │ +│ │ Entity APIs extend GenericDBApi: │ │ +│ │ UsersDBApi, ProjectsDBApi, AssetsDBApi, TourPagesDBApi, etc. │ │ +│ └────────────────────────────────────────────────────────────────────┘ │ +└──────────────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌──────────────────────────────────────────────────────────────────────────┐ +│ Sequelize Models │ +│ (src/db/models/*.ts) │ +│ Entity models + file model, loaded dynamically via loader.ts │ +└──────────────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌──────────────────────────────────────────────────────────────────────────┐ +│ PostgreSQL │ +└──────────────────────────────────────────────────────────────────────────┘ +``` + +--- + +## Directory Structure + +``` +backend/src/ +├── index.ts # Application entry point +├── config.ts # Environment configuration +├── helpers.ts # Utility functions (wrapAsync, JWT, UUID validation) +├── types/ # Reusable strict TypeScript contracts for migrated code +│ +├── auth/ +│ └── auth.ts # Passport strategies (JWT, Google, Microsoft) +│ +├── middlewares/ +│ ├── check-permissions.ts # RBAC permission checking +│ ├── runtime-context.ts # Runtime environment context (dev/stage/production) +│ ├── runtime-public.ts # Public runtime access control & field sanitization +│ ├── rateLimiter.ts # Rate limiting (auth, API, upload, download) +│ └── upload.ts # File upload handling (multer) +│ +├── routes/ +│ ├── auth.ts # Authentication routes (custom) +│ ├── file.ts # File upload/download routes (custom) +│ ├── publish.ts # Publishing workflow routes (custom) +│ ├── search.ts # Full-text search routes (custom) +│ ├── runtime-context.ts # Runtime context detection (custom) +│ └── [entity].ts # Entity CRUD routes +│ +├── services/ +│ ├── auth.ts # Authentication service +│ ├── publish.ts # Publishing workflow (dev→stage→production) +│ ├── file.ts # Unified file storage service +│ ├── search.ts # Full-text search service +│ ├── file/ # File storage providers (S3, Local) +│ │ ├── index.ts # Module exports & provider factory +│ │ ├── BaseStorageProvider.ts # Abstract base class +│ │ ├── S3StorageProvider.ts # AWS S3 implementation +│ │ ├── LocalStorageProvider.ts # Local filesystem implementation +│ │ └── UploadSessionManager.ts # Chunked upload session management +│ ├── email/ +│ │ ├── index.ts # Email sender (Nodemailer/SES) +│ │ └── list/ # Email templates +│ ├── notifications/ +│ │ ├── helpers.ts # Notification helpers +│ │ └── errors/ # Error classes (ValidationError, ForbiddenError) +│ └── [entity].ts # Entity services +│ +├── factories/ +│ ├── router.factory.ts # Route generator (createEntityRouter) +│ └── service.factory.ts # Service generator (createEntityService) +│ +├── db/ +│ ├── db-config.ts # Database configuration (env-based) +│ ├── umzug.ts # Migration and seeder runner +│ ├── models/ +│ │ ├── index.ts # Model registry entrypoint +│ │ ├── loader.ts # Model loader +│ │ └── [entity].ts # Sequelize models +│ ├── api/ +│ │ ├── base.api.ts # GenericDBApi base class +│ │ ├── runtime-context.ts # Runtime filtering helpers +│ │ └── [entity].ts # Entity DB APIs +│ ├── migrations/ # Applied migration history +│ └── seeders/ # Typed seed data files +│ +└── utils/ + ├── index.ts # Utils barrel export + ├── errors.ts # Error classes (AppError, NotFoundError, etc.) + ├── logger.ts # Pino logger configuration + └── env-validation.ts # Environment variable validation +``` + +--- + +## Design Patterns + +### 1. Factory Pattern + +**Router Factory** (`factories/router.factory.js`) + +Generates standardized CRUD routes for entities: + +```javascript +const { createEntityRouter } = require('../factories/router.factory'); + +// Creates routes: POST /, PUT /:id, DELETE /:id, GET /, GET /:id, GET /count, GET /autocomplete +module.exports = createEntityRouter('assets', AssetsService, AssetsDBApi, { + permissionEntity: 'assets', + csvFields: ['id', 'name', 'asset_type', 'createdAt'], +}); +``` + +**Service Factory** (`factories/service.factory.js`) + +Generates transaction-wrapped service classes: + +```javascript +const { createEntityService } = require('../factories/service.factory'); + +// Creates: create(), update(), remove(), deleteByIds(), bulkImport() +module.exports = createEntityService(AssetsDBApi, { entityName: 'Asset' }); +``` + +### 2. Template Method Pattern + +**GenericDBApi** (`db/api/base.api.js`) + +Base class with configurable hooks for entity-specific behavior: + +```javascript +class AssetsDBApi extends GenericDBApi { + // Required: Define the Sequelize model + static get MODEL() { return db.assets; } + + // Configurable behavior via static getters + static get SEARCHABLE_FIELDS() { return ['name', 'cdn_url']; } + static get RANGE_FIELDS() { return ['size_mb', 'width_px']; } + static get ENUM_FIELDS() { return ['asset_type', 'is_public']; } + static get JSON_FIELDS() { return ['settings_json']; } + static get FIELD_DEFAULTS() { return { type: { default: 'general' } }; } + static get ASSOCIATIONS() { return [{ field: 'project', setter: 'setProject' }]; } + static get FIND_BY_INCLUDES() { return [{ association: 'project' }]; } + static get FIND_ALL_INCLUDES() { return [{ model: db.projects, as: 'project' }]; } + + // Custom field transformation + static getFieldMapping(data) { + return { + name: data.name || null, + asset_type: data.asset_type || null, + type: data.type || 'general', + // ... + }; + } +} +``` + +### 3. Strategy Pattern + +**File Storage Providers** (`services/file/`) + +Two concrete implementations with pluggable architecture: + +``` +BaseStorageProvider (abstract) + │ + ├── S3StorageProvider # AWS S3 implementation (with timeout/retry) + └── LocalStorageProvider # Local filesystem implementation +``` + +The storage provider base, S3 provider, and local provider are migrated TS/ESM modules. The S3 implementation uses official AWS SDK v3 types; shared provider-domain contracts are in `src/types/file.ts`. + +Interface: +- `upload(key, data, options)` → `{ key, url }` +- `download(key)` → `{ body, contentType }` +- `delete(key)` → `void` +- `deleteMany(keys)` → `void` +- `exists(key)` → `boolean` +- `list(prefix)` → `string[]` +- `getSignedUrl(key, expiresIn)` → `string` + +### 4. Middleware Chain Pattern + +Request flow through middleware stack: + +``` +Request → requestLogger → runtimeContextMiddleware → rateLimiter + → passport.authenticate('jwt') → checkCrudPermissions + → Route Handler → Service → DB API → Database + → Response +``` + +--- + +## Layers in Detail + +### Entry Point (`src/index.ts`) + +Application bootstrap: + +1. **Security**: Helmet, CORS configuration +2. **Logging**: Request logger middleware (Pino) +3. **Authentication**: Passport JWT initialization +4. **Rate Limiting**: Per-route rate limiters +5. **Body Parsing**: JSON (1mb limit), applied after file routes +6. **Runtime Context**: Environment detection middleware +7. **Route Mounting**: Entity routes with auth/permissions +8. **Error Handling**: Generic error handler +9. **Static Files**: Public directory serving + +```javascript +// Key route mounting patterns +app.use('/api/auth', authRoutes); // No JWT required +app.use('/api/users', jwtAuth, usersRoutes); // JWT required + +// Runtime public routes (production content accessible without auth) +const mountRuntimeEntityRoute = (path, entityName, router) => { + app.use(path, + requireRuntimeReadOrAuth, // JWT or public production + blockNonPublicRuntimeListEndpoints, // Block non-list endpoints + sanitizePublicRuntimeListResponse(entityName), // Filter sensitive fields + router + ); +}; +mountRuntimeEntityRoute('/api/projects', 'projects', projectsRoutes); +mountRuntimeEntityRoute('/api/tour_pages', 'tour_pages', tour_pagesRoutes); +``` + +### Routes Layer + +**Factory-Generated Routes** provide standard CRUD: + +| Method | Path | Description | +|--------|------|-------------| +| POST | `/` | Create record | +| POST | `/bulk-import` | Bulk import from CSV | +| PUT | `/:id` | Update record | +| DELETE | `/:id` | Delete record | +| POST | `/deleteByIds` | Bulk delete | +| GET | `/` | List with pagination & filters | +| GET | `/count` | Count only | +| GET | `/autocomplete` | Autocomplete search | +| GET | `/:id` | Get single record | + +**Custom Routes** (auth, file, publish, search, runtime-context): + +| Route | Endpoints | +|-------|-----------| +| `/api/auth` | signin, signup, me, password-reset, verify-email, Google/Microsoft OAuth | +| `/api/file` | upload, download, presign, upload-sessions (chunked) | +| `/api/publish` | publish (stage→production), save-to-stage (dev→stage) | +| `/api/search` | Global full-text search | +| `/api/runtime-context` | Runtime environment detection | + +### Service Layer + +**Transaction Management**: Services wrap operations in transactions: + +```javascript +static async create({ data, currentUser, transaction: externalTransaction, runtimeContext }) { + const transaction = externalTransaction || await db.sequelize.transaction(); + const ownsTransaction = !externalTransaction; + try { + const record = await DBApi.create({ data, currentUser, transaction, runtimeContext }); + if (ownsTransaction) await transaction.commit(); + return record; + } catch (error) { + if (ownsTransaction) await transaction.rollback(); + throw error; + } +} +``` + +**Publish Service** (`services/publish.ts`): + +Implements the dev→stage→production workflow with: +- Transaction locking to prevent concurrent publishes +- Source key tracking for content lineage +- Bulk copy operations for pages and audio tracks + +### Database API Layer + +**Query Building** in `findAll()`: + +| Filter Type | Example | SQL | +|-------------|---------|-----| +| Text search | `?name=foo` | `name ILIKE '%foo%'` | +| Range | `?size_mbRange=[0,100]` | `size_mb >= 0 AND size_mb <= 100` | +| Enum | `?asset_type=image` | `asset_type = 'image'` | +| Relation | `?project=uuid` | JOIN with projects table | +| Sort | `?field=name&sort=asc` | `ORDER BY name ASC` | +| Pagination | `?page=1&limit=10` | `OFFSET 0 LIMIT 10` | + +--- + +## Authentication & Authorization + +### Authentication Flow + +``` +┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ +│ POST /signin │ │ Passport JWT │ │ Protected │ +│ → JWT Token │────▶│ Middleware │────▶│ Route │ +└─────────────────┘ └─────────────────┘ └─────────────────┘ + │ + ▼ + req.currentUser = { + id, email, app_role, + custom_permissions + } +``` + +### Authorization (RBAC) + +**Permission Check Flow** (`middlewares/check-permissions.ts`): + +1. Self-access bypass (user accessing own resource) +2. Check custom permissions (user-specific) +3. Check role permissions (from app_role) +4. Fallback to Public role for unauthenticated + +**Permission Naming Convention**: +- `CREATE_` - Create records +- `READ_` - Read records +- `UPDATE_` - Modify records +- `DELETE_` - Delete records + +```javascript +// Auto-generated from HTTP method +const permissionName = `${METHOD_MAP[req.method]}_${name.toUpperCase()}`; +// POST /api/assets → CREATE_ASSETS +// GET /api/assets → READ_ASSETS +``` + +### Runtime Public Access + +For production content accessible without authentication: + +```javascript +const requireRuntimeReadOrAuth = (req, res, next) => { + const isPublicEnvironment = req.runtimeContext?.headerEnvironment === 'production'; + const isReadOnlyRequest = ['GET', 'OPTIONS'].includes(req.method); + + if (isPublicEnvironment && isReadOnlyRequest && !hasAuthHeader) { + req.isRuntimePublicRequest = true; + return next(); // Allow without JWT + } + + return jwtAuth(req, res, next); // Require JWT +}; +``` + +--- + +## Rate Limiting + +Pre-configured limiters (`middlewares/rateLimiter.ts`): + +| Limiter | Window | Max Requests | Use Case | +|---------|--------|--------------|----------| +| `authLimiter` | 15 min | 10 | Authentication endpoints | +| `passwordResetLimiter` | 1 hour | 5 | Password reset | +| `apiLimiter` | 1 min | 100 | General API | +| `uploadLimiter` | 1 min | 10 | File uploads | +| `downloadLimiter` | 1 min | 200 | File downloads | +| `searchLimiter` | 1 min | 30 | Search queries | + +Headers returned: +- `X-RateLimit-Limit`: Maximum requests +- `X-RateLimit-Remaining`: Remaining requests +- `X-RateLimit-Reset`: Reset time (ISO timestamp) +- `Retry-After`: Seconds until reset (when limited) + +--- + +## File Storage + +**Storage Provider Selection**: + +```javascript +const provider = config.fileStorage.provider || + (hasS3Credentials ? 's3' : hasGCloudCredentials ? 'gcloud' : 'local'); +``` + +**S3 Operations**: + +| Operation | Method | Description | +|-----------|--------|-------------| +| Upload | `upload(key, data, options)` | Put object with metadata | +| Download | `download(key)` | Get object stream | +| Presign | `getSignedUrl(key, expiresIn)` | Generate presigned URL | +| Delete | `delete(key)` / `deleteMany(keys)` | Remove objects | +| Check | `exists(key)` | Head object | +| List | `list(prefix)` | List objects with prefix | + +**Chunked Uploads** (`UploadSessionManager`): + +For large files, supports multipart upload sessions: +1. `POST /upload-sessions/init` - Create session +2. `POST /upload-sessions/:id/chunk` - Upload chunk +3. `POST /upload-sessions/:id/finalize` - Complete upload + +--- + +## Error Handling + +**Error Classes** (`utils/errors.js`): + +```javascript +class AppError extends Error { + constructor(message, statusCode = 500, details = null) { + super(message); + this.statusCode = statusCode; + this.details = details; + this.isOperational = true; + } +} + +class NotFoundError extends AppError { statusCode = 404 } +class ValidationError extends AppError { statusCode = 400 } +class ForbiddenError extends AppError { statusCode = 403 } +class UnauthorizedError extends AppError { statusCode = 401 } +class ConflictError extends AppError { statusCode = 409 } +``` + +**Async Handler** (`helpers.ts`): + +```javascript +// Wraps async route handlers to catch errors +static wrapAsync(fn) { + return function(req, res, next) { + fn(req, res, next).catch(next); + }; +} +``` + +**Common Error Handler** (`helpers.ts`): + +```javascript +static commonErrorHandler(error, req, res, _next) { + const statusCode = error.code || error.status; + + if ([400, 401, 403, 404, 409, 422].includes(statusCode)) { + return res.status(statusCode).send(error.message); + } + + console.error(error); + return res.status(500).send('Internal server error'); +} +``` + +--- + +## Logging + +**Pino Logger** (`utils/logger.js`): + +Logger initialization is a bootstrap exception: it may read `process.env` +directly because importing `config.ts` would create an initialization cycle. + +```javascript +const logger = pino({ + level: process.env.LOG_LEVEL || 'info', + transport: isDevelopment + ? { target: 'pino-pretty', options: { colorize: true } } + : undefined, + base: { + service: 'tour-builder-api', + env: process.env.NODE_ENV || 'development', + }, +}); +``` + +**Request Logging**: + +```javascript +function requestLogger(req, res, next) { + const requestId = req.headers['x-request-id'] || crypto.randomUUID(); + req.log = logger.child({ requestId }); + req.requestId = requestId; + res.setHeader('X-Request-Id', requestId); + + res.on('finish', () => { + req.log.info({ + method: req.method, + url: req.originalUrl, + status: res.statusCode, + duration: Date.now() - start, + }, 'Request completed'); + }); +} +``` + +--- + +## Configuration + +**Environment Variables** (`config.ts`): + +| Variable | Description | Default | +|----------|-------------|---------| +| `SECRET_KEY` | JWT signing key | UUID-based default | +| `ADMIN_EMAIL` | Admin user email | `admin@flatlogic.com` | +| `ADMIN_PASS` | Admin user password | Generated | +| `AWS_S3_BUCKET` | S3 bucket name | - | +| `AWS_S3_REGION` | S3 region | `us-east-1` | +| `AWS_ACCESS_KEY_ID` | AWS access key | - | +| `AWS_SECRET_ACCESS_KEY` | AWS secret key | - | +| `GOOGLE_CLIENT_ID` | Google OAuth client ID | - | +| `GOOGLE_CLIENT_SECRET` | Google OAuth client secret | - | +| `MS_CLIENT_ID` | Microsoft OAuth client ID | - | +| `MS_CLIENT_SECRET` | Microsoft OAuth client secret | - | +| `EMAIL_USER` | SMTP username | - | +| `EMAIL_PASS` | SMTP password | - | +| `LOG_LEVEL` | Logging level | `info` | + +**Database Configuration** (`db/db-config.ts`): + +| Environment | Database | Logging | +|-------------|----------|---------| +| `production` | `DB_*` env vars | Disabled | +| `development` | `db_tour_builder_platform` | Console | +| `dev_stage` | `DB_*` env vars | Console | + +--- + +## API Documentation + +Swagger/OpenAPI documentation is available at `/api-docs`. + +The served document is centralized in `backend/src/openapi/document.ts`. The +OpenAPI module defines shared schemas, common parameters, reusable responses, +and generated standard CRUD paths for every `createEntityRouter` resource. This +keeps the documented factory contract aligned with the route factory endpoints: +create, bulk import, update, delete, bulk delete, list, count, autocomplete, +and get by ID. + +```javascript +const specs = createOpenApiDocument({ + serverUrl: config.server.swaggerServerUrl, +}); +``` + +When adding a new route, update `backend/src/openapi/document.ts` in the same +change. For new factory-backed entities, add the entity schema and `CrudResource` +entry; for custom routes, add an explicit path item. + +--- + +## Health Check + +```javascript +GET /api/health + +{ + "status": "ok", // or "degraded" + "timestamp": "2026-03-29T...", + "uptime": 12345.678, + "environment": "production", + "database": "connected" // or "disconnected" +} +``` + +--- + +## Key Implementation Files + +| File | Purpose | +|------|---------| +| `src/index.ts` | Application entry, middleware setup, route mounting | +| `src/config.ts` | Environment configuration | +| `src/helpers.ts` | wrapAsync, commonErrorHandler, jwtSign, isUuidV4 | +| `src/auth/auth.ts` | Passport strategies (JWT, Google, Microsoft) | +| `src/factories/router.factory.ts` | Route generator for entities | +| `src/factories/service.factory.ts` | Service generator for entities | +| `src/db/api/base.api.ts` | GenericDBApi base class | +| `src/middlewares/check-permissions.ts` | RBAC permission checking | +| `src/middlewares/rateLimiter.ts` | Rate limiting configuration | +| `src/middlewares/runtime-context.ts` | Runtime environment detection | +| `src/middlewares/runtime-public.ts` | Public runtime access control & field sanitization | +| `src/services/publish.ts` | Publishing workflow service | +| `src/services/file/S3StorageProvider.ts` | S3 storage implementation using official AWS SDK v3 types | +| `src/utils/logger.ts` | Pino logger configuration | +| `src/utils/errors.ts` | Error class definitions | + +--- + +## Best Practices Implemented + +1. **Factory Patterns** - Reduce boilerplate for CRUD operations +2. **Template Method** - Configurable base class for DB operations +3. **Strategy Pattern** - Pluggable storage providers +4. **Middleware Chain** - Composable request processing +5. **Transaction Management** - Consistent rollback on errors +6. **Rate Limiting** - Protection against abuse +7. **Structured Logging** - JSON logs with request IDs +8. **Environment-Based Config** - Secure credential handling +9. **Soft Deletes** - Paranoid models for data recovery +10. **RBAC** - Fine-grained permission control diff --git a/backend/docs/database-schema.md b/backend/docs/database-schema.md new file mode 100644 index 0000000..a144a12 --- /dev/null +++ b/backend/docs/database-schema.md @@ -0,0 +1,1025 @@ +# Database Schema Documentation + +This document provides a comprehensive analysis of the Tour Builder Platform database schema, including all models, fields, relationships, constraints, and configurations. + +## Overview + +- **Database**: PostgreSQL +- **ORM**: Sequelize v6 +- **Table Naming**: `freezeTableName: true` - table names match model names exactly +- **Soft Delete**: All models use `paranoid: true` - records are soft-deleted via `deletedAt` timestamp +- **Timestamps**: All models include `createdAt`, `updatedAt`, and `deletedAt` columns +- **Primary Keys**: All models use UUID v4 as primary key + +## Database Configuration + +Configuration is environment-based in `backend/src/db/db-config.ts`: + +| Environment | Database | Logging | +|-------------|----------|---------| +| production | `DB_*` env vars | Disabled | +| development | `db_tour_builder_platform` | Console | +| dev_stage | `DB_*` env vars | Console | + +**Migration Settings:** +- Migration storage: `sequelize` (SequelizeMeta table) +- Seeder storage: `sequelize` + +--- + +## Entity-Relationship Overview + +``` +┌──────────────┐ ┌─────────────────┐ ┌──────────────────┐ +│ users │────<│project_memberships│>───│ projects │ +└──────────────┘ └─────────────────┘ └──────────────────┘ + │ │ + │ app_roleId │ hasMany + ▼ ▼ +┌──────────────┐ ┌──────────────────┐ +│ roles │ │ tour_pages │ +└──────────────┘ │ (ui_schema_json) │ + │ └──────────────────┘ + │ belongsToMany + ▼ +┌──────────────┐ +│ permissions │ +└──────────────┘ + +┌──────────────┐ +│ assets │────< asset_variants +└──────────────┘ + +┌────────────────────────┐ snapshotted from ┌────────────────────────┐ +│ project_element_defaults│ ─────────────────────── │ element_type_defaults │ +└────────────────────────┘ (source_element_id) └────────────────────────┘ + │ │ + │ belongsTo │ (global defaults) + ▼ │ +┌──────────────────┐ │ +│ projects │<────────────────────────────────────────┘ +└──────────────────┘ (templates for new projects) +``` + +**Note:** Page elements, navigation links, and transition videos are stored directly in `tour_pages.ui_schema_json` rather than in separate tables. This simplifies the data model and avoids ID remapping issues when publishing between environments. + +--- + +## Core Models + +### 1. users + +User accounts for authentication and authorization. + +| Field | Type | Constraints | Description | +|-------|------|-------------|-------------| +| `id` | UUID | PK, default: UUIDv4 | Primary identifier | +| `firstName` | TEXT | nullable | User's first name | +| `lastName` | TEXT | nullable | User's last name | +| `phoneNumber` | TEXT | nullable | Contact phone | +| `email` | TEXT | NOT NULL, UNIQUE | Email (validated) | +| `disabled` | BOOLEAN | NOT NULL, default: false | Account disabled flag | +| `password` | TEXT | NOT NULL | Bcrypt hashed password | +| `emailVerified` | BOOLEAN | NOT NULL, default: false | Email verification status | +| `emailVerificationToken` | TEXT | nullable | Token for email verification | +| `emailVerificationTokenExpiresAt` | DATE | nullable | Token expiry | +| `passwordResetToken` | TEXT | nullable | Password reset token | +| `passwordResetTokenExpiresAt` | DATE | nullable | Reset token expiry | +| `provider` | TEXT | NOT NULL, default: 'local' | Auth provider (local, google, microsoft) | +| `importHash` | STRING(255) | UNIQUE, nullable | Import deduplication key | +| `app_roleId` | UUID | FK → roles.id | User's application role | +| `createdById` | UUID | FK → users.id | Record creator | +| `updatedById` | UUID | FK → users.id | Last modifier | + +**Indexes:** +- `email` (unique) +- `app_roleId` +- `deletedAt` + +**Associations:** +- `belongsTo` roles (as `app_role`) +- `hasMany` project_memberships +- `hasMany` presigned_url_requests +- `hasMany` publish_events +- `hasMany` access_logs +- `belongsToMany` permissions (as `custom_permissions`) +- `hasMany` file (as `avatar`, polymorphic) + +**Hooks:** +- `beforeCreate`: Trims string fields, auto-generates password for OAuth users, sets `emailVerified: true` for OAuth +- `beforeUpdate`: Trims string fields + +--- + +### 2. roles + +Application-level roles for RBAC. + +| Field | Type | Constraints | Description | +|-------|------|-------------|-------------| +| `id` | UUID | PK, default: UUIDv4 | Primary identifier | +| `name` | TEXT | NOT NULL, len: 1-100 | Role name | +| `role_customization` | TEXT | nullable | Custom role settings | +| `importHash` | STRING(255) | UNIQUE, nullable | Import deduplication | + +**Associations:** +- `hasMany` users (as `users_app_role`) +- `belongsToMany` permissions (through `rolesPermissionsPermissions`) + +**Seeded Roles:** +- Administrator (full access) +- Platform Owner +- Account Manager +- Tour Designer +- Content Reviewer +- Analytics Viewer +- Public (minimal access) + +--- + +### 3. permissions + +Individual permission definitions for RBAC. + +| Field | Type | Constraints | Description | +|-------|------|-------------|-------------| +| `id` | UUID | PK, default: UUIDv4 | Primary identifier | +| `name` | TEXT | NOT NULL, UNIQUE, len: 1-100 | Permission name | +| `importHash` | STRING(255) | UNIQUE, nullable | Import deduplication | + +**Permission Naming Convention:** +- `CREATE_` - Create records +- `READ_` - Read records +- `UPDATE_` - Modify records +- `DELETE_` - Delete records +- `READ_API_DOCS` - Access API documentation +- `CREATE_SEARCH` - Perform searches + +**Seeded Entities with CRUD Permissions:** +users, roles, permissions, projects, project_memberships, assets, asset_variants, presigned_url_requests, tour_pages, project_audio_tracks, publish_events, pwa_caches, access_logs, element_type_defaults, project_element_defaults + +Private production presentation access is intentionally not part of the generic +CRUD permission seed set. It is represented by project visibility plus explicit +customer grants in `production_presentation_access`. + +--- + +### 4. projects + +Virtual tour projects - the main organizational unit. + +| Field | Type | Constraints | Description | +|-------|------|-------------|-------------| +| `id` | UUID | PK, default: UUIDv4 | Primary identifier | +| `name` | TEXT | NOT NULL, len: 1-255 | Project name | +| `slug` | TEXT | NOT NULL, UNIQUE, regex: `^[a-z0-9_-]+$/i`, len: 1-255 | URL-safe identifier | +| `description` | TEXT | nullable | Project description | +| `logo_url` | TEXT | nullable | Project logo URL | +| `favicon_url` | TEXT | nullable | Favicon URL | +| `og_image_url` | TEXT | nullable | Open Graph image URL | +| `production_presentation_visibility` | ENUM | NOT NULL, default: 'public' | Production runtime visibility: `public`, `private` | +| `design_width` | INTEGER | nullable, default: 1920 | Design canvas width (px) | +| `design_height` | INTEGER | nullable, default: 1080 | Design canvas height (px) | +| `importHash` | STRING(255) | UNIQUE, nullable | Import deduplication | + +**Indexes:** +- `slug` (unique) +- `deletedAt` + +**Associations:** +- `hasMany` project_memberships +- `hasMany` assets +- `hasMany` presigned_url_requests +- `hasMany` tour_pages +- `hasMany` project_audio_tracks +- `hasMany` project_element_defaults +- `hasMany` publish_events +- `hasMany` pwa_caches +- `hasMany` access_logs +- `hasMany` production_presentation_access +- `hasMany` project_ui_control_settings + +**Cascade Behavior:** All child records are deleted when project is deleted. + +**Auto-Snapshot on Create:** When a project is created, all `element_type_defaults` records are automatically snapshotted to `project_element_defaults` for the new project. + +### Global UI Controls + +Global UI controls configure the system-owned fullscreen, sound, and offline +buttons. Defaults live in `global_ui_control_defaults`, project/environment +overrides live in `project_ui_control_settings`, and page overrides live in +`tour_pages.global_ui_controls_settings_json`. + +Sizes are stored as canvas-width-relative percentage fields +(`buttonSizePercent`, `iconSizePercent`, `borderRadiusPercent`). Positions are +stored as canvas-relative percentages (`xPercent`, `yPercent`) for stable +placement across devices. + +Icons, background colors, and border colors are state-specific: +`defaultIconUrl`/`activeIconUrl`, `defaultBackgroundColor`/`activeBackgroundColor`, +and `defaultBorderColor`/`activeBorderColor`. + +#### `global_ui_control_defaults` + +Singleton table for platform-wide defaults. + +| Field | Type | Constraints | Description | +|-------|------|-------------|-------------| +| `id` | UUID | PK, default UUIDv4 | Primary identifier | +| `settings_json` | JSON | NOT NULL | Defaults for `offline`, `fullscreen`, `sound` | +| `createdById` | UUID | nullable FK users | Creator | +| `updatedById` | UUID | nullable FK users | Last updater | +| `createdAt`, `updatedAt`, `deletedAt` | DATE | paranoid timestamps | Lifecycle fields | + +#### `project_ui_control_settings` + +Project/environment-level overrides. + +| Field | Type | Constraints | Description | +|-------|------|-------------|-------------| +| `id` | UUID | PK, default UUIDv4 | Primary identifier | +| `projectId` | UUID | NOT NULL, FK projects, cascade delete | Owning project | +| `environment` | ENUM | NOT NULL: `dev`, `stage`, `production` | Content environment | +| `source_key` | TEXT | nullable | Snapshot/publish/clone provenance | +| `settings_json` | JSON | NOT NULL | Project-level UI-control overrides | +| `importHash` | STRING(255) | nullable unique | Import deduplication | +| `createdById`, `updatedById` | UUID | nullable FK users | Audit users | +| `createdAt`, `updatedAt`, `deletedAt` | DATE | paranoid timestamps | Lifecycle fields | + +**Indexes:** +- unique partial index on `("projectId", environment)` where `deletedAt IS NULL` +- index on `deletedAt` + +--- + +### 5. production_presentation_access + +Explicit customer access grants for private production presentations. + +| Field | Type | Constraints | Description | +|-------|------|-------------|-------------| +| `id` | UUID | PK, default: UUIDv4 | Primary identifier | +| `projectId` | UUID | FK → projects.id, NOT NULL | Private production presentation project | +| `userId` | UUID | FK → users.id, NOT NULL | Customer user allowed to view the presentation | +| `createdById` | UUID | FK → users.id | Record creator | +| `updatedById` | UUID | FK → users.id | Last modifier | +| `importHash` | STRING(255) | UNIQUE, nullable | Import deduplication | + +**Indexes:** +- `projectId` +- `userId` +- `projectId, userId` (unique for active rows where `deletedAt IS NULL`) + +**Access Rules:** +- Public production projects do not require rows in this table. +- Staff users with any RBAC permission can view every private production presentation. +- Public-role customer users with no RBAC permissions need an active row for each private production presentation. + +--- + +### 6. project_memberships + +Junction table linking users to projects with access levels. + +| Field | Type | Constraints | Description | +|-------|------|-------------|-------------| +| `id` | UUID | PK, default: UUIDv4 | Primary identifier | +| `access_level` | ENUM | NOT NULL, default: 'viewer' | Access level: `owner`, `editor`, `reviewer`, `viewer` | +| `is_active` | BOOLEAN | NOT NULL, default: false | Membership active status | +| `invited_at` | DATE | nullable | Invitation timestamp | +| `accepted_at` | DATE | nullable | Acceptance timestamp | +| `projectId` | UUID | FK → projects.id | Associated project | +| `userId` | UUID | FK → users.id | Associated user | +| `importHash` | STRING(255) | UNIQUE, nullable | Import deduplication | + +**Indexes:** +- `projectId` +- `userId` +- `projectId, userId` (unique composite) +- `is_active` +- `deletedAt` + +**Cascade Behavior:** Deleted when associated project or user is deleted. + +--- + +## Tour Content Models + +### 7. tour_pages + +Individual pages/scenes within a tour. + +| Field | Type | Constraints | Description | +|-------|------|-------------|-------------| +| `id` | UUID | PK, default: UUIDv4 | Primary identifier | +| `environment` | ENUM | NOT NULL, default: 'dev' | Environment: `dev`, `stage`, `production` | +| `source_key` | TEXT | nullable | Reference to source version when published | +| `name` | TEXT | NOT NULL, len: 1-255 | Page display name | +| `slug` | TEXT | NOT NULL, regex: `^[a-z0-9_-]+$/i`, len: 1-255 | URL-safe identifier | +| `sort_order` | INTEGER | NOT NULL, default: 0 | Presentation display order; first sorted page is the entry page | +| `background_image_url` | TEXT | nullable | Background image URL | +| `background_video_url` | TEXT | nullable | Background video URL | +| `background_embed_url` | TEXT | nullable | Background 360/embed URL | +| `background_audio_url` | TEXT | nullable | Background audio URL | +| `background_loop` | BOOLEAN | NOT NULL, default: false | Loop background media | +| `background_video_autoplay` | BOOLEAN | NOT NULL, default: true | Autoplay background video | +| `background_video_loop` | BOOLEAN | NOT NULL, default: true | Loop background video | +| `background_video_muted` | BOOLEAN | NOT NULL, default: true | Mute background video | +| `background_video_start_time` | DECIMAL(10,1) | nullable | Background video start time (seconds) | +| `background_video_end_time` | DECIMAL(10,1) | nullable | Background video end time (seconds) | +| `background_video_play_once` | BOOLEAN | NOT NULL, default: false | Play video only once per session (show last frame on revisit) | +| `background_audio_autoplay` | BOOLEAN | NOT NULL, default: true | Autoplay background audio | +| `background_audio_loop` | BOOLEAN | NOT NULL, default: true | Loop background audio | +| `background_audio_start_time` | DECIMAL(10,1) | nullable | Background audio start time (seconds) | +| `background_audio_end_time` | DECIMAL(10,1) | nullable | Background audio end time (seconds) | +| `design_width` | INTEGER | nullable, default: null | Design canvas width (px) - copied from project on save | +| `design_height` | INTEGER | nullable, default: null | Design canvas height (px) - copied from project on save | +| `requires_auth` | BOOLEAN | NOT NULL, default: false | Requires authentication | +| `ui_schema_json` | JSON | nullable | UI element schema | +| `projectId` | UUID | FK → projects.id | Parent project | +| `importHash` | STRING(255) | UNIQUE, nullable | Import deduplication | + +**Indexes:** +- `projectId` +- `projectId, environment, slug` (unique composite) +- `projectId, environment, sort_order` +- `deletedAt` + +**Ordering semantics:** +- Constructor page reordering updates `sort_order` for dev pages only. +- Constructor page duplication creates a new dev row with a + unique slug, fresh page ID, fresh inline element IDs in `ui_schema_json`, and + `sort_order = max(project dev sort_order) + 1`. +- Constructor page deletion removes the dev row through the standard + `DELETE /api/tour_pages/:id` endpoint; stage and production keep their + previous copies until Save to Stage and Publish run. +- Runtime loaders sort by `sort_order`; the first page after sorting is the + presentation entry page. +- Save to Stage copies dev `sort_order` to stage. Publish copies stage + `sort_order` to production. + +**Associations:** +- `belongsTo` projects (as `project`) + +**UI Schema JSON Structure:** +The `ui_schema_json` field contains all page elements and navigation configuration: +```json +{ + "elements": [{ + "id": "unique-element-id", + "type": "navigation_next", + "name": "Next Page Button", + "xPercent": 90, + "yPercent": 85, + "widthPercent": 8, + "heightPercent": 10, + "targetPageSlug": "page-2", + "transitionVideoUrl": "assets/.../transition.mp4", + "iconUrl": "assets/.../icon.png", + "styleJson": { ... } + }] +} +``` + +**Element Types:** +- `navigation_next` - Forward navigation button +- `navigation_prev` - Back navigation button +- `spot` - Hotspot/clickable area +- `description` - Text description +- `tooltip` - Hover tooltip +- `gallery` - Image gallery +- `carousel` - Image carousel +- `logo` - Logo element +- `video_player` - Video player +- `audio_player` - Audio player +- `popup` - Popup/modal + +--- + +## Asset Management Models + +### 7. assets + +Media files (images, videos, audio, documents) used in tours. + +| Field | Type | Constraints | Description | +|-------|------|-------------|-------------| +| `id` | UUID | PK, default: UUIDv4 | Primary identifier | +| `name` | TEXT | nullable, len: 0-255 | Asset display name | +| `asset_type` | ENUM | NOT NULL | Media type: `image`, `video`, `audio`, `file` | +| `type` | ENUM | NOT NULL, default: 'general' | Usage type (see values below) | +| `cdn_url` | TEXT | nullable | Public CDN URL | +| `storage_key` | TEXT | nullable | S3/storage key | +| `mime_type` | TEXT | nullable, validated | MIME type (e.g., `image/png`) | +| `size_mb` | DECIMAL | nullable | File size in MB | +| `width_px` | INTEGER | nullable | Width in pixels | +| `height_px` | INTEGER | nullable | Height in pixels | +| `duration_sec` | DECIMAL | nullable | Duration for audio/video | +| `frame_rate` | DECIMAL | nullable | Video FPS from backend ffprobe | +| `checksum` | TEXT | nullable | File checksum | +| `is_public` | BOOLEAN | NOT NULL, default: false | Publicly accessible | +| `projectId` | UUID | FK → projects.id | Parent project | +| `importHash` | STRING(255) | UNIQUE, nullable | Import deduplication | + +**Asset Usage Types:** +- `icon` - UI icons +- `background_image` - Page backgrounds +- `audio` - Audio files +- `video` - Video files +- `transition` - Transition videos +- `logo` - Project logos +- `favicon` - Favicon images +- `document` - PDF/documents +- `general` - General assets + +**Indexes:** +- `projectId` +- `storage_key` partial active-row index (`assets_storage_key_active`, where `deletedAt IS NULL` and `storage_key IS NOT NULL`) for transition/reversed-video lookup by canonical storage key +- `asset_type` +- `type` +- `is_public` +- `deletedAt` + +**Associations:** +- `belongsTo` projects (as `project`) +- `hasMany` asset_variants (as `asset_variants_asset`) + +--- + +### 8. asset_variants + +Processed variants of assets (thumbnails, transcoded videos, reversed videos, etc.). + +| Field | Type | Constraints | Description | +|-------|------|-------------|-------------| +| `id` | UUID | PK, default: UUIDv4 | Primary identifier | +| `variant_type` | ENUM | nullable | Variant type (see values below) | +| `cdn_url` | TEXT | nullable, len: 0-2048, URL validated | Variant CDN URL | +| `storage_key` | TEXT | nullable | Private storage path (e.g., `assets/{assetId}/reversed.mp4`) | +| `width_px` | INTEGER | nullable, min: 0 | Width in pixels | +| `height_px` | INTEGER | nullable, min: 0 | Height in pixels | +| `size_mb` | DECIMAL | nullable, min: 0 | File size in MB | +| `assetId` | UUID | FK → assets.id | Parent asset | +| `importHash` | STRING(255) | UNIQUE, nullable | Import deduplication | + +**Variant Types:** +- `thumbnail` - Small preview +- `preview` - Medium preview +- `webp` - WebP format +- `mp4_low` - Low-quality MP4 +- `mp4_high` - High-quality MP4 +- `original` - Original file +- `reversed` - Reversed video for back navigation transitions (generated server-side with FFmpeg) + +**Cascade Behavior:** Deleted when parent asset is deleted. + +**Indexes:** +- `assetId, variant_type` partial active-row index (`asset_variants_asset_id_variant_type_active`, where `deletedAt IS NULL`) for asset variant joins and reversed-variant lookup + +--- + +### 9. presigned_url_requests + +Tracks presigned URL requests for secure uploads/downloads. + +| Field | Type | Constraints | Description | +|-------|------|-------------|-------------| +| `id` | UUID | PK, default: UUIDv4 | Primary identifier | +| `purpose` | ENUM | nullable | Purpose: `upload`, `download` | +| `asset_type` | ENUM | nullable | Asset type: `image`, `video`, `audio`, `file` | +| `requested_key` | TEXT | nullable, len: 0-1024 | Requested storage key | +| `mime_type` | TEXT | nullable, len: 0-255, validated | Expected MIME type | +| `requested_size_mb` | DECIMAL | nullable, min: 0 | Expected file size | +| `expires_at` | DATE | nullable | URL expiration time | +| `status` | TEXT | nullable | Request status | +| `projectId` | UUID | FK → projects.id | Associated project | +| `userId` | UUID | FK → users.id | Requesting user | +| `importHash` | STRING(255) | UNIQUE, nullable | Import deduplication | + +--- + +## Audio & Media Models + +### 10. project_audio_tracks + +Background audio tracks for projects. + +| Field | Type | Constraints | Description | +|-------|------|-------------|-------------| +| `id` | UUID | PK, default: UUIDv4 | Primary identifier | +| `environment` | ENUM | NOT NULL, default: 'dev' | Environment: `dev`, `stage`, `production` | +| `source_key` | TEXT | nullable | Reference to source version when published | +| `name` | TEXT | nullable, len: 0-255 | Track name | +| `slug` | TEXT | nullable | URL-safe identifier | +| `url` | TEXT | nullable | Audio file URL | +| `loop` | BOOLEAN | NOT NULL, default: false | Loop playback | +| `volume` | DECIMAL | nullable, min: 0, max: 1 | Volume level (0.0-1.0) | +| `sort_order` | INTEGER | nullable | Playback order | +| `is_enabled` | BOOLEAN | NOT NULL, default: false | Track enabled | +| `projectId` | UUID | FK → projects.id | Parent project | +| `importHash` | STRING(255) | UNIQUE, nullable | Import deduplication | + +**Note:** The `environment` field is NOT NULL with default 'dev' for consistency with `tour_pages`. + +--- + +### 11. project_transition_settings + +Environment-aware project-level transition settings for CSS-based page transitions. Settings cascade: Element → Project → Global → Hardcoded defaults. + +| Field | Type | Constraints | Description | +|-------|------|-------------|-------------| +| `id` | UUID | PK, default: UUIDv4 | Primary identifier | +| `environment` | ENUM | NOT NULL | Environment: `dev`, `stage`, `production` | +| `source_key` | TEXT | nullable | Reference to source record when published | +| `transition_type` | TEXT | NOT NULL, default: 'fade' | CSS transition type (`fade`, `none`) | +| `duration_ms` | INTEGER | NOT NULL, default: 700 | Transition duration in milliseconds | +| `easing` | TEXT | NOT NULL, default: 'ease-in-out' | CSS easing function | +| `overlay_color` | TEXT | NOT NULL, default: '#000000' | Transition overlay color | +| `projectId` | UUID | FK → projects.id, NOT NULL | Parent project | +| `createdById` | UUID | FK → users.id, nullable | Creator user | +| `updatedById` | UUID | FK → users.id, nullable | Last updater | +| `importHash` | STRING(255) | UNIQUE, nullable | Import deduplication | + +**Indexes:** +- `project_transition_settings_project_env_unique` - UNIQUE on (projectId, environment) WHERE deletedAt IS NULL + +**Associations:** +- `belongsTo` projects (as `project`) - CASCADE on delete +- `belongsTo` users (as `createdBy`, `updatedBy`) + +**Publishing Integration:** Copied between environments during Save to Stage (dev → stage) and Publish (stage → production). The `source_key` tracks lineage. + +**Cascade Resolution:** When determining transition settings: +1. Element-level settings (from `ui_schema_json`) +2. Project-level settings (this table, environment-specific) +3. Global defaults (`global_transition_defaults`) +4. Hardcoded fallback (fade, 700ms, ease-in-out, #000000) + +--- + +## Publishing & Caching Models + +### 12. publish_events + +Records of content publishing between environments. + +| Field | Type | Constraints | Description | +|-------|------|-------------|-------------| +| `id` | UUID | PK, default: UUIDv4 | Primary identifier | +| `title` | STRING | nullable, len: 0-255 | Event title | +| `description` | TEXT | nullable, len: 0-5000 | Event description | +| `from_environment` | ENUM | NOT NULL | Source: `dev`, `stage`, `production` | +| `to_environment` | ENUM | NOT NULL | Target: `dev`, `stage`, `production` | +| `started_at` | DATE | nullable | Start timestamp | +| `finished_at` | DATE | nullable | Completion timestamp | +| `status` | ENUM | NOT NULL, default: 'queued' | Status: `queued`, `running`, `success`, `failed` | +| `error_message` | TEXT | nullable | Error details | +| `pages_copied` | INTEGER | nullable, min: 0 | Pages copied count | +| `audios_copied` | INTEGER | nullable, min: 0 | Audio tracks copied count | +| `projectId` | UUID | FK → projects.id | Published project | +| `userId` | UUID | FK → users.id | Publishing user | +| `importHash` | STRING(255) | UNIQUE, nullable | Import deduplication | + +**Indexes:** +- `projectId` +- `userId` +- `status` +- `started_at` + +**Cascade Behavior:** `userId` set to NULL when user is deleted (preserves audit trail). + +--- + +### 13. pwa_caches + +PWA cache configurations for offline support. + +| Field | Type | Constraints | Description | +|-------|------|-------------|-------------| +| `id` | UUID | PK, default: UUIDv4 | Primary identifier | +| `environment` | ENUM | nullable | Environment: `dev`, `stage`, `production` | +| `cache_version` | TEXT | nullable, len: 0-255 | Cache version string | +| `manifest_json` | JSON | nullable | PWA manifest configuration | +| `asset_list_json` | JSON | nullable | List of cached assets | +| `generated_at` | DATE | nullable | Generation timestamp | +| `is_active` | BOOLEAN | NOT NULL, default: false | Cache active status | +| `projectId` | UUID | FK → projects.id | Parent project | +| `importHash` | STRING(255) | UNIQUE, nullable | Import deduplication | + +--- + +## Audit & Logging Models + +### 14. access_logs + +Audit trail for tour access. + +| Field | Type | Constraints | Description | +|-------|------|-------------|-------------| +| `id` | UUID | PK, default: UUIDv4 | Primary identifier | +| `environment` | ENUM | NOT NULL | Access context: `admin`, `stage`, `production` | +| `path` | TEXT | nullable, len: 0-2048 | Accessed path | +| `ip_address` | TEXT | nullable, len: 0-45 | Client IP (IPv4/IPv6) | +| `user_agent` | TEXT | nullable, len: 0-1024 | Browser user agent | +| `accessed_at` | DATE | NOT NULL, default: NOW | Access timestamp | +| `projectId` | UUID | FK → projects.id | Accessed project | +| `userId` | UUID | FK → users.id | Accessing user (if authenticated) | +| `importHash` | STRING(255) | UNIQUE, nullable | Import deduplication | + +**Indexes:** +- `projectId` +- `environment` +- `userId` +- `accessed_at` + +**Cascade Behavior:** `userId` set to NULL when user is deleted (preserves audit trail). + +--- + +## Element Default Settings Models + +These models implement a two-tier settings hierarchy for UI elements: +1. **Global defaults** (`element_type_defaults`) - Platform-wide default settings per element type +2. **Project defaults** (`project_element_defaults`) - Project-specific overrides, snapshotted from global on project creation + +Instance element settings are stored directly in `tour_pages.ui_schema_json` as part of each element's configuration. + +### 15. element_type_defaults + +Global platform-wide default settings for each element type. These serve as templates that are snapshotted to new projects. + +**Note:** This table was renamed from `ui_elements` to `element_type_defaults` for clarity. + +| Field | Type | Constraints | Description | +|-------|------|-------------|-------------| +| `id` | UUID | PK, default: UUIDv4 | Primary identifier | +| `element_type` | TEXT | NOT NULL, UNIQUE, len: 1-100 | Element type identifier | +| `name` | TEXT | NOT NULL, len: 1-255 | Display name | +| `sort_order` | INTEGER | NOT NULL, default: 0 | Display order in UI | +| `is_active` | VIRTUAL | getter: true | Virtual active field | +| `settings_json` | TEXT | nullable | Default settings JSON | +| `importHash` | STRING(255) | UNIQUE, nullable | Import deduplication | + +**Field Aliasing:** The model exposes `default_settings_json` as a property name, but it maps to the `settings_json` column in the database via `field: 'settings_json'`. This provides a clearer API name while maintaining backward compatibility with the database schema. + +**Virtual Field:** `is_active` is a VIRTUAL field that always returns `true` (computed, not stored in database). + +**Indexes:** +- `element_type` (unique) +- `sort_order` +- `deletedAt` + +**Associations:** +- `hasMany` project_element_defaults (as `project_defaults`) + +**Auto-Initialization:** The API includes an `ensureInitialized()` method that automatically seeds default records if the table is empty. This runs before any CRUD operation. + +**Seeded Element Types (11 types auto-seeded):** +- `navigation_next` - Forward navigation button (sort_order: 1) +- `navigation_prev` - Back navigation button (sort_order: 2) +- `tooltip` - Hover tooltip (sort_order: 3) +- `description` - Text description (sort_order: 4) +- `gallery` - Image gallery (sort_order: 5) +- `carousel` - Image carousel (sort_order: 6) +- `video_player` - Video player (sort_order: 7) +- `audio_player` - Audio player (sort_order: 8) +- `spot` - Hotspot/clickable area (sort_order: 9) +- `logo` - Logo element (sort_order: 10) +- `popup` - Popup/modal (sort_order: 11) + +--- + +### 16. project_element_defaults + +Project-specific element default settings. Created automatically when a project is created by snapshotting all global `element_type_defaults`. Can be customized per-project without affecting global defaults or other projects. + +| Field | Type | Constraints | Description | +|-------|------|-------------|-------------| +| `id` | UUID | PK, default: UUIDv4 | Primary identifier | +| `element_type` | TEXT | NOT NULL, len: 1-100 | Element type identifier | +| `name` | TEXT | nullable, len: 0-255 | Custom display name | +| `sort_order` | INTEGER | NOT NULL, default: 0 | Display order in UI | +| `settings_json` | TEXT | nullable | Project-specific settings JSON | +| `source_element_id` | UUID | FK → element_type_defaults.id, nullable | Reference to global default (for reset/diff) | +| `snapshot_version` | INTEGER | NOT NULL, default: 1 | Version counter (incremented on reset) | +| `projectId` | UUID | FK → projects.id, NOT NULL | Parent project | +| `importHash` | STRING(255) | UNIQUE, nullable | Import deduplication | + +**Indexes:** +- `projectId` +- `projectId, element_type` (unique composite) +- `element_type` +- `source_element_id` +- `deletedAt` + +**Associations:** +- `belongsTo` projects (as `project`) - CASCADE on delete +- `belongsTo` element_type_defaults (as `source_element`) - SET NULL on delete + +**Custom findAll with Project Filtering:** +The API implements a custom `findAll()` method that supports filtering by project: +- Query params: `?projectId=` or `?project=` +- Supports multiple values: `?projectId=uuid1|uuid2` +- Can filter by project UUID or project name (case-insensitive) +- Includes related `project` and `source_element` associations +- Default ordering: `sort_order ASC` + +**Auto-Snapshot Behavior:** +When a new project is created, all records from `element_type_defaults` are automatically copied to `project_element_defaults` for that project. This ensures each project starts with consistent defaults while allowing customization. + +**Reset Functionality:** +Projects can reset their element defaults back to the current global defaults via the `/api/project-element-defaults/:id/reset` endpoint. This increments `snapshot_version` and copies current global settings. + +**Diff Functionality:** +The `/api/project-element-defaults/:id/diff` endpoint compares project defaults with global defaults to show customizations. + +--- + +### 16. file (table: `files`) + +Polymorphic file attachments (e.g., user avatars). + +**Note:** This model does NOT have `freezeTableName: true`, so the table name is pluralized to `files`. + +| Field | Type | Constraints | Description | +|-------|------|-------------|-------------| +| `id` | UUID | PK, default: UUIDv4 | Primary identifier | +| `belongsTo` | STRING(255) | nullable | Parent table name | +| `belongsToId` | UUID | nullable | Parent record ID | +| `belongsToColumn` | STRING(255) | nullable | Column name | +| `name` | STRING(2083) | NOT NULL | File name | +| `sizeInBytes` | INTEGER | nullable | File size | +| `privateUrl` | STRING(2083) | nullable | Private storage URL | +| `publicUrl` | STRING(2083) | NOT NULL | Public access URL | + +**Polymorphic Usage:** +Files are attached to records via `belongsTo`/`belongsToId`/`belongsToColumn` columns. +Example: User avatar has `belongsTo: 'users'`, `belongsToId: `, `belongsToColumn: 'avatar'` + +--- + +## Junction Tables + +### rolesPermissionsPermissions + +Many-to-many relationship between roles and permissions. + +| Field | Type | Constraints | +|-------|------|-------------| +| `roles_permissionsId` | UUID | PK, FK → roles.id | +| `permissionId` | UUID | PK, FK → permissions.id | +| `createdAt` | TIMESTAMP WITH TIME ZONE | NOT NULL | +| `updatedAt` | TIMESTAMP WITH TIME ZONE | NOT NULL | + +**Indexes:** +- `permissionId` + +### usersCustom_permissionsPermissions + +Many-to-many relationship for user custom permissions. + +| Field | Type | Constraints | +|-------|------|-------------| +| `users_custom_permissionsId` | UUID | FK → users.id | +| `permissionId` | UUID | FK → permissions.id | + +--- + +## Database Indexes Summary + +| Table | Index | Fields | Type | +|-------|-------|--------|------| +| users | email | email | unique | +| users | app_roleId | app_roleId | - | +| users | deletedAt | deletedAt | - | +| projects | slug | slug | unique | +| projects | deletedAt | deletedAt | - | +| production_presentation_access | projectId | projectId | - | +| production_presentation_access | userId | userId | - | +| production_presentation_access | composite | projectId, userId | unique active rows | +| project_memberships | composite | projectId, userId | unique | +| tour_pages | composite | projectId, environment, slug | unique | +| tour_pages | sort | projectId, environment, sort_order | - | +| assets | projectId | projectId | - | +| assets | asset_type | asset_type | - | +| assets | type | type | - | +| assets | is_public | is_public | - | +| element_type_defaults | element_type | element_type | unique | +| element_type_defaults | sort_order | sort_order | - | +| element_type_defaults | deletedAt | deletedAt | - | +| project_element_defaults | projectId | projectId | - | +| project_element_defaults | composite | projectId, element_type | unique | +| project_element_defaults | element_type | element_type | - | +| project_element_defaults | source_element_id | source_element_id | - | +| project_element_defaults | deletedAt | deletedAt | - | +| publish_events | status | status | - | +| publish_events | started_at | started_at | - | +| access_logs | accessed_at | accessed_at | - | + +--- + +## Foreign Key Constraints + +All foreign key constraints are enforced at the database level via migration `20260319000001-add-foreign-key-constraints.js`. + +| Child Table | Column | Parent Table | On Delete | On Update | +|-------------|--------|--------------|-----------|-----------| +| asset_variants | assetId | assets | CASCADE | CASCADE | +| assets | projectId | projects | CASCADE | CASCADE | +| tour_pages | projectId | projects | CASCADE | CASCADE | +| project_memberships | projectId | projects | CASCADE | CASCADE | +| project_memberships | userId | users | CASCADE | CASCADE | +| production_presentation_access | projectId | projects | CASCADE | CASCADE | +| production_presentation_access | userId | users | CASCADE | CASCADE | +| production_presentation_access | createdById | users | SET NULL | CASCADE | +| production_presentation_access | updatedById | users | SET NULL | CASCADE | +| presigned_url_requests | projectId | projects | CASCADE | CASCADE | +| presigned_url_requests | userId | users | CASCADE | CASCADE | +| project_audio_tracks | projectId | projects | CASCADE | CASCADE | +| project_element_defaults | projectId | projects | CASCADE | CASCADE | +| project_element_defaults | source_element_id | element_type_defaults | SET NULL | CASCADE | +| publish_events | projectId | projects | CASCADE | CASCADE | +| publish_events | userId | users | SET NULL | CASCADE | +| pwa_caches | projectId | projects | CASCADE | CASCADE | +| access_logs | projectId | projects | CASCADE | CASCADE | +| access_logs | userId | users | SET NULL | CASCADE | +| users | app_roleId | roles | SET NULL | CASCADE | + +--- + +## Migration History + +| Migration | Description | +|-----------|-------------| +| `20260319000001-add-foreign-key-constraints.js` | Adds all FK constraints to enforce referential integrity | +| `20260319000002-remove-redundant-deletion-columns.js` | Removes deprecated `is_deleted` and `deleted_at_time` columns from assets and projects | +| `20260326000001-rename-ui-elements-to-element-type-defaults.js` | Renames `ui_elements` table to `element_type_defaults` for clarity | +| `20260326000002-convert-element-type-enum-to-text.js` | Converts element type from ENUM to TEXT for flexibility | +| `20260326000003-create-project-element-defaults.js` | Creates `project_element_defaults` table for project-specific element settings | +| `20260326000004-backfill-project-element-defaults.js` | Backfills `project_element_defaults` for existing projects by snapshotting global defaults | +| `20260326000005-fix-project-audio-tracks-environment.js` | Fixes `project_audio_tracks.environment` to NOT NULL with default 'dev' | +| `20260326000006-copy-dev-to-stage.js` | Copies existing dev content to stage environment for all projects (initializes dev→stage workflow) | +| `20260326043002-enforce-environment-not-null.js` | Enforces NOT NULL constraint on environment columns in tour_pages and transitions | +| `20260326050442-remove-project-phase-column.js` | Removes redundant `phase` column from projects table (environment is on tour_pages) | +| `20260326054410-remove-entry-page-slug-column.js` | Removes `entry_page_slug` from projects (entry page is first by sort_order) | +| `20260326060000-convert-targetpageid-to-slug.js` | Converts `targetPageId` to `targetPageSlug` in ui_schema_json for environment-safe navigation | +| `20260326060001-drop-page-elements-table.js` | Drops unused `page_elements` table (data stored in ui_schema_json) | +| `20260326060002-drop-page-links-table.js` | Drops unused `page_links` table (navigation stored in ui_schema_json) | +| `20260326060003-drop-transitions-table.js` | Drops unused `transitions` table (transitionVideoUrl stored in ui_schema_json) | +| `20260326171017-add-missing-element-type-defaults.js` | Adds missing element types (spot, logo, popup) to element_type_defaults and backfills project_element_defaults | +| `20260327000001-sync-all-element-type-defaults.js` | Syncs all 11 element types with correct sort_order and backfills missing project_element_defaults for all projects | +| `20260331024423-remove-unused-theme-columns-from-projects.js` | Removes unused `theme_config_json`, `custom_css_json`, `cdn_base_url` columns from projects table | +| `20260331054340-remove-duplicate-element-type-defaults.js` | Removes duplicate element_type_defaults records created during earlier migrations | +| `20260331063424-cleanup-invalid-element-type-defaults.js` | Cleans up invalid element_type_defaults entries and ensures data integrity | +| `20260403000001-add-background-video-settings.js` | Adds background video playback settings to tour_pages (autoplay, loop, muted, start_time, end_time) | +| `20260409000001-add-design-dimensions-to-projects.js` | Adds design_width and design_height columns to projects table for canvas scaling | +| `20260409111309-add-design-dimensions-to-tour-pages.js` | Adds design_width and design_height columns to tour_pages table for presentation isolation | +| `20260422000001-add-background-video-play-once.js` | Adds background_video_play_once column to tour_pages for session-scoped single playback | +| `20260605000001-add-background-audio-settings.js` | Adds background audio playback settings to tour_pages (autoplay, loop, start_time, end_time) | +| `20260613000001-add-background-embed-url-to-tour-pages.js` | Adds background_embed_url to tour_pages for 360/embed page backgrounds | +| `20260626000001-add-private-production-presentation-access.js` | Adds project production visibility and customer access grants for private production presentations | +| `20260626000002-grant-account-manager-create-users.js` | Grants `CREATE_USERS` to Account Manager for customer viewer creation | + +--- + +## Seeders + +| Seeder | Description | +|--------|-------------| +| `20200430130759-admin-user.js` | Creates initial admin and test users | +| `20200430130760-user-roles.js` | Creates roles, permissions, and role-permission assignments | +| `20231127130745-sample-data.js` | Creates sample projects, pages, assets, and other demo data | + +**Initial Users:** +- Admin: `admin@flatlogic.com` (admin password from config) +- John Doe: `john@doe.com` (user password from config) +- Client: `client@hello.com` (user password from config) + +--- + +## Base DB API Patterns + +All entity DB APIs extend `GenericDBApi` which provides: + +### Configurable Properties +- `MODEL` - Sequelize model reference (required, must be defined in subclass) +- `TABLE_NAME` - Derived from MODEL.getTableName() +- `SEARCHABLE_FIELDS` - Text search fields (ILIKE) +- `RANGE_FIELDS` - Date/number range filters +- `ENUM_FIELDS` - Exact match filters +- `RELATION_FILTERS` - Related entity filters +- `CSV_FIELDS` - Export field list (default: `['id', 'createdAt']`) +- `AUTOCOMPLETE_FIELD` - Field for autocomplete (default: 'name') +- `ASSOCIATIONS` - Many-to-many relationships +- `FIND_BY_INCLUDES` - Eager loading for findBy +- `FIND_ALL_INCLUDES` - Eager loading for findAll +- `getFieldMapping(data)` - Transform input data before save (default: returns data unchanged) + +### Standard Methods +- `create(data, options)` - Create record with associations +- `bulkImport(data, options)` - Bulk create records +- `update({ id, data, currentUser, transaction, runtimeContext })` - Update record +- `deleteByIds({ ids, currentUser, transaction, runtimeContext })` - Soft delete multiple records +- `remove({ id, currentUser, transaction, runtimeContext })` - Soft delete single record +- `findBy(where, options)` - Find single record +- `findAll(filter, options)` - Paginated list with filters +- `findAllAutocomplete({ query, limit, offset }, options)` - Autocomplete search +- `toCSV(rows)` - Export to CSV + +--- + +## Environment-Based Content + +The platform uses an environment-based content model for publishing workflow: + +1. **dev** - Development environment (editable in constructor) +2. **stage** - Staging/preview environment (for review before production) +3. **production** - Live production environment (public-facing) + +### Content Flow + +``` +┌─────────┐ Save to Stage ┌─────────┐ Publish ┌────────────┐ +│ dev │ ─────────────────── │ stage │ ───────────── │ production │ +└─────────┘ └─────────┘ └────────────┘ + ▲ + │ + Constructor + edits +``` + +### Tables with Environment Column +- `tour_pages` - Pages with `environment` column +- `project_audio_tracks` - Audio tracks with `environment` column +- `project_transition_settings` - CSS transition settings with `environment` column + +### Source Key Tracking +When content is copied between environments, the `source_key` field stores the ID of the source record. This enables: +- Tracking content lineage +- Identifying which stage records came from dev +- Rolling back changes if needed + +### Environment Access Control + +Access to content is controlled by environment with strict isolation: + +| Environment | Authentication | Access | Use Case | +|-------------|----------------|--------|----------| +| **dev** | Required (JWT) | Admin/Constructor only | Editing in constructor | +| **stage** | Required (JWT) | Authenticated users | Review workspace before publish | +| **production** | Public (no auth) | Anyone | Published public tours | + +**Security Layers:** + +1. **Backend Middleware** (`requireRuntimeReadOrAuth`): Only `production` environment allows unauthenticated GET requests. Stage and dev require JWT authentication. + +2. **Runtime Context API** (`getRuntimeEnvironment`): Blocks `dev` environment from being accessed via `X-Runtime-Environment` header. Only `production` and `stage` are allowed. + +3. **Database Filtering** (`applyRuntimeEnvironment`): Applies environment filter to all queries when runtime context is present. + +4. **Frontend Routes**: + - `/p/[slug]` → production (LayoutGuest) + - `/p/[slug]/stage` → stage (LayoutAuthenticated) + - `/constructor` → dev (LayoutAuthenticated) + +**Navigation uses `targetPageSlug`** (not `targetPageId`) in `ui_schema_json` to ensure navigation works correctly across environments since page IDs differ between dev, stage, and production. + +--- + +## Data Types Reference + +| Sequelize Type | PostgreSQL Type | Usage | +|----------------|-----------------|-------| +| UUID | uuid | Primary keys, foreign keys | +| TEXT | text | Long strings (unlimited) | +| STRING(n) | varchar(n) | Limited strings | +| INTEGER | integer | Whole numbers | +| DECIMAL | numeric | Precise decimals | +| BOOLEAN | boolean | True/false | +| DATE | timestamp with time zone | Dates and times | +| JSON | jsonb | Structured data | +| ENUM | enum type | Fixed value sets | +| VIRTUAL | (none) | Computed fields | + +--- + +## Common Model Options + +Most models share these Sequelize options: +```javascript +{ + timestamps: true, // Adds createdAt, updatedAt + paranoid: true, // Soft delete via deletedAt + freezeTableName: true, // Table name = model name +} +``` + +**Exception:** The `file` model does not set `freezeTableName: true`, so its table name is `files` (pluralized by Sequelize default). + +All records include audit fields: +- `createdAt` - Creation timestamp +- `updatedAt` - Last modification timestamp +- `deletedAt` - Soft deletion timestamp (null if not deleted) +- `createdById` - Creating user's ID +- `updatedById` - Last modifying user's ID diff --git a/backend/docs/modules/auth.md b/backend/docs/modules/auth.md new file mode 100644 index 0000000..b16fad2 --- /dev/null +++ b/backend/docs/modules/auth.md @@ -0,0 +1,942 @@ +# Backend Auth Module Documentation + +## Overview + +The Auth module provides comprehensive authentication and authorization for the application. It supports local email/password authentication, OAuth 2.0 (Google, Microsoft), JWT-based session management, email verification, and password reset flows. + +**Files:** +| File | Purpose | +|------|---------| +| `src/auth/auth.ts` | Passport.js strategy configurations (JWT, Google, Microsoft) | +| `src/services/auth.ts` | Auth business logic (signin, password reset/update, email verification) | +| `src/routes/auth.ts` | REST API endpoints for authentication | +| `src/helpers.ts` | JWT signing utility (`jwtSign`) | +| `src/db/api/users.js` | User database operations (tokens, password updates) | +| `src/middlewares/rateLimiter.js` | Auth-specific rate limiters | + +--- + +## Architecture Diagram + +``` +┌──────────────────────────────────────────────────────────────────────┐ +│ Frontend/Client │ +│ ┌────────────────┐ ┌────────────────┐ ┌────────────────────────┐ │ +│ │ Login Form │ │ Signup Form │ │ OAuth Buttons │ │ +│ │ (email/pass) │ │ (email/pass) │ │ (Google/Microsoft) │ │ +│ └───────┬────────┘ └───────┬────────┘ └───────────┬────────────┘ │ +└──────────┼───────────────────┼───────────────────────┼───────────────┘ + │ │ │ + ▼ ▼ ▼ +┌──────────────────────────────────────────────────────────────────────┐ +│ Routes Layer (routes/auth.ts) │ +│ │ +│ ┌──────────────────────────────────────────────────────────────┐ │ +│ │ Rate Limiters: │ │ +│ │ • authLimiter (10 req/15min) → /signin/local │ │ +│ │ • passwordResetLimiter (5 req/hour) → /send-password-reset │ │ +│ └──────────────────────────────────────────────────────────────┘ │ +│ │ +│ ┌───────────────┐ ┌───────────────┐ ┌───────────────────────┐ │ +│ │ POST /signin │ │ PUT /reset │ │ GET /signin/google │ │ +│ │ /local │ │ │ │ GET /signin/microsoft │ │ +│ └───────┬───────┘ └───────┬───────┘ └───────────┬───────────┘ │ +│ │ │ │ │ +└──────────┼──────────────────┼──────────────────────┼─────────────────┘ + │ │ │ + ▼ ▼ ▼ +┌──────────────────────────────────────────────────────────────────────┐ +│ Service Layer (services/auth.ts) │ +│ ┌────────────────────────────────────────────────────────────────┐ │ +│ │ Auth Class (Static Methods) │ │ +│ │ • signup(email, password, options, host) │ │ +│ │ • signin(email, password) │ │ +│ │ • verifyEmail(token) │ │ +│ │ • passwordUpdate(currentPassword, newPassword, options) │ │ +│ │ • passwordReset(token, password) │ │ +│ │ • sendEmailAddressVerificationEmail(email, host) │ │ +│ │ • sendPasswordResetEmail(email, type, host) │ │ +│ │ • updateProfile(data, currentUser) │ │ +│ └────────────────────────────────────────────────────────────────┘ │ +└──────────────────────────────────────────────────────────────────────┘ + │ │ │ + ▼ ▼ ▼ +┌──────────────────────────────────────────────────────────────────────┐ +│ Passport Layer (auth/auth.ts) │ +│ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────────┐ │ +│ │ JWT Strategy │ │ Google Strategy │ │ Microsoft Strategy │ │ +│ │ (API Auth) │ │ (OAuth 2.0) │ │ (OAuth 2.0) │ │ +│ └─────────────────┘ └─────────────────┘ └─────────────────────┘ │ +│ │ │ │ +│ └──────────┬───────────┘ │ +│ │ │ +│ ┌──────────▼──────────┐ │ +│ │ socialStrategy() │ │ +│ │ • findOrCreate │ │ +│ │ • Generate JWT │ │ +│ └─────────────────────┘ │ +└──────────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌──────────────────────────────────────────────────────────────────────┐ +│ Database Layer (db/api/users.js) │ +│ ┌────────────────────────────────────────────────────────────────┐ │ +│ │ UsersDBApi (Static Methods) │ │ +│ │ • findBy({ email }) │ │ +│ │ • createFromAuth(data) │ │ +│ │ • updatePassword(id, password) │ │ +│ │ • generateEmailVerificationToken(email) │ │ +│ │ • generatePasswordResetToken(email) │ │ +│ │ • findByEmailVerificationToken(token) │ │ +│ │ • findByPasswordResetToken(token) │ │ +│ │ • markEmailVerified(id) │ │ +│ └────────────────────────────────────────────────────────────────┘ │ +└──────────────────────────────────────────────────────────────────────┘ +``` + +--- + +## Authentication Strategies + +### 1. JWT Strategy (Primary) + +Used for API authentication on all protected routes. + +**Configuration (auth/auth.ts):** +```javascript +passport.use( + new JWTstrategy({ + passReqToCallback: true, + secretOrKey: config.secret_key, + jwtFromRequest: ExtractJWT.fromAuthHeaderAsBearerToken(), + }, async (req, token, done) => { + const user = await UsersDBApi.findBy({ email: token.user.email }); + + if (user && user.disabled) { + return done(new Error(`User '${user.email}' is disabled`)); + } + + req.currentUser = user; + return done(null, user); + }) +); +``` + +**Token Structure:** +```javascript +{ + user: { + id: "uuid", + email: "user@example.com" + }, + iat: 1234567890, // Issued at + exp: 1234589490 // Expires in 6 hours +} +``` + +**Usage:** +```javascript +// Protect route with JWT +router.get('/me', passport.authenticate('jwt', { session: false }), handler); + +// Access user in handler +const currentUser = req.currentUser; +``` + +### 2. Google OAuth Strategy + +**Configuration (auth/auth.ts):** +```javascript +passport.use( + new GoogleStrategy({ + clientID: config.google.clientId, + clientSecret: config.google.clientSecret, + callbackURL: config.apiUrl + '/auth/signin/google/callback', + passReqToCallback: true, + }, (request, accessToken, refreshToken, profile, done) => { + socialStrategy(profile.email, profile, providers.GOOGLE, done); + }) +); +``` + +**Environment Variables:** +| Variable | Description | +|----------|-------------| +| `GOOGLE_CLIENT_ID` | Google OAuth client ID | +| `GOOGLE_CLIENT_SECRET` | Google OAuth client secret | + +**OAuth Scopes:** `profile`, `email` + +### 3. Microsoft OAuth Strategy + +**Configuration (auth/auth.ts):** +```javascript +passport.use( + new MicrosoftStrategy({ + clientID: config.microsoft.clientId, + clientSecret: config.microsoft.clientSecret, + callbackURL: config.apiUrl + '/auth/signin/microsoft/callback', + passReqToCallback: true, + }, (request, accessToken, refreshToken, profile, done) => { + const email = profile._json.mail || profile._json.userPrincipalName; + socialStrategy(email, profile, providers.MICROSOFT, done); + }) +); +``` + +**Environment Variables:** +| Variable | Description | +|----------|-------------| +| `MS_CLIENT_ID` | Microsoft OAuth client ID | +| `MS_CLIENT_SECRET` | Microsoft OAuth client secret | + +**OAuth Scopes:** `https://graph.microsoft.com/user.read`, `openid` + +### Social Strategy Helper + +Common logic for OAuth providers: + +```javascript +function socialStrategy(email, profile, provider, done) { + db.users.findOrCreate({ where: { email, provider } }).then(([user]) => { + const body = { + id: user.id, + email: user.email, + name: profile.displayName, + }; + const token = helpers.jwtSign({ user: body }); + return done(null, { token }); + }); +} +``` + +--- + +## File Details + +### 1. services/auth.ts + +Core authentication business logic. + +#### Class: Auth + +```typescript +class Auth { + static async signin(email, password) + static async verifyEmail(token, options) + static async passwordUpdate(currentPassword, newPassword, options) + static async passwordReset(token, password, options) + static async sendEmailAddressVerificationEmail(email, host) + static async sendPasswordResetEmail(email, type, host) + static async updateProfile(data, currentUser) +} +``` + +#### Method: signup(email, password, options, host) + +Registers a new user or updates password for existing unverified user. + +**Flow:** +``` +1. Check if user exists by email + ├── User exists with authenticationUid → Error: emailAlreadyInUse + ├── User exists but disabled → Error: userDisabled + └── User exists without authenticationUid → Update password +2. If new user: + ├── Hash password (bcrypt, 12 rounds) + ├── Create user via UsersDBApi.createFromAuth() + └── Assign default "User" role +3. Send verification email (if EmailSender configured) +4. Return signed JWT token +``` + +**Returns:** JWT token string + +#### Method: signin(email, password) + +Authenticates user with email and password. + +**Flow:** +``` +1. Find user by email + └── Not found → Error: userNotFound +2. Check if disabled + └── Disabled → Error: userDisabled +3. Verify password exists + └── No password → Error: wrongPassword +4. Check email verification + └── Not verified (and email configured) → Error: userNotVerified +5. Compare password with bcrypt + └── Mismatch → Error: wrongPassword +6. Return signed JWT token +``` + +**Returns:** JWT token string + +#### Method: verifyEmail(token, options) + +Verifies user email address using token. + +**Flow:** +``` +1. Find user by email verification token + └── Not found or expired → Error: invalidToken +2. Mark email as verified +3. Return true +``` + +#### Method: passwordUpdate(currentPassword, newPassword, options) + +Updates password for authenticated user. + +**Flow:** +``` +1. Verify currentUser exists + └── Not authenticated → ForbiddenError +2. Verify current password matches + └── Mismatch → Error: wrongPassword +3. Verify new password is different + └── Same → Error: samePassword +4. Hash new password and update +``` + +#### Method: passwordReset(token, password, options) + +Resets password using reset token. + +**Flow:** +``` +1. Find user by password reset token + └── Not found or expired → Error: invalidToken +2. Hash new password +3. Update user password +``` + +--- + +### 2. routes/auth.ts (327 lines) + +REST API endpoints for authentication. + +#### Endpoints Overview + +| Method | Path | Auth | Rate Limit | Description | +|--------|------|------|------------|-------------| +| POST | `/signin/local` | No | authLimiter | Login with email/password | +| GET | `/me` | JWT | - | Get current user | +| PUT | `/password-reset` | No | - | Reset password with token | +| PUT | `/password-update` | JWT | - | Change password | +| PUT | `/profile` | JWT | - | Update user profile | +| PUT | `/verify-email` | No | - | Verify email with token | +| POST | `/send-email-address-verification-email` | JWT | - | Resend verification email | +| POST | `/send-password-reset-email` | No | passwordResetLimiter | Send password reset email | +| GET | `/email-configured` | No | - | Check if email is configured | +| GET | `/signin/google` | No | - | Initiate Google OAuth | +| GET | `/signin/google/callback` | No | - | Google OAuth callback | +| GET | `/signin/microsoft` | No | - | Initiate Microsoft OAuth | +| GET | `/signin/microsoft/callback` | No | - | Microsoft OAuth callback | + +#### POST /api/auth/signin/local + +Login with email and password. + +**Request:** +```json +{ + "email": "user@example.com", + "password": "securepassword123" +} +``` + +**Response (200):** +```json +"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9..." +``` + +**Errors:** +| Code | Message | Cause | +|------|---------|-------| +| 400 | `auth.userNotFound` | User doesn't exist | +| 400 | `auth.userDisabled` | User account is disabled | +| 400 | `auth.wrongPassword` | Invalid password | +| 400 | `auth.userNotVerified` | Email not verified | +| 429 | Too Many Requests | Rate limit exceeded | + +#### Self-Registration + +Self-registration is disabled. `POST /api/auth/signup` is not registered. +New users are created through the authenticated Users flow and receive an +invitation/setup link. + +#### GET /api/auth/me + +Get current authenticated user. + +**Headers:** +``` +Authorization: Bearer +``` + +**Response (200):** +```json +{ + "id": "uuid", + "email": "user@example.com", + "firstName": "John", + "lastName": "Doe", + "emailVerified": true, + "disabled": false, + "app_role": { + "id": "uuid", + "name": "Administrator", + "permissions": [...] + }, + "custom_permissions": [], + "avatar": [...], + "createdAt": "2024-01-01T00:00:00.000Z" +} +``` + +**Note:** Password field is omitted from response. + +#### PUT /api/auth/password-reset + +Reset password using token from email. + +**Request:** +```json +{ + "token": "abc123...", + "password": "newSecurePassword456" +} +``` + +**Response (200):** +```json +{ "success": true } +``` + +#### PUT /api/auth/password-update + +Change password for authenticated user. + +**Headers:** +``` +Authorization: Bearer +``` + +**Request:** +```json +{ + "currentPassword": "oldPassword123", + "newPassword": "newPassword456" +} +``` + +**Errors:** +| Code | Message | Cause | +|------|---------|-------| +| 400 | `auth.wrongPassword` | Current password incorrect | +| 400 | `auth.passwordUpdate.samePassword` | New password same as old | +| 403 | Forbidden | Not authenticated | + +#### PUT /api/auth/profile + +Update user profile. + +**Headers:** +``` +Authorization: Bearer +``` + +**Request:** +```json +{ + "profile": { + "firstName": "John", + "lastName": "Smith", + "phoneNumber": "+1234567890" + } +} +``` + +#### OAuth Endpoints + +**GET /api/auth/signin/google** +- Redirects to Google OAuth consent screen +- Query param: `app` (passed as state) + +**GET /api/auth/signin/google/callback** +- Handles Google OAuth callback +- Redirects to: `{uiUrl}/login?token={jwt}` + +**GET /api/auth/signin/microsoft** +- Redirects to Microsoft OAuth consent screen +- Query param: `app` (passed as state) + +**GET /api/auth/signin/microsoft/callback** +- Handles Microsoft OAuth callback +- Redirects to: `{uiUrl}/login?token={jwt}` + +--- + +### 3. helpers.js (32 lines) + +JWT and utility functions. + +#### Method: jwtSign(data) + +Signs JWT token with application secret. + +```javascript +static jwtSign(data) { + return jwt.sign(data, config.secret_key, { expiresIn: '6h' }); +} +``` + +**Configuration:** +| Setting | Value | Description | +|---------|-------|-------------| +| Secret Key | `config.secret_key` | From `SECRET_KEY` env var | +| Expiration | `6h` | Token valid for 6 hours | +| Algorithm | `HS256` | Default HMAC SHA-256 | + +--- + +### 4. db/api/users.js (Authentication Methods) + +User database operations related to authentication. + +#### Method: createFromAuth(data) + +Creates new user during signup. + +```javascript +static async createFromAuth(data, options) { + const users = await db.users.create({ + email: data.email, + firstName: data.firstName, + authenticationUid: data.authenticationUid, + password: data.password, + }, { transaction }); + + // Assign default "User" role + const app_role = await db.roles.findOne({ + where: { name: config.roles?.user || 'User' }, + }); + await users.setApp_role(app_role?.id); + + return users; +} +``` + +#### Method: generateEmailVerificationToken(email) + +Generates secure token for email verification. + +```javascript +static async generateEmailVerificationToken(email, options) { + const token = crypto.randomBytes(20).toString('hex'); + const tokenExpiresAt = Date.now() + (24 * 60 * 60 * 1000); // 24 hours + + await users.update({ + emailVerificationToken: token, + emailVerificationTokenExpiresAt: tokenExpiresAt, + }); + + return token; +} +``` + +**Token Properties:** +| Property | Value | +|----------|-------| +| Length | 40 hex characters | +| Expiry | 24 hours | +| Storage | `emailVerificationToken` column | + +#### Method: generatePasswordResetToken(email) + +Generates secure token for password reset. + +Same implementation as `generateEmailVerificationToken` but stores in: +- `passwordResetToken` +- `passwordResetTokenExpiresAt` + +#### Method: findByEmailVerificationToken(token) + +Finds user by valid (non-expired) verification token. + +```javascript +static async findByEmailVerificationToken(token, options) { + return db.users.findOne({ + where: { + emailVerificationToken: token, + emailVerificationTokenExpiresAt: { + [Op.gt]: Date.now(), + }, + }, + }); +} +``` + +#### Method: markEmailVerified(id) + +Marks user email as verified. + +```javascript +static async markEmailVerified(id, options) { + const users = await db.users.findByPk(id); + await users.update({ emailVerified: true }); + return true; +} +``` + +--- + +## Rate Limiting + +Authentication endpoints have dedicated rate limiters defined in `middlewares/rateLimiter.js`. + +### Auth Limiter (Login) + +```javascript +const authLimiter = createRateLimiter({ + keyPrefix: 'auth', + windowMs: 15 * 60 * 1000, // 15 minutes + max: 10, + message: 'Too many authentication attempts. Please try again later.', + skipFailedRequests: false, // Count ALL attempts +}); +``` + +| Setting | Value | +|---------|-------| +| Window | 15 minutes | +| Max Requests | 10 | +| Applied To | `/signin/local` | + +### Signup Limiter + +Self-registration is disabled, so no signup limiter is registered. + +### Password Reset Limiter + +```javascript +const passwordResetLimiter = createRateLimiter({ + keyPrefix: 'password-reset', + windowMs: 60 * 60 * 1000, // 1 hour + max: 5, + message: 'Too many password reset requests. Please try again later.', +}); +``` + +| Setting | Value | +|---------|-------| +| Window | 1 hour | +| Max Requests | 5 | +| Applied To | `/send-password-reset-email` | + +### Rate Limit Response + +```json +{ + "error": "Too Many Requests", + "message": "Too many authentication attempts. Please try again later.", + "retryAfter": 300 +} +``` + +**Headers:** +``` +X-RateLimit-Limit: 10 +X-RateLimit-Remaining: 0 +X-RateLimit-Reset: 2024-01-01T00:15:00.000Z +Retry-After: 300 +``` + +--- + +## Password Security + +### Hashing Configuration + +```javascript +// config.ts +bcrypt: { + saltRounds: 12, +} +``` + +| Setting | Value | Security Impact | +|---------|-------|-----------------| +| Algorithm | bcrypt | Industry standard | +| Salt Rounds | 12 | ~200ms hash time | +| Salt | Auto-generated | Per-password unique | + +### Password Validation + +Passwords are: +1. Hashed before storage (never stored in plain text) +2. Compared using `bcrypt.compare()` (timing-attack safe) +3. Required for local authentication + +--- + +## Email Integration + +Email functionality is conditional based on configuration. + +### Email Configuration Check + +```javascript +if (EmailSender.isConfigured) { + await this.sendEmailAddressVerificationEmail(user.email, host); +} +``` + +When email is NOT configured: +- Signup succeeds without verification email +- Users are auto-verified on signin +- Password reset emails not sent + +### Verification Email + +**Email Class:** `EmailAddressVerificationEmail` + +**Link Format:** +``` +{host}/verify-email?token={token} +``` + +### Password Reset Email + +**Email Classes:** +- `PasswordResetEmail` - Standard reset +- `InvitationEmail` - New user invitation + +**Link Format:** +``` +{host}/password-reset?token={token} +``` + +--- + +## Configuration Reference + +### Environment Variables + +| Variable | Required | Default | Description | +|----------|----------|---------|-------------| +| `SECRET_KEY` | Yes | `88dbeaf8-e906-405e-9e41-c3baadeda5c6` | JWT signing secret | +| `GOOGLE_CLIENT_ID` | No | - | Google OAuth client ID | +| `GOOGLE_CLIENT_SECRET` | No | - | Google OAuth client secret | +| `MS_CLIENT_ID` | No | - | Microsoft OAuth client ID | +| `MS_CLIENT_SECRET` | No | - | Microsoft OAuth client secret | +| `ADMIN_EMAIL` | No | `admin@flatlogic.com` | Default admin email | +| `ADMIN_PASS` | No | `88dbeaf8` | Default admin password | +| `USER_PASS` | No | `c3baadeda5c6` | Default user password | + +### config.ts Settings + +```javascript +{ + bcrypt: { saltRounds: 12 }, + secret_key: env.SECRET_KEY, + providers: { + LOCAL: 'local', + GOOGLE: 'google', + MICROSOFT: 'microsoft', + }, + google: { + clientId: env.GOOGLE_CLIENT_ID, + clientSecret: env.GOOGLE_CLIENT_SECRET, + }, + microsoft: { + clientId: env.MS_CLIENT_ID, + clientSecret: env.MS_CLIENT_SECRET, + }, + roles: { + admin: 'Administrator', + user: 'Analytics Viewer', + }, +} +``` + +--- + +## Authentication Flows + +### Local Login Flow + +``` +┌────────┐ ┌─────────┐ ┌─────────────┐ ┌──────────┐ +│ Client │────▶│ Router │────▶│ AuthService │────▶│ UsersDB │ +└────────┘ └─────────┘ └─────────────┘ └──────────┘ + │ │ │ │ + │ POST /signin │ │ │ + │ {email,pass} │ │ │ + │──────────────▶│ │ │ + │ │ signin() │ │ + │ │───────────────▶│ │ + │ │ │ findBy({email}) │ + │ │ │──────────────────▶│ + │ │ │◀──────────────────│ + │ │ │ │ + │ │ │ bcrypt.compare() │ + │ │ │ │ + │ │ │ jwtSign() │ + │ │◀───────────────│ │ + │◀──────────────│ │ │ + │ JWT Token │ │ │ +``` + +### OAuth Login Flow + +``` +┌────────┐ ┌─────────┐ ┌──────────┐ ┌──────────┐ +│ Client │ │ Backend │ │ Provider │ │ UsersDB │ +└────────┘ └─────────┘ └──────────┘ └──────────┘ + │ │ │ │ + │ GET /signin/ │ │ │ + │ google │ │ │ + │──────────────▶│ │ │ + │ │ Redirect to │ │ + │◀──────────────│ consent screen │ │ + │──────────────────────────────▶│ │ + │ │ │ │ + │ │ User consents │ │ + │◀──────────────────────────────│ │ + │ │ │ │ + │ GET /callback │ │ │ + │ ?code=... │ │ │ + │──────────────▶│ │ │ + │ │ Exchange code │ │ + │ │───────────────▶│ │ + │ │ Profile data │ │ + │ │◀───────────────│ │ + │ │ │ │ + │ │ findOrCreate() │ │ + │ │───────────────────────────────▶│ + │ │◀───────────────────────────────│ + │ │ jwtSign() │ │ + │ │ │ │ + │ Redirect to │ │ │ + │ /login?token= │ │ │ + │◀──────────────│ │ │ +``` + +### Email Verification Flow + +``` +┌────────┐ ┌─────────┐ ┌─────────────┐ ┌───────┐ +│ Client │ │ Router │ │ AuthService │ │ Email │ +└────────┘ └─────────┘ └─────────────┘ └───────┘ + │ │ │ │ + │ POST /signup │ │ │ + │──────────────▶│ │ │ + │ │ signup() │ │ + │ │───────────────▶│ │ + │ │ │ generateToken()│ + │ │ │ sendEmail() │ + │ │ │───────────────▶│ + │ │◀───────────────│ │ + │◀──────────────│ │ │ + │ JWT Token │ │ │ + │ │ │ │ + │ User clicks email link │ │ + │ │ │ │ + │ PUT /verify- │ │ │ + │ email │ │ │ + │ {token} │ │ │ + │──────────────▶│ │ │ + │ │ verifyEmail() │ │ + │ │───────────────▶│ │ + │ │ │ markVerified() │ + │ │◀───────────────│ │ + │◀──────────────│ │ │ + │ Success │ │ │ +``` + +--- + +## Error Codes + +| Error Key | HTTP Status | Description | +|-----------|-------------|-------------| +| `auth.userNotFound` | 400 | User with email doesn't exist | +| `auth.userDisabled` | 400 | User account is disabled | +| `auth.wrongPassword` | 400 | Password doesn't match | +| `auth.userNotVerified` | 400 | Email not verified | +| `auth.emailAlreadyInUse` | 400 | Email already registered | +| `auth.passwordUpdate.samePassword` | 400 | New password same as current | +| `auth.passwordReset.error` | 400 | Token generation failed | +| `auth.passwordReset.invalidToken` | 400 | Invalid or expired reset token | +| `auth.emailAddressVerificationEmail.error` | 400 | Verification email failed | +| `auth.emailAddressVerificationEmail.invalidToken` | 400 | Invalid verification token | + +--- + +## Security Considerations + +1. **Password Storage:** bcrypt with 12 salt rounds +2. **JWT Expiration:** 6 hours (balances security and UX) +3. **Token Generation:** crypto.randomBytes(20) - 160-bit entropy +4. **Token Expiry:** 24 hours for email/password reset tokens +5. **Rate Limiting:** Prevents brute force attacks +6. **Disabled User Check:** Checked on both JWT validation and login +7. **Session-less:** No server-side session storage (stateless JWT) +8. **HTTPS Required:** OAuth callbacks require HTTPS in production + +--- + +## Testing + +### Test Local Login + +```bash +curl -X POST http://localhost:3000/api/auth/signin/local \ + -H "Content-Type: application/json" \ + -d '{"email": "admin@flatlogic.com", "password": "password"}' +``` + +### Test Get Current User + +```bash +curl http://localhost:3000/api/auth/me \ + -H "Authorization: Bearer " +``` + +### User Creation + +Use the authenticated Users API/UI to create invited users. + +## Dependencies + +| Package | Version | Purpose | +|---------|---------|---------| +| `passport` | ^0.6.0 | Authentication middleware | +| `passport-jwt` | ^4.0.0 | JWT strategy for Passport | +| `passport-google-oauth2` | ^0.2.0 | Google OAuth strategy | +| `passport-microsoft` | ^2.0.0 | Microsoft OAuth strategy | +| `@types/passport-jwt` | ^4.0.1 | Maintained TypeScript definitions for JWT Passport strategy | +| `@types/passport-google-oauth2` | ^0.1.10 | Maintained TypeScript definitions for Google OAuth Passport strategy | +| `@types/passport-microsoft` | ^2.1.1 | Maintained TypeScript definitions for Microsoft Passport strategy | +| `jsonwebtoken` | ^9.0.0 | JWT sign/verify | +| `bcrypt` | ^5.1.0 | Password hashing | +| `crypto` | built-in | Token generation | + +--- + +## Summary + +The Auth module provides: + +1. **JWT Authentication** - Stateless API authentication with 6-hour tokens +2. **Local Login** - Email/password authentication with bcrypt +3. **OAuth 2.0** - Google and Microsoft social login +4. **Email Verification** - Token-based email confirmation +5. **Password Reset** - Secure token-based password recovery +6. **Rate Limiting** - Protection against brute force attacks +7. **Profile Management** - User profile updates +8. **Role Assignment** - Default role on signup diff --git a/backend/docs/modules/core.md b/backend/docs/modules/core.md new file mode 100644 index 0000000..2126472 --- /dev/null +++ b/backend/docs/modules/core.md @@ -0,0 +1,781 @@ +# Core Module Documentation + +The Core module provides the foundational components of the backend application: the entry point, configuration management, and utility functions. + +## Overview + +| File | Purpose | Lines | +|------|---------|-------| +| `src/index.ts` | Application entry point, Express setup, middleware, route mounting | varies | +| `src/config.ts` | Environment configuration and settings | varies | +| `src/helpers.js` | Utility functions (wrapAsync, JWT, validation) | 32 | +| `src/types/` | Shared strict TypeScript contracts for migrated backend code | varies | +| `src/load-env.ts` | Central backend `.env` bootstrap for app and DB entrypoints | varies | + +--- + +## 1. Application Entry Point (`index.ts`) + +### Purpose + +The main entry point that bootstraps the Express application, configures middleware, mounts routes, and starts the HTTP server. + +### Dependencies + +```typescript +import bodyParser from 'body-parser'; +import cors from 'cors'; +import express from 'express'; +import helmet from 'helmet'; +import * as swaggerUI from 'swagger-ui-express'; + +import { authenticateJwt, authenticateJwtWithCallback } from './auth/passport-middleware.ts'; +import config from './config.ts'; +import { wrapAsync } from './helpers.ts'; +import { runtimeContextMiddleware } from './middlewares/runtime-context.ts'; +import { downloadLimiter, searchLimiter, uploadLimiter } from './middlewares/rateLimiter.ts'; +import { createOpenApiDocument } from './openapi/document.ts'; +import { + exitAfterLogging, + logger, + registerProcessErrorHandlers, + requestLogger, +} from './utils/logger.ts'; +``` + +> TypeScript migration note: `index.ts`, `config.ts`, `helpers.ts`, `middlewares/validate-request.ts`, `middlewares/runtime-context.ts`, `middlewares/runtime-public.ts`, `routes/runtime-context.ts`, `validators/request-schemas.ts`, `services/notifications/list.ts`, `services/notifications/helpers.ts`, `utils/logger.ts`, and `utils/env-validation.ts` are now migrated runtime TypeScript modules. New migrated backend TypeScript files use `backend/tsconfig.json` with `strict: true`, reusable named types from `src/types/`, and lint rules that reject `any`, non-null assertions, and unsafe type assertions. + +> Environment bootstrap note: `src/load-env.ts` is loaded by app config and DB entrypoints so short backend scripts can see `backend/.env` without repeating `NODE_OPTIONS=-r dotenv/config`. If `NODE_ENV` is absent, it defaults to `dev_stage`, matching the standard VM backend flow. + +### Application Bootstrap Sequence + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ APPLICATION BOOTSTRAP │ +└─────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ 1. EXPRESS INITIALIZATION │ +│ • Create Express app │ +│ • Enable trust proxy (for reverse proxies) │ +└─────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ 2. SECURITY MIDDLEWARE │ +│ • Helmet (CSP disabled, COEP disabled) │ +│ • CORS (origin: true - allow all origins) │ +└─────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ 3. AUTHENTICATION SETUP │ +│ • Load Passport strategies (JWT, Google, Microsoft) │ +│ • Create jwtAuth middleware │ +└─────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ 4. LOGGING │ +│ • Register process-level error handlers │ +│ • Apply requestLogger middleware (early for full coverage) │ +└─────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ 5. FILE ROUTES (BEFORE BODY PARSER) │ +│ • Mount file routes without JSON parsing │ +│ • Apply rate limiters (download: 200/min, upload: 10/min) │ +└─────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ 6. BODY PARSING │ +│ • JSON parser (1MB limit) │ +│ • URL-encoded parser (1MB limit) │ +└─────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ 7. RUNTIME CONTEXT │ +│ • Apply runtimeContextMiddleware │ +│ • Detect environment from X-Runtime-Environment header │ +└─────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ 8. ROUTE MOUNTING │ +│ • Public routes (health, auth, runtime-context) │ +│ • Protected routes (JWT required) │ +│ • Runtime public routes (production content without auth) │ +└─────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ 9. STATIC FILES & ERROR HANDLING │ +│ • Serve public directory if exists │ +│ • Generic error handler │ +└─────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ 10. SERVER START │ +│ • Listen on PORT (8080 or 3000 for dev_stage) │ +│ • Log startup message │ +└─────────────────────────────────────────────────────────────────┘ +``` + +### Middleware Stack + +```javascript +// Order matters - applied in this sequence: + +1. swaggerUI.serve // API documentation at /api-docs +2. helmet() // Security headers +3. cors({ origin: true }) // Cross-origin requests +4. requestLogger // Request logging (Pino) +5. downloadLimiter // Rate limit for /api/file/download, /api/file/presign +6. uploadLimiter // Rate limit for /api/file/upload* +7. bodyParser.json() // JSON parsing (1MB limit) +8. bodyParser.urlencoded() // Form data parsing +9. runtimeContextMiddleware // Environment detection +10. passport.authenticate() // JWT authentication (per-route) +11. checkPermissions // RBAC (per-route) +12. errorHandler // Generic error handling +``` + +### Route Mounting + +#### Public Routes (No Authentication) + +```javascript +app.get('/api/health', ...) // Health check +app.use('/api/auth', authRoutes) // Authentication +app.use('/api/runtime-context', ...) // Runtime context +app.use('/api/file', fileRoutes) // File download/presign (partial) +``` + +#### Protected Routes (JWT Required) + +```javascript +app.use('/api/users', jwtAuth, usersRoutes) +app.use('/api/roles', jwtAuth, rolesRoutes) +app.use('/api/permissions', jwtAuth, permissionsRoutes) +app.use('/api/project_memberships', jwtAuth, project_membershipsRoutes) +app.use('/api/assets', jwtAuth, assetsRoutes) +app.use('/api/asset_variants', jwtAuth, asset_variantsRoutes) +app.use('/api/presigned_url_requests', jwtAuth, presigned_url_requestsRoutes) +app.use('/api/publish_events', jwtAuth, publish_eventsRoutes) +app.use('/api/pwa_caches', jwtAuth, pwa_cachesRoutes) +app.use('/api/access_logs', jwtAuth, access_logsRoutes) +app.use('/api/element-type-defaults', jwtAuth, element_type_defaultsRoutes) +app.use('/api/ui-elements', jwtAuth, element_type_defaultsRoutes) // Alias +app.use('/api/project-element-defaults', jwtAuth, project_element_defaultsRoutes) +app.use('/api/publish', jwtAuth, publishRoutes) +app.use('/api/search', jwtAuth, searchLimiter, searchRoutes) +``` + +#### Runtime Public Routes (Production Content Without Auth) + +```javascript +// These routes use requireRuntimeReadOrAuth middleware +// Allows unauthenticated GET requests in production environment + +mountRuntimeEntityRoute('/api/projects', 'projects', projectsRoutes) +mountRuntimeEntityRoute('/api/tour_pages', 'tour_pages', tour_pagesRoutes) +mountRuntimeEntityRoute('/api/project_audio_tracks', 'project_audio_tracks', project_audio_tracksRoutes) +``` + +### Key Functions + +#### `requireRuntimeReadOrAuth` + +Middleware that allows public read access for production content: + +```javascript +const requireRuntimeReadOrAuth = (req, res, next) => { + const headerEnvironment = req.runtimeContext?.headerEnvironment; + const isReadOnlyRequest = ['GET', 'OPTIONS'].includes(req.method); + const hasAuthHeader = Boolean(req.headers.authorization); + + // Only production is public. Stage requires authentication. + const isPublicEnvironment = headerEnvironment === 'production'; + + if (isPublicEnvironment && isReadOnlyRequest && !hasAuthHeader) { + req.isRuntimePublicRequest = true; + return next(); // Allow without JWT + } + + req.isRuntimePublicRequest = false; + return jwtAuth(req, res, next); // Require JWT +}; +``` + +#### `mountRuntimeEntityRoute` + +Helper to mount routes with runtime public access middleware stack: + +```javascript +const mountRuntimeEntityRoute = (path, entityName, router) => { + app.use( + path, + requireRuntimeReadOrAuth, // JWT or public production + blockNonPublicRuntimeListEndpoints, // Block non-list for public + sanitizePublicRuntimeListResponse(entityName), // Filter sensitive fields + router, + ); +}; +``` + +#### `getBaseUrl` + +Utility to extract base URL for Swagger: + +```javascript +const getBaseUrl = (url) => { + if (!url) return ''; + return url.endsWith('/api') ? url.slice(0, -4) : url; +}; +``` + +### Health Check Endpoint + +```javascript +GET /api/health + +// Response (200 - healthy): +{ + "status": "ok", + "timestamp": "2026-03-30T12:00:00.000Z", + "uptime": 12345.678, + "environment": "production", + "database": "connected" +} + +// Response (503 - degraded): +{ + "status": "degraded", + "timestamp": "2026-03-30T12:00:00.000Z", + "uptime": 12345.678, + "environment": "production", + "database": "disconnected", + "databaseError": "Connection refused" +} +``` + +### Swagger/OpenAPI Configuration + +```javascript +const specs = createOpenApiDocument({ + serverUrl: config.server.swaggerServerUrl, +}); + +app.use('/api-docs', swaggerUI.serve, swaggerUI.setup(specs)); +``` + +The canonical OpenAPI source is `backend/src/openapi/document.ts`. It defines +shared schemas, common responses and parameters, generated factory CRUD paths, +and explicit custom-route paths. Route-local JSDoc comments are not the source +of truth for Swagger UI. + +### Error Handler + +```javascript +app.use((err, req, res, _next) => { + if (!res.headersSent) { + const requestLog = getRequestLogger(req) ?? logger; + requestLog.error( + { err, url: req.url, method: req.method }, + 'Express error middleware caught unhandled error', + ); + res.status(safeStatusCode).json({ + message: + safeStatusCode === 503 + ? 'Service temporarily unavailable' + : 'Internal server error', + }); + } +}); +``` + +Route-level `commonErrorHandler` also uses the request-scoped logger when +available and logs unexpected route failures as `Route handler failed`. Circuit +breaker rejections use status `503` instead of being collapsed to `500`. + +### Server Configuration + +```javascript +const PORT = config.server.port; + +const server = app.listen(PORT, () => { + logger.info( + { port: PORT, env: config.server.env }, + 'Server started', + ); +}); + +server.on('error', (err) => { + logger.error( + { err, port: PORT, env: config.server.env }, + 'Server failed to start', + ); + exitAfterLogging(); +}); +``` + +Startup errors such as `EADDRINUSE` happen on the Node `Server`, outside the +Express request lifecycle, so the generic Express error middleware cannot catch +them. The server-level `error` listener logs the failure and exits with a +non-zero status for nodemon/PM2 supervision. + +--- + +## 2. Configuration (`config.ts`) + +### Purpose + +Centralized configuration management with environment variable support and sensible defaults. + +### Configuration Structure + +```javascript +const config = { + // Google Cloud Storage + gcloud: { + bucket: 'fldemo-files', + hash: 'afeefb9d49f5b7977577876b99532ac7', + projectId: env.GC_PROJECT_ID, + clientEmail: env.GC_CLIENT_EMAIL, + privateKey: env.GC_PRIVATE_KEY, + }, + fileStorage: { + provider: env.FILE_STORAGE_PROVIDER, + }, + + // AWS S3 + s3: { + bucket: env.AWS_S3_BUCKET, + region: env.AWS_S3_REGION, + accessKeyId: env.AWS_ACCESS_KEY_ID, + secretAccessKey: env.AWS_SECRET_ACCESS_KEY, + prefix: env.AWS_S3_PREFIX, + connectionTimeout: env.AWS_S3_CONNECTION_TIMEOUT, + requestTimeout: env.AWS_S3_REQUEST_TIMEOUT, + maxAttempts: env.AWS_S3_MAX_ATTEMPTS, + maxSockets: env.AWS_S3_MAX_SOCKETS, + keepAlive: env.AWS_S3_KEEP_ALIVE !== 'false', + presignExpirySeconds: env.AWS_S3_PRESIGN_EXPIRY, + }, + + resilience: { + ffmpeg: { + reverseTimeoutMs: env.FFMPEG_REVERSE_TIMEOUT_MS, + ffprobeTimeoutMs: env.FFPROBE_TIMEOUT_MS, + breaker: { + failureThreshold: env.FFMPEG_BREAKER_FAILURE_THRESHOLD, + cooldownMs: env.FFMPEG_BREAKER_COOLDOWN_MS, + successThreshold: env.FFMPEG_BREAKER_SUCCESS_THRESHOLD, + }, + }, + fileStorage: { + breaker: { + failureThreshold: env.FILE_STORAGE_BREAKER_FAILURE_THRESHOLD, + cooldownMs: env.FILE_STORAGE_BREAKER_COOLDOWN_MS, + successThreshold: env.FILE_STORAGE_BREAKER_SUCCESS_THRESHOLD, + }, + }, + }, + + // Password hashing + bcrypt: { + saltRounds: 12, + }, + + // Default credentials + admin_pass: env.ADMIN_PASS, + user_pass: env.USER_PASS, + admin_email: env.ADMIN_EMAIL, + + // Authentication providers + providers: { + LOCAL: 'local', + GOOGLE: 'google', + MICROSOFT: 'microsoft', + }, + + // JWT + secret_key: env.SECRET_KEY, + + // Server URLs + remote: '', + port, + hostUI, + portUI, + + // Swagger + swaggerUI, + swaggerPort, + + // OAuth + google: { + clientId: env.GOOGLE_CLIENT_ID, + clientSecret: env.GOOGLE_CLIENT_SECRET, + }, + microsoft: { + clientId: env.MS_CLIENT_ID, + clientSecret: env.MS_CLIENT_SECRET, + }, + + // File uploads + uploadDir: os.tmpdir(), + + // Email (AWS SES) + email: { + from: 'Tour Builder Platform ', + host: 'email-smtp.us-east-1.amazonaws.com', + port: 587, + auth: { + user: env.EMAIL_USER, + pass: env.EMAIL_PASS, + }, + tls: { + rejectUnauthorized: env.EMAIL_TLS_REJECT_UNAUTHORIZED !== 'false', + }, + }, + + // Default roles + roles: { + admin: 'Administrator', + user: 'Analytics Viewer', + }, + server: { + env: env.NODE_ENV, + port: serverPort, + swaggerServerUrl, + }, + +}; +``` + +### Environment Variables Reference + +| Variable | Type | Default | Description | +|----------|------|---------|-------------| +| `NODE_ENV` | string | `development` | Environment: `development`, `production`, `dev_stage`, `test` | +| `PORT` | number | `8080` | Server port | +| `SECRET_KEY` | string | UUID | JWT signing key (min 16 chars) | +| `ADMIN_EMAIL` | string | `admin@flatlogic.com` | Admin user email | +| `ADMIN_PASS` | string | Generated | Admin user password | +| `USER_PASS` | string | Generated | Default user password | +| `AWS_S3_BUCKET` | string | - | S3 bucket name | +| `AWS_S3_REGION` | string | `us-east-1` | S3 region | +| `AWS_ACCESS_KEY_ID` | string | - | AWS access key | +| `AWS_SECRET_ACCESS_KEY` | string | - | AWS secret key | +| `AWS_S3_PREFIX` | string | Hash | S3 key prefix | +| `GOOGLE_CLIENT_ID` | string | - | Google OAuth client ID | +| `GOOGLE_CLIENT_SECRET` | string | - | Google OAuth client secret | +| `MS_CLIENT_ID` | string | - | Microsoft OAuth client ID | +| `MS_CLIENT_SECRET` | string | - | Microsoft OAuth client secret | +| `EMAIL_USER` | string | - | SMTP username | +| `EMAIL_PASS` | string | - | SMTP password | +| `EMAIL_TLS_REJECT_UNAUTHORIZED` | string | `true` | TLS validation | +| `LOG_LEVEL` | string | `info` | Pino log level | + +### Environment Validation + +The configuration uses Joi schema validation via `utils/env-validation.js`: + +```javascript +const Joi = require('joi'); + +const envSchema = Joi.object({ + NODE_ENV: Joi.string() + .valid('development', 'test', 'production', 'dev_stage') + .default('development'), + + PORT: Joi.number().default(8080), + + DB_HOST: Joi.string().default('localhost'), + DB_PORT: Joi.number().default(5432), + DB_NAME: Joi.string().default('db_tour_builder_platform'), + DB_USER: Joi.string().default('postgres'), + DB_PASS: Joi.string().allow('').default(''), + + SECRET_KEY: Joi.string() + .min(16) + .default('88dbeaf8-e906-405e-9e41-c3baadeda5c6'), + + // ... more validations +}).unknown(true); + +function validateEnv() { + const { error, value } = envSchema.validate(process.env, { + abortEarly: false, + stripUnknown: false, + }); + + if (error) { + const messages = error.details.map((d) => ` - ${d.message}`); + logger.error({ errors: messages }, 'Environment validation failed'); + + if (process.env.NODE_ENV === 'production') { + process.exit(1); // Fatal in production + } else { + logger.warn('Continuing with default values in non-production mode'); + } + } + + return value; +} +``` + +--- + +## 3. Helpers (`helpers.js`) + +### Purpose + +Utility class providing common functions used across the application. + +### Class Definition + +```javascript +const jwt = require('jsonwebtoken'); +const config = require('./config'); + +module.exports = class Helpers { + /** + * Wraps async route handlers to catch errors and pass to next() + * @param {Function} fn - Async function (req, res, next) => Promise + * @returns {Function} Wrapped function + */ + static wrapAsync(fn) { + return function (req, res, next) { + fn(req, res, next).catch(next); + }; + } + + /** + * Common error handler middleware + * @param {Error} error - Error object with code/status property + * @param {Request} req - Express request + * @param {Response} res - Express response + * @param {Function} _next - Next middleware (unused) + */ + static commonErrorHandler(error, req, res, _next) { + const statusCode = error.code || error.status; + + // Known HTTP error codes - return error message + if ([400, 401, 403, 404, 409, 422, 503].includes(statusCode)) { + return res.status(statusCode).send(error.message); + } + + // Unknown errors - log and return generic message + const requestLog = getRequestLogger(req) ?? logger; + requestLog.error({ err: error }, 'Route handler failed'); + return res.status(500).send('Internal server error'); + } + + /** + * Sign JWT token with 6-hour expiration + * @param {Object} data - Payload to sign + * @returns {string} JWT token + */ + static jwtSign(data) { + return jwt.sign(data, config.secret_key, { expiresIn: '6h' }); + } + + /** + * Validate UUID v4 format + * @param {string} value - String to validate + * @returns {boolean} True if valid UUID v4 + */ + static isUuidV4(value) { + return /^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i.test( + value, + ); + } +}; +``` + +### Usage Examples + +#### wrapAsync + +```javascript +// Without wrapAsync - need try/catch +router.get('/', async (req, res, next) => { + try { + const data = await Service.findAll(); + res.json(data); + } catch (error) { + next(error); + } +}); + +// With wrapAsync - cleaner code +const wrapAsync = require('../helpers').wrapAsync; + +router.get('/', wrapAsync(async (req, res) => { + const data = await Service.findAll(); + res.json(data); +})); +``` + +#### commonErrorHandler + +```javascript +const { commonErrorHandler } = require('../helpers'); + +// At end of route file +router.use('/', commonErrorHandler); + +// Errors with code/status are returned as-is +const error = new Error('Not found'); +error.code = 404; +throw error; // → 404 "Not found" + +// Unknown errors return 500 +throw new Error('Database connection failed'); // → 500 "Internal server error" +``` + +#### jwtSign + +```javascript +const { jwtSign } = require('./helpers'); + +// Create JWT token +const token = jwtSign({ + user: { + id: user.id, + email: user.email, + }, +}); + +// Token expires in 6 hours +// Token is signed with config.secret_key +``` + +#### isUuidV4 + +```javascript +const { isUuidV4 } = require('./helpers'); + +// Validate UUID format +isUuidV4('550e8400-e29b-41d4-a716-446655440000'); // true +isUuidV4('550e8400-e29b-31d4-a716-446655440000'); // false (version 3) +isUuidV4('not-a-uuid'); // false +isUuidV4(''); // false +``` + +--- + +## Module Dependencies + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ index.ts │ +│ (Application Entry) │ +└─────────────────────────────────────────────────────────────────┘ + │ + ┌───────────────────┼───────────────────┐ + ▼ ▼ ▼ +┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ +│ config.ts │ │ helpers.ts │ │ utils/logger │ +│ (Configuration) │ │ (Utilities) │ │ (Logging) │ +└─────────────────┘ └─────────────────┘ └─────────────────┘ + │ │ + ▼ ▼ +┌─────────────────┐ ┌─────────────────┐ +│ env-validation │ │ jsonwebtoken │ +│ (Joi) │ │ (JWT lib) │ +└─────────────────┘ └─────────────────┘ + +External Dependencies: + • express (web framework) + • cors (CORS middleware) + • helmet (security headers) + • passport (authentication) + • body-parser (request parsing) + • swagger-ui-express (API docs) + • jsonwebtoken (JWT signing) + • joi (schema validation) + • dotenv (environment loading) + • pino (logging) +``` + +--- + +## Best Practices Implemented + +### 1. Environment Validation + +- Joi schema validates all environment variables at startup +- Fails fast in production, warns in development +- Provides sensible defaults for optional variables + +### 2. Security + +- Helmet middleware for security headers +- CORS configured for cross-origin requests +- JWT authentication with 6-hour expiration +- Trust proxy enabled for reverse proxy support + +### 3. Error Handling + +- `wrapAsync` wrapper for async route handlers +- Centralized error handler with status code mapping +- Generic error handler logs and returns safe message + +### 4. Middleware Ordering + +- File routes mounted before body parser (binary uploads) +- Rate limiters applied before routes +- Authentication applied per-route (not globally) +- Error handler at the end of middleware stack + +### 5. API Documentation + +- Swagger/OpenAPI 3.0 generated from `backend/src/openapi/document.ts` +- Available at `/api-docs` endpoint +- Security scheme documented (Bearer JWT) +- `backend/tests/openapi-document.test.ts` checks key paths, factory CRUD + coverage, internal `$ref` resolution, and Swagger UI handler smoke behavior + +### 6. Logging + +- Pino structured logging +- Request logger for all routes +- Error logging with context (URL, method) + +--- + +## Configuration Precedence + +``` +1. Environment Variables (process.env.*) + ↓ (fallback) +2. .env file (loaded by dotenv) + ↓ (fallback) +3. Joi schema defaults + ↓ (fallback) +4. Hardcoded defaults in config.ts +``` + +--- + +## Server Modes + +| NODE_ENV | Port | Database | Swagger | Description | +|----------|------|----------|---------|-------------| +| `development` | 8080 | Local | localhost:8080 | Legacy local development | +| `dev_stage` | 3000 | Remote | localhost:3000 | Staging preview | +| `production` | 8080 | Remote | Disabled | Production deployment | +| `test` | 8080 | Test DB | Disabled | Automated testing | + +**Standard VM note:** the VM PM2 setup runs the backend with +`NODE_ENV=dev_stage`, so the backend listens on port `3000`. The frontend runs +separately on port `3001`, and Apache proxies public traffic from port `80`. +Do not use `8080` as the VM backend health check unless the PM2 definition has +been changed. See [deployment-vm.md](../../../documentation/deployment-vm.md). diff --git a/backend/docs/modules/db-api.md b/backend/docs/modules/db-api.md new file mode 100644 index 0000000..8173c1d --- /dev/null +++ b/backend/docs/modules/db-api.md @@ -0,0 +1,1109 @@ +# Backend DB API Module + +## Overview + +The DB API module provides the data access layer that sits between services and Sequelize models. It encapsulates query logic, filtering, pagination, and data transformations using a declarative configuration pattern via the `GenericDBApi` base class. + +**Location:** `backend/src/db/api/` + +**Files:** 20 files (1 base class + 18 entity APIs + 1 utility) + +| File | Class/Purpose | LOC | Extends GenericDBApi | +|------|---------------|-----|---------------------| +| `base.api.ts` | `GenericDBApi` - Base class | 726 | - | +| `users.ts` | `UsersDBApi` - User accounts | 979 | No (custom) | +| `projects.ts` | `ProjectsDBApi` - Projects | ~320 | Yes | +| `tour_pages.ts` | `Tour_pagesDBApi` - Tour pages | ~350 | Yes | +| `assets.ts` | `AssetsDBApi` - Media assets | ~92 | Yes | +| `asset_variants.ts` | `Asset_variantsDBApi` - Asset variants | 82 | Yes | +| `roles.ts` | `RolesDBApi` - RBAC roles | 71 | Yes | +| `permissions.ts` | `PermissionsDBApi` - RBAC permissions | 53 | Yes | +| `project_memberships.ts` | `Project_membershipsDBApi` - Team access | 86 | Yes | +| `element_type_defaults.ts` | `Element_type_defaultsDBApi` - Global defaults | ~409 | Yes | +| `project_element_defaults.ts` | `Project_element_defaultsDBApi` - Project defaults | ~410 | Yes | +| `project_audio_tracks.ts` | `Project_audio_tracksDBApi` - Audio tracks | ~199 | Yes | +| `project_transition_settings.ts` | `Project_transition_settingsDBApi` - Project transition settings | ~277 | Yes | +| `global_transition_defaults.ts` | `Global_transition_defaultsDBApi` - Global transition defaults | ~155 | Yes | +| `global_ui_control_defaults.ts` | `Global_ui_control_defaultsDBApi` - Global UI control defaults | ~160 | Yes | +| `project_ui_control_settings.ts` | `Project_ui_control_settingsDBApi` - Project UI control settings | ~150 | Yes | +| `publish_events.ts` | `Publish_eventsDBApi` - Publishing history | 101 | Yes | +| `pwa_caches.ts` | `Pwa_cachesDBApi` - PWA manifests | 76 | Yes | +| `access_logs.ts` | `Access_logsDBApi` - Audit trail | 88 | Yes | +| `presigned_url_requests.ts` | `Presigned_url_requestsDBApi` - S3 URL audit | 90 | Yes | +| `file.ts` | `FileDBApi` - Polymorphic file attachments | ~95 | No (custom) | +| `runtime-context.ts` | Runtime context helpers | 57 | - | + +--- + +## Architecture + +``` +┌─────────────────────────────────────────────────────────────────────────┐ +│ DB API Layer Architecture │ +└─────────────────────────────────────────────────────────────────────────┘ + + ┌─────────────────────┐ + │ Service Layer │ + │ (services/*.js) │ + └──────────┬──────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────────────┐ +│ DB API Layer │ +│ │ +│ ┌─────────────────────────────────────────────────────────────────┐ │ +│ │ GenericDBApi (base.api.ts) │ │ +│ │ │ │ +│ │ Static Getters (Configuration): │ │ +│ │ • MODEL, TABLE_NAME │ │ +│ │ • SEARCHABLE_FIELDS, RANGE_FIELDS, ENUM_FIELDS, UUID_FIELDS │ │ +│ │ • ASSOCIATIONS, RELATION_FILTERS │ │ +│ │ • FIND_BY_INCLUDES, FIND_ALL_INCLUDES │ │ +│ │ • JSON_FIELDS, FIELD_DEFAULTS, FIELD_TRANSFORMERS │ │ +│ │ • CSV_FIELDS, AUTOCOMPLETE_FIELD │ │ +│ │ │ │ +│ │ Methods (CRUD + Query): │ │ +│ │ • create(), bulkImport(), update() │ │ +│ │ • deleteByIds(), remove() │ │ +│ │ • findBy(), findAll(), findAllAutocomplete() │ │ +│ │ • getFieldMapping(), toCSV() │ │ +│ └─────────────────────────────────────────────────────────────────┘ │ +│ │ │ +│ ┌──────────────┼──────────────┐ │ +│ │ │ │ │ +│ ▼ ▼ ▼ │ +│ ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ │ +│ │ Simple Entity │ │ Runtime-Aware │ │ Fully Custom │ │ +│ │ API │ │ API │ │ API │ │ +│ │ │ │ │ │ │ │ +│ │ • PermissionsDB │ │ • Tour_pagesDB │ │ • UsersDBApi │ │ +│ │ • RolesDBApi │ │ • ProjectsDBApi │ │ • FileDBApi │ │ +│ │ • AssetsDBApi │ │ • AudioTracksDB │ │ │ │ +│ │ │ │ │ │ │ │ +│ │ Override only: │ │ Override: │ │ Fully custom │ │ +│ │ - Static getters │ │ - Static getters │ │ implementation │ │ +│ │ - getFieldMapping│ │ - findBy() │ │ │ │ +│ │ │ │ - findAll() │ │ │ │ +│ └──────────────────┘ └──────────────────┘ └──────────────────┘ │ +│ │ +│ ┌─────────────────────────────────────────────────────────────────┐ │ +│ │ Supporting Utilities │ │ +│ │ runtime-context.ts - Environment/project filtering helpers │ │ +│ │ ../utils.js - UUID validation, ILIKE query building │ │ +│ └─────────────────────────────────────────────────────────────────┘ │ +│ │ +└─────────────────────────────────────────────────────────────────────────┘ + │ + ▼ + ┌─────────────────────┐ + │ Sequelize Models │ + │ (models/*.js) │ + └─────────────────────┘ +``` + +--- + +## GenericDBApi Base Class + +**Location:** `backend/src/db/api/base.api.ts` + +The base class provides a Template Method pattern where subclasses configure behavior through static getters and optionally override methods for custom logic. + +### Static Getters (Configuration) + +| Getter | Type | Default | Description | +|--------|------|---------|-------------| +| `MODEL` | Model | (required) | Sequelize model reference | +| `TABLE_NAME` | string | From MODEL | Database table name | +| `SEARCHABLE_FIELDS` | string[] | `[]` | Fields for ILIKE text search | +| `RANGE_FIELDS` | string[] | `[]` | Fields for range queries (min/max) | +| `ENUM_FIELDS` | string[] | `[]` | Fields for exact match filtering | +| `UUID_FIELDS` | string[] | `[]` | UUID foreign key fields (validated before query) | +| `RELATION_FILTERS` | object[] | `[]` | Related entity filter configs | +| `ASSOCIATIONS` | object[] | `[]` | M:N or belongsTo setters | +| `FIND_BY_INCLUDES` | object[] | `[]` | Includes for findBy() | +| `FIND_ALL_INCLUDES` | object[] | `[]` | Includes for findAll() | +| `CSV_FIELDS` | string[] | `['id', 'createdAt']` | Fields for CSV export | +| `AUTOCOMPLETE_FIELD` | string | `'name'` | Field for autocomplete | +| `JSON_FIELDS` | string[] | `[]` | Fields to auto-stringify | +| `FIELD_DEFAULTS` | object | `{}` | Default values for fields | +| `FIELD_TRANSFORMERS` | object | `{}` | Custom field transformations | + +### Methods + +#### getFieldMapping(data) + +Transforms input data for database operations using declarative configuration: + +```javascript +static getFieldMapping(data) { + if (!data) return data; + const mapped = { ...data }; + + // 1. Apply field defaults + for (const [field, config] of Object.entries(this.FIELD_DEFAULTS)) { + if (mapped[field] === undefined) { + mapped[field] = config.default; + } else if (mapped[field] === null && config.nullDefault !== undefined) { + mapped[field] = config.nullDefault; + } + } + + // 2. Auto-stringify JSON fields + for (const field of this.JSON_FIELDS) { + if (mapped[field] !== undefined && mapped[field] !== null) { + if (typeof mapped[field] !== 'string') { + mapped[field] = JSON.stringify(mapped[field]); + } + } + } + + // 3. Apply custom transformers + for (const [field, transformer] of Object.entries(this.FIELD_TRANSFORMERS)) { + if (mapped[field] !== undefined) { + mapped[field] = transformer(mapped[field]); + } + } + + return mapped; +} +``` + +#### create(options) + +Creates a new record with associations: + +```javascript +static async create({ data, currentUser = { id: null }, transaction, runtimeContext }) { + + const mappedData = this.getFieldMapping(data); + + const record = await this.MODEL.create( + { + ...mappedData, + importHash: data.importHash || null, + createdById: currentUser.id, + updatedById: currentUser.id, + }, + { transaction }, + ); + + // Handle M:N and belongsTo associations + for (const assoc of this.ASSOCIATIONS) { + if (data[assoc.field] !== undefined) { + const setter = record[assoc.setter]; + await setter.call( + record, + data[assoc.field] || (assoc.isArray ? [] : null), + { transaction }, + ); + } + } + + return record; +} +``` + +#### update({ id, data, currentUser, transaction, runtimeContext }) + +Updates a record with partial data: + +```javascript +static async update({ id, data, currentUser = { id: null }, transaction }) { + + const record = await this.MODEL.findByPk(id, { transaction }); + + if (!record) { + throw { status: 404, message: `${this.TABLE_NAME} not found` }; + } + + const updatePayload = { updatedById: currentUser.id }; + const mappedData = this.getFieldMapping(data); + + // Only update defined fields + for (const [key, value] of Object.entries(mappedData)) { + if (value !== undefined) { + updatePayload[key] = value; + } + } + + await record.update(updatePayload, { transaction }); + + // Update associations + for (const assoc of this.ASSOCIATIONS) { + if (data[assoc.field] !== undefined) { + const setter = record[assoc.setter]; + await setter.call(record, data[assoc.field], { transaction }); + } + } + + return record; +} +``` + +#### findAll(filter, options) + +Full-featured query with filtering, pagination, and sorting: + +```javascript +static async findAll(filter = {}, options = {}) { + const limit = filter.limit || 0; + const currentPage = +filter.page || 0; + const offset = currentPage * limit; + + let where = {}; + let include = [...this.FIND_ALL_INCLUDES]; + + // ID exact match (with UUID validation) + if (filter.id) { + if (!Utils.isValidUuid(filter.id)) { + return { rows: [], count: 0 }; + } + where.id = filter.id; + } + + // Text search (ILIKE) + for (const field of this.SEARCHABLE_FIELDS) { + if (filter[field]) { + where[Op.and] = Utils.ilike(this.TABLE_NAME, field, filter[field]); + } + } + + // Range queries (fieldRange: [min, max]) + for (const field of this.RANGE_FIELDS) { + const rangeKey = `${field}Range`; + if (filter[rangeKey]) { + const [start, end] = filter[rangeKey]; + if (start) where[field] = { ...where[field], [Op.gte]: start }; + if (end) where[field] = { ...where[field], [Op.lte]: end }; + } + } + + // Enum exact match + for (const field of this.ENUM_FIELDS) { + if (filter[field] !== undefined) { + where[field] = filter[field]; + } + } + + // Relation filters (search by related entity ID or name) + for (const rel of this.RELATION_FILTERS) { + if (filter[rel.filterKey]) { + // ... adds required include with where clause + } + } + + // createdAt date range + if (filter.createdAtRange) { + // ... applies date range + } + + const queryOptions = { + where, + include, + distinct: true, + order: filter.field && filter.sort + ? [[filter.field, filter.sort]] + : [['createdAt', 'desc']], + transaction: options.transaction, + }; + + if (!options.countOnly) { + queryOptions.limit = limit ? Number(limit) : undefined; + queryOptions.offset = offset ? Number(offset) : undefined; + } + + const { rows, count } = await this.MODEL.findAndCountAll(queryOptions); + return { rows: options.countOnly ? [] : rows, count }; +} +``` + +#### Other Methods + +| Method | Description | +|--------|-------------| +| `bulkImport(data, options)` | Bulk create with timestamps offset | +| `deleteByIds({ ids, currentUser, transaction, runtimeContext })` | Soft delete multiple records | +| `remove({ id, currentUser, transaction, runtimeContext })` | Soft delete single record | +| `findBy(where, options)` | Find single record by criteria | +| `findAllAutocomplete({ query, limit, offset }, options)` | Autocomplete search | +| `toCSV(rows)` | Convert rows to CSV string | + +--- + +## Query Filter Patterns + +### SEARCHABLE_FIELDS - Text Search + +Case-insensitive partial match using ILIKE: + +```javascript +// Configuration +static get SEARCHABLE_FIELDS() { + return ['name', 'description', 'email']; +} + +// Usage +GET /api/users?name=john +// Generates: WHERE LOWER(users.name) LIKE '%john%' +``` + +### RANGE_FIELDS - Range Queries + +Min/max range filtering: + +```javascript +// Configuration +static get RANGE_FIELDS() { + return ['sort_order', 'price', 'created_at']; +} + +// Usage +GET /api/assets?sort_orderRange=[0,10] +// Generates: WHERE sort_order >= 0 AND sort_order <= 10 + +GET /api/users?createdAtRange=[2024-01-01,2024-12-31] +// Generates: WHERE createdAt >= '2024-01-01' AND createdAt <= '2024-12-31' +``` + +### ENUM_FIELDS - Exact Match + +Direct equality filtering: + +```javascript +// Configuration +static get ENUM_FIELDS() { + return ['environment', 'status', 'asset_type']; +} + +// Usage +GET /api/tour_pages?environment=production +// Generates: WHERE environment = 'production' +``` + +### RELATION_FILTERS - Related Entity Search + +Filter by related entity ID or searchable field: + +```javascript +// Configuration +static get RELATION_FILTERS() { + return [ + { + filterKey: 'project', // Query param name + model: db.projects, // Related model + as: 'project', // Association alias + searchField: 'name', // Field for text search + }, + ]; +} + +// Usage +GET /api/assets?project=abc123 +// Filters by project ID: abc123 + +GET /api/assets?project=my-tour +// Filters by project name containing "my-tour" + +GET /api/assets?project=abc123|my-tour +// Pipe-separated: matches either +``` + +--- + +## API Categories + +### 1. Simple Entity APIs + +Extend `GenericDBApi` with minimal configuration. Only override static getters and `getFieldMapping()`. + +**Example: PermissionsDBApi** + +```typescript +class PermissionsDBApi extends GenericDBApi { + static override get MODEL(): unknown { return db.permissions; } + static override get TABLE_NAME(): string { return 'permissions'; } + static override get SEARCHABLE_FIELDS(): string[] { return ['name']; } + static override get CSV_FIELDS(): string[] { return ['id', 'name', 'createdAt']; } + static override get AUTOCOMPLETE_FIELD(): string { return 'name'; } + + static override getFieldMapping(data: PermissionData): PermissionFieldMapping { + return { + id: data.id || undefined, + name: data.name || null, + }; + } +} +``` + +**Entities using this pattern:** +- `PermissionsDBApi` +- `AssetsDBApi` +- `Asset_variantsDBApi` +- `Publish_eventsDBApi` +- `Pwa_cachesDBApi` +- `Access_logsDBApi` +- `Presigned_url_requestsDBApi` + +### 2. Runtime-Aware APIs + +Override `findBy()` and `findAll()` to apply runtime environment/project filters for public presentation access. + +**Example: Tour_pagesDBApi** + +```javascript +class Tour_pagesDBApi extends GenericDBApi { + // ... configuration getters ... + + static async findBy(where, options = {}) { + const queryWhere = applyRuntimeEnvironment({ ...where }, options); + const projectInclude = applyRuntimeProjectFilter( + { model: db.projects, as: 'project' }, + options, + ); + + const record = await this.MODEL.findOne({ + where: queryWhere, + include: [projectInclude], + }); + + return record ? record.get({ plain: true }) : null; + } + + static async findAll(filter = {}, options = {}) { + // ... standard filtering logic ... + + // Apply runtime environment filter + where = applyRuntimeEnvironment(where, options); + + // Apply project slug filter to include + include[0] = applyRuntimeProjectFilter(include[0], options); + + // ... execute query ... + } +} +``` + +**Example: ProjectsDBApi** + +`ProjectsDBApi` uses runtime slug filtering but with two unique behaviors: + +1. **Auto-snapshot on create**: Copies global element defaults to new projects +2. **Slug filter skipped for ID lookups**: Prevents stale `X-Runtime-Project-Slug` headers from breaking ID-based queries + +```javascript +class ProjectsDBApi extends GenericDBApi { + // ... configuration getters ... + + static async findBy(where, options = {}) { + const runtimeProjectSlug = getRuntimeProjectSlug(options); + const queryWhere = { ...where }; + + // Runtime access: filter by project slug + // Skip if finding by ID (unambiguous lookup) + if (runtimeProjectSlug && !where.id) { + queryWhere.slug = runtimeProjectSlug; + } + + const record = await this.MODEL.findOne({ + where: queryWhere, + include: options.include ?? this.DEFAULT_INCLUDES, + }); + + return record ? record.get({ plain: true }) : null; + } + + static async create(options) { + const { transaction } = options; + // Create the project using parent's create + const project = await super.create(options); + + // Auto-snapshot global element defaults to the new project + await Project_element_defaultsDBApi.snapshotGlobalDefaults(project.id, { + ...options, + transaction, + }); + + return project; + } + + static async findAll(filter = {}, options = {}) { + // ... standard filtering logic ... + + // Runtime access: filter by project slug (no ID bypass needed for list) + const runtimeProjectSlug = getRuntimeProjectSlug(options); + if (runtimeProjectSlug) { + where.slug = runtimeProjectSlug; + } + + // ... execute query ... + } +} +``` + +**Entities using this pattern:** +- `Tour_pagesDBApi` - Environment filtering via `applyRuntimeEnvironment()` +- `ProjectsDBApi` - Slug filtering with ID bypass and auto-snapshot on create +- `Project_audio_tracksDBApi` - Environment filtering + +### 3. APIs with Custom Methods + +Extend base functionality with domain-specific methods. + +**Example: Element_type_defaultsDBApi** + +```javascript +class Element_type_defaultsDBApi extends GenericDBApi { + // ... configuration ... + + // Self-initialization pattern + static async ensureInitialized() { + if (!this.initializationPromise) { + this.initializationPromise = (async () => { + const count = await this.MODEL.count(); + if (count > 0) return; + + // Seed default rows + await this.MODEL.bulkCreate( + this.DEFAULT_ROWS.map((item) => this.getFieldMapping(item)), + ); + })(); + } + await this.initializationPromise; + } + + // Override all methods to ensure initialization + static async create(options) { + await this.ensureInitialized(); + return super.create(options); + } + + // ... similar overrides for other methods ... + + // Default data configuration + static get DEFAULT_ROWS() { + return [ + { element_type: 'navigation_next', name: 'Navigation Forward Button', ... }, + { element_type: 'tooltip', name: 'Tooltip', ... }, + // ... 11 default element types + ]; + } +} +``` + +**Example: Project_element_defaultsDBApi** + +```javascript +class Project_element_defaultsDBApi extends GenericDBApi { + // ... configuration ... + + // Snapshot global defaults to a project + static async snapshotGlobalDefaults(projectId, options = {}) { + const globalDefaults = await Element_type_defaultsDBApi.findAll({}); + + return this.MODEL.bulkCreate( + globalDefaults.rows.map((global) => ({ + projectId, + element_type: global.element_type, + settings_json: global.default_settings_json, + source_element_id: global.id, + snapshot_version: 1, + })), + { transaction: options.transaction }, + ); + } + + // Reset project default to current global + static async resetToGlobal(id, options = {}) { + const projectDefault = await this.MODEL.findByPk(id); + const globalDefault = await Element_type_defaultsDBApi.MODEL.findOne({ + where: { element_type: projectDefault.element_type }, + }); + + await projectDefault.update({ + settings_json: globalDefault.default_settings_json, + snapshot_version: projectDefault.snapshot_version + 1, + }); + + return projectDefault.reload(); + } + + // Compare project default with global + static async getDiffFromGlobal(id) { + // ... returns diff object + } +} +``` + +### 4. Fully Custom APIs + +Don't extend `GenericDBApi` due to significantly different requirements. + +**Example: UsersDBApi** + +Complex user management with: +- Password hashing (bcrypt) +- File avatar handling +- Token generation for email verification and password reset +- OAuth authentication support + +```typescript +class UsersDBApi { + static async create(options: UserCreateOptions): Promise { + const { data, currentUser = { id: null }, transaction } = options; + const users = await db.users.create({ + firstName: data.firstName || null, + lastName: data.lastName || null, + email: data.email || null, + password: data.password || null, // Already hashed by service + // ... + }, { transaction }); + + // Auto-assign default role + if (!data.app_role) { + const role = await db.roles.findOne({ where: { name: 'User' } }); + await users.setApp_role(role, { transaction }); + } + + // Handle avatar file + await FileDBApi.replaceRelationFiles( + { belongsTo: 'users', belongsToColumn: 'avatar', belongsToId: users.id }, + data.data.avatar, + options, + ); + + return users; + } + + static async update({ id, data, currentUser, transaction, runtimeContext }) { + const users = await db.users.findByPk(id, { transaction }); + + // Hash password if changed + if (data.password) { + data.password = bcrypt.hashSync(data.password, config.bcrypt.saltRounds); + } else { + data.password = users.password; + } + + // ... update fields ... + } + + // Token generation + static async generateEmailVerificationToken(email, options) { + return this._generateToken( + ['emailVerificationToken', 'emailVerificationTokenExpiresAt'], + email, + options, + ); + } + + static async _generateToken(keyNames, email, options) { + const users = await db.users.findOne({ where: { email: email.toLowerCase() } }); + const token = crypto.randomBytes(20).toString('hex'); + const tokenExpiresAt = Date.now() + (24 * 60 * 60 * 1000); // 24 hours + + await users.update({ + [keyNames[0]]: token, + [keyNames[1]]: tokenExpiresAt, + }); + + return token; + } + + // Token lookup + static async findByPasswordResetToken(token, options) { + return db.users.findOne({ + where: { + passwordResetToken: token, + passwordResetTokenExpiresAt: { [Op.gt]: Date.now() }, + }, + }); + } +}; +``` + +**Example: FileDBApi** + +Polymorphic file attachment handling: + +```typescript +export default class FileDBApi { + static async replaceRelationFiles(relation, rawFiles, options = {}) { + assert(relation.belongsTo); + assert(relation.belongsToColumn); + assert(relation.belongsToId); + + const files = Array.isArray(rawFiles) ? rawFiles : rawFiles ? [rawFiles] : []; + + await this._removeLegacyFiles(relation, files, options); + await this._addFiles(relation, files, options); + } + + static async _addFiles(relation, files, options) { + const inexistentFiles = files.filter((file) => !!file.new); + + for (const file of inexistentFiles) { + await db.file.create({ + belongsTo: relation.belongsTo, + belongsToColumn: relation.belongsToColumn, + belongsToId: relation.belongsToId, + name: file.name, + publicUrl: file.publicUrl, + privateUrl: file.privateUrl, + }, { transaction: options.transaction }); + } + } + + static async _removeLegacyFiles(relation, files, options) { + const filesToDelete = await db.file.findAll({ + where: { + belongsTo: relation.belongsTo, + belongsToId: relation.belongsToId, + belongsToColumn: relation.belongsToColumn, + id: { [Op.notIn]: files.filter((f) => !f.new).map((f) => f.id) }, + }, + }); + + for (const file of filesToDelete) { + await services.deleteFile(file.privateUrl); + await file.destroy({ transaction: options.transaction }); + } + } +}; +``` + +--- + +## Runtime Context Helpers + +**Location:** `backend/src/db/api/runtime-context.ts` + +Utilities for filtering queries based on runtime presentation context (public tour access). + +```javascript +// Get runtime environment from options.runtimeContext +function getRuntimeEnvironment(options = {}) { + const runtimeContext = options.runtimeContext || null; + if (!runtimeContext) return null; + + // SECURITY: Only allow 'production' and 'stage' from header + if (runtimeContext.headerEnvironment === 'production') return 'production'; + if (runtimeContext.headerEnvironment === 'stage') return 'stage'; + return null; +} + +// Get project slug from runtime context +function getRuntimeProjectSlug(options = {}) { + const runtimeContext = options.runtimeContext || null; + return runtimeContext?.headerProjectSlug || null; +} + +// Apply environment filter to where clause +function applyRuntimeEnvironment(where = {}, options = {}) { + const environment = getRuntimeEnvironment(options); + if (!environment) return where; + + return { ...where, environment }; +} + +// Apply project slug filter to include +function applyRuntimeProjectFilter(projectInclude = {}, options = {}) { + const projectSlug = getRuntimeProjectSlug(options); + if (!projectSlug) return projectInclude; + + return { + ...projectInclude, + required: true, + where: { ...(projectInclude.where || {}), slug: projectSlug }, + }; +} +``` + +**Usage:** + +```javascript +// In runtime-aware API +static async findAll(filter = {}, options = {}) { + let where = { /* ... */ }; + let include = [{ model: db.projects, as: 'project' }]; + + // Apply runtime filters + where = applyRuntimeEnvironment(where, options); + include[0] = applyRuntimeProjectFilter(include[0], options); + + // ... execute query +} +``` + +--- + +## DB Utils + +**Location:** `backend/src/db/utils.js` + +Utility functions for UUID validation and query building: + +```javascript +const validator = require('validator'); +const { v4: uuidv4 } = require('uuid'); + +module.exports = class Utils { + // Check if value is a valid UUID + static isValidUuid(value) { + return Boolean(value && validator.isUUID(String(value))); + } + + // Generate a new UUID v4 + static generateUuid() { + return uuidv4(); + } + + // Filter array to only valid UUIDs + static filterValidUuids(values) { + return values.filter((v) => this.isValidUuid(v)); + } + + // Case-insensitive LIKE query + static ilike(model, column, value) { + return Sequelize.where( + Sequelize.fn('lower', Sequelize.col(`${model}.${column}`)), + { [Sequelize.Op.like]: `%${value}%`.toLowerCase() }, + ); + } +}; +``` + +**UUID Utility Functions:** + +| Function | Purpose | Returns | +|----------|---------|---------| +| `isValidUuid(value)` | Check if valid UUID | `boolean` | +| `generateUuid()` | Create new UUID v4 | `string` | +| `filterValidUuids(values)` | Filter array to valid UUIDs only | `string[]` | +| `ilike(model, column, value)` | Case-insensitive search | Sequelize where clause | + +**UUID Validation Behavior:** +- Invalid single ID filter (`?id=xxx`) → returns `{ rows: [], count: 0 }` immediately +- Invalid UUID in relation filter (`?project=uuid|name`) → filters out invalid UUIDs for ID search, keeps all terms for text search +- Invalid UUID field filter (`?projectId=xxx`) → returns `{ rows: [], count: 0 }` immediately + +--- + +## Declarative Configuration Examples + +### Simple Entity + +```javascript +class AssetsDBApi extends GenericDBApi { + static get MODEL() { return db.assets; } + static get TABLE_NAME() { return 'assets'; } + + static get SEARCHABLE_FIELDS() { + return ['name', 'cdn_url', 'storage_key', 'mime_type', 'checksum']; + } + + static get RANGE_FIELDS() { + return ['size_mb', 'width_px', 'height_px', 'duration_sec']; + } + + static get ENUM_FIELDS() { + return ['asset_type', 'type', 'is_public']; + } + + // UUID foreign key fields - validated before querying + static get UUID_FIELDS() { + return ['projectId']; + } + + static get ASSOCIATIONS() { + return [{ field: 'project', setter: 'setProject', isArray: false }]; + } + + static get FIND_BY_INCLUDES() { + return [ + { association: 'asset_variants_asset' }, + { association: 'project' }, + ]; + } + + static get RELATION_FILTERS() { + return [ + { filterKey: 'project', model: db.projects, as: 'project', searchField: 'name' }, + ]; + } + + static getFieldMapping(data) { + return { + name: data.name || null, + asset_type: data.asset_type || null, + type: data.type || 'general', + cdn_url: data.cdn_url || null, + storage_key: data.storage_key || null, + mime_type: data.mime_type || null, + size_mb: data.size_mb || null, + width_px: data.width_px || null, + height_px: data.height_px || null, + duration_sec: data.duration_sec || null, + checksum: data.checksum || null, + is_public: data.is_public || false, + }; + } +} +``` + +### Foreign Key Fields in getFieldMapping + +**Important:** Foreign key fields (like `assetId`, `projectId`) must be included in `getFieldMapping()` if they are passed directly to `create()` or `update()`. Without this, the foreign key value is silently dropped and the record is created without the relationship. + +**Two ways to handle foreign keys:** + +1. **Direct field mapping** (preferred for programmatic use): +```javascript +// In getFieldMapping() +static getFieldMapping(data) { + return { + assetId: data.assetId || null, // Include FK field directly + variant_type: data.variant_type || null, + // ... + }; +} + +// In service +await Asset_variantsDBApi.create({ assetId: asset.id, ... }); +``` + +2. **Via ASSOCIATIONS setter** (used by frontend forms): +```javascript +// ASSOCIATIONS config uses 'asset' (relation name) +static get ASSOCIATIONS() { + return [{ field: 'asset', setter: 'setAsset', isArray: false }]; +} + +// Service passes 'asset' instead of 'assetId' +await Asset_variantsDBApi.create({ asset: asset.id, ... }); +// GenericDBApi.create() calls record.setAsset(asset.id) with record as `this` +``` + +**Common mistake:** Passing `assetId` when only `asset` association is configured (or vice versa) results in orphaned records with NULL foreign keys. + +**Implementation detail:** Sequelize association mixins rely on the model +instance as `this`. `GenericDBApi` must call association setters as bound +record methods (`setter.call(record, ...)`) rather than detached functions; +otherwise belongsTo setters such as `setProject`/`setAsset` receive an +undefined source instance and fail before the foreign key can be saved. + +### Entity with M:N Relations + +```typescript +class RolesDBApi extends GenericDBApi { + static override get MODEL(): unknown { return db.roles; } + + static override get ASSOCIATIONS(): RoleAssociationConfig[] { + return [{ field: 'permissions', setter: 'setPermissions', isArray: true }]; + } + + static override get FIND_BY_INCLUDES(): unknown[] { + return [ + { association: 'users_app_role' }, + { association: 'permissions' }, + ]; + } + + static override get FIND_ALL_INCLUDES(): unknown[] { + return [ + { model: db.permissions, as: 'permissions', required: false }, + ]; + } + + static get RELATION_FILTERS() { + return [ + { filterKey: 'permissions', model: db.permissions, as: 'permissions_filter', searchField: 'name' }, + ]; + } +} +``` + +### Entity with JSON Fields and Defaults + +```javascript +class Element_type_defaultsDBApi extends GenericDBApi { + // Auto-stringify JSON fields + static get JSON_FIELDS() { + return ['default_settings_json']; + } + + // Field defaults + static get FIELD_DEFAULTS() { + return { + element_type: { default: null }, + name: { default: null }, + sort_order: { default: 0 }, + }; + } + + static getFieldMapping(data) { + // Apply base class transformations first + const mapped = super.getFieldMapping(data); + + return { + id: mapped.id || undefined, + element_type: mapped.element_type, + name: mapped.name, + sort_order: mapped.sort_order, + default_settings_json: mapped.default_settings_json, + }; + } +} +``` + +--- + +## API Summary Table + +| API | Pattern | Getters | Custom Methods | Notes | +|-----|---------|---------|----------------|-------| +| `PermissionsDBApi` | Simple | 6 | 0 | Minimal config | +| `AssetsDBApi` | Simple | 9 | 0 | With associations | +| `RolesDBApi` | Simple | 10 | 0 | M:N permissions | +| `ProjectsDBApi` | Runtime-aware | 10 | 3 | Auto-snapshot on create, slug filter skipped for ID lookups | +| `Tour_pagesDBApi` | Runtime-aware | 9 | 0 | Environment filtering | +| `Project_audio_tracksDBApi` | Runtime-aware | 8 | 0 | Environment filtering | +| `Element_type_defaultsDBApi` | Self-init | 9 | 1 | Default seeding | +| `Project_element_defaultsDBApi` | Extended | 10 | 4 | Snapshot, reset, diff | +| `UsersDBApi` | Fully custom | - | 12 | Auth, tokens, files | +| `FileDBApi` | Fully custom | - | 3 | Polymorphic files | + +--- + +## Best Practices + +### When to Extend GenericDBApi + +- Standard CRUD operations needed +- Filtering by searchable/range/enum fields +- Pagination and sorting +- CSV export +- Autocomplete + +### When to Override Methods + +- Custom runtime environment filtering +- Special business logic during create/update +- Complex association handling +- Self-initialization patterns + +### When to Use Fully Custom API + +- Complex authentication logic (password hashing, tokens) +- Polymorphic associations (FileDBApi) +- Significantly different query patterns +- Multiple specialized methods + +--- + +## Related Documentation + +- [DB Models Module](./db-models.md) - Sequelize model definitions +- [Factories Module](./factories.md) - Service/router factories that use APIs +- [Services Module](./services.md) - Business logic layer +- [Middleware Module](./middleware.md) - Runtime context middleware diff --git a/backend/docs/modules/db-config.md b/backend/docs/modules/db-config.md new file mode 100644 index 0000000..bad02bf --- /dev/null +++ b/backend/docs/modules/db-config.md @@ -0,0 +1,505 @@ +# DB Config Module + +## Overview + +The DB Config module manages database connection settings, environment validation, and database utilities. It provides environment-aware configuration for PostgreSQL connections via Sequelize ORM. + +**Location:** `backend/src/db/` + +**Key Files:** +- `db-config.ts` - Typed ESM database connection settings per environment +- `umzug.ts` - Typed Umzug runner for migrations, seeders, create/drop +- `utils.ts` - Database utility functions +- `sync.ts` - Development database sync script +- `reset.ts` - Database reset script + +**Related Files:** +- `backend/src/config.ts` - Application configuration +- `backend/src/utils/env-validation.ts` - Environment variable validation + +--- + +## Architecture + +``` +backend/ +├── src/ +│ ├── config.ts # Application config +│ ├── utils/ +│ │ └── env-validation.ts # Joi schema validation +│ └── db/ +│ ├── db-config.ts # Typed database connection config +│ ├── umzug.ts # Typed migration/seed runner +│ ├── utils.ts # DB utilities +│ ├── sync.ts # Dev sync script (21 LOC) +│ ├── reset.ts # Reset script (18 LOC) +│ ├── models/ +│ │ ├── loader.ts # Typed model loader + Sequelize init +│ │ └── index.ts # ESM model registry entrypoint +│ ├── migrations/ # Schema migrations +│ └── seeders/ # Data seeders +``` + +--- + +## Database Configuration + +### db-config.ts + +`db-config.ts` exports environment-specific database settings for Sequelize. +It uses the official Sequelize `Options` type plus project-specific +`DatabaseConfigMap`/`DatabaseEnvironmentConfig` reusable contracts from +`backend/src/types/db-config.ts`. + +Database scripts use `backend/src/db/umzug.ts` directly. + +`DB_PORT` is parsed to a number for the Sequelize `Options` contract. If the +env var is absent or invalid, the `port` property is omitted. + +### Environment Comparison + +| Setting | Production | Development | Dev Stage | +|---------|------------|-------------|-----------| +| **Dialect** | postgres | postgres | postgres | +| **Credentials** | Env vars | Hardcoded | Env vars | +| **Logging** | Disabled | Pino debug | Pino debug | +| **Host** | Env var | localhost | Env var | +| **Migration Storage** | SequelizeMeta | SequelizeMeta | SequelizeMeta | +| **Seeder Storage** | SequelizeData | SequelizeData | SequelizeData | + +--- + +## Environment Variables + +### Database Variables + +| Variable | Required | Default | Description | +|----------|----------|---------|-------------| +| `NODE_ENV` | No | development | Environment selection | +| `DB_HOST` | Prod/Stage | localhost | Database host | +| `DB_PORT` | Prod/Stage | 5432 | Database port | +| `DB_NAME` | Prod/Stage | db_tour_builder_platform | Database name | +| `DB_USER` | Prod/Stage | postgres | Database username | +| `DB_PASS` | Prod/Stage | (empty) | Database password | + +### Environment Validation + +The `env-validation.ts` file uses Joi schema to validate all environment variables: + +```javascript +const Joi = require('joi'); + +const envSchema = Joi.object({ + NODE_ENV: Joi.string() + .valid('development', 'test', 'production', 'dev_stage') + .default('development'), + + PORT: Joi.number().default(8080), + + DB_HOST: Joi.string().default('localhost'), + DB_PORT: Joi.number().default(5432), + DB_NAME: Joi.string().default('db_tour_builder_platform'), + DB_USER: Joi.string().default('postgres'), + DB_PASS: Joi.string().allow('').default(''), + + SECRET_KEY: Joi.string() + .min(16) + .default('88dbeaf8-e906-405e-9e41-c3baadeda5c6'), + + // ... more variables +}).unknown(true); +``` + +### Complete Environment Variable Schema + +| Category | Variable | Validation | Default | +|----------|----------|------------|---------| +| **Server** | NODE_ENV | enum: development, test, production, dev_stage | development | +| | PORT | number | 8080 | +| **Database** | DB_HOST | string | localhost | +| | DB_PORT | number | 5432 | +| | DB_NAME | string | db_tour_builder_platform | +| | DB_USER | string | postgres | +| | DB_PASS | string (allow empty) | (empty) | +| **Auth** | SECRET_KEY | string, min 16 chars | (default UUID) | +| | ADMIN_PASS | string | 88dbeaf8 | +| | USER_PASS | string | c3baadeda5c6 | +| | ADMIN_EMAIL | email | admin@flatlogic.com | +| **OAuth** | GOOGLE_CLIENT_ID | string (allow empty) | (empty) | +| | GOOGLE_CLIENT_SECRET | string (allow empty) | (empty) | +| | MS_CLIENT_ID | string (allow empty) | (empty) | +| | MS_CLIENT_SECRET | string (allow empty) | (empty) | +| **AWS S3** | AWS_ACCESS_KEY_ID | string (allow empty) | (empty) | +| | AWS_SECRET_ACCESS_KEY | string (allow empty) | (empty) | +| | AWS_S3_BUCKET | string (allow empty) | (empty) | +| | AWS_S3_REGION | string | us-east-1 | +| | AWS_S3_PREFIX | string | (default hash) | +| | AWS_S3_CONNECTION_TIMEOUT | number (ms) | 5000 | +| | AWS_S3_REQUEST_TIMEOUT | number (ms) | 30000 | +| | AWS_S3_MAX_ATTEMPTS | number | 3 | +| | AWS_S3_MAX_SOCKETS | number | 50 | +| | AWS_S3_KEEP_ALIVE | boolean string | true | +| | AWS_S3_PRESIGN_EXPIRY | number (seconds) | 3600 | +| **Email** | EMAIL_USER | string (allow empty) | (empty) | +| | EMAIL_PASS | string (allow empty) | (empty) | +| | EMAIL_TLS_REJECT_UNAUTHORIZED | enum: true, false | true | +| **External APIs** | PEXELS_KEY | string (allow empty) | (empty) | +| **Logging** | LOG_LEVEL | enum: fatal, error, warn, info, debug, trace | info | + +### Validation Behavior + +```javascript +function validateEnv() { + const { error, value } = envSchema.validate(process.env, { + abortEarly: false, // Report all errors, not just first + stripUnknown: false, // Keep unknown env vars + }); + + if (error) { + const messages = error.details.map((d) => ` - ${d.message}`); + logger.error({ errors: messages }, 'Environment validation failed'); + + if (process.env.NODE_ENV === 'production') { + process.exit(1); // Fatal in production + } else { + logger.warn('Continuing with default values in non-production mode'); + } + } + + return value; +} +``` + +--- + +## Umzug Configuration + +The database command entrypoint is `backend/src/db/umzug.ts`. + +### Runtime Paths + +| Setting | Path | +|---------|------| +| Config | `src/db/db-config.ts` | +| Runner | `src/db/umzug.ts` | +| Models | `src/db/models/` | +| Seeders | `src/db/seeders/` | +| Migrations | `src/db/migrations/` | + +--- + +## Database Initialization + +### Model Loader + +`src/db/models/loader.ts` explicitly imports model factories and builds the +Sequelize registry. `src/db/models/index.ts` is the ESM entrypoint. The typed +service-specific overload facade lives in `src/types/db-models.ts`. + +### Initialization Flow + +``` +Server Start + │ + ▼ +config.ts loads + │ + ├─ dotenv.config() ─── Load .env file + │ + ├─ validateEnv() ─── Validate with Joi schema + │ │ + │ ├─ Production: Exit on error + │ └─ Development: Warn, use defaults + │ + └─ Export config object + │ + ▼ +db/models/index.ts loads + │ + ├─ Read NODE_ENV + │ + ├─ Load db-config.ts[env] + │ + ├─ Create Sequelize instance + │ + ├─ Initialize explicitly imported model factories + │ + ├─ Call model.associate() for relationships + │ + └─ Export db object { sequelize, Sequelize, models... } +``` + +--- + +## Database Utilities + +### utils.ts + +Database utility helpers are TypeScript/ESM source. Runtime DB access goes +through `src/db/models/index.ts`, whose typed facade is provided by +`src/types/db-models.ts`. + +### Usage Examples + +```typescript +import Utils from '../db/utils.ts'; + +// UUID validation +Utils.isValidUuid('550e8400-e29b-41d4-a716-446655440000'); // true +Utils.isValidUuid('not-a-uuid'); // false + +// Generate new UUID +const id = Utils.generateUuid(); // Returns new UUID v4 + +// Filter array to valid UUIDs only +const validIds = Utils.filterValidUuids(['uuid1', 'invalid', 'uuid2']); + +// Case-insensitive search +const where = { + [Op.or]: [ + Utils.ilike('users', 'firstName', searchTerm), + Utils.ilike('users', 'lastName', searchTerm), + Utils.ilike('users', 'email', searchTerm), + ] +}; +``` + +--- + +## Database Scripts + +### sync.ts (Development Only) + +Synchronizes models to database schema using Sequelize's `alter` mode. + +```javascript +async function syncDatabase() { + // Safety check - never run in production + if (process.env.NODE_ENV === 'production') { + console.error('ERROR: sync.ts should not be run in production. Use migrations instead.'); + process.exit(1); + } + + try { + console.log('Syncing database...'); + await db.sequelize.sync({ alter: true }); + console.log('Database synced successfully!'); + process.exit(0); + } catch (error) { + console.error('Error syncing database:', error); + process.exit(1); + } +} +``` + +**Usage:** +```bash +node src/db/sync.ts +``` + +**Sync Modes:** + +| Mode | Description | Use Case | +|------|-------------|----------| +| `{ force: true }` | Drop and recreate all tables | Fresh start | +| `{ alter: true }` | Modify tables to match models | Development | +| (none) | Create only missing tables | Safe default | + +### reset.ts + +Forcefully resets database and runs seeders. + +```javascript +db.sequelize + .sync({ force: true }) + .then(() => { + execSync('sequelize db:seed:all'); + console.log('OK'); + process.exit(); + }) + .catch((error) => { + console.error(error); + process.exit(1); + }); +``` + +**Usage:** +```bash +node src/db/reset.ts +``` + +**Warning:** This drops ALL tables and recreates them. Data loss! + +--- + +## Application Configuration + +### config.ts + +Centralized application configuration with environment-aware settings. + +```javascript +const env = validateEnv(); + +const config = { + gcloud: { + bucket: 'fldemo-files', + hash: 'afeefb9d49f5b7977577876b99532ac7', + projectId: env.GC_PROJECT_ID, + clientEmail: env.GC_CLIENT_EMAIL, + privateKey: env.GC_PRIVATE_KEY, + }, + fileStorage: { + provider: env.FILE_STORAGE_PROVIDER, + }, + s3: { + bucket: env.AWS_S3_BUCKET, + region: env.AWS_S3_REGION, + accessKeyId: env.AWS_ACCESS_KEY_ID, + secretAccessKey: env.AWS_SECRET_ACCESS_KEY, + prefix: env.AWS_S3_PREFIX, + connectionTimeout: env.AWS_S3_CONNECTION_TIMEOUT, + requestTimeout: env.AWS_S3_REQUEST_TIMEOUT, + maxAttempts: env.AWS_S3_MAX_ATTEMPTS, + maxSockets: env.AWS_S3_MAX_SOCKETS, + keepAlive: env.AWS_S3_KEEP_ALIVE !== 'false', + presignExpirySeconds: env.AWS_S3_PRESIGN_EXPIRY, + }, + resilience: { + ffmpeg: { reverseTimeoutMs, ffprobeTimeoutMs, breaker }, + fileStorage: { breaker }, + }, + server: { + env: env.NODE_ENV, + port: serverPort, + swaggerServerUrl, + }, + secret_key: env.SECRET_KEY, + admin_pass: env.ADMIN_PASS, + user_pass: env.USER_PASS, + admin_email: env.ADMIN_EMAIL, + + // Roles + roles: { + admin: 'Administrator', + user: 'Analytics Viewer', + }, + apiUrl: `${host}${port ? `:${port}` : ''}/api`, + swaggerUrl: `${swaggerUI}${swaggerPort}`, + uiUrl: `${hostUI}${portUI ? `:${portUI}` : ''}/#`, + backUrl: `${hostUI}${portUI ? `:${portUI}` : ''}`, +}; +``` + +--- + +## Configuration Categories + +### Storage Configuration + +| Provider | Variables | Purpose | +|----------|-----------|---------| +| **AWS S3** | AWS_S3_BUCKET, AWS_S3_REGION, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY | File storage | +| **GCloud** | (hardcoded bucket) | Legacy support | +| **Local** | uploadDir (os.tmpdir()) | Development fallback | + +### S3 Performance Tuning + +| Variable | Default | Description | +|----------|---------|-------------| +| AWS_S3_CONNECTION_TIMEOUT | 5000ms | TCP connection timeout | +| AWS_S3_REQUEST_TIMEOUT | 30000ms | Total request timeout | +| AWS_S3_MAX_ATTEMPTS | 3 | Retry attempts on failure | +| AWS_S3_MAX_SOCKETS | 50 | Connection pool size | +| AWS_S3_KEEP_ALIVE | true | Reuse TCP connections | +| AWS_S3_PRESIGN_EXPIRY | 3600s | Presigned URL validity (1 hour) | + +### Security Configuration + +| Setting | Value | Purpose | +|---------|-------|---------| +| bcrypt.saltRounds | 12 | Password hashing strength | +| SECRET_KEY | 16+ char string | JWT signing key | +| EMAIL_TLS_REJECT_UNAUTHORIZED | true/false | TLS certificate validation | + +### URL Configuration + +| URL | Development | Production | +|-----|-------------|------------| +| apiUrl | http://localhost:3000/api | (remote)/api | +| swaggerUrl | http://localhost:3000 | (remote) | +| uiUrl | http://localhost:3001/# | (remote)/# | +| backUrl | http://localhost:3001 | (remote) | + +--- + +## Running Commands + +### Development +```bash +cd backend +npm run start-dev +``` + +`start-dev` runs the standard `start` pipeline (`db:migrate`, `db:seed`, +`watch`) with `LOG_PRETTY=true`. `src/load-env.ts` loads `backend/.env` before +DB config selection; when `NODE_ENV` is absent it defaults to `dev_stage`, which +matches the standard VM backend flow and listens on port `3000`. + +### VM / Dev Stage +```bash +cd backend +npm run start +``` + +In the standard VM PM2 setup, `NODE_ENV=dev_stage` makes the backend listen on +port `3000`; the frontend runs separately on `3001` and Apache proxies public +traffic from port `80`. See [`deployment-vm.md`](../../../documentation/deployment-vm.md). + +Do not add `NODE_ENV=production` only to make local startup work. The current +flow loads `.env` through `src/load-env.ts` and defaults missing `NODE_ENV` to +`dev_stage`, matching the VM backend process. + +--- + +## Best Practices + +### 1. Never Commit Secrets +```bash +# .env file should be in .gitignore +# Use environment variables in deployment +``` + +### 2. Use Migrations in Production +```javascript +// Never use sync.ts or reset.ts in production +if (process.env.NODE_ENV === 'production') { + process.exit(1); +} +``` + +### 3. Validate Environment Early +```javascript +// config.ts loads validation at import time +import { validateEnv } from './utils/env-validation.ts'; +validateEnv(); // Called before app starts +``` + +### 4. Environment-Specific Logging +```javascript +// Production: logging disabled (performance) +// Development/dev_stage: SQL logs use structured Pino debug entries +logging: process.env.NODE_ENV === 'production' + ? false + : (sql) => logger.debug({ sql }, 'Sequelize query'), +``` + +--- + +## Related Documentation + +- [DB Models](./db-models.md) - Sequelize model definitions +- [DB API](./db-api.md) - Database access layer +- [DB Migrations](./db-migrations.md) - Schema evolution +- [DB Seeders](./db-seeders.md) - Initial data population +- [Core Module](./core.md) - Application entry point diff --git a/backend/docs/modules/db-migrations.md b/backend/docs/modules/db-migrations.md new file mode 100644 index 0000000..e023a3e --- /dev/null +++ b/backend/docs/modules/db-migrations.md @@ -0,0 +1,764 @@ +# DB Migrations Module + +## Overview + +The DB Migrations module manages database schema evolution using a typed +**Umzug v3** runner. It provides version-controlled, reversible database +changes that run automatically on server startup. + +**Location:** `backend/src/db/migrations/` + +**Files:** 34+ migration files (as of June 2026) + +**Migration safety policy:** Do not rewrite, rename, or reformat already +applied migration files. Production databases track migration names in +`SequelizeMeta`, and fresh databases must replay the same schema history. + +--- + +## Architecture + +``` +backend/ +├── package.json # Umzug-backed migration scripts +└── src/db/ + ├── db-config.ts # Database connection settings + ├── umzug.ts # Typed Umzug runner + └── migrations/ # Migration files + ├── package.json # CommonJS boundary for legacy migrations + ├── 20260319000001-*.js # Foreign key constraints + ├── 20260319000002-*.js # Column cleanup + ├── 20260326000001-*.js # Table rename + ├── 20260326000002-*.js # ENUM to TEXT conversion + ├── 20260326000003-*.js # Create table + ├── 20260326000004-*.js # Data backfill + ├── 20260326000005-*.js # Environment fix + ├── 20260326000006-*.js # Cross-environment copy + ├── 20260326043002-*.js # NOT NULL enforcement + ├── 20260326050442-*.js # Column removal + ├── 20260326054410-*.js # Column removal + ├── 20260326060000-*.js # JSON data transformation + ├── 20260326060001-*.js # Table drop (page_elements) + ├── 20260326060002-*.js # Table drop (page_links) + ├── 20260326060003-*.js # Table drop (transitions) + ├── 20260326171017-*.js # Missing data insertion + ├── 20260327000001-*.js # Full data sync + ├── 20260331024423-*.js # Column removal (theme/css) + ├── 20260331054340-*.js # Duplicate cleanup + ├── 20260331063424-*.js # Invalid data cleanup + ├── 20260403000001-*.js # Background video settings + ├── 20260409000001-*.js # Design dimensions (projects) + ├── 20260409111309-*.js # Design dimensions (tour_pages) + ├── 20260605000001-*.js # Background audio settings + ├── 20260626000001-*.js # Private production presentation access + ├── 20260626000002-*.js # Account manager user creation permission + ├── 20260628000001-*.js # Global UI-control settings tables + └── 20260628000005-*.js # Existing-project UI-control snapshots +``` + +--- + +## Configuration + +### Umzug Runner + +`backend/src/db/umzug.ts` owns migration and seeder execution. It uses official +Umzug types, `SequelizeStorage`, and the existing storage tables: + +| Flow | Files | Storage Table | Stored Names | +|------|-------|---------------|--------------| +| Migrations | `src/db/migrations/*.js` | `SequelizeMeta` | `*.js` | +| Seeders | `src/db/seeders/*.ts` in source, `dist/src/db/seeders/*.js` in build | `SequelizeData` | stable `*.js` names | + +Seeder files are typed ESM source, and the runner stores stable execution names +so already executed seeders are not treated as pending. + +### NPM Scripts +```bash +# Run pending migrations +npm run db:migrate + +# Undo last migration +npm run db:migrate:undo + +# Undo all migrations +npm run db:migrate:undo:all + +# Check migration status +npm run db:migrate:status + +# Full database reset (drop, create, migrate, seed) +npm run db:reset + +# Seed data +npm run db:seed +``` + +--- + +## Migration File Structure + +### Standard Template +```javascript +'use strict'; + +/** + * Migration: [Description] + * + * [Explanation of what this migration does and why] + */ +module.exports = { + async up(queryInterface, Sequelize) { + // Apply changes + }, + + async down(queryInterface, Sequelize) { + // Revert changes + }, +}; +``` + +Use one project-wide migration template for new schema changes. Do not modify +already applied migration files to match newer style choices. + +### Naming Convention +``` +YYYYMMDDHHMMSS-descriptive-name.js + +Examples: +- 20260319000001-add-foreign-key-constraints.js +- 20260326060000-convert-targetpageid-to-slug.js +- 20260326060001-drop-page-elements-table.js +- 20260628000001-create-ui-control-settings.js +- 20260628000005-snapshot-existing-project-ui-controls.js +``` + +--- + +## Migration Patterns + +### 1. Transaction Wrapper Pattern +**Purpose:** Ensure atomic operations - all changes succeed or all fail. + +```javascript +module.exports = { + async up(queryInterface) { + const transaction = await queryInterface.sequelize.transaction(); + + try { + // Multiple operations... + await queryInterface.addColumn('table', 'column', { ... }, { transaction }); + await queryInterface.addIndex('table', ['column'], { transaction }); + + await transaction.commit(); + } catch (error) { + await transaction.rollback(); + throw error; + } + }, +}; +``` + +**Used in:** Foreign key constraints, ENUM conversions, data transformations + +--- + +### 2. Idempotent Check Pattern +**Purpose:** Safely re-run migrations without errors. + +```javascript +// Check if table/column/constraint exists before modifying +const [results] = await queryInterface.sequelize.query( + `SELECT column_name FROM information_schema.columns + WHERE table_name = 'tableName' AND column_name = 'columnName'`, + { transaction } +); + +if (results.length > 0) { + console.log('Column already exists, skipping'); + return; +} + +// Proceed with modification +await queryInterface.addColumn('tableName', 'columnName', { ... }); +``` + +**Used in:** All migrations that add columns, constraints, or tables + +--- + +### 3. Helper Function Pattern +**Purpose:** Reduce repetition for bulk operations. + +```javascript +module.exports = { + async up(queryInterface) { + const transaction = await queryInterface.sequelize.transaction(); + + // Define reusable helper + const addForeignKey = async (tableName, columnName, references, onDelete) => { + const constraintName = `${tableName}_${columnName}_fkey`; + + // Check existence + const [results] = await queryInterface.sequelize.query( + `SELECT constraint_name FROM information_schema.table_constraints + WHERE table_name = '${tableName}' AND constraint_name = '${constraintName}'`, + { transaction } + ); + + if (results.length === 0) { + await queryInterface.addConstraint(tableName, { + fields: [columnName], + type: 'foreign key', + name: constraintName, + references, + onDelete, + onUpdate: 'CASCADE', + transaction, + }); + console.log(`Added FK: ${constraintName}`); + } + }; + + // Use helper multiple times + await addForeignKey('assets', 'projectId', { table: 'projects', field: 'id' }, 'CASCADE'); + await addForeignKey('tour_pages', 'projectId', { table: 'projects', field: 'id' }, 'CASCADE'); + // ... more FKs + }, +}; +``` + +**Used in:** Foreign key constraints, column removal, index management + +--- + +### 4. Safe Table Drop Pattern +**Purpose:** Prevent accidental data loss when dropping tables. + +```javascript +module.exports = { + async up(queryInterface) { + // Verify table is empty before dropping + const [results] = await queryInterface.sequelize.query( + 'SELECT COUNT(*) as count FROM table_name' + ); + const count = parseInt(results[0].count, 10); + + if (count > 0) { + throw new Error( + `Cannot drop table_name: it contains ${count} records. ` + + `Please migrate or delete them first.` + ); + } + + await queryInterface.dropTable('table_name'); + console.log('Dropped table_name (was empty)'); + }, + + async down(queryInterface, Sequelize) { + // Full table recreation with all columns, indexes, and constraints + await queryInterface.createTable('table_name', { + id: { type: Sequelize.UUID, ... }, + // ... all columns + }); + }, +}; +``` + +**Used in:** `drop-page-elements-table`, `drop-page-links-table`, `drop-transitions-table` + +--- + +### 5. ENUM to TEXT Conversion Pattern +**Purpose:** Convert restrictive ENUMs to flexible TEXT while preserving data. + +```javascript +module.exports = { + async up(queryInterface, Sequelize) { + const transaction = await queryInterface.sequelize.transaction(); + + try { + // 1. Create temporary TEXT column + await queryInterface.addColumn('table', 'column_text', { + type: Sequelize.TEXT, + allowNull: true, + }, { transaction }); + + // 2. Copy ENUM values to TEXT + await queryInterface.sequelize.query( + `UPDATE table SET column_text = column::TEXT`, + { transaction } + ); + + // 3. Drop old ENUM column + await queryInterface.removeColumn('table', 'column', { transaction }); + + // 4. Rename TEXT column + await queryInterface.renameColumn('table', 'column_text', 'column', { transaction }); + + // 5. Add NOT NULL constraint + await queryInterface.changeColumn('table', 'column', { + type: Sequelize.TEXT, + allowNull: false, + }, { transaction }); + + // 6. Drop ENUM type + await queryInterface.sequelize.query( + `DROP TYPE IF EXISTS "enum_table_column"`, + { transaction } + ); + + await transaction.commit(); + } catch (error) { + await transaction.rollback(); + throw error; + } + }, + + async down(queryInterface, Sequelize) { + // Recreate ENUM type + await queryInterface.sequelize.query(` + CREATE TYPE "enum_table_column" AS ENUM ('value1', 'value2', 'value3') + `); + // ... reverse the process + }, +}; +``` + +**Used in:** `convert-element-type-enum-to-text` + +--- + +### 6. Data Backfill Pattern +**Purpose:** Populate new tables/columns with data from existing records. + +```javascript +// Define default data +const DEFAULT_ELEMENT_TYPES = [ + { element_type: 'navigation_next', name: 'Forward Button', sort_order: 1, settings_json: {...} }, + { element_type: 'navigation_prev', name: 'Back Button', sort_order: 2, settings_json: {...} }, + // ... more defaults +]; + +module.exports = { + async up(queryInterface, Sequelize) { + // Get existing records + const [projects] = await queryInterface.sequelize.query( + `SELECT id FROM projects WHERE "deletedAt" IS NULL`, + { type: Sequelize.QueryTypes.SELECT } + ); + + // For each project, check and insert missing records + for (const project of projects) { + const [existing] = await queryInterface.sequelize.query( + `SELECT element_type FROM project_element_defaults + WHERE "projectId" = :projectId AND "deletedAt" IS NULL`, + { replacements: { projectId: project.id }, type: Sequelize.QueryTypes.SELECT } + ); + + const existingTypes = new Set(existing.map(d => d.element_type)); + + for (const defaultType of DEFAULT_ELEMENT_TYPES) { + if (!existingTypes.has(defaultType.element_type)) { + await queryInterface.sequelize.query(` + INSERT INTO project_element_defaults (...) + VALUES (gen_random_uuid(), :element_type, :name, ...) + `, { replacements: { ... } }); + } + } + } + }, + + async down(queryInterface) { + // Delete only records created by this migration + await queryInterface.sequelize.query( + `DELETE FROM project_element_defaults WHERE snapshot_version = 1` + ); + }, +}; +``` + +**Used in:** `backfill-project-element-defaults`, `sync-all-element-type-defaults` + +--- + +### 7. Cross-Environment Data Copy Pattern +**Purpose:** Copy content between environments (dev → stage → production). + +```javascript +module.exports = { + async up(queryInterface, Sequelize) { + const projects = await queryInterface.sequelize.query( + `SELECT id FROM projects WHERE "deletedAt" IS NULL`, + { type: Sequelize.QueryTypes.SELECT } + ); + + for (const project of projects) { + // Check if target environment already has content + const [stageCheck] = await queryInterface.sequelize.query( + `SELECT COUNT(*)::int as count FROM tour_pages + WHERE "projectId" = '${project.id}' AND environment = 'stage'` + ); + + if (stageCheck?.count > 0) continue; + + // Copy with INSERT...SELECT, generating new UUIDs + await queryInterface.sequelize.query(` + INSERT INTO tour_pages (id, slug, name, ..., environment, source_key, ...) + SELECT + gen_random_uuid(), + slug, name, ..., + 'stage', -- New environment + id::text, -- Track source record for rollback + ... + FROM tour_pages + WHERE "projectId" = '${project.id}' AND environment = 'dev' + `); + + // Copy related records using source_key for ID mapping + await queryInterface.sequelize.query(` + INSERT INTO page_elements (id, ..., "pageId", ...) + SELECT + gen_random_uuid(), ..., + stage_page.id, -- Map to new page ID + ... + FROM page_elements pe + INNER JOIN tour_pages dev_page ON pe."pageId" = dev_page.id + INNER JOIN tour_pages stage_page ON stage_page.source_key = dev_page.id::text + WHERE dev_page.environment = 'dev' + `); + } + }, + + async down(queryInterface) { + // Delete records with source_key (created by migration) + await queryInterface.sequelize.query( + `DELETE FROM tour_pages WHERE environment = 'stage' AND source_key IS NOT NULL` + ); + }, +}; +``` + +**Used in:** `copy-dev-to-stage` + +--- + +### 8. JSON Field Transformation Pattern +**Purpose:** Transform data stored in JSON columns. + +```javascript +module.exports = { + async up(queryInterface, Sequelize) { + const transaction = await queryInterface.sequelize.transaction(); + + try { + // Get all records with JSON data + const [records] = await queryInterface.sequelize.query( + `SELECT id, "projectId", environment, slug, json_column + FROM table_name WHERE json_column IS NOT NULL`, + { transaction } + ); + + // Build lookup maps for ID → slug transformations + const slugById = new Map(); + records.forEach(r => slugById.set(r.id, { projectId: r.projectId, slug: r.slug })); + + // Transform each record + for (const record of records) { + const jsonData = typeof record.json_column === 'string' + ? JSON.parse(record.json_column) + : record.json_column; + + let hasChanges = false; + + // Transform JSON structure + if (jsonData.elements) { + jsonData.elements.forEach(element => { + if (element.targetPageId) { + const target = slugById.get(element.targetPageId); + if (target) { + element.targetPageSlug = target.slug; + delete element.targetPageId; + hasChanges = true; + } + } + }); + } + + if (hasChanges) { + await queryInterface.sequelize.query( + `UPDATE table_name SET json_column = :json WHERE id = :id`, + { + replacements: { json: JSON.stringify(jsonData), id: record.id }, + transaction + } + ); + } + } + + await transaction.commit(); + } catch (error) { + await transaction.rollback(); + throw error; + } + }, +}; +``` + +**Used in:** `convert-targetpageid-to-slug` + +--- + +### 9. Constraint Enforcement Pattern +**Purpose:** Add NOT NULL constraints after fixing existing NULL values. + +```javascript +module.exports = { + async up(queryInterface) { + // First, fix any NULL values + await queryInterface.sequelize.query( + `UPDATE table_name SET column = 'default' WHERE column IS NULL` + ); + + // Then add NOT NULL constraint with default + await queryInterface.sequelize.query(` + ALTER TABLE table_name + ALTER COLUMN column SET NOT NULL, + ALTER COLUMN column SET DEFAULT 'default' + `); + }, + + async down(queryInterface) { + await queryInterface.sequelize.query(` + ALTER TABLE table_name + ALTER COLUMN column DROP NOT NULL, + ALTER COLUMN column DROP DEFAULT + `); + }, +}; +``` + +**Used in:** `enforce-environment-not-null` + +--- + +### 10. Safe Down Migration Pattern +**Purpose:** Handle cases where down migration isn't meaningful. + +```javascript +module.exports = { + async up(queryInterface, Sequelize) { + // Add/update missing data + // ... + }, + + async down(_queryInterface, _Sequelize) { + // This migration only adds missing data, not destructive + console.log('No down migration needed - this migration only adds missing data.'); + }, +}; +``` + +**Used in:** `sync-all-element-type-defaults` + +--- + +## Migration Categories + +### Schema Changes +| Migration | Description | +|-----------|-------------| +| `add-foreign-key-constraints` | Add FK constraints to all model associations | +| `create-project-element-defaults` | Create new table with indexes | +| `drop-page-elements-table` | Drop unused table | +| `drop-page-links-table` | Drop unused table | +| `drop-transitions-table` | Drop unused table | + +### Column Modifications +| Migration | Description | +|-----------|-------------| +| `remove-redundant-deletion-columns` | Remove `is_deleted`, `deleted_at_time` | +| `remove-project-phase-column` | Remove redundant `phase` column | +| `remove-entry-page-slug-column` | Remove unused column | +| `convert-element-type-enum-to-text` | ENUM → TEXT for flexibility | +| `enforce-environment-not-null` | Add NOT NULL constraint | +| `remove-unused-theme-columns-from-projects` | Remove `theme_config_json`, `custom_css_json`, `cdn_base_url` | +| `add-background-video-settings` | Add video playback settings (autoplay, loop, muted, start/end time) to tour_pages | +| `add-design-dimensions-to-projects` | Add `design_width`, `design_height` to projects table | +| `add-design-dimensions-to-tour-pages` | Add `design_width`, `design_height` to tour_pages table | + +### Table Renames +| Migration | Description | +|-----------|-------------| +| `rename-ui-elements-to-element-type-defaults` | Rename for clarity | + +### Data Migrations +| Migration | Description | +|-----------|-------------| +| `backfill-project-element-defaults` | Populate new table for existing projects | +| `copy-dev-to-stage` | Initialize stage environment | +| `convert-targetpageid-to-slug` | Transform JSON navigation references | +| `fix-project-audio-tracks-environment` | Fix environment values | +| `add-missing-element-type-defaults` | Insert missing default rows | +| `sync-all-element-type-defaults` | Full sync of all 11 element types | +| `remove-duplicate-element-type-defaults` | Remove duplicate records created during earlier migrations | +| `cleanup-invalid-element-type-defaults` | Clean up invalid entries and ensure data integrity | + +--- + +## Foreign Key Strategies + +| Strategy | When to Use | Example | +|----------|-------------|---------| +| `CASCADE` | Delete child when parent deleted | `assets.projectId → projects.id` | +| `SET NULL` | Preserve record, nullify FK | `publish_events.userId → users.id` (audit trail) | +| `SET NULL` + `allowNull: true` | Optional FK | `users.app_roleId → roles.id` | + +```javascript +// CASCADE - delete assets when project is deleted +await addForeignKey('assets', 'projectId', { table: 'projects', field: 'id' }, 'CASCADE'); + +// SET NULL - preserve audit log when user is deleted +await addForeignKey('access_logs', 'userId', { table: 'users', field: 'id' }, 'SET NULL'); +``` + +--- + +## Best Practices + +### 1. Always Use Transactions +```javascript +const transaction = await queryInterface.sequelize.transaction(); +try { + // ... operations + await transaction.commit(); +} catch (error) { + await transaction.rollback(); + throw error; +} +``` + +### 2. Check Before Modify +```javascript +// Always check existence before adding/removing +const tableExists = await queryInterface.sequelize.query( + `SELECT EXISTS (SELECT FROM information_schema.tables WHERE table_name = 'name')` +); +``` + +### 3. Log Progress +```javascript +console.log(`Migrating project ${projectId}: ${addedCount} records added`); +console.log('Migration complete: All foreign keys added'); +``` + +### 4. Safe Drops +```javascript +// Never drop non-empty tables silently +if (count > 0) { + throw new Error(`Cannot drop table: contains ${count} records`); +} +``` + +### 5. Reversible Operations +```javascript +// Down migration should restore previous state +async down(queryInterface, Sequelize) { + // Recreate everything that was dropped + await queryInterface.createTable('table_name', { ... }); + await queryInterface.addIndex('table_name', [...]); +} +``` + +### 6. Use Parameterized Queries +```javascript +// Good - prevents SQL injection +await queryInterface.sequelize.query( + `UPDATE table SET column = :value WHERE id = :id`, + { replacements: { value: 'safe', id: record.id } } +); + +// Avoid - SQL injection risk +await queryInterface.sequelize.query( + `UPDATE table SET column = '${unsafeValue}' WHERE id = '${unsafeId}'` +); +``` + +--- + +## Running Migrations + +### Development +```bash +cd backend +npm run db:migrate +``` + +### Server Startup +Migrations run automatically via `npm start`: +```json +{ + "scripts": { + "start": "npm run db:migrate && npm run db:seed && npm run watch" + } +} +``` + +### Migration Status +```bash +npm run db:migrate:status +``` + +### Undo Migrations +```bash +# Undo last migration +npm run db:migrate:undo + +# Undo all migrations (dangerous!) +npm run db:migrate:undo:all +``` + +### Create New Migration + +Create new migration files manually under `backend/src/db/migrations/` only +when a schema change is required. Keep every migration reversible or document an +explicit rollback/backup plan. + +--- + +## Current Migration Inventory + +| # | Timestamp | Name | Type | +|---|-----------|------|------| +| 1 | 20260319000001 | add-foreign-key-constraints | Schema | +| 2 | 20260319000002 | remove-redundant-deletion-columns | Column | +| 3 | 20260326000001 | rename-ui-elements-to-element-type-defaults | Rename | +| 4 | 20260326000002 | convert-element-type-enum-to-text | Column | +| 5 | 20260326000003 | create-project-element-defaults | Schema | +| 6 | 20260326000004 | backfill-project-element-defaults | Data | +| 7 | 20260326000005 | fix-project-audio-tracks-environment | Data | +| 8 | 20260326000006 | copy-dev-to-stage | Data | +| 9 | 20260326043002 | enforce-environment-not-null | Column | +| 10 | 20260326050442 | remove-project-phase-column | Column | +| 11 | 20260326054410 | remove-entry-page-slug-column | Column | +| 12 | 20260326060000 | convert-targetpageid-to-slug | Data | +| 13 | 20260326060001 | drop-page-elements-table | Schema | +| 14 | 20260326060002 | drop-page-links-table | Schema | +| 15 | 20260326060003 | drop-transitions-table | Schema | +| 16 | 20260326171017 | add-missing-element-type-defaults | Data | +| 17 | 20260327000001 | sync-all-element-type-defaults | Data | +| 18 | 20260331024423 | remove-unused-theme-columns-from-projects | Column | +| 19 | 20260331054340 | remove-duplicate-element-type-defaults | Data | +| 20 | 20260331063424 | cleanup-invalid-element-type-defaults | Data | +| 21 | 20260403000001 | add-background-video-settings | Column | +| 22 | 20260409000001 | add-design-dimensions-to-projects | Column | +| 23 | 20260409111309 | add-design-dimensions-to-tour-pages | Column | +| 24 | 20260605000001 | add-background-audio-settings | Column | + +--- + +## Related Documentation + +- [DB Models](./db-models.md) - Sequelize model definitions +- [DB API](./db-api.md) - Database access layer +- [Database Schema](../database-schema.md) - Complete schema reference diff --git a/backend/docs/modules/db-models.md b/backend/docs/modules/db-models.md new file mode 100644 index 0000000..9115ad7 --- /dev/null +++ b/backend/docs/modules/db-models.md @@ -0,0 +1,977 @@ +# Backend DB Models Module + +## Overview + +The DB Models module defines the Sequelize ORM models that map to PostgreSQL database tables. It provides the data layer for the Tour Builder Platform, handling entity definitions, relationships, validations, and lifecycle hooks. + +**Location:** `backend/src/db/models/` + +**Files:** 22 model-loader entries (20 models + loader + index bridge). During +the backend TS/ESM migration, model entries have a typed `.ts` source plus a +typed ESM source file. There is no model-level CommonJS compatibility facade. + +| File | Model | Purpose | LOC | +|------|-------|---------|-----| +| `index.ts` | - | ESM entrypoint re-exporting `loader.ts` | 1 | +| `loader.ts` | - | Typed model registry and Sequelize initialization | 128 | +| `users.ts` + `.js` bridge | `users` | User accounts with authentication | 246 | +| `projects.ts` + `.js` bridge | `projects` | Virtual tour projects | 211 | +| `production_presentation_access.ts` + `.js` bridge | `production_presentation_access` | Customer grants for private production presentations | 67 | +| `tour_pages.ts` + `.js` bridge | `tour_pages` | Individual tour pages with UI schema | 131 | +| `assets.ts` + `.js` bridge | `assets` | Uploaded media files | 169 | +| `asset_variants.ts` + `.js` bridge | `asset_variants` | Asset size/format variants | 103 | +| `roles.ts` + `roles.js` bridge | `roles` | RBAC roles | 85 | +| `permissions.ts` + `permissions.js` bridge | `permissions` | RBAC permissions | 52 | +| `project_memberships.ts` + `.js` bridge | `project_memberships` | User-project access | 89 | +| `publish_events.ts` + `.js` bridge | `publish_events` | Publishing history | 148 | +| `pwa_caches.ts` + `.js` bridge | `pwa_caches` | PWA offline cache manifests | 84 | +| `access_logs.ts` + `.js` bridge | `access_logs` | Activity audit trail | 105 | +| `element_type_defaults.ts` + `.js` bridge | `element_type_defaults` | Global UI element defaults | 91 | +| `project_element_defaults.ts` + `.js` bridge | `project_element_defaults` | Project-specific element defaults | 101 | +| `project_audio_tracks.ts` + `.js` bridge | `project_audio_tracks` | Background audio tracks | 103 | +| `project_transition_settings.ts` + `.js` bridge | `project_transition_settings` | Environment-aware CSS transition settings | 95 | +| `global_transition_defaults.ts` + `.js` bridge | `global_transition_defaults` | Platform defaults for CSS page transitions | 65 | +| `global_ui_control_defaults.ts` + `.js` bridge | `global_ui_control_defaults` | Platform defaults for fullscreen, sound, and offline controls | 33 | +| `project_ui_control_settings.ts` + `.js` bridge | `project_ui_control_settings` | Project/environment overrides for global UI controls | 61 | +| `presigned_url_requests.ts` + `.js` bridge | `presigned_url_requests` | S3 presigned URL audit | 118 | +| `file.ts` + `.js` bridge | `file` | Generic file attachments | 53 | + +--- + +## Architecture + +### TS/ESM Migration Boundary + +The model loader uses explicit typed ESM imports from `backend/src/db/models/`. +Migrated model definitions use this structure: + +- `model-name.ts` contains the typed ESM model factory. +- `loader.ts` explicitly imports every model factory and builds the Sequelize + registry without dynamic CommonJS discovery. +- `index.ts` is the ESM entrypoint to `loader.ts` used by DB consumers. +- `backend/src/types/db-models.ts` keeps the service-specific overload facade + as reusable TypeScript contracts. +- Shared model factory contracts live in `backend/src/types/sequelize-models.ts` + and use official Sequelize types. + +As of 2026-07-01, all model definitions and the model loader use typed ESM +source. `users.ts` keeps the hook payload typed through reusable +`UserModelInstance` and `UsersSequelizeModel` contracts without type +assertions. + +`npm run check:esm-boundaries` guards this boundary: new `.js` source or +CommonJS syntax in `.ts` fails the check unless it is an explicitly documented +bridge or immutable migration. + +``` +┌─────────────────────────────────────────────────────────────────────────┐ +│ Model Loading Flow │ +└─────────────────────────────────────────────────────────────────────────┘ + +┌─────────────────┐ +│ index.ts │ +│ (entrypoint) │ +└────────┬────────┘ + │ + │ 1. require('./loader.ts') + ▼ +┌─────────────────┐ +│ loader.ts │ +└────────┬────────┘ + ▼ +┌─────────────────────────────────────────────────────────────────────┐ +│ Model Files │ +│ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────────┐ │ +│ │ users │ │projects │ │ assets │ │ roles │ │ tour_pages │ │ +│ └────┬────┘ └────┬────┘ └────┬────┘ └────┬────┘ └──────┬──────┘ │ +│ │ │ │ │ │ │ +│ └───────────┴───────────┴───────────┴─────────────┘ │ +│ │ │ +└───────────────────────────────┼──────────────────────────────────────┘ + │ + 3. Call model.associate(db) for each model + ▼ +┌─────────────────────────────────────────────────────────────────────┐ +│ Associations Created │ +│ │ +│ users ─────────────┬─── belongsTo ───→ roles (app_role) │ +│ ├─── hasMany ─────→ project_memberships │ +│ ├─── hasMany ─────→ production_presentation_access│ +│ ├─── hasMany ─────→ publish_events │ +│ └─── belongsToMany → permissions (custom) │ +│ │ +│ projects ──────────┬─── hasMany ─────→ tour_pages │ +│ ├─── hasMany ─────→ assets │ +│ ├─── hasMany ─────→ project_memberships │ +│ ├─── hasMany ─────→ production_presentation_access│ +│ └─── hasMany ─────→ publish_events │ +│ │ +│ roles ─────────────┬─── belongsToMany → permissions │ +│ └─── hasMany ─────→ users │ +└─────────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────────┐ +│ Exported db Object │ +│ { │ +│ users, projects, tour_pages, assets, roles, permissions, │ +│ project_memberships, publish_events, pwa_caches, access_logs, │ +│ element_type_defaults, project_element_defaults, ... │ +│ sequelize, // Connection instance │ +│ Sequelize // Sequelize library │ +│ } │ +└─────────────────────────────────────────────────────────────────────┘ +``` + +--- + +## Model Loader + +**Locations:** +- `backend/src/db/models/loader.ts` +- `backend/src/db/models/index.ts` +- `backend/src/types/db-models.ts` + +`loader.ts` initializes Sequelize, explicitly imports every model factory, and +calls `associate()` for models that define associations. `index.ts` re-exports +the typed registry, and `src/types/db-models.ts` provides the overload surface +for service-specific model calls. + +--- + +## Database Configuration + +**Location:** `backend/src/db/db-config.ts` + +| Environment | Database | Logging | Notes | +|-------------|----------|---------|-------| +| `production` | From env vars | Disabled | Live production | +| `development` | `db_tour_builder_platform` | Console | Local dev | +| `dev_stage` | From env vars | Console | Staging server | + +--- + +## Model Definitions + +### Common Model Options + +All models share these Sequelize options: + +```javascript +{ + timestamps: true, // Adds createdAt, updatedAt + paranoid: true, // Soft delete with deletedAt + freezeTableName: true, // Use exact model name as table name +} +``` + +### Common Fields + +| Field | Type | Description | +|-------|------|-------------| +| `id` | `UUID` | Primary key (auto-generated UUIDv4) | +| `importHash` | `STRING(255)` | Unique hash for bulk import deduplication | +| `createdAt` | `DATE` | Auto-managed creation timestamp | +| `updatedAt` | `DATE` | Auto-managed update timestamp | +| `deletedAt` | `DATE` | Soft delete timestamp (paranoid mode) | + +### Common Associations + +Most models have these standard associations: + +```javascript +// Audit trail associations +db.MODEL.belongsTo(db.users, { as: 'createdBy' }); +db.MODEL.belongsTo(db.users, { as: 'updatedBy' }); +``` + +--- + +## Core Models + +### users + +**Purpose:** User accounts for authentication and authorization. + +| Field | Type | Nullable | Default | Validation | +|-------|------|----------|---------|------------| +| `id` | UUID | No | UUIDv4 | - | +| `firstName` | TEXT | Yes | - | Trimmed | +| `lastName` | TEXT | Yes | - | Trimmed | +| `phoneNumber` | TEXT | Yes | - | - | +| `email` | TEXT | No | - | isEmail, notEmpty, unique | +| `password` | TEXT | No | - | Hashed with bcrypt | +| `disabled` | BOOLEAN | No | false | - | +| `emailVerified` | BOOLEAN | No | false | - | +| `emailVerificationToken` | TEXT | Yes | - | - | +| `emailVerificationTokenExpiresAt` | DATE | Yes | - | - | +| `passwordResetToken` | TEXT | Yes | - | - | +| `passwordResetTokenExpiresAt` | DATE | Yes | - | - | +| `provider` | TEXT | No | 'local' | OAuth provider | +| `app_roleId` | UUID | Yes | - | FK to roles | + +**Indexes:** +- `email` (unique) +- `app_roleId` +- `deletedAt` + +**Associations:** +```javascript +users.belongsTo(roles, { as: 'app_role' }); +users.belongsToMany(permissions, { as: 'custom_permissions', through: 'usersCustom_permissionsPermissions' }); +users.hasMany(project_memberships, { as: 'project_memberships_user' }); +users.hasMany(presigned_url_requests, { as: 'presigned_url_requests_user' }); +users.hasMany(publish_events, { as: 'publish_events_user' }); +users.hasMany(access_logs, { as: 'access_logs_user' }); +users.hasMany(file, { as: 'avatar', scope: { belongsTo: 'users', belongsToColumn: 'avatar' } }); +``` + +**Hooks:** +```javascript +users.beforeCreate((user) => { + // Trim string fields + // Auto-verify email for OAuth providers + // Generate random password for OAuth if not provided +}); + +users.beforeUpdate((user) => { + // Trim string fields +}); +``` + +--- + +### projects + +**Purpose:** Virtual tour projects container. + +| Field | Type | Nullable | Default | Validation | +|-------|------|----------|---------|------------| +| `id` | UUID | No | UUIDv4 | - | +| `name` | TEXT | No | - | notEmpty, len[1,255] | +| `slug` | TEXT | No | - | notEmpty, unique, alphanumeric + dashes/underscores | +| `description` | TEXT | Yes | - | - | +| `logo_url` | TEXT | Yes | - | - | +| `favicon_url` | TEXT | Yes | - | - | +| `og_image_url` | TEXT | Yes | - | - | +| `production_presentation_visibility` | ENUM | No | public | public, private | + +**Indexes:** +- `slug` (unique) +- `deletedAt` + +**Associations:** +```javascript +projects.hasMany(project_memberships, { as: 'project_memberships_project', onDelete: 'CASCADE' }); +projects.hasMany(assets, { as: 'assets_project', onDelete: 'CASCADE' }); +projects.hasMany(presigned_url_requests, { as: 'presigned_url_requests_project', onDelete: 'CASCADE' }); +projects.hasMany(tour_pages, { as: 'tour_pages_project', onDelete: 'CASCADE' }); +projects.hasMany(project_audio_tracks, { as: 'project_audio_tracks_project', onDelete: 'CASCADE' }); +projects.hasMany(project_transition_settings, { as: 'project_transition_settings_project', onDelete: 'CASCADE' }); +projects.hasMany(publish_events, { as: 'publish_events_project', onDelete: 'CASCADE' }); +projects.hasMany(pwa_caches, { as: 'pwa_caches_project', onDelete: 'CASCADE' }); +projects.hasMany(access_logs, { as: 'access_logs_project', onDelete: 'CASCADE' }); +projects.hasMany(project_element_defaults, { as: 'project_element_defaults_project', onDelete: 'CASCADE' }); +projects.hasMany(production_presentation_access, { as: 'production_presentation_access_project', onDelete: 'CASCADE' }); +``` + +--- + +### production_presentation_access + +**Purpose:** Grants Public-role customer users access to selected private +production presentations. + +| Field | Type | Nullable | Default | Validation | +|-------|------|----------|---------|------------| +| `id` | UUID | No | UUIDv4 | - | +| `projectId` | UUID | No | - | FK to projects | +| `userId` | UUID | No | - | FK to users | +| `createdById` | UUID | Yes | - | FK to users | +| `updatedById` | UUID | Yes | - | FK to users | +| `importHash` | STRING(255) | Yes | - | unique | + +**Indexes:** +- `projectId` +- `userId` +- `projectId, userId` unique for active rows + +**Associations:** +```javascript +production_presentation_access.belongsTo(projects, { as: 'project', onDelete: 'CASCADE' }); +production_presentation_access.belongsTo(users, { as: 'user', onDelete: 'CASCADE' }); +production_presentation_access.belongsTo(users, { as: 'createdBy', onDelete: 'SET NULL' }); +production_presentation_access.belongsTo(users, { as: 'updatedBy', onDelete: 'SET NULL' }); +``` + +--- + +### tour_pages + +**Purpose:** Individual pages within a tour with UI elements schema. + +| Field | Type | Nullable | Default | Validation | +|-------|------|----------|---------|------------| +| `id` | UUID | No | UUIDv4 | - | +| `environment` | ENUM | No | 'dev' | dev, stage, production | +| `source_key` | TEXT | Yes | - | Original page ID for cloning | +| `name` | TEXT | No | - | notEmpty, len[1,255] | +| `slug` | TEXT | No | - | notEmpty, alphanumeric + dashes | +| `sort_order` | INTEGER | No | 0 | - | +| `background_image_url` | TEXT | Yes | - | - | +| `background_video_url` | TEXT | Yes | - | - | +| `background_audio_url` | TEXT | Yes | - | - | +| `background_loop` | BOOLEAN | No | false | - | +| `requires_auth` | BOOLEAN | No | false | - | +| `ui_schema_json` | JSON | Yes | - | Page elements, links, transitions | +| `projectId` | UUID | Yes | - | FK to projects | + +**Indexes:** +- `projectId` +- `[projectId, environment, slug]` (unique) - Composite unique per project+environment +- `[projectId, environment, sort_order]` - For ordering queries +- `deletedAt` + +**Note:** The `ui_schema_json` field stores all page elements (buttons, hotspots, galleries, tooltips, media players), navigation links, and transition configurations. + +--- + +### roles + +**Purpose:** RBAC role definitions. + +| Field | Type | Nullable | Default | Validation | +|-------|------|----------|---------|------------| +| `id` | UUID | No | UUIDv4 | - | +| `name` | TEXT | No | - | notEmpty, len[1,100] | +| `role_customization` | TEXT | Yes | - | Custom role metadata | + +**Associations:** +```javascript +roles.belongsToMany(permissions, { as: 'permissions', through: 'rolesPermissionsPermissions' }); +roles.hasMany(users, { as: 'users_app_role', onDelete: 'SET NULL' }); +``` + +--- + +### permissions + +**Purpose:** Individual permission definitions. + +| Field | Type | Nullable | Default | Validation | +|-------|------|----------|---------|------------| +| `id` | UUID | No | UUIDv4 | - | +| `name` | TEXT | No | - | notEmpty, unique, len[1,100] | + +**Permission Naming Convention:** `{ACTION}_{ENTITY}` (e.g., `READ_USERS`, `CREATE_ASSETS`) + +--- + +## Asset Models + +### assets + +**Purpose:** Uploaded media files (images, videos, audio, documents). + +| Field | Type | Nullable | Default | Validation | +|-------|------|----------|---------|------------| +| `id` | UUID | No | UUIDv4 | - | +| `name` | TEXT | Yes | - | len[0,255] | +| `asset_type` | ENUM | No | - | image, video, audio, file | +| `type` | ENUM | No | 'general' | icon, background_image, audio, video, transition, logo, favicon, document, general | +| `cdn_url` | TEXT | Yes | - | - | +| `storage_key` | TEXT | Yes | - | S3/storage path | +| `mime_type` | TEXT | Yes | - | MIME type format | +| `size_mb` | DECIMAL | Yes | - | - | +| `width_px` | INTEGER | Yes | - | Image/video width | +| `height_px` | INTEGER | Yes | - | Image/video height | +| `duration_sec` | DECIMAL | Yes | - | Audio/video duration | +| `checksum` | TEXT | Yes | - | File hash | +| `is_public` | BOOLEAN | No | false | - | +| `projectId` | UUID | Yes | - | FK to projects | + +**Indexes:** +- `projectId` +- `asset_type` +- `type` +- `is_public` +- `deletedAt` + +**Associations:** +```javascript +assets.hasMany(asset_variants, { as: 'asset_variants_asset', onDelete: 'CASCADE' }); +assets.belongsTo(projects, { as: 'project', onDelete: 'CASCADE' }); +``` + +--- + +### asset_variants + +**Purpose:** Optimized versions of assets (thumbnails, different formats). + +| Field | Type | Nullable | Default | Validation | +|-------|------|----------|---------|------------| +| `id` | UUID | No | UUIDv4 | - | +| `variant_type` | ENUM | Yes | - | thumbnail, preview, webp, mp4_low, mp4_high, original | +| `cdn_url` | TEXT | Yes | - | len[0,2048], URL format | +| `width_px` | INTEGER | Yes | - | min: 0 | +| `height_px` | INTEGER | Yes | - | min: 0 | +| `size_mb` | DECIMAL | Yes | - | min: 0 | +| `assetId` | UUID | Yes | - | FK to assets | + +**Associations:** +```javascript +asset_variants.belongsTo(assets, { as: 'asset', onDelete: 'CASCADE' }); +``` + +--- + +## Element Defaults Models + +### element_type_defaults + +**Purpose:** Global platform-wide default settings for UI element types. + +| Field | Type | Nullable | Default | Validation | +|-------|------|----------|---------|------------| +| `id` | UUID | No | UUIDv4 | - | +| `element_type` | TEXT | No | - | notEmpty, unique, len[1,100] | +| `name` | TEXT | No | - | notEmpty, len[1,255] | +| `sort_order` | INTEGER | No | 0 | - | +| `is_active` | VIRTUAL | - | true | Always returns true | +| `default_settings_json` | TEXT | Yes | - | Mapped from `settings_json` column | + +**Indexes:** +- `element_type` +- `sort_order` +- `deletedAt` + +**Associations:** +```javascript +element_type_defaults.hasMany(project_element_defaults, { + as: 'project_defaults', + foreignKey: 'source_element_id', + onDelete: 'SET NULL' +}); +``` + +**Element Types:** button, hotspot, gallery, tooltip, video_player, audio_player, image, text, link, form, iframe + +--- + +### project_element_defaults + +**Purpose:** Project-specific overrides for element defaults. + +| Field | Type | Nullable | Default | Validation | +|-------|------|----------|---------|------------| +| `id` | UUID | No | UUIDv4 | - | +| `element_type` | TEXT | No | - | notEmpty, len[1,100] | +| `name` | TEXT | Yes | - | len[0,255] | +| `sort_order` | INTEGER | No | 0 | - | +| `settings_json` | TEXT | Yes | - | Element configuration | +| `source_element_id` | UUID | Yes | - | FK to element_type_defaults | +| `snapshot_version` | INTEGER | No | 1 | Version tracking | +| `projectId` | UUID | No | - | FK to projects | + +**Indexes:** +- `projectId` +- `[projectId, element_type]` (unique) +- `element_type` +- `source_element_id` +- `deletedAt` + +**Associations:** +```javascript +project_element_defaults.belongsTo(projects, { as: 'project', onDelete: 'CASCADE' }); +project_element_defaults.belongsTo(element_type_defaults, { + as: 'source_element', + onDelete: 'SET NULL' +}); +``` + +--- + +## Publishing & Audit Models + +### publish_events + +**Purpose:** Track publishing actions between environments. + +| Field | Type | Nullable | Default | Validation | +|-------|------|----------|---------|------------| +| `id` | UUID | No | UUIDv4 | - | +| `title` | STRING | Yes | - | len[0,255] | +| `description` | TEXT | Yes | - | len[0,5000] | +| `from_environment` | ENUM | No | - | dev, stage, production | +| `to_environment` | ENUM | No | - | dev, stage, production | +| `started_at` | DATE | Yes | - | - | +| `finished_at` | DATE | Yes | - | - | +| `status` | ENUM | No | 'queued' | queued, running, success, failed | +| `error_message` | TEXT | Yes | - | - | +| `pages_copied` | INTEGER | Yes | - | min: 0 | +| `transitions_copied` | INTEGER | Yes | - | min: 0 | +| `audios_copied` | INTEGER | Yes | - | min: 0 | +| `projectId` | UUID | Yes | - | FK to projects | +| `userId` | UUID | Yes | - | FK to users | + +**Indexes:** +- `projectId` +- `userId` +- `status` +- `started_at` + +--- + +### access_logs + +**Purpose:** Audit trail for user activity. + +| Field | Type | Nullable | Default | Validation | +|-------|------|----------|---------|------------| +| `id` | UUID | No | UUIDv4 | - | +| `environment` | ENUM | No | - | admin, stage, production | +| `path` | TEXT | Yes | - | len[0,2048] | +| `ip_address` | TEXT | Yes | - | len[0,45] (IPv6 max) | +| `user_agent` | TEXT | Yes | - | len[0,1024] | +| `accessed_at` | DATE | No | NOW | - | +| `projectId` | UUID | Yes | - | FK to projects | +| `userId` | UUID | Yes | - | FK to users | + +**Indexes:** +- `projectId` +- `environment` +- `userId` +- `accessed_at` + +--- + +## Supporting Models + +### project_memberships + +**Purpose:** User access to projects with role-based permissions. + +| Field | Type | Nullable | Default | Validation | +|-------|------|----------|---------|------------| +| `id` | UUID | No | UUIDv4 | - | +| `access_level` | ENUM | No | 'viewer' | owner, editor, reviewer, viewer | +| `is_active` | BOOLEAN | No | false | - | +| `invited_at` | DATE | Yes | - | - | +| `accepted_at` | DATE | Yes | - | - | +| `projectId` | UUID | Yes | - | FK to projects | +| `userId` | UUID | Yes | - | FK to users | + +**Indexes:** +- `projectId` +- `userId` +- `[projectId, userId]` (unique) - One membership per user per project +- `is_active` +- `deletedAt` + +--- + +### project_audio_tracks + +**Purpose:** Background audio tracks for projects. + +| Field | Type | Nullable | Default | Validation | +|-------|------|----------|---------|------------| +| `id` | UUID | No | UUIDv4 | - | +| `environment` | ENUM | Yes | - | dev, stage, production | +| `source_key` | TEXT | Yes | - | Original track ID for cloning | +| `name` | TEXT | Yes | - | len[0,255] | +| `slug` | TEXT | Yes | - | - | +| `url` | TEXT | Yes | - | - | +| `loop` | BOOLEAN | No | false | - | +| `volume` | DECIMAL | Yes | - | min: 0, max: 1 | +| `sort_order` | INTEGER | Yes | - | - | +| `is_enabled` | BOOLEAN | No | false | - | +| `projectId` | UUID | Yes | - | FK to projects | + +--- + +### project_transition_settings + +**Purpose:** Environment-aware CSS transition settings for page navigation. + +| Field | Type | Nullable | Default | Validation | +|-------|------|----------|---------|------------| +| `id` | UUID | No | UUIDv4 | - | +| `environment` | ENUM | No | - | dev, stage, production | +| `source_key` | TEXT | Yes | - | Original settings ID for cloning | +| `transition_type` | TEXT | No | 'fade' | CSS transition type | +| `duration_ms` | INTEGER | No | 700 | Transition duration in ms | +| `easing` | TEXT | No | 'ease-in-out' | CSS easing function | +| `overlay_color` | TEXT | No | '#000000' | Transition overlay color | +| `projectId` | UUID | No | - | FK to projects | +| `createdById` | UUID | Yes | - | FK to users | +| `updatedById` | UUID | Yes | - | FK to users | + +**Indexes:** +- `[projectId, environment]` (unique where deletedAt IS NULL) +- `projectId` +- `deletedAt` + +**Associations:** +```javascript +project_transition_settings.belongsTo(projects, { as: 'project', onDelete: 'CASCADE' }); +project_transition_settings.belongsTo(users, { as: 'createdBy' }); +project_transition_settings.belongsTo(users, { as: 'updatedBy' }); +``` + +**Publishing Integration:** Copied between environments during Save to Stage (dev → stage) and Publish (stage → production). Uses `source_key` to track lineage. + +--- + +### pwa_caches + +**Purpose:** PWA offline cache manifest tracking. + +| Field | Type | Nullable | Default | Validation | +|-------|------|----------|---------|------------| +| `id` | UUID | No | UUIDv4 | - | +| `environment` | ENUM | Yes | - | dev, stage, production | +| `cache_version` | TEXT | Yes | - | len[0,255] | +| `manifest_json` | JSON | Yes | - | PWA manifest | +| `asset_list_json` | JSON | Yes | - | Cached asset URLs | +| `generated_at` | DATE | Yes | - | - | +| `is_active` | BOOLEAN | No | false | - | +| `projectId` | UUID | Yes | - | FK to projects | + +--- + +### presigned_url_requests + +**Purpose:** Audit log for S3 presigned URL requests. + +| Field | Type | Nullable | Default | Validation | +|-------|------|----------|---------|------------| +| `id` | UUID | No | UUIDv4 | - | +| `purpose` | ENUM | Yes | - | upload, download | +| `asset_type` | ENUM | Yes | - | image, video, audio, file | +| `requested_key` | TEXT | Yes | - | len[0,1024] | +| `mime_type` | TEXT | Yes | - | MIME format, len[0,255] | +| `requested_size_mb` | DECIMAL | Yes | - | min: 0 | +| `expires_at` | DATE | Yes | - | - | +| `status` | TEXT | Yes | - | - | +| `projectId` | UUID | Yes | - | FK to projects | +| `userId` | UUID | Yes | - | FK to users | + +--- + +### file + +**Purpose:** Generic file attachments (user avatars, etc.). + +| Field | Type | Nullable | Default | Validation | +|-------|------|----------|---------|------------| +| `id` | UUID | No | UUIDv4 | - | +| `belongsTo` | STRING(255) | Yes | - | Parent table name | +| `belongsToId` | UUID | Yes | - | Parent record ID | +| `belongsToColumn` | STRING(255) | Yes | - | Parent column name | +| `name` | STRING(2083) | No | - | notEmpty | +| `sizeInBytes` | INTEGER | Yes | - | - | +| `privateUrl` | STRING(2083) | Yes | - | - | +| `publicUrl` | STRING(2083) | No | - | notEmpty | + +**Usage Pattern:** Polymorphic association via `belongsTo`, `belongsToId`, `belongsToColumn` fields and scoped `hasMany` on parent models: + +```javascript +// In users model: +db.users.hasMany(db.file, { + as: 'avatar', + foreignKey: 'belongsToId', + scope: { + belongsTo: 'users', + belongsToColumn: 'avatar', + }, +}); +``` + +--- + +## Entity Relationship Diagram + +``` +┌─────────────────────────────────────────────────────────────────────────────┐ +│ Entity Relationships │ +└─────────────────────────────────────────────────────────────────────────────┘ + + ┌─────────────┐ + │ roles │ + │ │ + │ name │ + │ permissions │◄─── M:N ───┐ + └──────┬──────┘ │ + │ │ + │ 1:N │ + │ │ +┌─────────────────────────┐ ┌──────▼──────┐ ┌───────┴───────┐ +│ element_type_defaults │ │ users │ │ permissions │ +│ │ │ │ │ │ +│ element_type │ │ email │ │ name │ +│ name │ │ password │ │ │ +│ settings_json │ │ app_role ───┼───┘ │ +└───────────┬─────────────┘ └──────┬──────┘ │ + │ │ │ + │ 1:N │ 1:N │ + │ │ │ + ▼ ▼ │ +┌───────────────────────┐ ┌────────────────────┐ │ +│project_element_defaults│ │project_memberships │ │ +│ │ │ │ │ +│ element_type │ │ access_level │ │ +│ settings_json │ │ is_active │ │ +│ source_element_id ────┼───┘ │ │ +│ projectId ────────────┼───┐ │ │ +└───────────────────────┘ │ │ │ + │ │ │ + │ N:1 │ N:1 │ + │ │ │ + ▼ ▼ │ + ┌─────────────────────────────────────┐ │ + │ projects │ │ + │ │ │ + │ name, slug, description │ │ + │ logo_url, favicon_url, og_image_url │ │ + └───────────────┬─────────────────────┘ │ + │ │ + ┌─────────────┬───────────────┼───────────────┬─────────┘ + │ │ │ │ + │ 1:N │ 1:N │ 1:N │ 1:N + ▼ ▼ ▼ ▼ +┌───────────────┐ ┌───────────┐ ┌───────────────┐ ┌───────────────┐ +│ tour_pages │ │ assets │ │publish_events │ │ pwa_caches │ +│ │ │ │ │ │ │ │ +│ environment │ │ asset_type│ │ status │ │ environment │ +│ name, slug │ │ cdn_url │ │ from/to_env │ │ manifest_json │ +│ ui_schema_json│ │ storage_key│ │ pages_copied │ │ asset_list │ +└───────────────┘ └─────┬─────┘ └───────────────┘ └───────────────┘ + │ + │ 1:N + ▼ + ┌─────────────────┐ + │ asset_variants │ + │ │ + │ variant_type │ + │ cdn_url │ + │ width_px │ + └─────────────────┘ + + +Additional Models (project-scoped): + • project_audio_tracks → projects (N:1) + • project_transition_settings → projects (N:1) + • production_presentation_access → projects (N:1), users (N:1) + • access_logs → projects (N:1), users (N:1) + • presigned_url_requests → projects (N:1), users (N:1) +``` + +--- + +## Model Patterns + +### 1. Soft Delete (Paranoid Mode) + +All models use `paranoid: true`, which adds a `deletedAt` column and modifies queries: + +```javascript +// This: +await Model.destroy({ where: { id } }); + +// Does NOT delete, but sets deletedAt = NOW() +// All find queries automatically filter deletedAt IS NULL +``` + +### 2. Environment Enum + +Several models use the environment enum for multi-environment content: + +```javascript +environment: { + type: DataTypes.ENUM, + values: ['dev', 'stage', 'production'], +} +``` + +**Used by:** tour_pages, project_audio_tracks, project_transition_settings, pwa_caches, publish_events + +### 3. Import Hash Deduplication + +The `importHash` field prevents duplicate imports: + +```javascript +importHash: { + type: DataTypes.STRING(255), + allowNull: true, + unique: true, +} +``` + +### 4. Cascade Delete Patterns + +| Relationship | onDelete | Use Case | +|--------------|----------|----------| +| `CASCADE` | Delete children when parent deleted | Projects → tour_pages | +| `SET NULL` | Keep children, null the FK | Roles → users | + +### 5. Composite Unique Constraints + +For scoped uniqueness: + +```javascript +indexes: [ + { fields: ['projectId', 'environment', 'slug'], unique: true }, +] +``` + +### 6. JSON Fields + +Complex configurations stored as JSON: + +```javascript +ui_schema_json: { type: DataTypes.JSON } // Parsed JSON column +settings_json: { type: DataTypes.TEXT } // Stringified JSON text +``` + +### 7. Virtual Fields + +Computed fields that don't exist in the database: + +```javascript +is_active: { + type: DataTypes.VIRTUAL, + get() { + return true; + }, +} +``` + +--- + +## Validation Patterns + +### String Length + +```javascript +name: { + type: DataTypes.TEXT, + validate: { + len: { + args: [1, 255], + msg: 'Name must be between 1 and 255 characters', + }, + }, +} +``` + +### Required Fields + +```javascript +email: { + type: DataTypes.TEXT, + allowNull: false, + validate: { + notEmpty: { msg: 'Email is required' }, + }, +} +``` + +### Email Format + +```javascript +email: { + validate: { + isEmail: { msg: 'Must be a valid email address' }, + }, +} +``` + +### Custom Validators + +```javascript +cdn_url: { + validate: { + isUrlOrEmpty(value) { + if (value && value.length > 0 && !/^https?:\/\/.+/.test(value)) { + throw new Error('CDN URL must be a valid URL'); + } + }, + }, +} +``` + +### Numeric Range + +```javascript +volume: { + type: DataTypes.DECIMAL, + validate: { + min: { args: [0], msg: 'Volume must be at least 0' }, + max: { args: [1], msg: 'Volume must be at most 1' }, + }, +} +``` + +--- + +## Usage Examples + +### Importing Models + +```javascript +const db = require('./db/models'); + +// Access models +const user = await db.users.findByPk(id); +const project = await db.projects.findOne({ where: { slug } }); + +// Access Sequelize instance +const transaction = await db.sequelize.transaction(); + +// Access Sequelize operators +const { Op } = db.Sequelize; +const users = await db.users.findAll({ + where: { email: { [Op.like]: '%@example.com' } } +}); +``` + +### Creating Records with Associations + +```javascript +const project = await db.projects.create({ + name: 'My Tour', + slug: 'my-tour', + createdById: userId, +}); + +const page = await db.tour_pages.create({ + name: 'Home', + slug: 'home', + projectId: project.id, + environment: 'dev', + createdById: userId, +}); +``` + +### Querying with Includes + +```javascript +const project = await db.projects.findOne({ + where: { id: projectId }, + include: [ + { association: 'tour_pages_project', where: { environment: 'production' } }, + { association: 'project_memberships_project', include: ['user'] }, + ], +}); +``` + +--- + +## Related Documentation + +- [Database Schema](../database-schema.md) - Complete schema reference +- [Factories Module](./factories.md) - Service and router factories that use models +- [Services Module](./services.md) - Business logic using models +- [API Endpoints](../api-endpoints.md) - REST API exposing models diff --git a/backend/docs/modules/db-seeders.md b/backend/docs/modules/db-seeders.md new file mode 100644 index 0000000..2d2b042 --- /dev/null +++ b/backend/docs/modules/db-seeders.md @@ -0,0 +1,575 @@ +# DB Seeders Module + +## Overview + +The DB Seeders module provides initial data population for the database using +the typed **Umzug v3** runner in `backend/src/db/umzug.ts`. Seeders create +essential system data (users, roles, permissions) and optional sample data for +development and testing. + +**Location:** `backend/src/db/seeders/` + +**Files:** 3 TypeScript ESM seeder entries. There are no JavaScript seeder +bridges after the backend ESM migration. + +--- + +## Architecture + +``` +backend/src/db/seeders/ +├── 20200430130759-admin-user.ts # Initial admin users +├── 20200430130760-user-roles.ts # RBAC roles + permissions +└── 20231127130745-sample-data.ts # Sample data, opt-in +``` + +**Execution Order:** Seeders run in timestamp order (oldest first). + +--- + +## Configuration + +### NPM Scripts +```bash +# Run pending seeders +npm run db:seed + +# Undo all seeders +npm run db:seed:undo + +# Full database reset (includes seeding) +npm run db:reset +``` + +### Server Startup +Seeders run automatically via `npm start`: +```json +{ + "scripts": { + "start": "npm run db:migrate && npm run db:seed && npm run watch" + } +} +``` + +--- + +## Seeder Files + +### 1. Admin User Seeder (`20200430130759-admin-user.ts`) + +**Purpose:** Create initial system users. + +The implementation uses reusable contracts from +`backend/src/types/db-seeders.ts`. Umzug executes the TypeScript source in +development and the compiled JavaScript file from `dist/` in production builds. + +**Users Created:** + +| User | Email | Role Assignment | +|------|-------|-----------------| +| Admin | `config.admin_email` | Administrator | +| John | john@doe.com | Account Manager | +| Client | client@hello.com | Platform Owner | + +**Key Features:** +- Uses bcrypt for password hashing with configured salt rounds +- Hardcoded UUIDs for consistent user IDs +- Reads credentials from `config.ts` (environment variables) +- Idempotent on repeated `db:seed`: existing seed user IDs are selected first, + and only missing rows are inserted. This prevents duplicate-key failures when + `SequelizeData` lacks the seeder entry but users already exist. + +```typescript +const existingRows = + await queryInterface.sequelize.query( + 'SELECT "id" FROM "users" WHERE "id" IN (:ids)', + { + replacements: { ids: seedUserIds }, + type: QueryTypes.SELECT, + }, + ); + +const existingIds = new Set(existingRows.map((row) => row.id)); +const rowsToInsert = createAdminUserRows().filter( + (row) => !existingIds.has(row.id), +); + +if (rowsToInsert.length > 0) { + await queryInterface.bulkInsert('users', rowsToInsert); +} +``` + +--- + +### 2. User Roles Seeder (`20200430130760-user-roles.ts`) + +**Purpose:** Create the complete RBAC (Role-Based Access Control) system. + +The implementation uses reusable contracts from +`backend/src/types/db-seeders.ts`. Umzug stores stable seeder execution names +in `SequelizeData` so repeated startup seeding does not rerun completed seed +data. + +**Data Created:** + +| Data Type | Count | Description | +|-----------|-------|-------------| +| Roles | 7 | User role definitions | +| Permissions | 54 | CRUD permissions for 13 entities + special | +| Role-Permission Links | 200+ | M:N relationships | +| Join Table | 1 | `rolesPermissionsPermissions` table | + +**Key Features:** +- Uses stable named role and permission definitions, then reuses existing DB IDs + when the same role/permission name is already present. +- Inserts only missing roles, permissions, and role-permission links. This keeps + repeated `db:seed` safe. +- Keeps the existing startup behavior that assigns system users to seeded roles + by email after RBAC data exists. + +#### Roles +```javascript +const roles = [ + 'Administrator', // Full system access + 'PlatformOwner', // Full project/content access + 'AccountManager', // User and project management + 'TourDesigner', // Content creation and editing + 'ContentReviewer', // Read + limited update access + 'AnalyticsViewer', // Read-only access + 'Public', // Public/unauthenticated access +]; +``` + +#### Permission Generation Pattern +```javascript +// Generates CREATE, READ, UPDATE, DELETE permissions per entity +function createPermissions(name) { + return [ + { name: `CREATE_${name.toUpperCase()}` }, + { name: `READ_${name.toUpperCase()}` }, + { name: `UPDATE_${name.toUpperCase()}` }, + { name: `DELETE_${name.toUpperCase()}` }, + ]; +} + +const entities = [ + 'users', 'roles', 'permissions', 'projects', 'project_memberships', + 'assets', 'asset_variants', 'presigned_url_requests', 'tour_pages', + 'project_audio_tracks', 'publish_events', 'pwa_caches', 'access_logs' +]; + +// Creates 52 permissions (13 entities × 4 CRUD operations) +await queryInterface.bulkInsert('permissions', entities.flatMap(createPermissions)); + +// Plus special permissions +await queryInterface.bulkInsert('permissions', [ + { name: 'READ_API_DOCS' }, + { name: 'CREATE_SEARCH' }, +]); +``` + +#### ID Map Pattern +```javascript +// Consistent UUID generation using key-based map +const idMap = new Map(); + +function getId(key) { + if (idMap.has(key)) { + return idMap.get(key); + } + const id = uuid(); + idMap.set(key, id); + return id; +} + +// Usage - same key always returns same UUID within seeder run +getId('Administrator') // Returns consistent UUID +getId('CREATE_USERS') // Returns consistent UUID +``` + +#### Permission Matrix + +| Role | Users | Projects | Assets | Tour Pages | Access Logs | +|------|-------|----------|--------|------------|-------------| +| **Administrator** | CRUD | CRUD | CRUD | CRUD | CRUD | +| **PlatformOwner** | CRUD | CRUD | CRUD | CRUD | CRUD | +| **AccountManager** | RU | CRU | CRU | CRU | R | +| **TourDesigner** | R | RU | CRU | CRU | R | +| **ContentReviewer** | R | RU | RU | RU | R | +| **AnalyticsViewer** | R | R | R | R | R | +| **Public** | - | - | - | - | - | + +**Legend:** C=Create, R=Read, U=Update, D=Delete + +#### Join Table Creation +```javascript +// Creates M:N relationship table directly in seeder +await queryInterface.sequelize.query(` + CREATE TABLE IF NOT EXISTS "rolesPermissionsPermissions" ( + "createdAt" timestamp with time zone not null, + "updatedAt" timestamp with time zone not null, + "roles_permissionsId" uuid not null, + "permissionId" uuid not null, + primary key ("roles_permissionsId", "permissionId"), + constraint "rolesPermissionsPermissions_roles_permissions_fk" + foreign key ("roles_permissionsId") references "roles"("id") + on delete cascade on update cascade, + constraint "rolesPermissionsPermissions_permission_fk" + foreign key ("permissionId") references "permissions"("id") + on delete cascade on update cascade + ); +`); +``` + +--- + +### 3. Sample Data Seeder (`20231127130745-sample-data.ts`) + +**Purpose:** Create sample data for development and testing. + +The implementation lives in `20231127130745-sample-data.ts`. Demo-only +Sequelize operations use reusable `SampleDataModel` and +`SampleDataAssociationRecord` contracts from +`backend/src/types/db-seeders.ts`; the production service model overloads stay +separate. Umzug records the seeder under its legacy `.js` name for storage +compatibility only. + +**Opt-In Activation:** +```bash +# Enable sample data seeding +export ENABLE_SAMPLE_DATA=true +npm run db:seed +``` + +**Check in Code:** +```typescript +const sampleDataSeeder: SequelizeSeeder = { + async up() { + if (process.env.ENABLE_SAMPLE_DATA !== 'true') return; + // ... seed data + }, +}; +``` + +**Data Created:** + +| Entity | Records | Description | +|--------|---------|-------------| +| Projects | 3 | Sample tour projects | +| Project Memberships | 3 | User-project associations | +| Assets | 3 | Images, videos, audio | +| Asset Variants | 3 | Thumbnail/preview variants | +| Presigned URL Requests | 3 | Upload/download requests | +| Tour Pages | 3 | Sample tour pages | +| Project Audio Tracks | 3 | Background audio | +| Publish Events | 3 | Deployment history | +| PWA Caches | 3 | Offline cache configs | +| Access Logs | 3 | Visitor tracking | + +#### Sample Projects +```javascript +const ProjectsData = [ + { + name: 'Cardiff Arena Tour', + slug: 'cardiff-arena', + description: 'Interactive arena tour for visitors and event planners.', + logo_url: 'https://cdn.platform.com/cardiff/logo.png', + favicon_url: 'https://cdn.platform.com/cardiff/favicon.ico', + og_image_url: 'https://cdn.platform.com/cardiff/og.jpg', + }, + { + name: 'Riverside Park Walkthrough', + slug: 'riverside-park', + description: 'Offline-ready guided walkthrough for the city park.', + // ... + }, + { + name: 'Mall Central Experience', + slug: 'mall-central', + description: 'Retail complex presentation with navigation and galleries.', + // ... + }, +]; +``` + +#### Association Helper Pattern +```javascript +// Associates records after bulk creation using Sequelize model methods +async function associateAssetWithProject() { + const relatedProject = await Projects.findOne({ + offset: Math.floor(Math.random() * (await Projects.count())), + }); + const asset = await Assets.findOne({ + order: [['id', 'ASC']], + offset: 0, + }); + if (asset?.setProject) { + await asset.setProject(relatedProject); + } +} +``` + +--- + +## Seeder Patterns + +### 1. bulkInsert Pattern +**Purpose:** Insert multiple records efficiently. + +```javascript +await queryInterface.bulkInsert('tableName', [ + { + id: uuid(), + field1: 'value1', + createdAt: new Date(), + updatedAt: new Date(), + }, + // ... more records +]); +``` + +### 2. bulkDelete Pattern +**Purpose:** Remove seeded data during rollback. + +```javascript +async down(queryInterface, Sequelize) { + await queryInterface.bulkDelete('users', { + id: { [Sequelize.Op.in]: ids } + }); +} +``` + +### 3. Conditional Execution Pattern +**Purpose:** Enable/disable seeders based on environment. + +Seeder-only environment gates are intentionally allowed to read `process.env` +directly. Runtime app configuration must go through `backend/src/config.ts`, +but seeders and scripts are bootstrap/data-loading entrypoints and are listed +as exceptions in `AGENTS.md`. + +The resilience/config hardening work does not require seeder changes because it +does not introduce database schema, RBAC permissions, seed users, roles, default +records, or sample-data entities. + +```javascript +up: async () => { + if (process.env.ENABLE_SAMPLE_DATA !== 'true') { + return; // Skip seeding + } + // ... proceed with seeding +} +``` + +### 4. ID Consistency Pattern +**Purpose:** Use deterministic IDs for reliable down() migrations. + +```javascript +// Hardcoded IDs for rollback capability +const ids = [ + '193bf4b5-9f07-4bd5-9a43-e7e41f3e96af', + 'af5a87be-8f9c-4630-902a-37a60b7005ba', +]; + +// OR: Map-based consistent generation +const idMap = new Map(); +function getId(key) { + if (!idMap.has(key)) { + idMap.set(key, uuid()); + } + return idMap.get(key); +} +``` + +### 5. Model-Based Association Pattern +**Purpose:** Create relationships using Sequelize models after bulk insert. + +```javascript +// Use model methods for associations (more readable) +const project = await Projects.findOne({ where: { slug: 'test' } }); +const asset = await Assets.findOne({ order: [['id', 'ASC']] }); +await asset.setProject(project); +``` + +### 6. Raw SQL Pattern +**Purpose:** Create structures not managed by Sequelize models. + +```javascript +// Create join table directly via SQL +await queryInterface.sequelize.query(` + CREATE TABLE IF NOT EXISTS "rolesPermissionsPermissions" ( + ... + ); +`); + +// Create indexes +await queryInterface.sequelize.query( + 'CREATE INDEX IF NOT EXISTS "index_name" ON "tableName" ("columnName");' +); +``` + +--- + +## Execution Flow + +``` +npm run db:seed + │ + ▼ +┌────────────────────────────────────┐ +│ 20200430130759-admin-user.ts │ +│ ├─ Typed ESM source │ +│ ├─ Hash passwords │ +│ └─ Insert missing seed users │ +└────────────────────────────────────┘ + │ + ▼ +┌────────────────────────────────────┐ +│ 20200430130760-user-roles.ts │ +│ ├─ Typed ESM source │ +│ ├─ Insert missing roles │ +│ ├─ Insert missing permissions │ +│ ├─ Create join table │ +│ ├─ Insert missing RBAC links │ +│ └─ Update user app_roleId │ +└────────────────────────────────────┘ + │ + ▼ +┌────────────────────────────────────┐ +│ 20231127130745-sample-data.ts │ +│ ├─ Check ENABLE_SAMPLE_DATA │ +│ ├─ Skip if not enabled │ +│ ├─ Insert sample records │ +│ └─ Create associations │ +└────────────────────────────────────┘ +``` + +--- + +## Best Practices + +### 1. Use Consistent IDs +```javascript +// Good - allows rollback +const ids = ['uuid-1', 'uuid-2']; +await queryInterface.bulkInsert('table', records.map((r, i) => ({ id: ids[i], ...r }))); + +// Down migration can target specific IDs +await queryInterface.bulkDelete('table', { id: { [Op.in]: ids } }); +``` + +### 2. Always Include Timestamps +```javascript +{ + field: 'value', + createdAt: new Date(), + updatedAt: new Date(), +} +``` + +### 3. Handle Errors +```javascript +try { + await queryInterface.bulkInsert('users', [...]); +} catch (error) { + console.error('Error during bulkInsert:', error); + throw error; +} +``` + +### 4. Environment-Aware Seeding +```javascript +// Production - only essential data +// Development - include sample data +if (process.env.ENABLE_SAMPLE_DATA !== 'true') { + return; +} +``` + +### 5. Idempotent Where Possible +```javascript +// Use IF NOT EXISTS for table/index creation +await queryInterface.sequelize.query(` + CREATE TABLE IF NOT EXISTS "tableName" (...) +`); + +await queryInterface.sequelize.query(` + CREATE INDEX IF NOT EXISTS "indexName" ON "tableName" (...) +`); +``` + +--- + +## Data Dependencies + +``` +admin-user.ts + │ + │ Creates missing users with stable IDs + ▼ +user-roles.ts + │ + │ Creates/reuses roles and permissions, then references user emails to assign roles + │ UPDATE users SET app_roleId = '...' WHERE email = '...' + ▼ +sample-data.ts + │ + │ Uses Sequelize models to find existing users/projects + │ Creates associations via model methods + ▼ +``` + +--- + +## Running Seeders + +### Development Setup +```bash +cd backend +npm run db:seed +``` + +### With Sample Data +```bash +export ENABLE_SAMPLE_DATA=true +npm run db:seed +``` + +### Fresh Database +```bash +npm run db:reset # drop, create, migrate, seed +``` + +### Undo Seeders +```bash +npm run db:seed:undo # Runs all down() methods in reverse order +``` + +--- + +## Seeder Inventory + +| # | Timestamp | Name | Records | Required | +|---|-----------|------|---------|----------| +| 1 | 20200430130759 | admin-user | 3 users | Yes | +| 2 | 20200430130760 | user-roles | 7 roles, 54 permissions, 200+ links | Yes | +| 3 | 20231127130745 | sample-data | 30+ sample records | No (opt-in) | + +--- + +## Environment Variables + +| Variable | Purpose | Default | +|----------|---------|---------| +| `ENABLE_SAMPLE_DATA` | Enable sample data seeder | `false` | +| `ADMIN_EMAIL` | Admin user email | (from config) | +| `ADMIN_PASS` | Admin user password | (from config) | +| `USER_PASS` | Default user password | (from config) | + +--- + +## Related Documentation + +- [DB Migrations](./db-migrations.md) - Schema evolution +- [DB Models](./db-models.md) - Sequelize model definitions +- [RBAC System](../../../documentation/rbac-system.md) - Role-based access control details +- [Authentication](./auth.md) - User authentication diff --git a/backend/docs/modules/email.md b/backend/docs/modules/email.md new file mode 100644 index 0000000..417b328 --- /dev/null +++ b/backend/docs/modules/email.md @@ -0,0 +1,961 @@ +# Backend Email Module + +## Overview + +The Email module provides transactional email functionality for user authentication flows including email verification, password reset, and user invitations. It uses **Nodemailer** with **AWS SES** (Simple Email Service) as the SMTP transport. + +**Location:** `backend/src/services/email/` + +**Total Files:** 7 + +--- + +## Architecture Diagram + +``` +┌─────────────────────────────────────────────────────────────────────┐ +│ Auth Service │ +│ (services/auth.ts) │ +│ │ +│ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────────┐ │ +│ │ signup() │ │ signin() │ │ sendPasswordReset() │ │ +│ └────────┬────────┘ └─────────────────┘ └──────────┬──────────┘ │ +│ │ │ │ +│ │ sends verification │ sends reset │ +│ ▼ ▼ │ +└───────────┼───────────────────────────────────────────┼─────────────┘ + │ │ + ▼ ▼ +┌─────────────────────────────────────────────────────────────────────┐ +│ Email Module │ +│ (services/email/) │ +│ │ +│ ┌─────────────────────────────────────────────────────────────┐ │ +│ │ EmailSender │ │ +│ │ (index.js) │ │ +│ │ • Nodemailer transport │ │ +│ │ • AWS SES configuration │ │ +│ │ • send() method │ │ +│ └─────────────────────────────────────────────────────────────┘ │ +│ │ │ +│ ┌──────────────────┼──────────────────┐ │ +│ ▼ ▼ ▼ │ +│ ┌────────────────┐ ┌────────────────┐ ┌────────────────┐ │ +│ │PasswordReset │ │ AddressVerif. │ │ Invitation │ │ +│ │ Email │ │ Email │ │ Email │ │ +│ └───────┬────────┘ └───────┬────────┘ └───────┬────────┘ │ +│ │ │ │ │ +│ ▼ ▼ ▼ │ +│ ┌────────────────┐ ┌────────────────┐ ┌────────────────┐ │ +│ │ passwordReset │ │ addressVerif. │ │ invitation │ │ +│ │ Email.html │ │ Email.html │ │ Template.html │ │ +│ └────────────────┘ └────────────────┘ └────────────────┘ │ +└─────────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────────┐ +│ AWS SES (SMTP) │ +│ email-smtp.us-east-1.amazonaws.com:587 │ +└─────────────────────────────────────────────────────────────────────┘ +``` + +--- + +## Directory Structure + +``` +services/email/ +├── index.js # EmailSender class (45 LOC) +├── list/ # Email template classes +│ ├── passwordReset.ts # Password reset email +│ ├── addressVerification.ts # Email verification email +│ └── invitation.ts # User invitation email +└── htmlTemplates/ # HTML email templates + ├── passwordReset/ + │ └── passwordResetEmail.html (52 LOC) + ├── addressVerification/ + │ └── emailAddressVerification.html (52 LOC) + └── invitation/ + └── invitationTemplate.html (55 LOC) +``` + +--- + +## Core Components + +### EmailSender Class (`index.ts`) + +The main email sending service using Nodemailer. + +```typescript +import nodemailer from 'nodemailer'; +import type SMTPTransport from 'nodemailer/lib/smtp-transport'; + +import config from '../../config.ts'; +import type { EmailSendResult, EmailTemplate } from '../../types/index.js'; + +export default class EmailSender { + constructor(private readonly email: EmailTemplate) {} + + async send(): Promise { + const htmlContent = await this.email.html(); + const transporter = nodemailer.createTransport(this.transportConfig); + const mailOptions: SMTPTransport.MailOptions = { + from: this.from, + to: this.email.to, + subject: this.email.subject, + html: htmlContent, + headers: { + 'X-SES-CONFIGURATION-SET': 'flatlogic-app', + }, + }; + + return transporter.sendMail(mailOptions); + } + + static get isConfigured(): boolean { + return Boolean(config.email.auth.pass && config.email.auth.user); + } + + get transportConfig(): SMTPTransport.Options { + return config.email; + } + + get from(): string { + return config.email.from; + } +} +``` + +**Key Methods:** + +| Method | Type | Description | +|--------|------|-------------| +| `constructor(email)` | Instance | Accepts email template object | +| `send()` | Async | Sends email via Nodemailer | +| `isConfigured` | Static getter | Checks if SMTP credentials exist | +| `transportConfig` | Getter | Returns SMTP config | +| `from` | Getter | Returns sender address | + +--- + +## Email Templates + +### Template Interface + +All email templates implement the same interface: + +```typescript +interface EmailTemplate { + to: string; + subject: string; + html(): Promise | string; +} +``` + +### 1. Password Reset Email (`list/passwordReset.ts`) + +Sent when user requests password reset. + +```typescript +export default class PasswordResetEmail implements EmailTemplate { + constructor({ to, link }: LinkEmailTemplateOptions) { + this.to = to; + this.link = link; + } + + get subject() { + return getNotification( + 'emails.passwordReset.subject', + getNotification('app.title') + ); + // → "Reset your password for Tour Builder Platform" + } + + async html() { + const template = await fs.readFile(templatePath, 'utf8'); + return template + .replace(/{appTitle}/g, appTitle) + .replace(/{resetUrl}/g, this.link) + .replace(/{accountName}/g, this.to); + } +}; +``` + +**Template Variables:** +- `{appTitle}` - Application name +- `{resetUrl}` - Password reset link +- `{accountName}` - User email address + +### 2. Email Verification (`list/addressVerification.ts`) + +Sent after user registration. + +```typescript +export default class EmailAddressVerificationEmail implements EmailTemplate { + constructor({ to, link }: LinkEmailTemplateOptions) { + this.to = to; + this.link = link; + } + + get subject() { + return getNotification( + 'emails.emailAddressVerification.subject', + getNotification('app.title') + ); + // → "Verify your email for Tour Builder Platform" + } + + async html() { + const template = await fs.readFile(templatePath, 'utf8'); + return template + .replace(/{appTitle}/g, appTitle) + .replace(/{signupUrl}/g, this.link) + .replace(/{to}/g, this.to); + } +}; +``` + +**Template Variables:** +- `{appTitle}` - Application name +- `{signupUrl}` - Email verification link +- `{to}` - User email address + +### 3. User Invitation (`list/invitation.ts`) + +Sent when admin invites new user. + +```typescript +export default class InvitationEmail implements EmailTemplate { + constructor({ to, host }: InvitationEmailTemplateOptions) { + this.to = to; + this.host = host; + } + + get subject() { + return getNotification( + 'emails.invitation.subject', + getNotification('app.title') + ); + // → "You've been invited to Tour Builder Platform" + } + + async html() { + const template = await fs.readFile(templatePath, 'utf8'); + const signupUrl = `${this.host}&invitation=true`; + return template + .replace(/{appTitle}/g, appTitle) + .replace(/{signupUrl}/g, signupUrl) + .replace(/{to}/g, this.to); + } +}; +``` + +**Template Variables:** +- `{appTitle}` - Application name +- `{signupUrl}` - Account setup link with `&invitation=true` +- `{to}` - User email address + +--- + +## HTML Email Templates + +### Template Structure + +All HTML templates follow consistent styling: + +```html + + + + + + + + + +``` + +### Template Comparison + +| Template | Header Text | Call-to-Action | Button Style | +|----------|-------------|----------------|--------------| +| Password Reset | "Reset your password for {appTitle}" | Link | Text link | +| Email Verification | "Verify your email for {appTitle}!" | Link | Text link | +| Invitation | "Welcome to {appTitle}!" | Button | Primary button | + +--- + +## Configuration + +### SMTP Settings (`config.ts`) + +```javascript +email: { + from: 'Tour Builder Platform ', + host: 'email-smtp.us-east-1.amazonaws.com', + port: 587, + auth: { + user: process.env.EMAIL_USER || '', + pass: process.env.EMAIL_PASS, + }, + tls: { + rejectUnauthorized: process.env.EMAIL_TLS_REJECT_UNAUTHORIZED !== 'false', + }, +} +``` + +### Environment Variables + +| Variable | Required | Description | +|----------|----------|-------------| +| `EMAIL_USER` | Yes | SMTP username (AWS SES IAM user) | +| `EMAIL_PASS` | Yes | SMTP password (AWS SES IAM credentials) | +| `EMAIL_TLS_REJECT_UNAUTHORIZED` | No | Set to `'false'` to skip TLS verification | + +### AWS SES Configuration + +The system uses AWS SES in the `us-east-1` region: + +``` +Host: email-smtp.us-east-1.amazonaws.com +Port: 587 (STARTTLS) +Authentication: SMTP credentials from IAM +Configuration Set: flatlogic-app +``` + +--- + +## Integration Points + +### Auth Service Integration + +The Auth service (`services/auth.ts`) is the primary consumer: + +```typescript +import EmailAddressVerificationEmail from './email/list/addressVerification.ts'; +import InvitationEmail from './email/list/invitation.ts'; +const PasswordResetEmail = require('./email/list/passwordReset.ts').default; +const EmailSender = require('./email/index.ts').default; + +class Auth { + // Called during signup + static async sendEmailAddressVerificationEmail(email, host) { + const token = await UsersDBApi.generateEmailVerificationToken(email); + const link = `${host}/verify-email?token=${token}`; + + const emailObj = new EmailAddressVerificationEmail({ to: email, link }); + return new EmailSender(emailObj).send(); + } + + // Called for password reset or invitation + static async sendPasswordResetEmail(email, type = 'register', host) { + const token = await UsersDBApi.generatePasswordResetToken(email); + const link = `${host}/password-reset?token=${token}`; + + const emailObj = type === 'invitation' + ? new InvitationEmail({ to: email, host: link }) + : new PasswordResetEmail({ to: email, link }); + + return new EmailSender(emailObj).send(); + } +} +``` + +### Users Service Integration + +User invitations are sent when creating users: + +```javascript +// services/users.ts +static async create({ data, currentUser, sendInvitationEmails = true, host }) { + // ... create user ... + + if (sendInvitationEmails) { + AuthService.sendPasswordResetEmail(email, 'invitation', host); + } +} + +static async bulkImport(req, res, sendInvitationEmails = true, host) { + // ... import users from CSV ... + + if (!sendInvitationEmails) { + emailsToInvite.forEach((email) => { + AuthService.sendPasswordResetEmail(email, 'invitation', host); + }); + } +} +``` + +### Auth Routes Integration + +Auth routes expose email functionality: + +```javascript +// routes/auth.js + +// Check if email is configured (public) +router.get('/email-configured', (req, res) => { + const payload = EmailSender.isConfigured; + res.status(200).send(payload); +}); + +// Resend verification email (authenticated) +router.put('/send-email-address-verification-email', jwtAuth, async (req, res) => { + await AuthService.sendEmailAddressVerificationEmail(req.currentUser.email); + res.status(200).send(true); +}); + +// Request password reset (public) +router.put('/send-password-reset-email', async (req, res) => { + const host = getRequestHost(req); + await AuthService.sendPasswordResetEmail(req.body.email, 'register', host); + res.status(200).send(true); +}); +``` + +--- + +## Token Management + +### Token Generation + +Tokens are generated in `db/api/users.js`: + +```javascript +static async _generateToken(keyNames, email, options) { + const users = await db.users.findOne({ + where: { email: email.toLowerCase() }, + transaction, + }); + + const token = crypto.randomBytes(20).toString('hex'); + const TOKEN_EXPIRY_MS = 24 * 60 * 60 * 1000; // 24 hours + const tokenExpiresAt = Date.now() + TOKEN_EXPIRY_MS; + + if (users) { + await users.update({ + [keyNames[0]]: token, // emailVerificationToken or passwordResetToken + [keyNames[1]]: tokenExpiresAt, // ...ExpiresAt + updatedById: currentUser.id, + }, { transaction }); + } + + return token; +} + +static async generateEmailVerificationToken(email, options) { + return this._generateToken( + ['emailVerificationToken', 'emailVerificationTokenExpiresAt'], + email, options + ); +} + +static async generatePasswordResetToken(email, options) { + return this._generateToken( + ['passwordResetToken', 'passwordResetTokenExpiresAt'], + email, options + ); +} +``` + +### Token Validation + +```javascript +static async findByPasswordResetToken(token, options) { + return db.users.findOne({ + where: { + passwordResetToken: token, + passwordResetTokenExpiresAt: { + [db.Sequelize.Op.gt]: Date.now(), // Not expired + }, + }, + transaction, + }); +} + +static async findByEmailVerificationToken(token, options) { + return db.users.findOne({ + where: { + emailVerificationToken: token, + emailVerificationTokenExpiresAt: { + [db.Sequelize.Op.gt]: Date.now(), + }, + }, + transaction, + }); +} + +static async markEmailVerified(id, options) { + const user = await db.users.findByPk(id, { transaction }); + await user.update({ + emailVerified: true, + emailVerificationToken: null, + emailVerificationTokenExpiresAt: null, + }, { transaction }); +} +``` + +### Token Properties + +| Token Type | Field | Expiry Field | TTL | +|------------|-------|--------------|-----| +| Email Verification | `emailVerificationToken` | `emailVerificationTokenExpiresAt` | 24 hours | +| Password Reset | `passwordResetToken` | `passwordResetTokenExpiresAt` | 24 hours | + +--- + +## Email Flows + +### 1. User Registration Flow + +``` +┌────────────┐ ┌──────────────┐ ┌─────────────┐ +│ User │────▶│ POST /auth │────▶│ Auth.signup │ +│ Signs Up │ │ /signup │ │ │ +└────────────┘ └──────────────┘ └──────┬──────┘ + │ + ▼ + ┌──────────────────────────────────────────┐ + │ if (EmailSender.isConfigured) { │ + │ await sendEmailAddressVerification(); │ + │ } │ + └───────────────────┬──────────────────────┘ + │ + ┌──────────────────┼──────────────────┐ + ▼ ▼ + ┌────────────────┐ ┌────────────────┐ + │ Generate Token │ │ Skip │ + │ (24hr expiry) │ │ (not configured)│ + └───────┬────────┘ └────────────────┘ + │ + ▼ + ┌────────────────┐ + │ Send Email │ + │ (verification) │ + └───────┬────────┘ + │ + ▼ + ┌────────────────┐ + │ User clicks │ + │ /verify-email │ + │ ?token=xxx │ + └───────┬────────┘ + │ + ▼ + ┌────────────────┐ + │ Auth.verifyEmail│ + │ markEmailVerified│ + └────────────────┘ +``` + +### 2. Password Reset Flow + +``` +┌────────────┐ ┌────────────────────┐ ┌───────────────────────┐ +│ User │────▶│ PUT /auth/send- │────▶│ Auth.sendPassword │ +│ Forgot Pwd │ │ password-reset-email│ │ ResetEmail() │ +└────────────┘ └────────────────────┘ └───────────┬───────────┘ + │ + ▼ + ┌───────────────────────┐ + │ generatePasswordReset │ + │ Token (24hr) │ + └───────────┬───────────┘ + │ + ▼ + ┌───────────────────────┐ + │ Send Email │ + │ (PasswordResetEmail) │ + └───────────┬───────────┘ + │ + ▼ + ┌───────────────────────┐ + │ User clicks │ + │ /password-reset │ + │ ?token=xxx │ + └───────────┬───────────┘ + │ + ▼ + ┌───────────────────────┐ + │ PUT /auth/password- │ + │ reset │ + │ { token, password } │ + └───────────────────────┘ +``` + +### 3. User Invitation Flow + +``` +┌────────────┐ ┌──────────────┐ ┌─────────────────┐ +│ Admin │────▶│ POST /users │────▶│ UsersService │ +│ Creates │ │ { email } │ │ .create() │ +│ User │ └──────────────┘ └────────┬────────┘ +└────────────┘ │ + ▼ + ┌─────────────────────────┐ + │ AuthService.sendPassword│ + │ ResetEmail(email, │ + │ 'invitation', host) │ + └────────────┬────────────┘ + │ + ▼ + ┌─────────────────────────┐ + │ InvitationEmail │ + │ (Welcome to {appTitle}!)│ + └────────────┬────────────┘ + │ + ▼ + ┌─────────────────────────┐ + │ User clicks link │ + │ /password-reset?token= │ + │ xxx&invitation=true │ + └────────────┬────────────┘ + │ + ▼ + ┌─────────────────────────┐ + │ Sets password │ + │ (account activated) │ + └─────────────────────────┘ +``` + +--- + +## Email Behavior Modes + +### Email Configured Mode + +When `EMAIL_USER` and `EMAIL_PASS` are set: +- Email verification required before login +- Password reset emails sent on request +- User invitations include email + +### Email Not Configured Mode + +When credentials are missing: +- `EmailSender.isConfigured` returns `false` +- Users auto-verified on signin: `user.emailVerified = true` +- Password reset/invitation silently skipped +- Frontend adapts UI accordingly + +```javascript +// Auth service adapts behavior +static async signin(email, password) { + // ... + if (!EmailSender.isConfigured) { + user.emailVerified = true; // Auto-verify when email disabled + } + + if (!user.emailVerified) { + throw new ValidationError('auth.userNotVerified'); + } + // ... +} +``` + +--- + +## Notification Catalog + +Email subjects use the notification system (`services/notifications/list.ts`): + +```javascript +emails: { + invitation: { + subject: "You've been invited to {0}", + body: ` +

Hello,

+

You've been invited to {0} set password for your {1} account.

+

{2}

+

Thanks,

+

Your {0} team

+ `, + }, + emailAddressVerification: { + subject: "Verify your email for {0}", + body: ` +

Hello,

+

Follow this link to verify your email address.

+

{0}

+

If you didn't ask to verify this address, you can ignore this email.

+

Thanks,

+

Your {1} team

+ `, + }, + passwordReset: { + subject: "Reset your password for {0}", + body: ` +

Hello,

+

Follow this link to reset your {0} password for your {1} account.

+

{2}

+

If you didn't ask to reset your password, you can ignore this email.

+

Thanks,

+

Your {0} team

+ `, + }, +} +``` + +--- + +## Error Handling + +### Email-Related Errors + +| Error Code | Message | When Thrown | +|------------|---------|-------------| +| `auth.emailAddressVerificationEmail.error` | "Email not recognized" | Token generation fails | +| `auth.emailAddressVerificationEmail.invalidToken` | "Email verification link is invalid or has expired" | Invalid/expired verification token | +| `auth.passwordReset.error` | "Email not recognized" | Password reset token generation fails | +| `auth.passwordReset.invalidToken` | "Password reset link is invalid or has expired" | Invalid/expired reset token | +| `auth.userNotVerified` | "Sorry, your email has not been verified yet" | Login without email verification | + +### Error Flow + +```javascript +static async sendEmailAddressVerificationEmail(email, host) { + let link; + try { + const token = await UsersDBApi.generateEmailVerificationToken(email); + link = `${host}/verify-email?token=${token}`; + } catch (error) { + console.error(error); + throw new ValidationError('auth.emailAddressVerificationEmail.error'); + } + + // Create and send email + const emailObj = new EmailAddressVerificationEmail({ to: email, link }); + return new EmailSender(emailObj).send(); +} +``` + +--- + +## Testing + +### Unit Testing Email Templates + +```javascript +describe('PasswordResetEmail', () => { + it('should generate correct subject', () => { + const email = new PasswordResetEmail({ + to: 'user@example.com', + link: 'https://app.com/reset?token=abc', + }); + expect(email.subject).toBe('Reset your password for Tour Builder Platform'); + }); + + it('should render HTML with placeholders', async () => { + const email = new PasswordResetEmail({ + to: 'user@example.com', + link: 'https://app.com/reset?token=abc', + }); + const html = await email.html(); + + expect(html).toContain('Tour Builder Platform'); + expect(html).toContain('https://app.com/reset?token=abc'); + expect(html).toContain('user@example.com'); + }); +}); +``` + +### Integration Testing Email Sending + +```javascript +describe('EmailSender', () => { + it('should validate required fields', async () => { + const sender = new EmailSender({}); + await expect(sender.send()).rejects.toThrow('email.to is required'); + }); + + it('should send email via nodemailer', async () => { + const mockTransport = { sendMail: jest.fn().mockResolvedValue({}) }; + jest.spyOn(nodemailer, 'createTransport').mockReturnValue(mockTransport); + + const email = new PasswordResetEmail({ + to: 'test@example.com', + link: 'https://example.com', + }); + const sender = new EmailSender(email); + await sender.send(); + + expect(mockTransport.sendMail).toHaveBeenCalledWith( + expect.objectContaining({ + to: 'test@example.com', + subject: expect.stringContaining('Reset your password'), + }) + ); + }); +}); +``` + +### Testing Without SMTP + +Set credentials to empty for local development: + +```bash +EMAIL_USER= +EMAIL_PASS= +``` + +Result: +- `EmailSender.isConfigured` returns `false` +- Users auto-verified on login +- No emails sent + +--- + +## Customization + +### Adding New Email Template + +1. **Create HTML template:** + +```html + + + + + + + + + + +``` + +2. **Create email class:** + +```javascript +// services/email/list/welcome.ts +import { promises as fs } from 'fs'; +import path from 'path'; + +import { getNotification } from '../../notifications/helpers.ts'; +import type { EmailTemplate } from '../../../types/index.js'; + +interface WelcomeEmailOptions { + to: string; + userName: string; +} + +export default class WelcomeEmail implements EmailTemplate { + to: string; + private readonly userName: string; + + constructor({ to, userName }: WelcomeEmailOptions) { + this.to = to; + this.userName = userName; + } + + get subject(): string { + return getNotification('emails.welcome.subject', getNotification('app.title')); + } + + async html(): Promise { + const templatePath = path.resolve( + process.cwd(), + 'src/services/email/htmlTemplates/welcome/welcomeEmail.html', + ); + const template = await fs.readFile(templatePath, 'utf8'); + const appTitle = getNotification('app.title'); + + return template + .replace(/{appTitle}/g, appTitle) + .replace(/{userName}/g, this.userName); + } +}; +``` + +3. **Add notification entry:** + +```typescript +// services/notifications/list.ts +emails: { + // ... existing ... + welcome: { + subject: "Welcome to {0}!", + body: "...", + }, +} +``` + +4. **Use in service:** + +```typescript +import WelcomeEmail from './email/list/welcome.ts'; + +const email = new WelcomeEmail({ to: 'user@example.com', userName: 'John' }); +await new EmailSender(email).send(); +``` + +--- + +## Dependencies + +| Package | Version | Purpose | +|---------|---------|---------| +| `nodemailer` | ^6.x | SMTP transport | +| `assert` | built-in | Input validation | +| `fs.promises` | built-in | Template file reading | +| `path` | built-in | Template path resolution | + +--- + +## Related Documentation + +- [Auth Module](./auth.md) - Authentication service integration +- [Services Module](./services.md) - Service layer overview +- [Notifications Module](./notifications.md) - Error messages and i18n +- [User Management](../../../documentation/user-management.md) - User invitation flow diff --git a/backend/docs/modules/factories.md b/backend/docs/modules/factories.md new file mode 100644 index 0000000..6837f39 --- /dev/null +++ b/backend/docs/modules/factories.md @@ -0,0 +1,846 @@ +# Backend Factories Module + +## Overview + +The Factories module provides code generation patterns that eliminate boilerplate for standard CRUD operations across the backend. Two factory functions generate consistent, standardized router and service classes for all entity types. + +**Location:** `backend/src/factories/` + +**Files:** +| File | Purpose | LOC | +|------|---------|-----| +| `router.factory.ts` | Generates Express routers with CRUD endpoints | 429 | +| `service.factory.ts` | Generates service classes with transaction handling | 350 | + +--- + +## Architecture + +``` +┌─────────────────────────────────────────────────────────────────────────┐ +│ Factory Pattern Flow │ +└─────────────────────────────────────────────────────────────────────────┘ + +┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ +│ Route File │ │ Service File │ │ DB API File │ +│ (3-5 lines) │ │ (3-5 lines) │ │ (extends base) │ +└────────┬────────┘ └────────┬────────┘ └────────┬────────┘ + │ │ │ + ▼ ▼ ▼ +┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ +│ createEntity │ │ createEntity │ │ GenericDBApi │ +│ Router() │ │ Service() │ │ (base.api.ts) │ +└────────┬────────┘ └────────┬────────┘ └────────┬────────┘ + │ │ │ + └──────────────────────┼──────────────────────┘ + │ + ▼ + ┌───────────────────────┐ + │ Sequelize Model │ + │ (db/models/*.js) │ + └───────────────────────┘ + +Boilerplate Reduction: +- Without factories: ~300-500 lines per entity +- With factories: ~10-20 lines per entity (97% reduction) +``` + +--- + +## Factory Files + +### router.factory.ts + +**Purpose:** Generates Express routers with standardized CRUD endpoints, permission checking, CSV export, and error handling. + +**Location:** `backend/src/factories/router.factory.ts` + +#### Function Signature + +```javascript +function createEntityRouter(entityName, Service, DBApi, options = {}) +``` + +#### Parameters + +| Parameter | Type | Description | +|-----------|------|-------------| +| `entityName` | `string` | Entity name for routes and permissions | +| `Service` | `class` | Service class with CRUD methods | +| `DBApi` | `class` | Database API class extending GenericDBApi | +| `options` | `object` | Configuration options | + +#### Options + +| Option | Type | Default | Description | +|--------|------|---------|-------------| +| `permissionEntity` | `string` | `entityName` | Override permission entity name | +| `csvFields` | `string[]` | `DBApi.CSV_FIELDS` | Fields to include in CSV export | +| `customRoutes` | `function` | `null` | Callback to add custom routes | + +#### Generated Endpoints + +| Method | Endpoint | Description | +|--------|----------|-------------| +| `POST` | `/` | Create new record | +| `POST` | `/bulk-import` | Bulk import from CSV | +| `PUT` | `/:id` | Update record by ID | +| `DELETE` | `/:id` | Delete record by ID | +| `POST` | `/deleteByIds` | Delete multiple records | +| `GET` | `/` | List all records (with filters, pagination) | +| `GET` | `/count` | Get record count | +| `GET` | `/autocomplete` | Get autocomplete suggestions | +| `GET` | `/:id` | Get single record by ID | + +#### Implementation + +```javascript +const express = require('express'); +const { wrapAsync, commonErrorHandler, isUuidV4 } = require('../helpers'); +const { checkCrudPermissions } = require('../middlewares/check-permissions'); +const { parse } = require('json2csv'); + +function createEntityRouter(entityName, Service, DBApi, options = {}) { + const router = express.Router(); + + // Apply CRUD permission middleware for all routes + const permissionEntity = options.permissionEntity || entityName; + router.use(checkCrudPermissions(permissionEntity)); + + // POST / - Create + router.post('/', wrapAsync(async (req, res) => { + const referer = req.headers.referer || `${req.protocol}://${req.hostname}${req.originalUrl}`; + const link = new URL(referer); + const payload = await Service.create({ + data: req.body.data, + currentUser: req.currentUser, + runtimeContext: req.runtimeContext, + sendInvitationEmails: true, + host: link.host, + }); + res.status(200).send(payload); + })); + + // POST /bulk-import - Bulk CSV import + router.post('/bulk-import', wrapAsync(async (req, res) => { + const referer = req.headers.referer || `${req.protocol}://${req.hostname}${req.originalUrl}`; + const link = new URL(referer); + await Service.bulkImport(req, res, true, link.host); + res.status(200).send(true); + })); + + // PUT /:id - Update + router.put('/:id', wrapAsync(async (req, res) => { + assertRouteIdMatchesBody(req); + await Service.update({ + id: req.params.id, + data: req.body.data, + currentUser: req.currentUser, + runtimeContext: req.runtimeContext, + }); + res.status(200).send(true); + })); + + // DELETE /:id - Delete single + router.delete('/:id', wrapAsync(async (req, res) => { + await Service.remove({ + id: req.params.id, + currentUser: req.currentUser, + runtimeContext: req.runtimeContext, + }); + res.status(200).send(true); + })); + + // POST /deleteByIds - Delete multiple + router.post('/deleteByIds', wrapAsync(async (req, res) => { + await Service.deleteByIds({ + ids: req.body.data, + currentUser: req.currentUser, + runtimeContext: req.runtimeContext, + }); + res.status(200).send(true); + })); + + // GET / - List all with optional CSV export + router.get('/', wrapAsync(async (req, res) => { + const filetype = req.query.filetype; + const currentUser = req.currentUser; + const runtimeContext = req.runtimeContext; + + const payload = await DBApi.findAll(normalizeQuery(req.query, DBApi, { + csv: filetype === 'csv', + }), { currentUser, runtimeContext }); + + if (filetype === 'csv') { + const fields = options.csvFields || DBApi.CSV_FIELDS || ['id', 'createdAt']; + const opts = { fields }; + try { + const csv = parse(payload.rows, opts); + res.status(200).attachment('export.csv').send(csv); + } catch (err) { + logger.error({ err, entityName }, 'CSV export error'); + res.status(500).send('CSV export error'); + } + } else { + res.status(200).send(payload); + } + })); + + // GET /count - Count only + router.get('/count', wrapAsync(async (req, res) => { + const currentUser = req.currentUser; + const runtimeContext = req.runtimeContext; + const payload = await DBApi.findAll(normalizeQuery(req.query, DBApi), { countOnly: true, currentUser, runtimeContext }); + res.status(200).send(payload); + })); + + // GET /autocomplete - Autocomplete search + router.get('/autocomplete', wrapAsync(async (req, res) => { + const payload = await DBApi.findAllAutocomplete({ + query: req.query.query, + limit, + offset: req.query.offset, + }); + res.status(200).send(payload); + })); + + // GET /:id - Find by ID + router.get('/:id', wrapAsync(async (req, res) => { + if (!isUuidV4(req.params.id)) { + return res.status(400).send(`Invalid ${entityName} id`); + } + const runtimeContext = req.runtimeContext; + const payload = await DBApi.findBy({ id: req.params.id }, { runtimeContext }); + res.status(200).send(payload); + })); + + // Custom routes hook + if (options.customRoutes) { + options.customRoutes(router, Service, DBApi); + } + + // Error handler + router.use('/', commonErrorHandler); + + return router; +} + +module.exports = { createEntityRouter, isUuidV4 }; +``` + +Generic CRUD query safety: + +- `PUT /:id` uses `req.params.id` as the canonical id and rejects mismatched + body ids. +- List and count queries default to `limit=50`, max `limit=1000`, and sanitize + sort direction to `ASC` or `DESC`. +- Sort fields are accepted only when present in the model's `rawAttributes` or + `DBApi.SORTABLE_FIELDS`. +- CSV export uses the same auth path as list and is capped at `limit=1000`. +- Autocomplete defaults to `limit=20` and is capped at `limit=50`. + +#### Exports + +| Export | Type | Description | +|--------|------|-------------| +| `createEntityRouter` | `function` | Factory function | +| `isUuidV4` | `function` | UUID validation helper | + +--- + +### service.factory.ts + +**Purpose:** Generates service classes with standardized CRUD operations wrapped in database transactions. + +**Location:** `backend/src/factories/service.factory.ts` + +#### Function Signature + +```javascript +function createEntityService(DBApi, options = {}) +``` + +#### Parameters + +| Parameter | Type | Description | +|-----------|------|-------------| +| `DBApi` | `class` | Database API class extending GenericDBApi | +| `options` | `object` | Configuration options | + +#### Options + +| Option | Type | Default | Description | +|--------|------|---------|-------------| +| `entityName` | `string` | `'Entity'` | Name used in error messages | + +#### Generated Methods + +| Method | Description | +|--------|-------------| +| `create({ data, currentUser, transaction, runtimeContext })` | Create record with transaction | +| `bulkImport(req, res)` | Bulk import from CSV with transaction | +| `update({ id, data, currentUser, transaction, runtimeContext })` | Update record with transaction | +| `deleteByIds({ ids, currentUser, transaction, runtimeContext })` | Delete multiple with transaction | +| `remove({ id, currentUser, transaction, runtimeContext })` | Delete single with transaction | + +#### Implementation + +```javascript +const db = require('../db/models'); +const processFile = require('../middlewares/upload'); +const ValidationError = require('../services/notifications/errors/validation'); +const csv = require('csv-parser'); +const stream = require('stream'); + +function createEntityService(DBApi, options = {}) { + const entityName = options.entityName || 'Entity'; + + return class GenericService { + static async create({ data, currentUser, transaction: externalTransaction, runtimeContext }) { + const transaction = externalTransaction || await db.sequelize.transaction(); + const ownsTransaction = !externalTransaction; + try { + const record = await DBApi.create({ data, currentUser, transaction, runtimeContext }); + if (ownsTransaction) await transaction.commit(); + return record; + } catch (error) { + if (ownsTransaction) await transaction.rollback(); + throw error; + } + } + + static async bulkImport(req, res) { + const transaction = await db.sequelize.transaction(); + try { + await processFile(req, res); + const bufferStream = new stream.PassThrough(); + const results = []; + + await bufferStream.end(Buffer.from(req.file.buffer, 'utf-8')); + + await new Promise((resolve, reject) => { + bufferStream + .pipe(csv()) + .on('data', (data) => results.push(data)) + .on('end', () => resolve()) + .on('error', (error) => reject(error)); + }); + + await DBApi.bulkImport(results, { + transaction, + ignoreDuplicates: true, + validate: true, + currentUser: req.currentUser, + }); + + await transaction.commit(); + } catch (error) { + await transaction.rollback(); + throw error; + } + } + + static async update({ id, data, currentUser, transaction: externalTransaction, runtimeContext }) { + const transaction = externalTransaction || await db.sequelize.transaction(); + const ownsTransaction = !externalTransaction; + try { + const record = await DBApi.findBy({ id }, { transaction, runtimeContext }); + + if (!record) { + throw new ValidationError(`${entityName}NotFound`); + } + + const updated = await DBApi.update({ id, data, currentUser, transaction, runtimeContext }); + if (ownsTransaction) await transaction.commit(); + return updated; + } catch (error) { + if (ownsTransaction) await transaction.rollback(); + throw error; + } + } + + static async deleteByIds({ ids, currentUser, transaction: externalTransaction, runtimeContext }) { + const transaction = externalTransaction || await db.sequelize.transaction(); + const ownsTransaction = !externalTransaction; + try { + await DBApi.deleteByIds({ ids, currentUser, transaction, runtimeContext }); + if (ownsTransaction) await transaction.commit(); + } catch (error) { + if (ownsTransaction) await transaction.rollback(); + throw error; + } + } + + static async remove({ id, currentUser, transaction: externalTransaction, runtimeContext }) { + const transaction = externalTransaction || await db.sequelize.transaction(); + const ownsTransaction = !externalTransaction; + try { + await DBApi.remove({ id, currentUser, transaction, runtimeContext }); + if (ownsTransaction) await transaction.commit(); + } catch (error) { + if (ownsTransaction) await transaction.rollback(); + throw error; + } + } + }; +} + +module.exports = { createEntityService }; +``` + +#### Transaction Pattern + +All service methods follow the same transaction pattern: + +```javascript +static async methodName(params) { + const transaction = await db.sequelize.transaction(); + try { + // ... database operations with { transaction } + await transaction.commit(); + return result; + } catch (error) { + await transaction.rollback(); + throw error; + } +} +``` + +--- + +## Supporting Components + +### helpers.js + +**Location:** `backend/src/helpers.js` + +Helper utilities used by the router factory: + +```javascript +module.exports = class Helpers { + // Wrap async route handlers to propagate errors + static wrapAsync(fn) { + return function (req, res, next) { + fn(req, res, next).catch(next); + }; + } + + // Centralized error response handler + static commonErrorHandler(error, req, res, _next) { + const statusCode = error.code || error.status; + + if ([400, 401, 403, 404, 409, 422].includes(statusCode)) { + return res.status(statusCode).send(error.message); + } + + console.error(error); + return res.status(500).send('Internal server error'); + } + + // JWT token signing + static jwtSign(data) { + return jwt.sign(data, config.secret_key, { expiresIn: '6h' }); + } + + // UUID v4 validation + static isUuidV4(value) { + return /^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i.test(value); + } +}; +``` + +### check-permissions.ts + +**Location:** `backend/src/middlewares/check-permissions.ts` + +Permission checking middleware used by router factory: + +```javascript +// HTTP method to permission action mapping +const METHOD_MAP = { + POST: 'CREATE', + GET: 'READ', + PUT: 'UPDATE', + PATCH: 'UPDATE', + DELETE: 'DELETE', +}; + +// Entities accessible publicly in runtime mode +const RUNTIME_PUBLIC_READ_ENTITIES = new Set([ + 'PROJECTS', + 'TOUR_PAGES', + 'PAGE_ELEMENTS', + 'PAGE_LINKS', + 'TRANSITIONS', + 'PROJECT_AUDIO_TRACKS', +]); + +// Generate permission name from HTTP method and entity +function checkCrudPermissions(name) { + return (req, res, next) => { + // Skip auth for public runtime read requests + const isRuntimePublicRead = + req.isRuntimePublicRequest === true && + req.method === 'GET' && + RUNTIME_PUBLIC_READ_ENTITIES.has(name.toUpperCase()); + + if (isRuntimePublicRead) { + return next(); + } + + // Build permission name: e.g., 'READ_ASSETS', 'CREATE_USERS' + const permissionName = `${METHOD_MAP[req.method]}_${name.toUpperCase()}`; + return checkPermissions(permissionName)(req, res, next); + }; +} +``` + +--- + +## GenericDBApi (Base Class) + +**Location:** `backend/src/db/api/base.api.ts` + +The DB API base class that all entity APIs extend. Provides declarative configuration for CRUD operations. + +### Static Getters (Override in Subclasses) + +| Getter | Type | Description | +|--------|------|-------------| +| `MODEL` | `Model` | Sequelize model reference (required) | +| `TABLE_NAME` | `string` | Database table name | +| `SEARCHABLE_FIELDS` | `string[]` | Fields for text search (ILIKE) | +| `RANGE_FIELDS` | `string[]` | Fields for range filtering | +| `ENUM_FIELDS` | `string[]` | Fields for exact match filtering | +| `RELATION_FILTERS` | `object[]` | Related entity filters | +| `CSV_FIELDS` | `string[]` | Fields for CSV export | +| `AUTOCOMPLETE_FIELD` | `string` | Field for autocomplete | +| `ASSOCIATIONS` | `object[]` | Related entity setters | +| `FIND_BY_INCLUDES` | `object[]` | Includes for findBy | +| `FIND_ALL_INCLUDES` | `object[]` | Includes for findAll | +| `JSON_FIELDS` | `string[]` | Fields to auto-stringify | +| `FIELD_TRANSFORMERS` | `object` | Custom field transformers | +| `FIELD_DEFAULTS` | `object` | Default values for fields | + +### Methods + +| Method | Description | +|--------|-------------| +| `getFieldMapping(data)` | Transform input data for database | +| `create(data, options)` | Create record | +| `bulkImport(data, options)` | Bulk create records | +| `update({ id, data, currentUser, transaction, runtimeContext })` | Update record | +| `deleteByIds({ ids, currentUser, transaction, runtimeContext })` | Soft delete multiple | +| `remove({ id, currentUser, transaction, runtimeContext })` | Soft delete single | +| `findBy(where, options)` | Find single by criteria | +| `findAll(filter, options)` | Find all with pagination/filters | +| `findAllAutocomplete({ query, limit, offset }, options)` | Autocomplete search | +| `toCSV(rows)` | Convert to CSV string | + +--- + +## Usage Examples + +### Basic Entity (Minimal Configuration) + +**Route (assets.ts):** +```typescript +import AssetsDBApi from '../db/api/assets.ts'; +import { createEntityRouter } from '../factories/router.factory.ts'; +import AssetsService from '../services/assets.ts'; + +// 1 line: generates 9 CRUD endpoints +export default createEntityRouter('assets', AssetsService, AssetsDBApi); +``` + +**Service (assets.ts):** +```typescript +import AssetsDBApi from '../db/api/assets.ts'; +import { createEntityService } from '../factories/service.factory.ts'; + +// 1 line: generates service class with 5 transaction-wrapped methods +export default createEntityService(AssetsDBApi, { + entityName: 'assets', +}); +``` + +**DB API (assets.js):** +```javascript +const GenericDBApi = require('./base.api'); +const db = require('../models'); + +class AssetsDBApi extends GenericDBApi { + static get MODEL() { + return db.assets; + } + + static get SEARCHABLE_FIELDS() { + return ['name', 'cdn_url', 'storage_key', 'mime_type', 'checksum']; + } + + static get RANGE_FIELDS() { + return ['size_mb', 'width_px', 'height_px', 'duration_sec']; + } + + static get ENUM_FIELDS() { + return ['asset_type', 'type', 'is_public']; + } + + static get ASSOCIATIONS() { + return [{ field: 'project', setter: 'setProject', isArray: false }]; + } + + static getFieldMapping(data) { + return { + name: data.name || null, + asset_type: data.asset_type || null, + type: data.type || 'general', + cdn_url: data.cdn_url || null, + // ... other fields + }; + } +} + +module.exports = AssetsDBApi; +``` + +### Entity with Custom Routes + +**Route (project_element_defaults.ts):** +```typescript +import Service from '../services/project_element_defaults.ts'; +import DBApi from '../db/api/project_element_defaults.ts'; +import { createEntityRouter } from '../factories/router.factory.ts'; +import { wrapAsync } from '../helpers.ts'; + +// Create base router +const baseRouter = createEntityRouter( + 'project_element_defaults', + Service, + DBApi, + { permissionEntity: 'page_elements' } // Override permission entity +); + +// Add custom endpoint +baseRouter.post('/:id/reset', wrapAsync(async (req, res) => { + const payload = await Service.resetToGlobal(req.params.id, { + currentUser: req.currentUser, + }); + res.status(200).json(payload); +})); + +// Add another custom endpoint +baseRouter.get('/:id/diff', wrapAsync(async (req, res) => { + const payload = await Service.getDiffFromGlobal(req.params.id); + res.status(200).json(payload); +})); + +export default baseRouter; +``` + +### Service with Extended Methods + +**Service (project_element_defaults.ts):** +```typescript +import DBApi from '../db/api/project_element_defaults.ts'; +import { createEntityService } from '../factories/service.factory.ts'; + +// Create base service class +const BaseService = createEntityService(DBApi, { + entityName: 'project_element_defaults', +}); + +// Extend with custom methods +export default class Project_element_defaultsService extends BaseService { + static resetToGlobal(id, options = {}) { + return DBApi.resetToGlobal(id, options); + } + + static getDiffFromGlobal(id) { + return DBApi.getDiffFromGlobal(id); + } + + static snapshotGlobalDefaults(projectId, options = {}) { + return DBApi.snapshotGlobalDefaults(projectId, options); + } +} +``` + +### Using customRoutes Callback + +Alternative approach using the options callback: + +```javascript +module.exports = createEntityRouter('entities', Service, DBApi, { + customRoutes: (router, Service, DBApi) => { + router.post('/:id/custom-action', wrapAsync(async (req, res) => { + const result = await Service.customAction(req.params.id); + res.status(200).json(result); + })); + + router.get('/stats', wrapAsync(async (req, res) => { + const stats = await DBApi.getStatistics(); + res.status(200).json(stats); + })); + } +}); +``` + +--- + +## Entity Usage Summary + +### Entities Using Router Factory (13) + +| Entity | Permission Override | Custom Routes | +|--------|--------------------|--------------| +| `access_logs` | - | No | +| `asset_variants` | - | No | +| `assets` | - | No | +| `element_type_defaults` | - | No | +| `permissions` | - | No | +| `presigned_url_requests` | - | No | +| `project_audio_tracks` | - | No | +| `project_element_defaults` | `page_elements` | Yes (reset, diff) | +| `project_memberships` | - | No | +| `publish_events` | - | No | +| `pwa_caches` | - | No | +| `roles` | - | No | +| `tour_pages` | - | No | + +### Entities Using Service Factory (11) + +| Entity | Custom Methods | +|--------|---------------| +| `access_logs` | No | +| `asset_variants` | No | +| `assets` | No | +| `element_type_defaults` | No | +| `permissions` | No | +| `presigned_url_requests` | No | +| `pwa_caches` | No | +| `publish_events` | No | +| `tour_pages` | No | +| `project_element_defaults` | Yes (resetToGlobal, getDiffFromGlobal, snapshotGlobalDefaults) | +| `project_memberships` | No | + +### Entities NOT Using Factories + +Some entities have custom implementations due to specialized requirements: + +| Entity | Reason | +|--------|--------| +| `users` | Complex auth, password hashing, token management | +| `projects` | Publishing workflow, complex business logic | +| `auth` | Authentication flows (login, OAuth, password reset) | +| `file` | File upload/download, S3/GCloud/Local storage | +| `search` | Full-text search across multiple entities | +| `publish` | Multi-step publishing workflow | + +--- + +## Generated Endpoints Flow + +``` +HTTP Request + │ + ▼ +┌─────────────────────────────────┐ +│ JWT Authentication │ (from index.js) +│ passport.authenticate('jwt') │ +└─────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────┐ +│ checkCrudPermissions() │ (from router.factory.ts) +│ - Maps HTTP method to action │ +│ - Builds permission name │ +│ - Checks user role permissions │ +└─────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────┐ +│ Route Handler │ (from router.factory.ts) +│ - wrapAsync() for error catch │ +│ - Calls Service method │ +└─────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────┐ +│ Service Method │ (from service.factory.ts) +│ - Starts transaction │ +│ - Calls DBApi method │ +│ - Commits or rollbacks │ +└─────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────┐ +│ DBApi Method │ (from base.api.ts + entity) +│ - getFieldMapping() transform │ +│ - Sequelize model operation │ +│ - Returns result │ +└─────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────┐ +│ commonErrorHandler() │ (from helpers.js) +│ - Formats error response │ +│ - Appropriate HTTP status │ +└─────────────────────────────────┘ + │ + ▼ +HTTP Response +``` + +--- + +## Design Patterns + +### Factory Pattern +Both `createEntityRouter` and `createEntityService` implement the Factory pattern, creating objects (router, service class) without specifying their exact classes. + +### Template Method Pattern +`GenericDBApi.getFieldMapping()` uses the Template Method pattern - the base class defines the algorithm skeleton, while subclasses can override specific steps via static getters (`JSON_FIELDS`, `FIELD_TRANSFORMERS`, `FIELD_DEFAULTS`). + +### Strategy Pattern +The permission checking system uses Strategy pattern - different entities can have different permission strategies by overriding `permissionEntity` option. + +### Decorator Pattern +The router factory decorates Express routers with permission middleware and error handling. + +--- + +## Best Practices + +### When to Use Factories + +Use factories when: +- Entity requires standard CRUD operations +- No complex business logic beyond data transformation +- Permissions follow standard READ/CREATE/UPDATE/DELETE pattern +- No special authentication requirements + +### When NOT to Use Factories + +Don't use factories when: +- Complex multi-step workflows (use custom service) +- Special authentication (OAuth flows, password reset) +- External API integration (file storage, AI) +- Cross-entity transactions +- Custom endpoint patterns + +### Extending Factory-Generated Code + +1. **Add custom routes:** Extend the base router after factory call +2. **Add custom service methods:** Extend the generated class +3. **Override permissions:** Use `permissionEntity` option +4. **Customize data transformation:** Override `getFieldMapping()` in DBApi + +--- + +## Related Documentation + +- [Services Module](./services.md) - Business logic layer +- [Routes Module](./routes.md) - All route files +- [Middleware Module](./middleware.md) - Permission checking +- [Database Schema](../database-schema.md) - Model definitions diff --git a/backend/docs/modules/middleware.md b/backend/docs/modules/middleware.md new file mode 100644 index 0000000..858520b --- /dev/null +++ b/backend/docs/modules/middleware.md @@ -0,0 +1,842 @@ +# Backend Middleware Module Documentation + +## Overview + +The Middleware module provides cross-cutting concerns for the Express application including rate limiting, permission checking, runtime context management, file uploads, and public access control. + +**Files:** +| File | Lines | Purpose | +|------|-------|---------| +| `src/middlewares/rateLimiter.js` | 268 | Configurable rate limiting with in-memory store | +| `src/middlewares/check-permissions.ts` | RBAC permission checking through AccessPolicy | +| `src/middlewares/runtime-context.ts` | 34 | Runtime environment context from headers | +| `src/middlewares/runtime-public.ts` | 200 | Public runtime access control and response sanitization | +| `src/middlewares/upload.ts` | 34 | Multer-based file upload handling | + +--- + +## Architecture Diagram + +``` +┌──────────────────────────────────────────────────────────────────────┐ +│ Incoming Request │ +└──────────────────────────────────┬───────────────────────────────────┘ + │ + ▼ +┌──────────────────────────────────────────────────────────────────────┐ +│ Express Middleware Stack │ +│ │ +│ ┌────────────────────────────────────────────────────────────────┐ │ +│ │ 1. helmet() - Security headers │ │ +│ └────────────────────────────────────────────────────────────────┘ │ +│ │ │ +│ ▼ │ +│ ┌────────────────────────────────────────────────────────────────┐ │ +│ │ 2. cors() - Cross-origin resource sharing │ │ +│ └────────────────────────────────────────────────────────────────┘ │ +│ │ │ +│ ▼ │ +│ ┌────────────────────────────────────────────────────────────────┐ │ +│ │ 3. requestLogger - Pino HTTP logging │ │ +│ └────────────────────────────────────────────────────────────────┘ │ +│ │ │ +│ ▼ │ +│ ┌────────────────────────────────────────────────────────────────┐ │ +│ │ 4. Rate Limiters (route-specific) │ │ +│ │ • downloadLimiter → /api/file/download, /api/file/presign │ │ +│ │ • uploadLimiter → /api/file/upload │ │ +│ │ • searchLimiter → /api/search │ │ +│ │ • authLimiter → /api/auth/signin │ │ +│ └────────────────────────────────────────────────────────────────┘ │ +│ │ │ +│ ▼ │ +│ ┌────────────────────────────────────────────────────────────────┐ │ +│ │ 5. bodyParser.json() - JSON body parsing (after file routes) │ │ +│ └────────────────────────────────────────────────────────────────┘ │ +│ │ │ +│ ▼ │ +│ ┌────────────────────────────────────────────────────────────────┐ │ +│ │ 6. runtimeContextMiddleware - Environment context │ │ +│ │ Reads: X-Runtime-Environment, X-Runtime-Project-Slug │ │ +│ └────────────────────────────────────────────────────────────────┘ │ +│ │ │ +│ ▼ │ +│ ┌────────────────────────────────────────────────────────────────┐ │ +│ │ 7. JWT Authentication (route-specific) │ │ +│ │ passport.authenticate('jwt', { session: false }) │ │ +│ └────────────────────────────────────────────────────────────────┘ │ +│ │ │ +│ ▼ │ +│ ┌────────────────────────────────────────────────────────────────┐ │ +│ │ 8. Runtime Public Middleware (route-specific) │ │ +│ │ • blockNonPublicRuntimeListEndpoints │ │ +│ │ • sanitizePublicRuntimeListResponse │ │ +│ └────────────────────────────────────────────────────────────────┘ │ +│ │ │ +│ ▼ │ +│ ┌────────────────────────────────────────────────────────────────┐ │ +│ │ 9. checkCrudPermissions / checkPermissions (route-specific) │ │ +│ └────────────────────────────────────────────────────────────────┘ │ +│ │ │ +│ ▼ │ +│ ┌────────────────────────────────────────────────────────────────┐ │ +│ │ 10. Route Handler │ │ +│ └────────────────────────────────────────────────────────────────┘ │ +│ │ │ +│ ▼ │ +│ ┌────────────────────────────────────────────────────────────────┐ │ +│ │ 11. Error Handler (commonErrorHandler) │ │ +│ └────────────────────────────────────────────────────────────────┘ │ +└──────────────────────────────────────────────────────────────────────┘ +``` + +--- + +## File Details + +### 1. rateLimiter.js (268 lines) + +In-memory rate limiting middleware with configurable windows and limits. + +#### Storage Architecture + +```javascript +// In-memory store (Map) +const rateLimitStore = new Map(); + +// Entry structure +{ + count: number, // Request count in window + expiresAt: number, // Window expiration timestamp + resetTime: string // ISO timestamp for headers +} + +// Automatic cleanup every 5 minutes +setInterval(() => { + for (const [key, entry] of rateLimitStore.entries()) { + if (entry.expiresAt <= now) { + rateLimitStore.delete(key); + } + } +}, 5 * 60 * 1000); +``` + +#### Factory: createRateLimiter(options) + +Creates a configurable rate limiter middleware. + +**Parameters:** +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `keyPrefix` | string | `'rate-limit'` | Prefix for rate limit keys | +| `windowMs` | number | `900000` (15min) | Time window in milliseconds | +| `max` | number | `100` | Maximum requests per window | +| `message` | string | `'Too many requests...'` | Error message on limit | +| `skipFailedRequests` | boolean | `false` | Don't count 4xx/5xx responses | +| `keyGenerator` | function | `null` | Custom key generator `(req) => string` | +| `skip` | function | `null` | Skip rate limiting `(req) => boolean` | + +**Returns:** Express middleware function + +**Response Headers:** +``` +X-RateLimit-Limit: 100 +X-RateLimit-Remaining: 99 +X-RateLimit-Reset: 2024-01-01T00:15:00.000Z +Retry-After: 300 (only when limit exceeded) +``` + +**Rate Limit Exceeded Response (429):** +```json +{ + "error": "Too Many Requests", + "message": "Too many requests. Please try again later.", + "retryAfter": 300 +} +``` + +#### Factory: createAuthenticatedRateLimiter(options) + +Creates rate limiter that uses IP + User ID as key. + +```javascript +const createAuthenticatedRateLimiter = (options = {}) => { + return createRateLimiter({ + ...options, + keyGenerator: (req) => { + const userId = req.currentUser?.id || 'anonymous'; + const ip = req.ip || 'unknown'; + return `${ip}:${userId}`; + }, + }); +}; +``` + +#### Pre-configured Limiters + +| Limiter | Key Prefix | Window | Max | Skip Failed | Use Case | +|---------|------------|--------|-----|-------------|----------| +| `authLimiter` | `auth` | 15 min | 10 | No | Login attempts | +| `passwordResetLimiter` | `password-reset` | 1 hour | 5 | No | Password reset | +| `apiLimiter` | `api` | 1 min | 100 | Yes | General API | +| `uploadLimiter` | `upload` | 1 min | 10 | No | File uploads | +| `downloadLimiter` | `download` | 1 min | 200 | Yes | File downloads | +| `searchLimiter` | `search` | 1 min | 30 | No | Search queries | + +#### Route Mapping + +```javascript +// index.js +app.use('/api/file/download', downloadLimiter); +app.use('/api/file/presign', downloadLimiter); +app.use('/api/file/upload', uploadLimiter); +app.use('/api/file/upload-sessions', uploadLimiter); +app.use('/api/search', jwtAuth, searchLimiter, searchRoutes); + +// routes/auth.js +router.post('/signin/local', signinLimiter, handler); +router.post('/send-password-reset-email', passwordResetLimiter, handler); +``` + +#### Development Mode + +Rate limiting is skipped for localhost in development: + +```javascript +if ( + config.server.env === 'development' && + (req.ip === '127.0.0.1' || req.ip === '::1') +) { + return next(); // Skip rate limiting +} +``` + +--- + +### 2. check-permissions.ts (194 lines) + +Role-based access control (RBAC) middleware. Permission decisions are delegated +to `src/services/access-policy.ts` so role/custom permission resolution and +Public-user hardening stay centralized. + +#### Public Role Caching + +```javascript +let publicRoleCache = null; + +// Fetched on module load (startup) +async function fetchAndCachePublicRole() { + publicRoleCache = await RolesDBApi.findBy({ name: 'Public' }); +} + +// Called immediately when module is imported +fetchAndCachePublicRole(); +``` + +#### Function: checkPermissions(permission) + +Creates middleware that checks if user has specific permission. + +**Permission Check Flow:** +``` +1. AccessPolicy.hasPermission(user, permission) + ├── Public users are always denied admin API permissions + └── Internal users use app_role.permissions + custom_permissions + +2. Public role fallback + └── Unauthenticated/no-role requests use cached Public role, but Public role permissions are ignored + +3. Role lacks permission → 403 Forbidden +``` + +Self-access bypass is not part of `checkPermissions`. It is explicitly limited +to `GET`, `PUT`, and `PATCH` on the authenticated user's own `/api/users/:id` +route in `checkCrudPermissions`. + +**Usage:** +```javascript +const { checkPermissions } = require('./middlewares/check-permissions'); + +// Check specific permission +router.get('/admin', checkPermissions('ADMIN_ACCESS'), handler); + +// Check entity permission +router.get('/users', checkPermissions('READ_USERS'), handler); +``` + +**Error Response (403):** +```json +{ + "message": "Forbidden" +} +``` + +#### Function: checkCrudPermissions(name) + +Creates middleware that maps HTTP method to CRUD permission. + +**Method Mapping:** +| HTTP Method | Permission Prefix | +|-------------|-------------------| +| `POST` | `CREATE_` | +| `GET` | `READ_` | +| `PUT` | `UPDATE_` | +| `PATCH` | `UPDATE_` | +| `DELETE` | `DELETE_` | + +**Permission Name Format:** `{METHOD}_{ENTITY}` + +Examples: +- `GET /api/users` → `READ_USERS` +- `POST /api/projects` → `CREATE_PROJECTS` +- `DELETE /api/assets/123` → `DELETE_ASSETS` + +Routes can set `req.permissionNameOverride` before `checkCrudPermissions` when +the HTTP verb does not describe the domain operation. The middleware uses the +override as the exact permission name and otherwise falls back to +`{METHOD}_{ENTITY}`. For example, environment-level resets for project runtime +settings use `DELETE` to remove an override row, but the user-facing operation +is "use inherited defaults", so those routes require `UPDATE_PAGE_ELEMENTS` +rather than `DELETE_PAGE_ELEMENTS`. + +**Usage:** +```javascript +const { checkCrudPermissions } = require('./middlewares/check-permissions'); + +// In router factory +router.get('/', checkCrudPermissions('users'), listHandler); +router.post('/', checkCrudPermissions('users'), createHandler); +router.delete('/:id', checkCrudPermissions('users'), deleteHandler); + +// For reset/update semantics implemented as DELETE +router.use((req, _res, next) => { + if (req.method === 'DELETE' && req.path.startsWith('/project/')) { + req.permissionNameOverride = 'UPDATE_PAGE_ELEMENTS'; + } + next(); +}); +router.use(checkCrudPermissions('page_elements')); +``` + +#### Runtime Public Read Bypass + +Certain entities allow public read access in production runtime: + +```javascript +const RUNTIME_PUBLIC_READ_ENTITIES = new Set([ + 'PROJECTS', + 'TOUR_PAGES', + 'PAGE_ELEMENTS', + 'PAGE_LINKS', + 'TRANSITIONS', + 'PROJECT_AUDIO_TRACKS', + 'GLOBAL_TRANSITION_DEFAULTS', + 'PROJECT_TRANSITION_SETTINGS', +]); + +// Bypass permission check for public runtime reads +const isRuntimePublicRead = + req.isRuntimePublicRequest === true && + req.method === 'GET' && + RUNTIME_PUBLIC_READ_ENTITIES.has(name.toUpperCase()); + +if (isRuntimePublicRead) { + return next(); // Skip permission check +} +``` + +**⚠️ Middleware Ordering Requirement:** + +For public read bypass to work, the middleware that sets `req.isRuntimePublicRequest = true` **MUST run before** `checkCrudPermissions`. Common mistake: + +```javascript +// ❌ WRONG - allowPublicRead runs AFTER checkCrudPermissions +router.use(checkCrudPermissions('entity')); +router.get('/', allowPublicRead, handler); // Too late! + +// ✅ CORRECT - allowPublicRead runs BEFORE checkCrudPermissions +router.use(allowPublicRead); +router.use(checkCrudPermissions('entity')); +router.get('/', handler); +``` + +When using `router.use()`, middleware is applied to ALL routes before route-specific middleware runs. + +--- + +### 3. runtime-context.ts (34 lines) + +Middleware that extracts runtime environment context from request headers. + +#### Function: runtimeContextMiddleware + +Reads environment and project slug from headers for route-based access. + +**Headers:** +| Header | Values | Description | +|--------|--------|-------------| +| `X-Runtime-Environment` | `production`, `stage`, `dev` | Content environment | +| `X-Runtime-Project-Slug` | string | Project identifier | + +**Context Object:** +```javascript +req.runtimeContext = { + mode: 'admin', // Default mode + projectSlug: null, // Extracted from path or header + headerEnvironment: 'production', // From X-Runtime-Environment + headerProjectSlug: 'my-tour' // From X-Runtime-Project-Slug +}; +``` + +**Usage in Routes:** +```javascript +// index.js +app.use(runtimeContextMiddleware); + +// Access in handlers +const env = req.runtimeContext?.headerEnvironment; +if (env === 'production') { + // Filter for production content +} +``` + +**Route-Based Environment Access:** +| Route | Environment | Access | +|-------|-------------|--------| +| `/p/[slug]` | `production` | Public (no auth) | +| `/p/[slug]/stage` | `stage` | Authenticated only | +| `/constructor?projectId=` | `dev` | Authenticated only | + +--- + +### 4. runtime-public.ts (200 lines) + +Middleware for controlling public runtime access and sanitizing responses. + +#### Allowed Fields (Whitelist) + +Only these fields are returned for public runtime requests: + +```javascript +const PUBLIC_RUNTIME_ENTITY_FIELDS = { + projects: [ + 'id', 'name', 'slug', 'description', 'logo_url', 'favicon_url', 'og_image_url', + ], + tour_pages: [ + 'id', 'projectId', 'environment', 'source_key', 'name', 'slug', + 'sort_order', 'background_image_url', 'background_video_url', + 'background_audio_url', 'background_loop', 'requires_auth', 'ui_schema_json', + ], + project_audio_tracks: [ + 'id', 'projectId', 'environment', 'source_key', 'name', 'slug', + 'url', 'loop', 'volume', 'sort_order', 'is_enabled', + ], +}; +``` + +#### Function: blockNonPublicRuntimeListEndpoints + +Restricts public runtime requests to list endpoints only. + +```javascript +const blockNonPublicRuntimeListEndpoints = (req, res, next) => { + if (!isPublicRuntimeReadRequest(req)) { + return next(); // Not a public request, continue + } + + // Only allow root path (list endpoint) + if (req.path !== '/') { + return res.status(404).send({ message: 'Not found' }); + } + + // Block CSV exports + if (req.query.filetype === 'csv') { + return res.status(404).send({ message: 'Not found' }); + } + + return next(); +}; +``` + +**Blocked:** +- Individual record access: `GET /api/projects/123` → 404 +- CSV exports: `GET /api/projects?filetype=csv` → 404 + +**Allowed:** +- List endpoints: `GET /api/projects/` → Continue + +#### Function: sanitizePublicRuntimeListResponse(entityName) + +Filters response data to only include whitelisted fields. + +```javascript +const sanitizePublicRuntimeListResponse = (entityName) => { + const fields = PUBLIC_RUNTIME_ENTITY_FIELDS[entityName] || []; + + return (req, res, next) => { + // Intercept res.send() + const originalSend = res.send.bind(res); + + res.send = (body) => { + if (Array.isArray(body.rows)) { + const sanitizedRows = body.rows.map((row) => pickFields(row, fields)); + return originalSend({ ...body, rows: sanitizedRows }); + } + return originalSend(body); + }; + + return next(); + }; +}; +``` + +**Before Sanitization:** +```json +{ + "rows": [{ + "id": "123", + "name": "My Tour", + "slug": "my-tour", + "createdAt": "2024-01-01", + "createdById": "user-456", + "internalNotes": "sensitive data" + }] +} +``` + +**After Sanitization:** +```json +{ + "rows": [{ + "id": "123", + "name": "My Tour", + "slug": "my-tour" + }] +} +``` + +#### Usage in index.js + +```javascript +const mountRuntimeEntityRoute = (path, entityName, router) => { + app.use( + path, + requireRuntimeReadOrAuth, // JWT or public access + blockNonPublicRuntimeListEndpoints, // Block non-list endpoints + sanitizePublicRuntimeListResponse(entityName), // Filter fields + router, + ); +}; + +mountRuntimeEntityRoute('/api/projects', 'projects', projectsRoutes); +mountRuntimeEntityRoute('/api/tour_pages', 'tour_pages', tour_pagesRoutes); +mountRuntimeEntityRoute('/api/project_audio_tracks', 'project_audio_tracks', ...); +``` + +--- + +### 5. upload.ts (34 lines) + +Simple Multer-based file upload middleware. + +```javascript +const util = require('util'); +const Multer = require('multer'); + +let processFile = Multer({ + storage: Multer.memoryStorage(), +}).single('file'); + +let processFileMiddleware = util.promisify(processFile); +module.exports = processFileMiddleware; +``` + +**Configuration:** +| Setting | Value | +|---------|-------| +| Storage | Memory (Buffer) | +| Field Name | `file` | +| Max Files | 1 (single) | + +**Usage:** +```javascript +const upload = require('./middlewares/upload'); + +router.post('/upload', async (req, res) => { + await upload(req, res); + // req.file contains: + // - buffer: File data + // - originalname: Original filename + // - mimetype: MIME type + // - size: File size in bytes +}); +``` + +**Note:** This middleware is primarily used for legacy uploads. The main file upload system uses chunked uploads without this middleware. + +--- + +## Middleware Composition Patterns + +### Pattern 1: Route-Level Rate Limiting + +```javascript +// Apply limiter before route handler +app.use('/api/file/upload', uploadLimiter); +app.use('/api/file', fileRoutes); +``` + +### Pattern 2: Inline Middleware Chain + +```javascript +// Multiple middlewares in route definition +router.post( + '/signin/local', + signinLimiter, // Rate limit + wrapAsync(async (req, res) => { ... }), +); +``` + +### Pattern 3: JWT + Feature Middleware + +```javascript +// JWT auth + rate limit + routes +app.use('/api/search', jwtAuth, searchLimiter, searchRoutes); +``` + +### Pattern 4: Conditional Auth (Runtime) + +```javascript +const requireRuntimeReadOrAuth = (req, res, next) => { + const headerEnvironment = req.runtimeContext?.headerEnvironment; + const isReadOnlyRequest = ['GET', 'OPTIONS'].includes(req.method); + const hasAuthHeader = Boolean(req.headers.authorization); + const isPublicEnvironment = headerEnvironment === 'production'; + + if (isPublicEnvironment && isReadOnlyRequest && !hasAuthHeader) { + req.isRuntimePublicRequest = true; + return next(); // Allow without auth + } + + req.isRuntimePublicRequest = false; + return jwtAuth(req, res, next); // Require auth +}; +``` + +### Pattern 5: Response Interception + +```javascript +// Intercept and modify response before sending +const sanitizeResponse = (req, res, next) => { + const originalSend = res.send.bind(res); + + res.send = (body) => { + const modified = transformBody(body); + return originalSend(modified); + }; + + return next(); +}; +``` + +--- + +## Request Flow Examples + +### Example 1: Authenticated API Request + +``` +GET /api/users +Authorization: Bearer + +1. helmet() → Security headers +2. cors() → CORS headers +3. requestLogger → Log request +4. bodyParser.json() → Parse body +5. runtimeContextMiddleware → Set req.runtimeContext +6. jwtAuth → Validate JWT, set req.currentUser +7. checkCrudPermissions('users') → Check READ_USERS permission +8. Route handler → Return users +``` + +### Example 2: Public Runtime Request + +``` +GET /api/projects +X-Runtime-Environment: production + +1. helmet() → Security headers +2. cors() → CORS headers +3. requestLogger → Log request +4. bodyParser.json() → Parse body +5. runtimeContextMiddleware → Set req.runtimeContext.headerEnvironment = 'production' +6. requireRuntimeReadOrAuth → Set req.isRuntimePublicRequest = true, skip JWT +7. blockNonPublicRuntimeListEndpoints → Allow (path is '/') +8. sanitizePublicRuntimeListResponse('projects') → Filter response fields +9. checkCrudPermissions('projects') → Skip (isRuntimePublicRequest) +10. Route handler → Return sanitized projects +``` + +### Example 3: Rate Limited Upload + +``` +POST /api/file/upload +Authorization: Bearer +Content-Type: multipart/form-data + +1. uploadLimiter → Check rate limit (10/min) + ├── Under limit → Continue + └── Over limit → 429 Too Many Requests +2. fileRoutes handles request (own body parsing) +``` + +--- + +## Error Handling + +### Rate Limit Errors + +```javascript +// 429 Too Many Requests +{ + "error": "Too Many Requests", + "message": "Too many requests. Please try again later.", + "retryAfter": 300 +} +``` + +### Permission Errors + +```javascript +// 403 Forbidden (via ValidationError) +{ + "message": "Role 'User' denied access to 'DELETE_USERS'." +} +``` + +### Public Access Errors + +```javascript +// 404 Not Found (blocked endpoint) +{ + "message": "Not found" +} +``` + +--- + +## Configuration + +### Environment Variables + +| Variable | Affects | Description | +|----------|---------|-------------| +| `NODE_ENV` | Rate limiting | Skip localhost in development | + +### Constants + +```javascript +// rateLimiter.js +const CLEANUP_INTERVAL_MS = 5 * 60 * 1000; // 5 minutes + +// check-permissions.ts +const METHOD_MAP = { + POST: 'CREATE', + GET: 'READ', + PUT: 'UPDATE', + PATCH: 'UPDATE', + DELETE: 'DELETE', +}; + +const RUNTIME_PUBLIC_READ_ENTITIES = new Set([ + 'PROJECTS', 'TOUR_PAGES', 'PAGE_ELEMENTS', + 'PAGE_LINKS', 'TRANSITIONS', 'PROJECT_AUDIO_TRACKS', +]); + +// runtime-public.ts +const PUBLIC_RUNTIME_ALLOWED_PATH = '/'; +``` + +--- + +## Dependencies + +| Package | Version | Purpose | +|---------|---------|---------| +| `multer` | ^1.4.5 | Multipart form data parsing | +| `util` | built-in | Promisify multer | + +**Internal Dependencies:** +- `../utils/logger` - Pino logger for rate limit logging +- `../services/notifications/errors/validation` - ValidationError class +- `../db/api/roles` - RolesDBApi for Public role + +--- + +## Security Considerations + +1. **Rate Limiting:** Prevents brute force and DoS attacks +2. **Permission Checking:** RBAC with role hierarchy +3. **Public Role Fallback:** Unauthenticated users get minimal permissions +4. **Response Sanitization:** Prevents data leakage in public runtime +5. **Self-Access Bypass:** Users can always access their own resources +6. **Memory Store:** Not suitable for horizontal scaling (use Redis) + +--- + +## Testing + +### Test Rate Limiting + +```bash +# Should succeed (under limit) +for i in {1..10}; do + curl -X POST http://localhost:3000/api/auth/signin/local \ + -H "Content-Type: application/json" \ + -d '{"email": "test@test.com", "password": "wrong"}' +done + +# Should return 429 (over limit) +curl -X POST http://localhost:3000/api/auth/signin/local \ + -H "Content-Type: application/json" \ + -d '{"email": "test@test.com", "password": "wrong"}' +``` + +### Test Public Runtime Access + +```bash +# Should return sanitized projects +curl http://localhost:3000/api/projects \ + -H "X-Runtime-Environment: production" + +# Should return 404 (individual record blocked) +curl http://localhost:3000/api/projects/123 \ + -H "X-Runtime-Environment: production" +``` + +### Test Permission Check + +```bash +# Should return 403 if user lacks permission +curl http://localhost:3000/api/users \ + -H "Authorization: Bearer " +``` + +--- + +## Summary + +The Middleware module provides: + +1. **rateLimiter.js** - 8 pre-configured rate limiters with in-memory store +2. **check-permissions.ts** - RBAC through AccessPolicy with user-route-only self access +3. **runtime-context.ts** - Runtime environment context from headers +4. **runtime-public.ts** - Public access control and response sanitization +5. **upload.ts** - Simple Multer-based file upload + +**Key Features:** +- Configurable rate limiting per endpoint type +- Role-based permission checking with method-to-CRUD mapping +- Public runtime access for production presentations +- Response field filtering for public access +- Memory-based storage (scales vertically, needs Redis for horizontal) diff --git a/backend/docs/modules/notifications.md b/backend/docs/modules/notifications.md new file mode 100644 index 0000000..e696775 --- /dev/null +++ b/backend/docs/modules/notifications.md @@ -0,0 +1,801 @@ +# Backend Notifications Module + +## Overview + +The Notifications module provides a centralized **error handling system** and **internationalization (i18n) catalog** for the backend. It includes custom error classes with HTTP status codes and a message resolution system that supports parameter substitution. + +**Location:** `backend/src/services/notifications/` + +**Total Files:** 4 + +--- + +## Architecture Diagram + +``` +┌─────────────────────────────────────────────────────────────────────┐ +│ Services Layer │ +│ (auth.js, users.js, projects.js, roles.js, search.js, etc.) │ +│ │ +│ throw new ValidationError('auth.userNotFound'); │ +│ throw new ForbiddenError('auth.forbidden'); │ +└───────────────────────────────┬─────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────────┐ +│ Notifications Module │ +│ │ +│ ┌─────────────────────────────────────────────────────────────┐ │ +│ │ Error Classes │ │ +│ │ (errors/) │ │ +│ │ ┌─────────────────────┐ ┌─────────────────────┐ │ │ +│ │ │ ValidationError │ │ ForbiddenError │ │ │ +│ │ │ (HTTP 400) │ │ (HTTP 403) │ │ │ +│ │ └──────────┬──────────┘ └──────────┬──────────┘ │ │ +│ │ │ │ │ │ +│ │ └────────────┬───────────┘ │ │ +│ │ │ │ │ +│ │ ▼ │ │ +│ │ ┌─────────────────────────┐ │ │ +│ │ │ helpers.js │ │ │ +│ │ │ • getNotification() │ │ │ +│ │ │ • isNotification() │ │ │ +│ │ └────────────┬────────────┘ │ │ +│ │ │ │ │ +│ │ ▼ │ │ +│ │ ┌─────────────────────────┐ │ │ +│ │ │ list.js │ │ │ +│ │ │ (Message Catalog) │ │ │ +│ │ │ • auth.* │ │ │ +│ │ │ • iam.* │ │ │ +│ │ │ • importer.* │ │ │ +│ │ │ • errors.* │ │ │ +│ │ │ • emails.* │ │ │ +│ │ └─────────────────────────┘ │ │ +│ └─────────────────────────────────────────────────────────────┘ │ +└─────────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────────┐ +│ Error Handler Middleware │ +│ (Generic error handling in index.js) │ +│ │ +│ app.use((err, req, res, _next) => { │ +│ res.status(err.code || 500).json({ message: err.message }); │ +│ }); │ +└─────────────────────────────────────────────────────────────────────┘ +``` + +--- + +## Directory Structure + +``` +services/notifications/ +├── helpers.js # Message resolution functions (31 LOC) +├── list.js # Message catalog / i18n strings (101 LOC) +└── errors/ + ├── validation.js # ValidationError class - 400 (17 LOC) + └── forbidden.js # ForbiddenError class - 403 (17 LOC) +``` + +--- + +## Core Components + +### Message Catalog (`list.js`) + +Centralized storage for all user-facing messages and email content. + +```javascript +const errors = { + app: { + title: 'Tour Builder Platform', + }, + + auth: { + userDisabled: 'Your account is disabled', + forbidden: 'Forbidden', + unauthorized: 'Unauthorized', + userNotFound: "Sorry, we don't recognize your credentials", + wrongPassword: "Sorry, we don't recognize your credentials", + weakPassword: 'This password is too weak', + emailAlreadyInUse: 'Email is already in use', + invalidEmail: 'Please provide a valid email', + passwordReset: { + invalidToken: 'Password reset link is invalid or has expired', + error: 'Email not recognized', + }, + passwordUpdate: { + samePassword: "You can't use the same password. Please create new password", + }, + userNotVerified: 'Sorry, your email has not been verified yet', + emailAddressVerificationEmail: { + invalidToken: 'Email verification link is invalid or has expired', + error: 'Email not recognized', + }, + }, + + iam: { + errors: { + userAlreadyExists: 'User with this email already exists', + userNotFound: 'User not found', + disablingHimself: "You can't disable yourself", + revokingOwnPermission: "You can't revoke your own owner permission", + deletingHimself: "You can't delete yourself", + emailRequired: 'Email is required', + slugAlreadyExists: 'This slug is already in use by another project', + searchQueryRequired: 'Search query is required', + }, + }, + + importer: { + errors: { + invalidFileEmpty: 'The file is empty', + invalidFileExcel: 'Only excel (.xlsx) files are allowed', + invalidFileUpload: 'Invalid file. Make sure you are using the last version of the template.', + importHashRequired: 'Import hash is required', + importHashExistent: 'Data has already been imported', + userEmailMissing: 'Some items in the CSV do not have an email', + }, + }, + + errors: { + forbidden: { + message: 'Forbidden', + }, + validation: { + message: 'An error occurred', + }, + searchQueryRequired: { + message: 'Search query is required', + }, + }, + + emails: { + invitation: { + subject: "You've been invited to {0}", + body: '...', + }, + emailAddressVerification: { + subject: 'Verify your email for {0}', + body: '...', + }, + passwordReset: { + subject: 'Reset your password for {0}', + body: '...', + }, + }, +}; + +module.exports = errors; +``` + +### Message Categories + +| Category | Purpose | Example Key | +|----------|---------|-------------| +| `app` | Application metadata | `app.title` | +| `auth` | Authentication errors | `auth.userNotFound` | +| `iam` | Identity/access management | `iam.errors.userAlreadyExists` | +| `importer` | CSV/file import errors | `importer.errors.invalidFileEmpty` | +| `errors` | Generic error messages | `errors.validation.message` | +| `emails` | Email subjects/bodies | `emails.invitation.subject` | + +--- + +### Helper Functions (`helpers.js`) + +Functions for resolving and formatting notification messages. + +```javascript +const _get = require('lodash/get'); +const errors = require('./list'); + +/** + * Format message with positional arguments + * @param {string} message - Message with {0}, {1}, etc. placeholders + * @param {Array} args - Arguments to substitute + * @returns {string} Formatted message + */ +function format(message, args) { + if (!message) { + return null; + } + + return message.replace(/{(\d+)}/g, function (match, number) { + return typeof args[number] != 'undefined' ? args[number] : match; + }); +} + +/** + * Check if a key exists in the notification catalog + * @param {string} key - Dot-notation path (e.g., 'auth.userNotFound') + * @returns {boolean} + */ +const isNotification = (key) => { + const message = _get(errors, key); + return !!message; +}; + +/** + * Get notification message by key with optional parameter substitution + * @param {string} key - Dot-notation path + * @param {...any} args - Values to substitute for {0}, {1}, etc. + * @returns {string} Resolved message or original key if not found + */ +const getNotification = (key, ...args) => { + const message = _get(errors, key); + + if (!message) { + return key; // Return raw key as fallback + } + + return format(message, args); +}; + +exports.getNotification = getNotification; +exports.isNotification = isNotification; +``` + +**Usage Examples:** + +```javascript +const { getNotification, isNotification } = require('./notifications/helpers'); + +// Simple lookup +getNotification('auth.userNotFound'); +// → "Sorry, we don't recognize your credentials" + +// With parameter substitution +getNotification('emails.invitation.subject', 'Tour Builder Platform'); +// → "You've been invited to Tour Builder Platform" + +// Check if key exists +isNotification('auth.userNotFound'); // → true +isNotification('custom.message'); // → false + +// Unknown key returns the key itself +getNotification('unknown.key'); +// → "unknown.key" +``` + +--- + +### Error Classes + +#### ValidationError (`errors/validation.js`) + +HTTP 400 Bad Request error for validation failures. + +```javascript +const { getNotification, isNotification } = require('../helpers'); + +module.exports = class ValidationError extends Error { + constructor(messageCode) { + let message; + + // Try to resolve from notification catalog + if (messageCode && isNotification(messageCode)) { + message = getNotification(messageCode); + } + + // Fallback to generic validation message + message = message || getNotification('errors.validation.message'); + + super(message); + this.code = 400; + } +}; +``` + +**Properties:** +- `message` - Human-readable error message +- `code` - HTTP status code (400) + +**Usage:** +```javascript +const ValidationError = require('./notifications/errors/validation'); + +// With catalog key +throw new ValidationError('auth.userNotFound'); +// → Error: "Sorry, we don't recognize your credentials" (code: 400) + +// With unknown key (uses fallback) +throw new ValidationError('unknown.error'); +// → Error: "An error occurred" (code: 400) + +// Without argument +throw new ValidationError(); +// → Error: "An error occurred" (code: 400) +``` + +#### ForbiddenError (`errors/forbidden.js`) + +HTTP 403 Forbidden error for authorization failures. + +```javascript +const { getNotification, isNotification } = require('../helpers'); + +module.exports = class ForbiddenError extends Error { + constructor(messageCode) { + let message; + + // Try to resolve from notification catalog + if (messageCode && isNotification(messageCode)) { + message = getNotification(messageCode); + } + + // Fallback to generic forbidden message + message = message || getNotification('errors.forbidden.message'); + + super(message); + this.code = 403; + } +}; +``` + +**Properties:** +- `message` - Human-readable error message +- `code` - HTTP status code (403) + +**Usage:** +```javascript +const ForbiddenError = require('./notifications/errors/forbidden'); + +// With catalog key +throw new ForbiddenError('auth.forbidden'); +// → Error: "Forbidden" (code: 403) + +// Without argument +throw new ForbiddenError(); +// → Error: "Forbidden" (code: 403) +``` + +--- + +## Complete Message Reference + +### Authentication Messages (`auth.*`) + +| Key | Message | Used In | +|-----|---------|---------| +| `auth.userDisabled` | "Your account is disabled" | signin | +| `auth.forbidden` | "Forbidden" | authorization failures | +| `auth.unauthorized` | "Unauthorized" | missing authentication | +| `auth.userNotFound` | "Sorry, we don't recognize your credentials" | signin | +| `auth.wrongPassword` | "Sorry, we don't recognize your credentials" | signin, password update | +| `auth.weakPassword` | "This password is too weak" | signup, password reset | +| `auth.emailAlreadyInUse` | "Email is already in use" | signup | +| `auth.invalidEmail` | "Please provide a valid email" | signup | +| `auth.userNotVerified` | "Sorry, your email has not been verified yet" | signin | +| `auth.passwordReset.invalidToken` | "Password reset link is invalid or has expired" | password reset | +| `auth.passwordReset.error` | "Email not recognized" | password reset request | +| `auth.passwordUpdate.samePassword` | "You can't use the same password..." | password update | +| `auth.emailAddressVerificationEmail.invalidToken` | "Email verification link is invalid or has expired" | email verification | +| `auth.emailAddressVerificationEmail.error` | "Email not recognized" | email verification | + +### IAM Messages (`iam.errors.*`) + +| Key | Message | Used In | +|-----|---------|---------| +| `iam.errors.userAlreadyExists` | "User with this email already exists" | user creation | +| `iam.errors.userNotFound` | "User not found" | user update/delete | +| `iam.errors.disablingHimself` | "You can't disable yourself" | user disable | +| `iam.errors.revokingOwnPermission` | "You can't revoke your own owner permission" | permission revoke | +| `iam.errors.deletingHimself` | "You can't delete yourself" | user delete | +| `iam.errors.emailRequired` | "Email is required" | user creation | +| `iam.errors.slugAlreadyExists` | "This slug is already in use by another project" | project create/update | +| `iam.errors.searchQueryRequired` | "Search query is required" | search | + +### Importer Messages (`importer.errors.*`) + +| Key | Message | Used In | +|-----|---------|---------| +| `importer.errors.invalidFileEmpty` | "The file is empty" | CSV import | +| `importer.errors.invalidFileExcel` | "Only excel (.xlsx) files are allowed" | file import | +| `importer.errors.invalidFileUpload` | "Invalid file..." | file import | +| `importer.errors.importHashRequired` | "Import hash is required" | bulk import | +| `importer.errors.importHashExistent` | "Data has already been imported" | duplicate import | +| `importer.errors.userEmailMissing` | "Some items in the CSV do not have an email" | user CSV import | + +### Generic Messages (`errors.*`) + +| Key | Message | Used In | +|-----|---------|---------| +| `errors.forbidden.message` | "Forbidden" | ForbiddenError default | +| `errors.validation.message` | "An error occurred" | ValidationError default | +| `errors.searchQueryRequired.message` | "Search query is required" | search validation | + +### Email Messages (`emails.*`) + +| Key | Message | Used In | +|-----|---------|---------| +| `emails.invitation.subject` | "You've been invited to {0}" | user invitation | +| `emails.emailAddressVerification.subject` | "Verify your email for {0}" | email verification | +| `emails.passwordReset.subject` | "Reset your password for {0}" | password reset | + +--- + +## Integration Points + +### Services Using Notifications + +| Service | Errors Used | Common Keys | +|---------|-------------|-------------| +| `auth.js` | ValidationError, ForbiddenError | auth.*, iam.* | +| `users.ts` | ValidationError | iam.errors.* | +| `projects.ts` | ValidationError | projectsNotFound | +| `roles.ts` | ValidationError | rolesNotFound, Public role permission validation | +| `search.js` | ValidationError | auth.unauthorized, auth.forbidden | +| `project_audio_tracks.ts` | ValidationError | project_audio_tracksNotFound | + +### Email Templates Using Notifications + +| Template | Helper Usage | +|----------|--------------| +| `passwordReset.js` | `getNotification('emails.passwordReset.subject')` | +| `addressVerification.js` | `getNotification('emails.emailAddressVerification.subject')` | +| `invitation.js` | `getNotification('emails.invitation.subject')` | + +### Middleware Using Notifications + +| Middleware | Error Used | Purpose | +|------------|------------|---------| +| `check-permissions.ts` | ValidationError | Permission denied responses | + +### Service Factory Using Notifications + +```javascript +// factories/service.factory.js +const ValidationError = require('../services/notifications/errors/validation'); + +// Used in update() when entity not found +if (!record) { + throw new ValidationError(`${entityName}NotFound`); +} +``` + +--- + +## Error Flow + +### Error Creation and Propagation + +``` +┌─────────────────────────────────────────────────────────────────────┐ +│ Service Layer │ +│ │ +│ const ValidationError = require('./notifications/errors/validation');│ +│ │ +│ if (!user) { │ +│ throw new ValidationError('auth.userNotFound'); │ +│ } │ +│ │ │ +└─────────────────┼───────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────────┐ +│ Notifications Module │ +│ │ +│ 1. isNotification('auth.userNotFound') → true │ +│ 2. getNotification('auth.userNotFound') │ +│ → lodash.get(errors, 'auth.userNotFound') │ +│ → "Sorry, we don't recognize your credentials" │ +│ 3. new Error(message) with code = 400 │ +│ │ │ +└─────────────────┼───────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────────┐ +│ Route Handler │ +│ │ +│ try { │ +│ await service.signin(email, password); │ +│ } catch (error) { │ +│ // Error propagates to error handler │ +│ } │ +│ │ │ +└─────────────────┼───────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────────┐ +│ Error Handler Middleware │ +│ (index.js) │ +│ │ +│ app.use((err, req, res, _next) => { │ +│ logger.error({ err, url: req.url }, 'Unhandled error'); │ +│ res.status(err.code || 500).json({ │ +│ message: err.message || 'Internal server error' │ +│ }); │ +│ }); │ +│ │ +│ → Response: { "message": "Sorry, we don't recognize your credentials" }│ +│ → Status: 400 │ +└─────────────────────────────────────────────────────────────────────┘ +``` + +### Message Resolution Flow + +``` +throw new ValidationError('auth.passwordReset.invalidToken') + │ + ▼ +┌─────────────────────────────────────┐ +│ isNotification(messageCode) │ +│ lodash.get(errors, key) │ +│ → "Password reset link..." │ +│ → true │ +└─────────────────┬───────────────────┘ + │ (exists) + ▼ +┌─────────────────────────────────────┐ +│ getNotification(messageCode) │ +│ → "Password reset link is │ +│ invalid or has expired" │ +└─────────────────┬───────────────────┘ + │ + ▼ +┌─────────────────────────────────────┐ +│ new Error(message) │ +│ this.message = "Password reset..."│ +│ this.code = 400 │ +└─────────────────────────────────────┘ +``` + +--- + +## Comparison: Notifications vs Utils Errors + +The backend has two error systems that serve different purposes: + +### `services/notifications/errors/` (Primary) + +- **Used by**: Services, middleware, service factory +- **Integration**: Uses notification catalog for i18n +- **Message resolution**: Dynamic via `getNotification()` +- **Properties**: `message`, `code` + +```javascript +// notifications/errors/validation.js +class ValidationError extends Error { + constructor(messageCode) { + const message = isNotification(messageCode) + ? getNotification(messageCode) + : getNotification('errors.validation.message'); + super(message); + this.code = 400; + } +} +``` + +### `utils/errors.js` (Utility) + +- **Used by**: Lower-level utilities +- **Integration**: Direct message strings +- **Message resolution**: Static, passed at construction +- **Properties**: `message`, `statusCode`, `details`, `isOperational` + +```javascript +// utils/errors.js +class AppError extends Error { + constructor(message, statusCode = 500, details = null) { + super(message); + this.statusCode = statusCode; + this.details = details; + this.isOperational = true; + } +} + +class ValidationError extends AppError { + constructor(message, details = null) { + super(message, 400, details); + } +} +``` + +### When to Use Which + +| Scenario | Use | +|----------|-----| +| Service business logic errors | `notifications/errors/ValidationError` | +| Authorization failures | `notifications/errors/ForbiddenError` | +| Email subject/body text | `getNotification()` | +| Low-level utility errors | `utils/errors.js` | +| Errors needing details object | `utils/errors.js` | + +--- + +## Adding New Messages + +### 1. Add to Catalog + +```typescript +// services/notifications/list.ts +const notifications = { + // ... existing ... + + projects: { + errors: { + notFound: 'Project not found', + slugTaken: 'A project with this slug already exists', + invalidSlug: 'Project slug must contain only letters, numbers, and hyphens', + }, + }, +}; +``` + +### 2. Use in Service + +```javascript +// services/projects.ts +const ValidationError = require('./notifications/errors/validation'); + +async function createProject(data) { + const existing = await findBySlug(data.slug); + if (existing) { + throw new ValidationError('projects.errors.slugTaken'); + } + // ... +} +``` + +### 3. With Parameters + +```javascript +// Add to catalog +const errors = { + projects: { + errors: { + tooManyPages: 'Project cannot have more than {0} pages', + }, + }, +}; + +// Use with parameter +const MAX_PAGES = 100; +if (pageCount > MAX_PAGES) { + const message = getNotification('projects.errors.tooManyPages', MAX_PAGES); + throw new ValidationError(message); +} +// → "Project cannot have more than 100 pages" +``` + +--- + +## Testing + +### Unit Testing Error Classes + +```javascript +describe('ValidationError', () => { + it('should resolve message from catalog', () => { + const error = new ValidationError('auth.userNotFound'); + expect(error.message).toBe("Sorry, we don't recognize your credentials"); + expect(error.code).toBe(400); + }); + + it('should use fallback for unknown keys', () => { + const error = new ValidationError('unknown.key'); + expect(error.message).toBe('An error occurred'); + expect(error.code).toBe(400); + }); + + it('should use fallback without argument', () => { + const error = new ValidationError(); + expect(error.message).toBe('An error occurred'); + }); +}); + +describe('ForbiddenError', () => { + it('should resolve message from catalog', () => { + const error = new ForbiddenError('auth.forbidden'); + expect(error.message).toBe('Forbidden'); + expect(error.code).toBe(403); + }); +}); +``` + +### Unit Testing Helpers + +```javascript +const { getNotification, isNotification } = require('./helpers'); + +describe('getNotification', () => { + it('should return message for valid key', () => { + expect(getNotification('auth.userNotFound')).toBe( + "Sorry, we don't recognize your credentials" + ); + }); + + it('should substitute parameters', () => { + expect(getNotification('emails.invitation.subject', 'My App')).toBe( + "You've been invited to My App" + ); + }); + + it('should return key for unknown path', () => { + expect(getNotification('unknown.path')).toBe('unknown.path'); + }); +}); + +describe('isNotification', () => { + it('should return true for existing keys', () => { + expect(isNotification('auth.userNotFound')).toBe(true); + }); + + it('should return false for unknown keys', () => { + expect(isNotification('unknown.key')).toBe(false); + }); +}); +``` + +--- + +## Dependencies + +| Package | Version | Purpose | +|---------|---------|---------| +| `lodash/get` | ^4.x | Deep object property access | + +--- + +## Best Practices + +### 1. Use Catalog Keys, Not Raw Strings + +```javascript +// Good - uses catalog for consistency +throw new ValidationError('auth.userNotFound'); + +// Avoid - raw strings bypass i18n +throw new ValidationError('User was not found'); +``` + +### 2. Group Related Messages + +```javascript +// Good - organized by domain +auth: { + passwordReset: { + invalidToken: '...', + error: '...', + } +} + +// Avoid - flat structure +authPasswordResetInvalidToken: '...', +authPasswordResetError: '...', +``` + +### 3. Security-Conscious Messages + +```javascript +// Good - doesn't reveal if email exists +userNotFound: "Sorry, we don't recognize your credentials", +wrongPassword: "Sorry, we don't recognize your credentials", + +// Avoid - reveals email existence +userNotFound: "No account with this email exists", +wrongPassword: "Password is incorrect", +``` + +### 4. Parameter Substitution for Dynamic Content + +```javascript +// Good - parameterized +subject: "You've been invited to {0}", +getNotification('emails.invitation.subject', appName); + +// Avoid - hardcoded +subject: "You've been invited to Tour Builder", +``` + +--- + +## Related Documentation + +- [Services Module](./services.md) - Service layer error handling +- [Auth Module](./auth.md) - Authentication errors +- [Email Module](./email.md) - Email message templates +- [Middleware Module](./middleware.md) - Permission error handling diff --git a/backend/docs/modules/routes.md b/backend/docs/modules/routes.md new file mode 100644 index 0000000..93fe596 --- /dev/null +++ b/backend/docs/modules/routes.md @@ -0,0 +1,757 @@ +# Backend Routes Module Documentation + +## Overview + +The Routes module defines all HTTP endpoints for the application. It uses a factory pattern for entity CRUD operations and custom routes for specialized functionality. + +**Directory:** `src/routes/` + +**Files (25 total):** +| File | Lines | Pattern | Description | +|------|-------|---------|-------------| +| `auth.ts` | 327 | Custom | Authentication endpoints | +| `file.ts` | 150 | Custom | File upload/download endpoints | +| `publish.ts` | 107 | Custom | Publishing workflow endpoints | +| `search.ts` | 64 | Custom | Global search endpoint | +| `runtime-context.ts` | 16 | Custom | Runtime context inspection | +| `projects.ts` | 46 | Hybrid | Projects CRUD + custom clone endpoint | +| `users.ts` | 64 | Hybrid | Users CRUD via factory + sanitized GET by ID | +| `tour_pages.ts` | 380 | Manual CRUD | Tour pages CRUD plus reorder, duplicate, reverse-video status | +| `roles.ts` | 141 | Factory | Roles CRUD via factory | +| `permissions.ts` | 188 | Factory | Permissions CRUD via factory | +| `assets.ts` | 155 | Factory | Assets CRUD via factory | +| `asset_variants.ts` | 147 | Factory | Asset variants CRUD via factory | +| `access_logs.ts` | 145 | Factory | Access logs CRUD via factory | +| `project_memberships.ts` | 145 | Factory | Project memberships CRUD via factory | +| `project_audio_tracks.ts` | 151 | Factory | Project audio tracks CRUD via factory | +| `global_transition_defaults.ts` | 145 | Custom | Runtime-readable global transition defaults | +| `global_ui_control_defaults.ts` | 79 | Custom | Runtime-readable global UI-control defaults | +| `project_transition_settings.ts` | 212 | Custom | Project/environment transition overrides | +| `project_ui_control_settings.ts` | 125 | Custom | Project/environment global UI-control overrides | +| `presigned_url_requests.ts` | 150 | Factory | Presigned URL requests CRUD via factory | +| `publish_events.ts` | 157 | Factory | Publish events CRUD via factory | +| `pwa_caches.ts` | 148 | Factory | PWA caches CRUD via factory | +| `element_type_defaults.ts` | 12 | Factory | Element type defaults via factory | +| `project_element_defaults.ts` | 92 | Hybrid | Project element defaults CRUD + custom (reset, diff) | + +--- + +## Architecture Diagram + +``` +┌──────────────────────────────────────────────────────────────────────┐ +│ Route Patterns │ +├──────────────────────────────────────────────────────────────────────┤ +│ │ +│ ┌─────────────────────────────────────────────────────────────────┐ │ +│ │ Factory Pattern (createEntityRouter) │ │ +│ │ │ │ +│ │ routes/roles.ts ──────────────┐ │ │ +│ │ routes/permissions.ts ────────┤ │ │ +│ │ routes/assets.ts ─────────────┼───▶ factories/router.factory.ts│ │ +│ │ routes/asset_variants.ts ─────┤ │ │ +│ │ routes/access_logs.ts ────────┤ Generates: │ │ +│ │ routes/pwa_caches.ts ─────────┤ • POST / │ │ +│ │ routes/publish_events.ts ─────┤ • POST /bulk-import │ │ +│ │ routes/element_type_defaults.ts • PUT /:id │ │ +│ │ routes/presigned_url_requests.ts • DELETE /:id │ │ +│ │ routes/project_memberships.ts • POST /deleteByIds │ │ +│ │ routes/project_audio_tracks.ts • GET / │ │ +│ │ ... (11 entities) • GET /count │ │ +│ │ • GET /autocomplete │ │ +│ │ • GET /:id │ │ +│ └─────────────────────────────────────────────────────────────────┘ │ +│ │ +│ ┌─────────────────────────────────────────────────────────────────┐ │ +│ │ Hybrid Pattern (Factory + Custom Routes) │ │ +│ │ │ │ +│ │ routes/projects.ts ──────────────────┐ │ │ +│ │ • Standard CRUD (manual) │ │ │ +│ │ + POST /:id/clone │ │ │ +│ │ + GET /:id/offline-manifest │ │ │ +│ │ │ │ │ +│ │ routes/project_element_defaults.ts ──┘ │ │ +│ │ • Standard CRUD (factory) │ │ +│ │ + POST /:id/reset │ │ +│ │ + GET /:id/diff │ │ +│ └─────────────────────────────────────────────────────────────────┘ │ +│ │ +│ ┌─────────────────────────────────────────────────────────────────┐ │ +│ │ Custom Pattern (Full Manual Implementation) │ │ +│ │ │ │ +│ │ routes/auth.ts ───────── Authentication flows │ │ +│ │ routes/file.ts ───────── File upload/download │ │ +│ │ routes/publish.ts ────── Publishing workflow │ │ +│ │ routes/search.ts ─────── Global search │ │ +│ │ routes/runtime-context.ts ─ Runtime inspection │ │ +│ └─────────────────────────────────────────────────────────────────┘ │ +└──────────────────────────────────────────────────────────────────────┘ +``` + +--- + +## Route Factory + +### factories/router.factory.ts + +Generates standardized CRUD routes for entities. + +```javascript +const router = createEntityRouter( + 'tour_pages', // Entity name + Tour_pagesService, // Service class + Tour_pagesDBApi, // Database API class + { + permissionEntity: 'tour_pages', // Permission entity name (optional) + csvFields: ['id', 'name'], // CSV export fields (optional) + validation: { // Request validation overrides (optional) + create: customCreateSchema, + update: customUpdateSchema, + }, + customRoutes: (router, Service, DBApi) => { // Custom routes (optional) + router.post('/custom', handler); + } + } +); +``` + +#### Generated Endpoints + +| Method | Path | Description | +|--------|------|-------------| +| `POST` | `/` | Create new item | +| `POST` | `/bulk-import` | Bulk import items | +| `PUT` | `/:id` | Update item by ID | +| `DELETE` | `/:id` | Delete item by ID | +| `POST` | `/deleteByIds` | Delete multiple items | +| `GET` | `/` | List items (with pagination, filters) | +| `GET` | `/count` | Count items matching filters | +| `GET` | `/autocomplete` | Autocomplete search | +| `GET` | `/:id` | Get single item by ID | + +#### Factory Features + +**Permission Checking:** +```javascript +router.use(checkCrudPermissions(permissionEntity)); +// Maps HTTP methods to permissions: +// GET → READ_ENTITY +// POST → CREATE_ENTITY +// PUT/PATCH → UPDATE_ENTITY +// DELETE → DELETE_ENTITY +``` + +**Request Validation:** +```typescript +import { validateRequest } from '../middlewares/validate-request.ts'; +import { crud as crudSchemas } from '../validators/request-schemas.ts'; + +router.get( + '/:id', + validateRequest(crudSchemas.findOne), + wrapAsync(async (req, res) => { + const payload = await DBApi.findBy({ id: req.params.id }); + res.status(200).send(payload); + }), +); +``` + +Factory CRUD routes validate request bodies, params, and common query controls before service/DB calls. List and count routes validate `limit`, `page`, `field`, `sort`, and `filetype` while keeping existing entity filter query parameters. Entity routers can pass `validation` overrides for stricter contracts, as `users` and `projects` do. + +**CSV Export:** +```javascript +// GET /?filetype=csv +if (filetype === 'csv') { + const csv = parse(payload.rows, { fields }); + res.status(200).attachment('export.csv').send(csv); +} +``` + +**Runtime Context:** +```javascript +const runtimeContext = req.runtimeContext; +const payload = await DBApi.findAll(req.query, { + currentUser, + runtimeContext, +}); +``` + +--- + +## Route Mounting (index.js) + +Routes are mounted with authentication and rate limiting: + +```javascript +// No auth required +app.use('/api/auth', authRoutes); +app.use('/api/runtime-context', runtimeContextRoutes); +app.get('/api/health', healthHandler); + +// File routes (before body parser) +app.use('/api/file/download', downloadLimiter); +app.use('/api/file/presign', downloadLimiter); +app.use('/api/file/upload', uploadLimiter); +app.use('/api/file', fileRoutes); + +// JWT auth required +app.use('/api/users', jwtAuth, usersRoutes); +app.use('/api/roles', jwtAuth, rolesRoutes); +app.use('/api/permissions', jwtAuth, permissionsRoutes); +app.use('/api/project_memberships', jwtAuth, project_membershipsRoutes); +app.use('/api/assets', jwtAuth, assetsRoutes); +app.use('/api/asset_variants', jwtAuth, asset_variantsRoutes); +app.use('/api/presigned_url_requests', jwtAuth, presigned_url_requestsRoutes); +app.use('/api/publish_events', jwtAuth, publish_eventsRoutes); +app.use('/api/pwa_caches', jwtAuth, pwa_cachesRoutes); +app.use('/api/access_logs', jwtAuth, access_logsRoutes); +app.use('/api/element-type-defaults', jwtAuth, element_type_defaultsRoutes); +app.use('/api/project-element-defaults', jwtAuth, project_element_defaultsRoutes); +app.use('/api/publish', jwtAuth, publishRoutes); + +// JWT + Rate limiting +app.use('/api/search', jwtAuth, searchLimiter, searchRoutes); + +// Runtime public access (production environment) +mountRuntimeEntityRoute('/api/projects', 'projects', projectsRoutes); +mountRuntimeEntityRoute('/api/tour_pages', 'tour_pages', tour_pagesRoutes); +mountRuntimeEntityRoute('/api/project_audio_tracks', 'project_audio_tracks', ...); +``` + +--- + +## Custom Routes Detail + +### 1. auth.ts (327 lines) + +Authentication and account management. + +| Method | Path | Auth | Description | +|--------|------|------|-------------| +| POST | `/signin/local` | No | Email/password login | +| POST | `/signup` | No | Register new user | +| GET | `/me` | JWT | Get current user | +| PUT | `/password-reset` | No | Reset password with token | +| PUT | `/password-update` | JWT | Change password | +| PUT | `/profile` | JWT | Update user profile | +| PUT | `/verify-email` | No | Verify email with token | +| POST | `/send-email-address-verification-email` | JWT | Resend verification | +| POST | `/send-password-reset-email` | No | Send reset email | +| GET | `/email-configured` | No | Check email config | +| GET | `/signin/google` | No | Google OAuth start | +| GET | `/signin/google/callback` | No | Google OAuth callback | +| GET | `/signin/microsoft` | No | Microsoft OAuth start | +| GET | `/signin/microsoft/callback` | No | Microsoft OAuth callback | + +`auth.ts` is a typed ESM route boundary. It uses reusable request body/query contracts from `backend/src/types/auth-routes.ts`, the typed `backend/src/services/auth.ts` service, and the shared Passport helpers in `backend/src/auth/passport-middleware.ts`. + +--- + +### 2. file.ts (150 lines) + +File upload and download operations. + +`file.ts` is a typed ESM route boundary. It calls the typed unified storage facade in `backend/src/services/file.ts`, while provider contracts and request/response shapes live in reusable `backend/src/types/file.ts` contracts. S3 and GCloud use official SDK-provided types instead of local SDK declarations. + +| Method | Path | Auth | Description | +|--------|------|------|-------------| +| GET | `/download` | No | Download file by privateUrl | +| POST | `/presign` | No | Generate presigned URLs (max 50) | +| POST | `/upload/:table/:field` | JWT | Legacy single file upload | +| POST | `/upload-sessions/init` | JWT | Initialize chunked upload | +| GET | `/upload-sessions/:sessionId` | JWT | Get upload session status | +| PUT | `/upload-sessions/:sessionId/chunks/:chunkIndex` | JWT | Upload chunk | +| POST | `/upload-sessions/:sessionId/finalize` | JWT | Finalize chunked upload | + +**Presigned URLs Request:** +```json +{ + "urls": ["assets/image.jpg", "assets/video.mp4"] +} +``` + +**Presigned URLs Response:** +```json +{ + "presignedUrls": { + "assets/image.jpg": "https://s3.../assets/image.jpg?X-Amz-...", + "assets/video.mp4": "https://s3.../assets/video.mp4?X-Amz-..." + } +} +``` + +--- + +### 3. publish.ts (107 lines) + +Publishing workflow for Dev → Stage → Production. + +| Method | Path | Auth | Description | +|--------|------|------|-------------| +| POST | `/` | JWT | Publish stage to production | +| POST | `/publish` | JWT | Alias for publish | +| POST | `/save-to-stage` | JWT | Save dev to stage | + +**Publish Request:** +```json +{ + "projectId": "uuid", + "title": "Release 1.0", + "description": "Initial release" +} +``` + +**Save to Stage Request:** +```json +{ + "projectId": "uuid" +} +``` + +--- + +### 4. search.ts (64 lines) + +Global full-text search across entities. + +| Method | Path | Auth | Description | +|--------|------|------|-------------| +| POST | `/` | JWT | Search across all entities | + +**Request:** +```json +{ + "searchQuery": "my search term" +} +``` + +**Response:** +```json +{ + "users": [...], + "projects": [...], + "tour_pages": [...] +} +``` + +### 5. runtime-context.ts (16 lines) + +Runtime context inspection for debugging. + +| Method | Path | Auth | Description | +|--------|------|------|-------------| +| GET | `/` | No | Get current runtime context | + +**Response:** +```json +{ + "mode": "admin", + "projectSlug": null, + "headerEnvironment": "production", + "headerProjectSlug": "my-tour" +} +``` + +--- + +## Entity Routes Detail + +### projects.ts (46 lines) + +Projects with factory CRUD plus a clone endpoint. + +**Standard CRUD Endpoints:** +| Method | Path | Description | +|--------|------|-------------| +| POST | `/` | Create project | +| POST | `/bulk-import` | Bulk import projects | +| PUT | `/:id` | Update project | +| DELETE | `/:id` | Delete project | +| POST | `/deleteByIds` | Delete multiple projects | +| GET | `/` | List projects | +| GET | `/count` | Count projects | +| GET | `/autocomplete` | Autocomplete search | +| GET | `/:id` | Get project by ID | + +**Custom Endpoints:** +| Method | Path | Description | +|--------|------|-------------| +| POST | `/:id/clone` | Clone project with all pages | +--- + +### tour_pages.ts (380 lines) + +Typed manual CRUD route for tour pages. In addition to standard create/update/delete/list/count/autocomplete/find-by-id operations, it owns page reorder, dev-page duplication, CSV export, and reverse-video status endpoints. The route uses reusable contracts from `backend/src/types/tour-pages.ts`. + +**Schema Fields:** +- `source_key`, `name`, `slug` +- `background_image_url`, `background_video_url`, `background_audio_url` +- `ui_schema_json`, `sort_order` + +**Endpoints:** +| Method | Path | Description | +|--------|------|-------------| +| POST | `/` | Create page | +| POST | `/bulk-import` | Bulk import pages | +| POST | `/reorder` | Reorder dev pages in a project/environment | +| POST | `/:id/duplicate` | Duplicate a dev page | +| PUT | `/:id` | Update page | +| DELETE | `/:id` | Remove page | +| POST | `/deleteByIds` | Remove multiple pages | +| GET | `/` | List pages with reverse video URL population and optional CSV export | +| POST | `/reverse-video-status` | Check generated reverse-video variants | +| GET | `/count` | Count pages | +| GET | `/autocomplete` | Page autocomplete | +| GET | `/:id` | Get one page with reverse video URL population | +- `POST /api/tour_pages/reorder` accepts `{ data: { projectId, environment, + orderedPageIds } }`, validates a complete page list for the project/dev + environment, and updates only `sort_order`. +- `POST /api/tour_pages/:id/duplicate` accepts `{ data: { projectId, + environment, name, slug } }`, duplicates a dev page into a new independent dev + page, appends it to the project order, deep-copies `ui_schema_json`, and + regenerates inline element IDs. +- Constructor page deletion uses the standard `DELETE /api/tour_pages/:id` + route after a frontend confirmation modal; no separate delete endpoint is + needed. +- Custom routes are protected by Passport JWT like other tour page mutations and + are implemented before `/:id` where needed so custom paths are not treated as + entity IDs. +- Non-dev environments are rejected for reorder and duplicate; stage and + production changes only flow through Save to Stage and Publish. + +--- + +### element_type_defaults.ts (12 lines) + +Minimal factory usage with permission override. + +```javascript +module.exports = createEntityRouter( + 'element_type_defaults', + Element_type_defaultsService, + Element_type_defaultsDBApi, + { + permissionEntity: 'page_elements', // Uses PAGE_ELEMENTS permissions + }, +); +``` + +**URL Aliases:** +- `/api/element-type-defaults` (primary) +- `/api/ui-elements` (backwards compatibility) + +--- + +### project_element_defaults.ts (92 lines) + +Factory-generated routes plus custom endpoints for resetting and comparing defaults. + +```javascript +const baseRouter = createEntityRouter( + 'project_element_defaults', + Project_element_defaultsService, + Project_element_defaultsDBApi, + { + permissionEntity: 'page_elements', + }, +); + +// Add custom endpoints +baseRouter.post('/:id/reset', ...); +baseRouter.get('/:id/diff', ...); +``` + +**Standard CRUD Endpoints:** (via factory) +| Method | Path | Description | +|--------|------|-------------| +| POST | `/` | Create project element default | +| PUT | `/:id` | Update project element default | +| DELETE | `/:id` | Delete project element default | +| GET | `/` | List project element defaults | +| GET | `/:id` | Get project element default by ID | + +**Custom Endpoints:** +| Method | Path | Description | +|--------|------|-------------| +| POST | `/:id/reset` | Reset project element default to global | +| GET | `/:id/diff` | Get diff from global element type default | + +**URL Alias:** +- `/api/project-element-defaults` (primary) + +--- + +## Request/Response Patterns + +### List Endpoint (GET /) + +**Query Parameters:** +| Param | Type | Description | +|-------|------|-------------| +| `page` | number | Page number (0-indexed) | +| `limit` | number | Items per page | +| `field` | string | Sort field | +| `sort` | string | Sort direction (`asc`/`desc`) | +| `filetype` | string | Export format (`csv`) | +| `[fieldName]` | string | Filter by field value | +| `[fieldName]Range` | array | Filter by range `[start, end]` | + +**Response:** +```json +{ + "rows": [...], + "count": 150 +} +``` + +### Create Endpoint (POST /) + +**Request:** +```json +{ + "data": { + "name": "New Item", + "field1": "value1" + } +} +``` + +**Response:** +```json +{ + "id": "uuid", + "name": "New Item", + "field1": "value1", + "createdAt": "2024-01-01T00:00:00.000Z" +} +``` + +### Update Endpoint (PUT /:id) + +**Request:** +```json +{ + "id": "uuid", + "data": { + "name": "Updated Name" + } +} +``` + +**Response:** +```json +true +``` + +### Delete Endpoints + +**Single Delete (DELETE /:id):** +```json +true +``` + +**Bulk Delete (POST /deleteByIds):** +```json +{ + "data": ["uuid1", "uuid2", "uuid3"] +} +``` + +--- + +## Swagger Documentation + +Swagger UI is served at `GET /api-docs`. The OpenAPI document is maintained in +`backend/src/openapi/document.ts`, not in route-local JSDoc comments. + +```javascript +const specs = createOpenApiDocument({ + serverUrl: config.server.swaggerServerUrl, +}); +``` + +The OpenAPI module contains: + +- Shared schemas for auth, users, projects, tour pages, assets, publishing, + runtime access, file upload, UI controls, transition settings, and audit + entities. +- A reusable factory CRUD path generator covering `POST /`, `POST /bulk-import`, + `PUT /:id`, `DELETE /:id`, `POST /deleteByIds`, `GET /`, `GET /count`, + `GET /autocomplete`, and `GET /:id`. +- Explicit path definitions for custom routes such as auth flows, publishing, + file upload/download, runtime access/context, project clone, tour page reorder + and duplicate, transition/UI-control settings, and project element default + reset/diff. + +When adding routes, update `backend/src/openapi/document.ts` alongside the route +implementation. For factory-backed entities, add a schema and `CrudResource` +entry instead of copying per-route Swagger boilerplate. + +**Access Swagger UI:** `http://localhost:3000/api-docs` in local development. + +On the standard VM, the backend runs in `NODE_ENV=dev_stage` on port `3000`; +Apache serves the public domain on port `80`. See +[`deployment-vm.md`](../../../documentation/deployment-vm.md) for VM health checks. + +--- + +## Error Handling + +### wrapAsync Helper + +All async route handlers use `wrapAsync` for error propagation: + +```javascript +router.post( + '/', + wrapAsync(async (req, res) => { + // Errors thrown here are caught and passed to error handler + const payload = await Service.create({ + data: req.body.data, + currentUser: req.currentUser, + runtimeContext: req.runtimeContext, + }); + res.status(200).send(payload); + }), +); +``` + +### commonErrorHandler + +Each route file includes error handler: + +```javascript +router.use('/', require('../helpers').commonErrorHandler); +``` + +**Error Response Mapping:** +| Status Code | Description | +|-------------|-------------| +| 400 | Bad Request | +| 401 | Unauthorized | +| 403 | Forbidden | +| 404 | Not Found | +| 409 | Conflict | +| 422 | Unprocessable Entity | +| 500 | Internal Server Error | + +--- + +## Route Categories + +### Public Routes (No Auth) + +| Route | Description | +|-------|-------------| +| `GET /api/health` | Health check | +| `POST /api/auth/signin/local` | Login | +| `GET /api/auth/signin/google` | Google OAuth | +| `GET /api/auth/signin/microsoft` | Microsoft OAuth | +| `GET /api/file/download` | File download | +| `POST /api/file/presign` | Generate presigned URLs | +| `GET /api/runtime-context` | Runtime context | + +### Runtime Public Routes (Production Environment) + +| Route | Description | +|-------|-------------| +| `GET /api/projects` | List projects (sanitized) | +| `GET /api/tour_pages` | List tour pages (sanitized) | +| `GET /api/project_audio_tracks` | List audio tracks (sanitized) | + +### Authenticated Routes (JWT Required) + +All other routes require JWT authentication via: +```javascript +app.use('/api/users', jwtAuth, usersRoutes); +``` + +### Rate Limited Routes + +| Route | Limiter | Config | +|-------|---------|--------| +| `/api/auth/signin/local` | authLimiter | 10/15min | +| `/api/auth/send-password-reset-email` | passwordResetLimiter | 5/hour | +| `/api/file/upload*` | uploadLimiter | 10/min | +| `/api/file/download`, `/presign` | downloadLimiter | 200/min | +| `/api/search` | searchLimiter | 30/min | + +## Dependencies + +| Package | Purpose | +|---------|---------| +| `express` | Router and middleware | +| `json2csv` | CSV export functionality | +| `passport` | JWT authentication | +| `body-parser` | JSON body parsing | + +--- + +## Testing + +### Test Entity CRUD + +```bash +# Create +curl -X POST http://localhost:3000/api/projects \ + -H "Authorization: Bearer " \ + -H "Content-Type: application/json" \ + -d '{"data": {"name": "Test Project", "slug": "test"}}' + +# List +curl http://localhost:3000/api/projects \ + -H "Authorization: Bearer " + +# Get by ID +curl http://localhost:3000/api/projects/ \ + -H "Authorization: Bearer " + +# Update +curl -X PUT http://localhost:3000/api/projects/ \ + -H "Authorization: Bearer " \ + -H "Content-Type: application/json" \ + -d '{"id": "", "data": {"name": "Updated Name"}}' + +# Delete +curl -X DELETE http://localhost:3000/api/projects/ \ + -H "Authorization: Bearer " +``` + +### Test Public Runtime + +```bash +curl http://localhost:3000/api/projects \ + -H "X-Runtime-Environment: production" +``` + +### Test Presigned URLs + +```bash +curl -X POST http://localhost:3000/api/file/presign \ + -H "Content-Type: application/json" \ + -d '{"urls": ["assets/test.jpg"]}' +``` + +--- + +## Summary + +The Routes module provides: + +1. **Factory Pattern** - 12 entities with standardized CRUD (createEntityRouter) +2. **Hybrid Pattern** - 2 entities with factory CRUD + custom endpoints (projects, project_element_defaults) +3. **Custom Routes** - 7 specialized route files for auth, files, search, etc. +4. **Swagger Documentation** - centralized OpenAPI document for all endpoints +5. **Permission Checking** - RBAC via checkCrudPermissions middleware +6. **Runtime Public Access** - Production environment public read access +7. **Rate Limiting** - Per-endpoint rate limiting +8. **CSV Export** - Export capability via `?filetype=csv` +9. **Request Validation** - Joi schemas via validateRequest before service calls +10. **Error Handling** - Centralized via wrapAsync and commonErrorHandler + +**Route Statistics:** +- 26 route files +- ~50+ unique endpoints +- 11 factory-generated entity routers +- 3 hybrid entity routers (factory + custom) +- 7 custom route implementations diff --git a/backend/docs/modules/services.md b/backend/docs/modules/services.md new file mode 100644 index 0000000..e6115e8 --- /dev/null +++ b/backend/docs/modules/services.md @@ -0,0 +1,1142 @@ +# Backend Services Module + +## Overview + +The Services module implements the **business logic layer** of the backend application. Services sit between routes (controllers) and the database API layer, encapsulating complex operations, transaction management, and cross-cutting concerns. + +**Location:** `backend/src/services/` + +**Total Files:** 37 (24 root services + 13 subdirectory files) + +--- + +## Architecture Diagram + +``` +┌─────────────────────────────────────────────────────────────────────┐ +│ Routes Layer │ +│ (HTTP Request Handling) │ +└─────────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────────┐ +│ Services Layer │ +│ ┌───────────────────────────────────────────────────────────────┐ │ +│ │ Service Categories │ │ +│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │ +│ │ │ Factory │ │ Custom │ │ Specialized │ │ │ +│ │ │ Services │ │ Services │ │ Modules │ │ │ +│ │ │ (9 files) │ │ (12 files) │ │ (3 dirs) │ │ │ +│ │ └─────────────┘ └─────────────┘ └─────────────┘ │ │ +│ └───────────────────────────────────────────────────────────────┘ │ +└─────────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────────┐ +│ Database API Layer │ +│ (Sequelize ORM Operations) │ +└─────────────────────────────────────────────────────────────────────┘ +``` + +--- + +## Service Categories + +### 1. Factory-Generated Services (Simple CRUD) + +Generated using `createEntityService()` from `factories/service.factory.ts`. These provide standardized CRUD operations with transaction handling. + +| Service | File | Entity | LOC | +|---------|------|--------|-----| +| tour_pages | `tour_pages.ts` | Tour Pages (includes reverse video generation) | ~1,300 | +| permissions | `permissions.ts` | Permissions | 6 | +| asset_variants | `asset_variants.ts` | Asset Variants | 6 | +| presigned_url_requests | `presigned_url_requests.ts` | Presigned URL Requests | 6 | +| publish_events | `publish_events.ts` | Publish Events | 6 | +| pwa_caches | `pwa_caches.ts` | PWA Caches | 6 | +| access_logs | `access_logs.ts` | Access Logs | 6 | +| element_type_defaults | `element_type_defaults.ts` | Element Type Defaults | 6 | +| project_memberships | `project_memberships.ts` | Project Memberships | 6 | +| global_transition_defaults | `global_transition_defaults.ts` | Global transition defaults | 6 | + +**Example - Factory Service:** +```javascript +// permissions.ts +import PermissionsDBApi from '../db/api/permissions.ts'; +import { createEntityService } from '../factories/service.factory.ts'; + +export default createEntityService(PermissionsDBApi, { + entityName: 'permissions', +}); +``` + +### 2. Custom Services (Business Logic) + +Services with domain-specific business logic beyond simple CRUD. + +| Service | File | Purpose | LOC | +|---------|------|---------|-----| +| assets | `assets.ts` | Asset management, MIME validation, embed URL validation, stored media metadata probing | ~300 | +| auth | `auth.ts` | Authentication, password reset, email verification | ~210 | +| users | `users.ts` | User management, invitation emails, Public viewer grants | ~350 | +| projects | `projects.ts` | Project cloning, slug generation, slug uniqueness validation | ~680 | +| roles | `roles.ts` | Role management, permission assignment, CSV import, Public-role hardening | ~170 | +| file | `file.ts` | Multi-provider file storage, downloadToBuffer, uploadBuffer, S3/GCloud circuit breaker for processing paths | ~1,600 | +| publish | `publish.ts` | Dev→Stage→Production publishing | ~400 | +| search | `search.ts` | Global full-text search | 178 | +| pwa_manifest | `pwa_manifest.js` | PWA offline manifest generation | 315 | +| project_audio_tracks | `project_audio_tracks.ts` | Audio track management | 117 | +| project_transition_settings | `project_transition_settings.ts` | Environment-aware transition settings | 209 | +| project_element_defaults | `project_element_defaults.ts` | Element defaults with reset/diff | 34 | +| global_ui_control_defaults | `global_ui_control_defaults.ts` | Global defaults CRUD service for system controls | 6 | +| project_ui_control_settings | `project_ui_control_settings.ts` | Transactional find/upsert/delete for project UI-control overrides | 51 | +| videoProcessing | `videoProcessing.ts` | FFmpeg video reversal for transition videos with single-worker queue, `-threads 1`, hard timeout, metadata logs, and circuit breaker | ~240 | + +### 3. Specialized Module Directories + +``` +services/ +├── file/ # Storage providers (Strategy Pattern) +│ ├── BaseStorageProvider.ts # Abstract interface +│ ├── S3StorageProvider.ts # AWS S3 implementation +│ ├── LocalStorageProvider.ts # Local filesystem +│ ├── UploadSessionManager.ts # Chunked uploads +│ └── index.js # Provider factory (27 LOC) +│ +├── email/ # Email sending +│ ├── index.js # EmailSender class (44 LOC) +│ └── list/ +│ ├── passwordReset.ts # Password reset email +│ ├── addressVerification.ts # Email verification +│ └── invitation.ts # User invitation email +│ +└── notifications/ # Error handling & i18n + ├── helpers.js # getNotification helper (30 LOC) + ├── list.js # Error message catalog (100 LOC) + └── errors/ + ├── validation.js # ValidationError class (16 LOC) + └── forbidden.js # ForbiddenError class (16 LOC) +``` + +--- + +## Service Factory Pattern + +### `factories/service.factory.ts` + +Generates standardized service classes with transaction handling. + +```javascript +function createEntityService(DBApi, options = {}) { + const entityName = options.entityName || 'Entity'; + + return class GenericService { + // Create with transaction + static async create({ data, currentUser, transaction, runtimeContext }) { + const transaction = await db.sequelize.transaction(); + try { + const record = await DBApi.create({ data, currentUser, transaction, runtimeContext }); + await transaction.commit(); + return record; + } catch (error) { + await transaction.rollback(); + throw error; + } + } + + // Bulk import from CSV + static async bulkImport(req, res) { ... } + + // Update with existence check + static async update({ id, data, currentUser, transaction, runtimeContext }) { ... } + + // Delete multiple by IDs + static async deleteByIds({ ids, currentUser, transaction, runtimeContext }) { ... } + + // Soft delete single + static async remove({ id, currentUser, transaction, runtimeContext }) { ... } + }; +} +``` + +**Generated Methods:** + +| Method | Signature | Description | +|--------|-----------|-------------| +| `create` | `(data, currentUser) → record` | Create with transaction | +| `bulkImport` | `(req, res) → void` | CSV import with validation | +| `update` | `(data, id, currentUser) → record` | Update with existence check | +| `deleteByIds` | `(ids, currentUser) → void` | Bulk soft delete | +| `remove` | `(id, currentUser) → void` | Single soft delete | + +--- + +## Core Services Detail + +### Tour Pages Service (`tour_pages.ts`) + +Extends the factory-generated CRUD service with page-specific operations. + +**Reorder operation:** +- `TourPagesService.reorder(data, currentUser)` accepts `projectId`, + `environment`, and `orderedPageIds`. +- Reordering is allowed only for `environment='dev'`; this preserves the + publishing model where stage and production are derived environments. +- The service loads all pages for the project/dev environment inside a + transaction, verifies that the ordered list contains every page exactly once, + and then updates `sort_order` sequentially. +- The operation updates only `sort_order`; it does not change page content, + slugs, backgrounds, `ui_schema_json`, navigation, transitions, or media. +- Stage receives the new order after Save to Stage; production receives it after + Publish. + +### Auth Service (`auth.ts`) + +Handles authentication, password management, and email verification. + +```typescript +class Auth { + // User login with bcrypt verification + static async signin(email, password) → JWT + + // Send email verification link + static async sendEmailAddressVerificationEmail(email, host) + + // Send password reset or invitation email + static async sendPasswordResetEmail(email, type, host) + + // Verify email from token + static async verifyEmail(token, options) → boolean + + // Update password (requires current password) + static async passwordUpdate(currentPassword, newPassword, options) + + // Reset password from token + static async passwordReset(token, password, options) + + // Update user profile + static async updateProfile(data, currentUser) +} +``` + +**Security Features:** +- bcrypt password hashing (`config.bcrypt.saltRounds`) +- JWT token generation via `helpers.jwtSign()` +- Email verification required for login +- Password reset tokens with expiration + +### Assets Service (`assets.ts`) + +Extended factory service with strict reusable TypeScript contracts, MIME type validation for asset uploads, embed URL validation, and stored audio/video metadata probing. + +```typescript +class AssetsService extends BaseService { + // Create asset with MIME type validation + static async create({ data, currentUser, transaction, runtimeContext }) + + // Update asset with MIME type validation + static async update({ id, data, currentUser, transaction, runtimeContext }) +} +``` + +**MIME Type Validation:** + +| Asset Type | Valid MIME Prefixes | Description | +|------------|---------------------|-------------| +| `image` | `image/` | JPEG, PNG, GIF, WebP, SVG, etc. | +| `video` | `video/` | MP4, WebM, MOV, etc. | +| `audio` | `audio/` | MP3, WAV, OGG, etc. | +| `embed` | n/a | MIME validation skipped; HTTPS embed URL domain is validated | + +**Validation Rules:** +- `asset_type` and `mime_type` must be consistent +- If `asset_type` is `image`, `mime_type` must start with `image/` +- If `asset_type` is `video`, `mime_type` must start with `video/` +- If `asset_type` is `audio`, `mime_type` must start with `audio/` +- Asset types not in the validation list (e.g., `file`) skip validation +- Missing `mime_type` is allowed (browser may not always send it) + +**Error Response:** +```javascript +throw new ValidationError( + `Invalid file type for ${assetType}. Expected ${patterns.description}, got "${mimeType}"` +); +``` + +--- + +### File Service (`file.js`) + +Unified file storage using Strategy Pattern for multiple backends. Features comprehensive error handling, +AbortController support for client disconnect handling, path validation for security, and structured Pino logging. + +```javascript +// Provider auto-detection +const getFileStorageProvider = () → 's3' | 'gcloud' | 'local' + +// Core operations (with AbortController support for S3) +const uploadFile = async (folder, req, res) → { url } +const downloadFile = async (req, res) → stream // Aborts on client disconnect +const deleteFile = async (privateUrl, { throwOnError }) → { success, error? } + +// Server-side file copy (S3 uses CopyObjectCommand, Local uses fs.copyFile) +const copyFile = async (sourceKey, destKey, options) → { url } | { key } +const copyFilesParallel = async (copies, options) → { succeeded, failed } + +// Chunked upload session management +const initUploadSession = async (req, res) → { sessionId, totalChunks } +const getUploadSession = async (req, res) → { status, uploadedChunks } +const uploadChunk = async (req, res) → { chunkIndex, uploadedChunks } +const finalizeUploadSession = async (req, res) → { url, privateUrl } + +// Presigned URL generation (S3 only, with path validation) +const generatePresignedUrls = async (urls) → { [url]: presignedUrl } + +// Utilities (exported for route layer) +const isValidPath = (urlPath) → boolean // Path traversal protection +const createErrorResponse = (message, code, details) → { message, code?, details? } +const getS3ErrorStatusCode = (error) → number // HTTP status code mapping +``` + +**Server-Side File Copy (S3 Native):** + +The `copyFile()` function uses provider-native copy operations for optimal performance: + +| Provider | Implementation | Performance | +|----------|----------------|-------------| +| S3 | `CopyObjectCommand` (server-side) | 15x faster, zero memory | +| Local | `fs.promises.copyFile` | Kernel-level copy | +| GCloud | Download + Upload (fallback) | Legacy behavior | + +```javascript +// Single file copy +const copyFile = async (sourceKey, destKey, { contentType }) → { url } + +// Parallel batch copy with concurrency control +const copyFilesParallel = async (copies, { concurrency = 10, continueOnError = true }) + → { succeeded: [{ sourceKey, destKey }], failed: [{ sourceKey, error }] } +``` + +**Benefits over download-then-upload:** +- **15x faster**: Server-side copy, no data through backend +- **Zero memory**: No file buffering in Node.js +- **No timeouts**: Works for large files (>100MB) +- **Reduced bandwidth**: No double network transfer + +**Error Response Format:** +```javascript +// Standardized across all file endpoints +{ + message: 'Human-readable error message', + code: 'ERROR_CODE', // For programmatic handling + details: { ... } // Optional additional context +} +``` + +**Request Cancellation:** +Downloads automatically abort S3 requests when client disconnects, preventing wasted bandwidth and server resources. + +**Provider Selection:** + +| Priority | Provider | Detection | +|----------|----------|-----------| +| 1 | `config.fileStorage.provider` | Validated `FILE_STORAGE_PROVIDER` override | +| 2 | S3 | `S3_BUCKET` + `S3_REGION` + credentials | +| 3 | GCloud | Validated `GC_PROJECT_ID` + `GC_CLIENT_EMAIL` + `GC_PRIVATE_KEY` | +| 4 | Local | Default fallback | + +External S3/GCloud operations used by processing paths are protected by the +shared file-storage circuit breaker. For S3, only retryable SDK/network errors +count toward opening the breaker; expected 4xx errors do not. + +**Chunked Upload Flow:** + +``` +1. POST /api/file/upload-sessions/init + ← { sessionId, totalChunks } + +2. PUT /api/file/upload-sessions/:sessionId/chunks/:chunkIndex + ← { uploadedChunks: N } + +3. POST /api/file/upload-sessions/:sessionId/finalize + ← { url, privateUrl } +``` + +**S3 Local Cache (Atomic Writes):** + +When S3 is the storage provider, downloads are cached locally to reduce S3 requests and improve latency. The cache uses **atomic writes** to prevent race conditions: + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ Request A Request B (concurrent) │ +├─────────────────────────────────────────────────────────────────┤ +│ 1. Check cache → miss 1. Check cache → miss │ +│ 2. Create .downloading 2. See .downloading → skip cache │ +│ 3. Stream to .tmp file 3. Stream directly from S3 │ +│ 4. Verify size matches 4. Complete │ +│ 5. Rename .tmp → final ↓ │ +│ 6. Delete .downloading Request C (later) │ +│ ↓ 1. Check cache → hit │ +│ 2. Serve from cache │ +└─────────────────────────────────────────────────────────────────┘ +``` + +**Cache validation:** +- File exists AND age < `S3_CACHE_MAX_AGE` (default: 1 hour) +- No `.downloading` marker file (indicates download in progress) +- Size matches `Content-Length` header (verified before rename) + +**Why atomic writes:** +Without atomic writes, concurrent requests could serve truncated cache files: +1. Request A starts downloading 12MB file, writes to cache +2. After 2MB written, Request B checks cache - file exists, age valid +3. Request B serves 2MB truncated file → **corrupted video/image** + +The atomic write pattern (write to `.tmp`, verify size, rename) prevents this. + +### Publish Service (`publish.ts`) + +Three-tier content environment publishing workflow. + +```javascript +module.exports = class PublishService { + // Acquire project lock and run callback + static async withProjectPublishLock(projectId, callback) + + // Copy stage content to production (blocking) + static async publishToProduction(projectId, currentUser, title, description) + + // Copy dev content to stage (non-blocking, returns immediately) + static async saveToStage(projectId, currentUser) + + // Generic environment copy + static async copyEnvironment(projectId, fromEnv, toEnv, currentUser, transaction) +} +``` + +**Non-Blocking vs Blocking:** +- `saveToStage()` - Returns immediately, copy runs in background via `setImmediate()` +- `publishToProduction()` - Waits for entire copy operation before returning + +**Publishing Flow:** + +``` +┌─────────┐ ┌─────────┐ ┌────────────┐ +│ DEV │──────▶│ STAGE │──────▶│ PRODUCTION │ +│(editing)│ │(preview)│ │ (public) │ +└─────────┘ └─────────┘ └────────────┘ + │ │ │ +saveToStage() publishToProduction() │ + │ │ │ + └──────────────────┴────────────────────┘ + Creates publish_event record +``` + +**Event Status Lifecycle:** +1. `queued` - Event created +2. `running` - Processing started +3. `success` / `failed` - Completed + +### Search Service (`search.ts`) + +Global full-text search with permission filtering. + +```typescript +export default class SearchService { + // Search across all permitted entities + static async search(searchQuery: string, currentUser: CurrentUser | undefined): Promise +} +``` + +**Searchable Tables:** + +| Table | Text Fields | Numeric Fields | +|-------|-------------|----------------| +| users | firstName, lastName, phoneNumber, email | - | +| projects | name, slug, description, logo_url, favicon_url, og_image_url | - | +| assets | name, cdn_url, storage_key, mime_type, checksum | size_mb, width_px, height_px, duration_sec | +| asset_variants | cdn_url | width_px, height_px, size_mb | +| presigned_url_requests | requested_key, mime_type, status | requested_size_mb | +| tour_pages | source_key, name, slug, background_image_url, background_video_url, background_audio_url, ui_schema_json | sort_order | +| project_audio_tracks | source_key, name, slug, url | volume, sort_order | +| publish_events | error_message | pages_copied, transitions_copied, audios_copied | +| pwa_caches | cache_version, manifest_json, asset_list_json | - | +| access_logs | path, ip_address, user_agent | - | + +**Permission Check:** +```javascript +// Only search tables user has READ permission for +if (!hasPermission(permissionSet, `READ_${tableName.toUpperCase()}`)) { + return []; +} +``` + +### Projects Service (`projects.ts`) + +Project management with cloning capabilities and slug uniqueness validation. + +```javascript +export default class ProjectsService { + // Normalize slug for URL safety + static normalizeSlug(value) → slug + + // Generate unique slug with -copy suffix + static async generateUniqueSlug(baseSlug, transaction) → uniqueSlug + + // Validate slug uniqueness before create/update (throws ValidationError if duplicate) + static async validateSlugUniqueness(slug, excludeId, transaction) → normalizedSlug + + // Create new project (validates slug uniqueness) + static async create({ data, currentUser, transaction, runtimeContext }) → project + + // Clone project with all assets and variants + static async cloneFromProject(sourceProjectId, currentUser) → clonedProject + + // Update project (validates slug uniqueness if changed) + static async update({ id, data, currentUser, transaction, runtimeContext }) → project +} +``` + +**Slug Validation:** +- `validateSlugUniqueness()` normalizes the slug and checks for duplicates +- Uses `excludeId` parameter to skip the current project during updates +- Checks soft-deleted projects (`paranoid: false`) to prevent conflicts +- Throws `ValidationError('iam.errors.slugAlreadyExists')` if duplicate found + +**Clone Process (Optimized with S3 Native Copy):** + +The clone process uses S3's native `CopyObjectCommand` for server-side file copying (15x faster than download-then-upload). Files are copied in parallel with configurable concurrency. + +``` +Phase A: Create cloned project record + ↓ +Phase B: Collect all copy operations (assets + non-reversed variants) + ↓ +Phase C: Execute parallel S3 copy (10 concurrent, continueOnError=true) + ↓ +Phase D: Build assetPathMap from copy results (failed → use original path) + ↓ +Phase E: Create asset/variant records, build assetIdMap (old → new asset IDs) + ↓ +Phase F: Copy reversed videos using asset ID mapping + └── Reversed videos use pattern: assets/{assetId}/reversed.mp4 + └── Copy from old asset ID path to new asset ID path + ↓ +Phase G: Clone tour_pages, audio_tracks, element_defaults + └── Transform ui_schema_json asset paths using assetPathMap +``` + +**Key Implementation Details:** + +| Phase | Operation | Notes | +|-------|-----------|-------| +| B-C | Parallel file copy | Uses `FileService.copyFilesParallel()` with S3 `CopyObjectCommand` | +| E | Asset ID mapping | Tracks `oldAssetId → newAssetId` for reversed video copying | +| F | Reversed video copy | Separate phase because reversed videos use asset-ID-based paths, not project-ID-based | +| G | Path transformation | `transformUiSchemaAssetPaths()` updates all asset URLs in `ui_schema_json` | + +**Reversed Video Storage Pattern:** +- Primary assets: `assets/{projectId}/{uuid}.ext` +- Reversed videos: `assets/{assetId}/reversed.mp4` (uses asset ID, not project ID) + +**Error Handling:** +- Failed file copies fall back to original storage path (cloned project still functional, shares assets with source) +- Most assets won't have reversed videos - this is expected (only navigation elements with transitions generate them) +- Transaction rollback on DB errors; orphaned S3 files acceptable (can be cleaned later) + +### Roles Service (`roles.ts`) + +Role management service for standard CRUD, CSV bulk import, and permission assignment. The service wraps DB writes in transactions when a caller does not provide one. + +```typescript +export default class RolesService { + static assertPublicRoleHasNoPermissions(data, existingRole) + static async create(options) + static async bulkImport(req, res) + static async update(options) + static async deleteByIds(options) + static async remove(options) +} +``` + +**Public Role Hardening:** +- Creating or updating a role named `Public` rejects non-empty permissions. +- Existing role lookup is done before update so renaming a role to `Public` cannot retain assigned permissions through the service boundary. +- This keeps customer viewer access separate from admin RBAC permissions. + +~~Generates offline manifests for PWA asset downloads.~~ + +### Project Element Defaults Service (`project_element_defaults.ts`) + +Extended factory service with custom methods for managing project-level element defaults. + +```javascript +class Project_element_defaultsService extends BaseService { + // Reset project default to current global default + static async resetToGlobal(id, options) → updated record + + // Get diff between project default and global default + static async getDiffFromGlobal(id) → { hasChanges, diff } + + // Snapshot all global defaults to a project (called on project creation) + static async snapshotGlobalDefaults(projectId, options) → created records +} +``` + +**Methods:** + +| Method | Description | +|--------|-------------| +| `resetToGlobal(id, options)` | Resets a project element default to match the current global element type default | +| `getDiffFromGlobal(id)` | Compares project element default with global default, returns differences | +| `snapshotGlobalDefaults(projectId, options)` | Creates project element defaults by copying all global element type defaults | + +**Use Cases:** +- **Project Creation:** `snapshotGlobalDefaults` is called to copy global defaults to new project +- **Reset to Global:** User can reset customized project defaults back to global values +- **Diff View:** UI can show which settings differ from global defaults + +--- + +## File Storage Module (`services/file/`) + +### Strategy Pattern Implementation + +``` +┌─────────────────────────────────────────────────────────────────────┐ +│ BaseStorageProvider │ +│ (Abstract Interface) │ +│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌───────────────┐ │ +│ │ upload │ │ download │ │ delete │ │ getSignedUrl │ │ +│ │ (abstract) │ │ (abstract) │ │ (abstract) │ │ (optional) │ │ +│ └─────────────┘ └─────────────┘ └─────────────┘ └───────────────┘ │ +└─────────────────────────────────────────────────────────────────────┘ + △ △ △ + │ │ │ +┌──────────┴─────────┐ ┌───────┴────────┐ ┌────────┴───────┐ +│ S3StorageProvider │ │LocalStorage │ │(GCloud inline) │ +│ │ │Provider │ │ │ +│ • AWS SDK v3 │ │ • fs module │ │ • @google-cloud│ +│ • Presigned URLs │ │ • MIME types │ │ /storage │ +│ • Batch delete │ │ • Recursive │ │ • Resumable │ +└────────────────────┘ └────────────────┘ └────────────────┘ +``` + +### BaseStorageProvider Interface + +```typescript +export default class BaseStorageProvider { + static get providerName(): string + upload(key, data, options): Promise + download(key): Promise + delete(key): Promise + deleteMany(keys): Promise + exists(key): Promise + list(prefix): Promise + getSignedUrl(key, expiresIn): Promise +} +``` + +### S3StorageProvider + +AWS S3 implementation using SDK v3 with robust timeout, retry, and error handling. + +`S3StorageProvider.ts` uses official AWS SDK v3 types for client config, command inputs, signed URL generation, streaming download bodies, and `S3ServiceException` metadata. Project-specific storage contracts live in `backend/src/types/file.ts`. + +```javascript +class S3StorageProvider extends BaseStorageProvider { + constructor({ + bucket, region, accessKeyId, secretAccessKey, prefix, + connectionTimeout, // Default: 5000ms + requestTimeout, // Default: 30000ms + maxAttempts, // Default: 3 (adaptive retry) + maxSockets, // Default: 50 (connection pool) + keepAlive // Default: true + }) + + // Static methods for error handling + static getErrorStatusCode(error) // Map S3 errors to HTTP status codes + static isRetryableError(error) // Check if error should be retried + + // Instance methods (all support AbortSignal for cancellation) + buildKey(key) // Add prefix to key + async upload(key, data, { signal }) // PutObjectCommand + async download(key, { signal }) // GetObjectCommand + async copy(sourceKey, destKey, { signal, contentType }) // CopyObjectCommand (server-side) + async delete(key, { signal }) // DeleteObjectCommand + async deleteMany(keys, { signal }) // DeleteObjectsCommand (batched 1000) + async exists(key, { signal }) // HeadObjectCommand + async list(prefix, { signal }) // ListObjectsV2Command (paginated) + async getSignedUrl(key, expiry) // @aws-sdk/s3-request-presigner + getConfig() // Get provider config for debugging + destroy() // Cleanup connection pool +} +``` + +**S3 Error to HTTP Status Code Mapping:** + +| S3 Error | HTTP Status | +|----------|-------------| +| NoSuchKey, NotFound, NoSuchBucket | 404 | +| AccessDenied, InvalidAccessKeyId | 403 | +| ExpiredToken | 401 | +| TimeoutError, RequestTimeout | 504 | +| NetworkingError, ServiceUnavailable | 503 | +| ThrottlingException | 429 | +| InternalError | 500 | + +### LocalStorageProvider + +Local filesystem implementation. + +`LocalStorageProvider.ts` shares the same typed storage contracts as S3 while using Node filesystem and stream APIs. + +```javascript +class LocalStorageProvider extends BaseStorageProvider { + constructor({ basePath = './uploads' }) + + buildPath(key) // path.join(basePath, key) + async upload(key, data) // fs.writeFileSync / stream.pipeline + async download(key) // fs.createReadStream + async copy(sourceKey, destKey) // fs.promises.copyFile (kernel-level) + async delete(key) // fs.unlinkSync + async exists(key) // fs.existsSync + async list(prefix) // fs.readdirSync (recursive) + getContentType(ext) // MIME type mapping +} +``` + +### UploadSessionManager + +Chunked upload session management for large files. + +`UploadSessionManager.ts` uses reusable upload-session metadata contracts from `backend/src/types/file.ts` and validates `meta.json` after parsing instead of relying on type assertions. + +```javascript +class UploadSessionManager { + constructor({ sessionDir, ttlMs = 24h }) + + createSession(options) // → sessionId (UUID) + readMeta(sessionId) // → session metadata + writeMeta(sessionId, payload) // Save session state + saveChunk(sessionId, index, data) // Save chunk file + chunkExists(sessionId, index) // Check chunk exists + isComplete(sessionId) // All chunks uploaded? + assembleChunks(sessionId, path) // Combine chunks + removeSession(sessionId) // Cleanup session + cleanupExpiredSessions() // Remove stale sessions +} +``` + +**Session Directory Structure:** +``` +upload_sessions/ +└── {sessionId}/ + ├── meta.json # Session metadata + └── chunks/ + ├── 0.part + ├── 1.part + └── N.part +``` + +--- + +## Email Module (`services/email/`) + +### EmailSender Class + +Core email sending using Nodemailer. + +```typescript +export default class EmailSender { + constructor(email: EmailTemplate) + async send(): Promise + static get isConfigured(): boolean + get transportConfig(): SMTPTransport.Options + get from(): string +} +``` + +**Configuration (`config.email`):** +```javascript +{ + host: 'email-smtp.us-east-1.amazonaws.com', + port: 587, + auth: { + user: process.env.EMAIL_USER || '', + pass: process.env.EMAIL_PASS || '' + }, + from: 'Tour Builder Platform ' +} +``` + +### Email Templates + +| Template | Class | Fields | +|----------|-------|--------| +| Password Reset | `PasswordResetEmail` | to, link | +| Email Verification | `EmailAddressVerificationEmail` | to, link | +| User Invitation | `InvitationEmail` | to, host | + +**Template Pattern:** +```typescript +export default class PasswordResetEmail implements EmailTemplate { + constructor({ to, link }: LinkEmailTemplateOptions) {} + + get subject(): string { + return getNotification('emails.passwordReset.subject', getNotification('app.title')); + } + + async html(): Promise { + const template = await fs.readFile(templatePath, 'utf8'); + return template + .replace(/{appTitle}/g, appTitle) + .replace(/{resetUrl}/g, this.link) + .replace(/{accountName}/g, this.to); + } +}; +``` + +--- + +## Notifications Module (`services/notifications/`) + +### Error Classes + +**ValidationError (400 Bad Request):** +```javascript +class ValidationError extends Error { + constructor(messageCode) { + const message = isNotification(messageCode) + ? getNotification(messageCode) + : getNotification('errors.validation.message'); + super(message); + this.code = 400; + } +} +``` + +**ForbiddenError (403 Forbidden):** +```javascript +class ForbiddenError extends Error { + constructor(messageCode) { + const message = isNotification(messageCode) + ? getNotification(messageCode) + : getNotification('errors.forbidden.message'); + super(message); + this.code = 403; + } +} +``` + +### Notification Catalog (`list.js`) + +Centralized error messages and i18n strings. + +```javascript +const errors = { + app: { + title: 'Tour Builder Platform', + }, + auth: { + userDisabled: 'Your account is disabled', + forbidden: 'Forbidden', + unauthorized: 'Unauthorized', + userNotFound: "Sorry, we don't recognize your credentials", + wrongPassword: "Sorry, we don't recognize your credentials", + // ... + }, + iam: { + errors: { + userAlreadyExists: 'User with this email already exists', + userNotFound: 'User not found', + // ... + } + }, + emails: { + invitation: { + subject: "You've been invited to {0}", + body: "..." + }, + // ... + } +}; +``` + +### Helper Functions + +```javascript +// Get notification with parameter substitution +getNotification('emails.invitation.subject', 'Tour Builder') +// → "You've been invited to Tour Builder" + +// Check if key exists in catalog +isNotification('auth.userNotFound') // → true +isNotification('custom.message') // → false +``` + +--- + +## Transaction Patterns + +### Standard Transaction Pattern + +All services use consistent transaction handling: + +```javascript +static async create({ data, currentUser, transaction: externalTransaction, runtimeContext }) { + const transaction = externalTransaction || await db.sequelize.transaction(); + const ownsTransaction = !externalTransaction; + try { + const record = await DBApi.create({ data, currentUser, transaction, runtimeContext }); + if (ownsTransaction) await transaction.commit(); + return record; + } catch (error) { + if (ownsTransaction) await transaction.rollback(); + throw error; + } +} +``` + +### Lock Pattern (Publish Service) + +```javascript +static async withProjectPublishLock(projectId, callback) { + return db.sequelize.transaction(async (transaction) => { + // Acquire row lock + const project = await db.projects.findByPk(projectId, { + transaction, + lock: transaction.LOCK.UPDATE, + }); + + // Check for concurrent operations + const runningEvent = await db.publish_events.findOne({ + where: { projectId, status: EVENT_STATUS.RUNNING }, + transaction, + lock: transaction.LOCK.UPDATE, + }); + + if (runningEvent) { + throw new Error('Publish is already running for this project'); + } + + return callback(transaction); + }); +} +``` + +--- + +## Dependencies + +### External Packages + +| Package | Usage | +|---------|-------| +| `bcrypt` | Password hashing | +| `nodemailer` | Email sending | +| `csv-parser` | CSV import parsing | +| `axios` | External API calls (widgets) | +| `uuid` | Upload session IDs | +| `@aws-sdk/client-s3` | S3 operations | +| `@aws-sdk/s3-request-presigner` | Presigned URLs | +| `@google-cloud/storage` | GCloud storage | +| `lodash/get` | Deep object access | + +### Internal Dependencies + +| Module | Services Using | +|--------|---------------| +| `db/models` | All services (transaction) | +| `db/api/*` | All entity services | +| `factories/service.factory` | 9 factory services | +| `config` | auth, file, users, roles | +| `helpers` | auth (jwtSign) | + +--- + +## Service Relationships + +``` +┌─────────────────────────────────────────────────────────────────────┐ +│ Service Graph │ +│ │ +│ ┌──────────┐ │ +│ │ auth │◄────────────────────┐ │ +│ └────┬─────┘ │ │ +│ │ uses │ sends invitations │ +│ ▼ │ │ +│ ┌──────────┐ ┌───────────┐ │ │ +│ │ users │────▶│ email │───┘ │ +│ │ (DBApi) │ └───────────┘ │ +│ └──────────┘ │ │ +│ │ uses templates │ +│ ▼ │ +│ ┌─────────────────┐ │ +│ │ notifications │ │ +│ │ (error catalog) │ │ +│ └─────────────────┘ │ +│ ▲ │ +│ │ throws errors │ +│ │ │ +│ ┌──────────┐ ┌─────┴─────┐ ┌──────────┐ │ +│ │ projects │ │ publish │ │ search │ │ +│ │ (clone) │ │ (env copy)│ │(fulltext)│ │ +│ └──────────┘ └───────────┘ └──────────┘ │ +│ │ +│ ┌──────────────────────────────────────────────────────────────┐ │ +│ │ File Service │ │ +│ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────────────┐ │ │ +│ │ │ S3Provider │ │LocalProvider │ │UploadSessionManager │ │ │ +│ │ └──────────────┘ └──────────────┘ └──────────────────────┘ │ │ +│ └──────────────────────────────────────────────────────────────┘ │ +└─────────────────────────────────────────────────────────────────────┘ +``` + +--- + +## Configuration + +### Environment Variables + +| Variable | Service | Description | +|----------|---------|-------------| +| `FILE_STORAGE_PROVIDER` | file | Force provider ('s3', 'gcloud', 'local') | +| `AWS_S3_BUCKET` | file | S3 bucket name | +| `AWS_S3_REGION` | file | AWS region (default: us-east-1) | +| `AWS_ACCESS_KEY_ID` | file | AWS access key | +| `AWS_SECRET_ACCESS_KEY` | file | AWS secret key | +| `AWS_S3_PREFIX` | file | Key prefix | +| `AWS_S3_CONNECTION_TIMEOUT` | file | S3 connection timeout in ms (default: 5000) | +| `AWS_S3_REQUEST_TIMEOUT` | file | S3 request timeout in ms (default: 30000) | +| `AWS_S3_MAX_ATTEMPTS` | file | S3 retry attempts (default: 3) | +| `AWS_S3_MAX_SOCKETS` | file | S3 connection pool size (default: 50) | +| `AWS_S3_KEEP_ALIVE` | file | Enable HTTP keep-alive (default: true) | +| `AWS_S3_PRESIGN_EXPIRY` | file | Presigned URL expiry in seconds (default: 3600) | +| `GC_PROJECT_ID` | file | GCloud project | +| `GC_CLIENT_EMAIL` | file | GCloud service account | +| `GC_PRIVATE_KEY` | file | GCloud private key | +| `FFMPEG_REVERSE_TIMEOUT_MS` | videoProcessing | Reverse-video hard timeout in ms (default: 600000) | +| `FFPROBE_TIMEOUT_MS` | videoProcessing | Metadata probe timeout in ms (default: 30000) | +| `FFMPEG_BREAKER_FAILURE_THRESHOLD` | videoProcessing | Failures before FFmpeg breaker opens (default: 3) | +| `FFMPEG_BREAKER_COOLDOWN_MS` | videoProcessing | FFmpeg breaker cooldown in ms (default: 120000) | +| `FFMPEG_BREAKER_SUCCESS_THRESHOLD` | videoProcessing | Half-open successes required to close FFmpeg breaker (default: 1) | +| `FILE_STORAGE_BREAKER_FAILURE_THRESHOLD` | file | Failures before storage breaker opens (default: 5) | +| `FILE_STORAGE_BREAKER_COOLDOWN_MS` | file | Storage breaker cooldown in ms (default: 30000) | +| `FILE_STORAGE_BREAKER_SUCCESS_THRESHOLD` | file | Half-open successes required to close storage breaker (default: 2) | +| `EMAIL_USER` | email | SMTP username | +| `EMAIL_PASS` | email | SMTP password | + +### Config References + +```javascript +// config.ts +module.exports = { + bcrypt: { saltRounds: 12 }, + email: { host, port, auth, from }, + s3: { + bucket, + region, + accessKeyId, + secretAccessKey, + prefix, + // Timeout/retry configuration + connectionTimeout, // 5000ms default + requestTimeout, // 30000ms default + maxAttempts, // 3 retries with adaptive backoff + // Connection pool + maxSockets, // 50 concurrent connections + keepAlive, // HTTP keep-alive enabled + // Presigned URLs + presignExpirySeconds, // 3600 (1 hour) default + }, + gcloud: { bucket, hash }, + uploadDir: './uploads', + flHost: 'https://flatlogic.host', // Widget service + project_uuid: '...', +}; +``` + +--- + +## Testing Guidelines + +### Unit Testing Services + +```javascript +// Mock transaction +jest.mock('../db/models', () => ({ + sequelize: { + transaction: jest.fn(() => ({ + commit: jest.fn(), + rollback: jest.fn(), + })), + }, +})); + +// Mock DB API +jest.mock('../db/api/assets'); + +describe('AssetsService', () => { + it('should create asset with transaction', async () => { + const result = await AssetsService.create({ + data: mockData, + currentUser: mockUser, + }); + expect(transaction.commit).toHaveBeenCalled(); + }); +}); +``` + +### Integration Testing + +```javascript +describe('PublishService', () => { + it('should copy dev to stage', async () => { + // Create test project with pages + const project = await createTestProject(); + await createTestPages(project.id, 'dev'); + + // Publish to stage + const result = await PublishService.saveToStage(project.id, mockUser); + + // Verify stage pages created + const stagePages = await findPages(project.id, 'stage'); + expect(stagePages.length).toBe(result.summary.pages_copied); + }); +}); +``` + +--- + +## Best Practices + +### 1. Transaction Handling +- Always wrap multi-step operations in transactions +- Use try/catch/rollback pattern consistently +- Pass transaction to all DB API calls + +### 2. Error Handling +- Use `ValidationError` for client errors (400) +- Use `ForbiddenError` for authorization failures (403) +- Use notification catalog for consistent messages + +### 3. Service Design +- Keep services focused on business logic +- Delegate DB operations to DB API layer +- Use factories for simple CRUD services + +### 4. File Operations +- Use Strategy Pattern for multi-provider support +- Implement chunked uploads for large files +- Clean up sessions on completion/failure + +--- + +## Related Documentation + +- [Backend Architecture](../backend-architecture.md) - Overall backend design +- [Database Schema](../database-schema.md) - Data models +- [API Endpoints](../api-endpoints.md) - REST API reference +- [Auth Module](./auth.md) - Authentication details +- [Routes Module](./routes.md) - Route implementations diff --git a/backend/docs/modules/utilities.md b/backend/docs/modules/utilities.md new file mode 100644 index 0000000..fd3001e --- /dev/null +++ b/backend/docs/modules/utilities.md @@ -0,0 +1,857 @@ +# Utilities Module + +## Overview + +The Utilities module provides centralized helper functions, error handling, logging, environment validation, and i18n message management. Utilities are organized across several locations based on their domain. + +**Locations:** +- `backend/src/utils/` - Core utilities (errors, logging, env validation, request context) +- `backend/src/helpers.ts` - Request helpers (async wrapper, error handler, JWT) +- `backend/src/db/utils.ts` - Database utilities +- `backend/src/services/notifications/` - i18n messages and legacy error classes + +--- + +## Architecture + +``` +backend/src/ +├── utils/ +│ ├── index.ts # Re-exports +│ ├── errors.ts # Error classes +│ ├── logger.ts # Pino logging +│ ├── request-context.ts # Request-scoped context storage +│ └── env-validation.ts # Joi env schema +├── helpers.ts # Request helpers +├── db/ +│ └── utils.ts # DB utilities +└── services/notifications/ + ├── list.ts # i18n message catalog + ├── helpers.ts # Message formatting + └── errors/ + ├── forbidden.ts # ForbiddenError + └── validation.ts # ValidationError +``` + +--- + +## Core Utilities (`utils/`) + +### Error Classes (`utils/errors.js`) + +Modern error hierarchy for consistent HTTP error responses. + +```javascript +class AppError extends Error { + constructor(message, statusCode = 500, details = null) { + super(message); + this.statusCode = statusCode; + this.details = details; + this.isOperational = true; // Distinguishes from programming errors + Error.captureStackTrace(this, this.constructor); + } +} +``` + +**Error Types:** + +| Class | Status Code | Default Message | Usage | +|-------|-------------|-----------------|-------| +| `AppError` | 500 | (custom) | Base class | +| `NotFoundError` | 404 | `{resource} not found` | Missing resources | +| `ValidationError` | 400 | (custom) | Invalid input | +| `ForbiddenError` | 403 | `Access denied` | Permission denied | +| `UnauthorizedError` | 401 | `Unauthorized` | Auth required | +| `ConflictError` | 409 | `Resource conflict` | Duplicate resources | + +**Usage:** +```javascript +const { NotFoundError, ValidationError, ForbiddenError } = require('./utils'); + +// In route handler +if (!user) { + throw new NotFoundError('User'); +} + +const currentUser = getCurrentUser(req); +if (!currentUser) { + throw new ForbiddenError(); +} + +if (errors.length > 0) { + throw new ValidationError('Invalid data', errors); +} +``` + +--- + +### Structured Logging (`utils/logger.ts`) + +Pino-based logging with request correlation and environment-aware formatting. +Logger initialization is a bootstrap exception where direct `process.env` +access is allowed; runtime services should use `config.ts` instead. + +```javascript +const pino = require('pino'); +const crypto = require('crypto'); + +const isDevelopment = process.env.NODE_ENV === 'development'; + +const logger = pino({ + level: process.env.LOG_LEVEL || 'info', + transport: isDevelopment + ? { target: 'pino-pretty', options: { colorize: true } } + : undefined, + base: { + service: 'tour-builder-api', + env: process.env.NODE_ENV || 'development', + }, +}); +``` + +**Log Levels:** +- `fatal` - Unrecoverable errors +- `error` - Errors requiring attention +- `warn` - Warning conditions (400-499 responses) +- `info` - Normal operations (default) +- `debug` - Detailed debugging +- `trace` - Very detailed tracing + +**Process-Level Failure Logging:** + +`registerProcessErrorHandlers()` is called from `src/index.ts` during backend +startup. It installs handlers for `uncaughtException` and `unhandledRejection`, +logs them through Pino at `fatal`, normalizes non-`Error` rejection reasons into +`Error` instances with `cause`, flushes Pino, and then exits with status `1`. +These failures happen outside the Express request lifecycle, so route error +middleware cannot catch them. + +```typescript +registerProcessErrorHandlers(); + +process.on('unhandledRejection', (reason) => { + logger.fatal( + { err: normalizeLoggedError(reason) }, + 'Unhandled promise rejection, shutting down process', + ); + exitAfterLogging(); +}); +``` + +**Request Logger Middleware:** +```javascript +function requestLogger(req, res, next) { + // Generate or use existing request ID + const requestId = req.headers['x-request-id'] || crypto.randomUUID(); + + // Store child logger in request context + setRequestLogger(req, logger.child({ requestId })); + setRequestId(req, requestId); + res.setHeader('X-Request-Id', requestId); + + const start = Date.now(); + res.on('finish', () => { + const duration = Date.now() - start; + const logData = { + method: req.method, + url: req.originalUrl || req.url, + status: res.statusCode, + duration, + userAgent: req.headers['user-agent'], + }; + + // Log level based on status code + if (res.statusCode >= 500) { + getRequestLogger(req)?.error(logData, 'Request completed with server error'); + } else if (res.statusCode >= 400) { + getRequestLogger(req)?.warn(logData, 'Request completed with client error'); + } else { + getRequestLogger(req)?.info(logData, 'Request completed'); + } + }); + + next(); +} +``` + +**Log Output Examples:** + +Development (pino-pretty): +``` +[12:34:56.789] INFO (tour-builder-api): Request completed + requestId: "abc-123" + method: "GET" + url: "/api/users" + status: 200 + duration: 45 +``` + +Production (JSON): +```json +{"level":30,"time":1711723456789,"service":"tour-builder-api","env":"production","requestId":"abc-123","method":"GET","url":"/api/users","status":200,"duration":45,"msg":"Request completed"} +``` + +**Usage:** +```javascript +const { logger, requestLogger } = require('./utils/logger'); + +// App setup +app.use(requestLogger); + +// Manual logging +logger.info({ userId: user.id }, 'User logged in'); +logger.error({ err }, 'Database connection failed'); + +// Request-scoped logging (in routes) +getRequestLogger(req)?.info({ data }, 'Processing request'); +``` + +--- + +### Request Context (`utils/request-context.ts`) + +Request-scoped data is stored outside the Express `Request` object through a +`WeakMap`. Do not add global +`Express.Request` augmentation for project fields such as `currentUser`, +`runtimeContext`, `log`, or runtime-public flags. Middleware should write with +`setCurrentUser`, `setRuntimeContext`, `setRequestLogger`, and related helpers; +routes/services should read through `getCurrentUser`, `getRuntimeContext`, +`getRequestLogger`, and `getRouteServiceContext`. + +### Environment Validation (`utils/env-validation.ts`) + +Joi-based validation ensuring all required environment variables are present with correct types. + +**Schema Definition:** +```javascript +const Joi = require('joi'); + +const envSchema = Joi.object({ + // Server + NODE_ENV: Joi.string() + .valid('development', 'test', 'production', 'dev_stage') + .default('development'), + PORT: Joi.number().default(8080), + + // Database + DB_HOST: Joi.string().default('localhost'), + DB_PORT: Joi.number().default(5432), + DB_NAME: Joi.string().default('db_tour_builder_platform'), + DB_USER: Joi.string().default('postgres'), + DB_PASS: Joi.string().allow('').default(''), + + // Authentication + SECRET_KEY: Joi.string() + .min(16) + .default('88dbeaf8-e906-405e-9e41-c3baadeda5c6'), + ADMIN_PASS: Joi.string().default('88dbeaf8'), + USER_PASS: Joi.string().default('c3baadeda5c6'), + ADMIN_EMAIL: Joi.string().email().default('admin@flatlogic.com'), + + // OAuth + GOOGLE_CLIENT_ID: Joi.string().allow('').default(''), + GOOGLE_CLIENT_SECRET: Joi.string().allow('').default(''), + MS_CLIENT_ID: Joi.string().allow('').default(''), + MS_CLIENT_SECRET: Joi.string().allow('').default(''), + + // AWS S3 + AWS_ACCESS_KEY_ID: Joi.string().allow('').default(''), + AWS_SECRET_ACCESS_KEY: Joi.string().allow('').default(''), + AWS_S3_BUCKET: Joi.string().allow('').default(''), + AWS_S3_REGION: Joi.string().default('us-east-1'), + AWS_S3_PREFIX: Joi.string().default('afeefb9d49f5b7977577876b99532ac7'), + + // Email + EMAIL_USER: Joi.string().allow('').default(''), + EMAIL_PASS: Joi.string().allow('').default(''), + EMAIL_TLS_REJECT_UNAUTHORIZED: Joi.string() + .valid('true', 'false') + .default('true'), + + // External APIs + PEXELS_KEY: Joi.string().allow('').default(''), + + // Logging + LOG_LEVEL: Joi.string() + .valid('fatal', 'error', 'warn', 'info', 'debug', 'trace') + .default('info'), +}).unknown(true); // Allow additional env vars +``` + +**Validation Function:** +```javascript +function validateEnv() { + const { error, value } = envSchema.validate(process.env, { + abortEarly: false, // Report all errors + stripUnknown: false, // Keep unknown vars + }); + + if (error) { + const messages = error.details.map((d) => ` - ${d.message}`); + logger.error({ errors: messages }, 'Environment validation failed'); + + // Strict in production, lenient in development + if (process.env.NODE_ENV === 'production') { + process.exit(1); + } else { + logger.warn('Continuing with default values in non-production mode'); + } + } + + return value; // Returns validated/defaulted values +} +``` + +**Environment Variable Categories:** + +| Category | Variables | Required | +|----------|-----------|----------| +| **Server** | `NODE_ENV`, `PORT` | Defaults | +| **Database** | `DB_HOST`, `DB_PORT`, `DB_NAME`, `DB_USER`, `DB_PASS` | Defaults | +| **Auth** | `SECRET_KEY`, `ADMIN_PASS`, `USER_PASS`, `ADMIN_EMAIL` | Defaults | +| **OAuth** | `GOOGLE_CLIENT_ID`, `GOOGLE_CLIENT_SECRET`, `MS_CLIENT_ID`, `MS_CLIENT_SECRET` | Optional | +| **AWS S3** | `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_S3_BUCKET`, `AWS_S3_REGION`, `AWS_S3_PREFIX` | Optional | +| **Email** | `EMAIL_USER`, `EMAIL_PASS`, `EMAIL_TLS_REJECT_UNAUTHORIZED` | Optional | +| **External APIs** | `PEXELS_KEY` | Optional | +| **Logging** | `LOG_LEVEL` | Defaults | + +--- + +### Index Re-exports (`utils/index.js`) + +Convenient re-export of utilities: + +```javascript +module.exports = { + ...require('./errors'), + ...require('./logger'), + envValidation: require('./env-validation'), +}; +``` + +**Exported:** +- `AppError`, `NotFoundError`, `ValidationError`, `ForbiddenError`, `UnauthorizedError`, `ConflictError` +- `logger`, `requestLogger`, `registerProcessErrorHandlers`, + `exitAfterLogging`, + `normalizeLoggedError` +- `envValidation.validateEnv`, `envValidation.envSchema` + +--- + +## Request Helpers (`helpers.js`) + +Core helper class for Express route handling. + +```javascript +const jwt = require('jsonwebtoken'); +const config = require('./config'); + +module.exports = class Helpers { + // Wrap async route handlers to catch errors + static wrapAsync(fn) { + return function (req, res, next) { + fn(req, res, next).catch(next); + }; + } + + // Centralized error response handler + static commonErrorHandler(error, req, res, _next) { + const statusCode = error.code || error.status; + + if ([400, 401, 403, 404, 409, 422].includes(statusCode)) { + return res.status(statusCode).send(error.message); + } + + console.error(error); + return res.status(500).send('Internal server error'); + } + + // Generate JWT token + static jwtSign(data) { + return jwt.sign(data, config.secret_key, { expiresIn: '6h' }); + } + + // Validate UUID v4 format + static isUuidV4(value) { + return /^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i.test( + value, + ); + } +}; +``` + +**Functions:** + +| Function | Purpose | Usage | +|----------|---------|-------| +| `wrapAsync(fn)` | Wraps async handlers to propagate errors | All async route handlers | +| `commonErrorHandler(err, req, res, next)` | Standardizes error responses | Route error middleware | +| `jwtSign(data)` | Creates JWT with 6h expiry | Auth service | +| `isUuidV4(value)` | Validates UUID v4 format | Route parameter validation | + +## Request Validation + +Request validation is centralized in: + +- `src/middlewares/validate-request.ts` - Joi middleware for `params`, `query`, and `body` +- `src/validators/request-schemas.ts` - shared schemas for CRUD, auth, users, projects, tour pages, publish, and file upload endpoints + +New external routes must validate all incoming `params`, `query`, and `body` before calling services. Factory CRUD routes use default schemas automatically and can override them through the `validation` option. + +```typescript +import { validateRequest } from '../middlewares/validate-request.ts'; +import { publish as publishSchemas } from '../validators/request-schemas.ts'; + +router.post( + '/save-to-stage', + validateRequest(publishSchemas.saveToStage), + wrapAsync(async (req, res) => { + const result = await PublishService.saveToStage( + req.body.projectId, + getCurrentUser(req), + ); + res.status(200).json(result); + }), +); +``` + +Request validation errors return JSON: + +```json +{ + "error": "Invalid request", + "details": [ + { + "path": "projectId", + "message": "\"projectId\" must be a valid GUID", + "type": "string.guid" + } + ] +} +``` + +Service/domain `ValidationError` responses keep the legacy plain-text format unless they are raised by request validation middleware. + +**Usage Pattern:** +```javascript +const { wrapAsync, commonErrorHandler, isUuidV4 } = require('../helpers'); + +// Async route handler +router.get('/users/:id', wrapAsync(async (req, res) => { + if (!isUuidV4(req.params.id)) { + return res.status(400).send('Invalid ID format'); + } + + const user = await UserService.findOne(req.params.id); + res.json(user); +})); + +// Register error handler at end of router +router.use('/', commonErrorHandler); +``` + +--- + +## Database Utilities (`db/utils.js`) + +Utilities specific to Sequelize database operations, including clean UUID validation functions. + +```javascript +const validator = require('validator'); +const { v4: uuidv4 } = require('uuid'); +const Sequelize = require('./models').Sequelize; + +module.exports = class Utils { + // Check if value is a valid UUID + static isValidUuid(value) { + return Boolean(value && validator.isUUID(String(value))); + } + + // Generate a new UUID v4 + static generateUuid() { + return uuidv4(); + } + + // Filter array to only valid UUIDs + static filterValidUuids(values) { + return values.filter((v) => this.isValidUuid(v)); + } + + // Case-insensitive LIKE query + static ilike(model, column, value) { + return Sequelize.where( + Sequelize.fn('lower', Sequelize.col(`${model}.${column}`)), + { [Sequelize.Op.like]: `%${value}%`.toLowerCase() }, + ); + } +}; +``` + +**Functions:** + +| Function | Purpose | Returns | +|----------|---------|---------| +| `isValidUuid(value)` | Check if value is a valid UUID | `boolean` | +| `generateUuid()` | Generate a new UUID v4 | `string` | +| `filterValidUuids(values)` | Filter array to only valid UUIDs | `string[]` | +| `ilike(model, column, value)` | Case-insensitive LIKE search | Sequelize where clause | + +**UUID Validation Behavior:** +- Invalid single ID filter (`?id=xxx`) → returns `{ rows: [], count: 0 }` immediately +- Invalid UUIDs in relation filters (`?project=uuid|name`) → filtered out for ID search, kept for text search +- Invalid UUID field filter (`?projectId=xxx`) → returns `{ rows: [], count: 0 }` immediately + +**Usage in DB API:** +```javascript +const Utils = require('../utils'); + +class GenericDBApi { + async findAll(filter = {}, options = {}) { + // Single ID validation + if (filter.id) { + if (!Utils.isValidUuid(filter.id)) { + return { rows: [], count: 0 }; + } + where.id = filter.id; + } + + // Relation filter with mixed UUID/text search + for (const rel of this.RELATION_FILTERS) { + if (filter[rel.filterKey]) { + const searchTerms = filter[rel.filterKey].split('|'); + const validUuids = Utils.filterValidUuids(searchTerms); + + // UUID search: only valid UUIDs + // Text search: all terms + } + } + } + + async findAllAutocomplete({ query, limit, offset }) { + const orConditions = [ + Utils.ilike(this.TABLE_NAME, this.AUTOCOMPLETE_FIELD, query), + ]; + + // Only add UUID search if query is a valid UUID + if (Utils.isValidUuid(query)) { + orConditions.unshift({ id: query }); + } + + // ... + } +} +``` + +--- + +## Notifications Module (`services/notifications/`) + +i18n message management and legacy error classes. + +### Message Catalog (`list.js`) + +Centralized message definitions with placeholders: + +```javascript +const errors = { + app: { + title: 'Tour Builder Platform', + }, + + auth: { + userDisabled: 'Your account is disabled', + forbidden: 'Forbidden', + unauthorized: 'Unauthorized', + userNotFound: `Sorry, we don't recognize your credentials`, + wrongPassword: `Sorry, we don't recognize your credentials`, + weakPassword: 'This password is too weak', + emailAlreadyInUse: 'Email is already in use', + invalidEmail: 'Please provide a valid email', + passwordReset: { + invalidToken: 'Password reset link is invalid or has expired', + error: `Email not recognized`, + }, + passwordUpdate: { + samePassword: `You can't use the same password. Please create new password`, + }, + userNotVerified: `Sorry, your email has not been verified yet`, + emailAddressVerificationEmail: { + invalidToken: 'Email verification link is invalid or has expired', + error: `Email not recognized`, + }, + }, + + iam: { + errors: { + userAlreadyExists: 'User with this email already exists', + userNotFound: 'User not found', + disablingHimself: `You can't disable yourself`, + revokingOwnPermission: `You can't revoke your own owner permission`, + deletingHimself: `You can't delete yourself`, + emailRequired: 'Email is required', + }, + }, + + importer: { + errors: { + invalidFileEmpty: 'The file is empty', + invalidFileExcel: 'Only excel (.xlsx) files are allowed', + invalidFileUpload: 'Invalid file. Make sure you are using the last version of the template.', + importHashRequired: 'Import hash is required', + importHashExistent: 'Data has already been imported', + userEmailMissing: 'Some items in the CSV do not have an email', + }, + }, + + errors: { + forbidden: { message: 'Forbidden' }, + validation: { message: 'An error occurred' }, + searchQueryRequired: { message: 'Search query is required' }, + }, + + emails: { + invitation: { + subject: `You've been invited to {0}`, + body: `

Hello,

You've been invited to {0}...

`, + }, + emailAddressVerification: { + subject: `Verify your email for {0}`, + body: `

Hello,

Follow this link to verify...

`, + }, + passwordReset: { + subject: `Reset your password for {0}`, + body: `

Hello,

Follow this link to reset...

`, + }, + }, +}; +``` + +**Message Categories:** + +| Category | Purpose | Examples | +|----------|---------|----------| +| `app` | Application metadata | `app.title` | +| `auth` | Authentication errors | `auth.userDisabled`, `auth.wrongPassword` | +| `iam` | User management errors | `iam.errors.userAlreadyExists` | +| `importer` | Import/export errors | `importer.errors.invalidFileEmpty` | +| `errors` | Generic errors | `errors.forbidden.message` | +| `emails` | Email templates | `emails.invitation.subject` | + +--- + +### Message Helpers (`helpers.js`) + +Functions for message formatting and lookup: + +```javascript +const _get = require('lodash/get'); +const errors = require('./list'); + +// Format message with placeholder substitution +function format(message, args) { + if (!message) return null; + return message.replace(/{(\d+)}/g, function (match, number) { + return typeof args[number] != 'undefined' ? args[number] : match; + }); +} + +// Check if key exists in catalog +const isNotification = (key) => { + const message = _get(errors, key); + return !!message; +}; + +// Get formatted message by key +const getNotification = (key, ...args) => { + const message = _get(errors, key); + if (!message) return key; + return format(message, args); +}; +``` + +**Usage:** +```javascript +const { getNotification, isNotification } = require('./helpers'); + +// Lookup message +getNotification('auth.userDisabled'); +// → 'Your account is disabled' + +// Format with placeholders +getNotification('emails.invitation.subject', 'Tour Builder'); +// → "You've been invited to Tour Builder" + +// Check existence +isNotification('auth.userDisabled'); // → true +isNotification('unknown.key'); // → false +``` + +--- + +### Legacy Error Classes (`errors/`) + +i18n-aware error classes (legacy pattern, prefer `utils/errors.js`): + +**ForbiddenError:** +```javascript +const { getNotification, isNotification } = require('../helpers'); + +module.exports = class ForbiddenError extends Error { + constructor(messageCode) { + let message; + + if (messageCode && isNotification(messageCode)) { + message = getNotification(messageCode); + } + + message = message || getNotification('errors.forbidden.message'); + + super(message); + this.code = 403; + } +}; +``` + +**ValidationError:** +```javascript +module.exports = class ValidationError extends Error { + constructor(messageCode) { + let message; + + if (messageCode && isNotification(messageCode)) { + message = getNotification(messageCode); + } + + message = message || getNotification('errors.validation.message'); + + super(message); + this.code = 400; + } +}; +``` + +**Usage:** +```javascript +const ForbiddenError = require('./services/notifications/errors/forbidden'); +const ValidationError = require('./services/notifications/errors/validation'); + +// With i18n key +throw new ForbiddenError('auth.forbidden'); +throw new ValidationError('iam.errors.emailRequired'); + +// With default message +throw new ForbiddenError(); // → 'Forbidden' +throw new ValidationError(); // → 'An error occurred' +``` + +--- + +## Error Handling Flow + +``` +┌─────────────────────────────────────────────────────────────┐ +│ Route Handler │ +│ │ +│ router.get('/users/:id', wrapAsync(async (req, res) => { │ +│ throw new NotFoundError('User'); │ +│ })); │ +└─────────────────────────────────────────────────────────────┘ + │ + wrapAsync catches + │ + ▼ +┌─────────────────────────────────────────────────────────────┐ +│ commonErrorHandler │ +│ │ +│ - Checks error.code or error.status │ +│ - Known codes (400-422): returns error.message │ +│ - Unknown: returns 'Internal server error' │ +└─────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────┐ +│ HTTP Response │ +│ │ +│ HTTP/1.1 404 Not Found │ +│ Content-Type: text/plain │ +│ │ +│ User not found │ +└─────────────────────────────────────────────────────────────┘ +``` + +--- + +## Usage Summary + +### Recommended Imports + +```javascript +// Modern error classes +const { NotFoundError, ValidationError, ForbiddenError } = require('./utils'); + +// Logging +const { logger, requestLogger } = require('./utils/logger'); + +// Request helpers +const { wrapAsync, commonErrorHandler, isUuidV4 } = require('./helpers'); + +// Database utilities +const Utils = require('./db/utils'); + +// i18n messages +const { getNotification } = require('./services/notifications/helpers'); +``` + +### Error Class Selection + +| Scenario | Recommended Class | +|----------|-------------------| +| Resource not found | `NotFoundError` from `utils/errors.js` | +| Invalid input | `ValidationError` from `utils/errors.js` | +| Permission denied | `ForbiddenError` from `utils/errors.js` | +| Auth required | `UnauthorizedError` from `utils/errors.js` | +| Duplicate resource | `ConflictError` from `utils/errors.js` | +| i18n error message | Legacy classes from `notifications/errors/` | + +--- + +## Environment Variables Reference + +| Variable | Type | Default | Description | +|----------|------|---------|-------------| +| `NODE_ENV` | string | `development` | `development`, `test`, `production`, `dev_stage` | +| `PORT` | number | `8080` | Server port | +| `DB_HOST` | string | `localhost` | PostgreSQL host | +| `DB_PORT` | number | `5432` | PostgreSQL port | +| `DB_NAME` | string | `db_tour_builder_platform` | Database name | +| `DB_USER` | string | `postgres` | Database user | +| `DB_PASS` | string | `` | Database password | +| `SECRET_KEY` | string | UUID | JWT signing key (min 16 chars) | +| `ADMIN_EMAIL` | email | `admin@flatlogic.com` | Admin account email | +| `ADMIN_PASS` | string | `88dbeaf8` | Admin account password | +| `USER_PASS` | string | `c3baadeda5c6` | Default user password | +| `GOOGLE_CLIENT_ID` | string | `` | Google OAuth client ID | +| `GOOGLE_CLIENT_SECRET` | string | `` | Google OAuth client secret | +| `MS_CLIENT_ID` | string | `` | Microsoft OAuth client ID | +| `MS_CLIENT_SECRET` | string | `` | Microsoft OAuth client secret | +| `AWS_ACCESS_KEY_ID` | string | `` | AWS access key | +| `AWS_SECRET_ACCESS_KEY` | string | `` | AWS secret key | +| `AWS_S3_BUCKET` | string | `` | S3 bucket name | +| `AWS_S3_REGION` | string | `us-east-1` | S3 region | +| `AWS_S3_PREFIX` | string | UUID | S3 key prefix | +| `EMAIL_USER` | string | `` | SMTP username | +| `EMAIL_PASS` | string | `` | SMTP password | +| `EMAIL_TLS_REJECT_UNAUTHORIZED` | string | `true` | TLS cert validation | +| `PEXELS_KEY` | string | `` | Pexels API key | +| `LOG_LEVEL` | string | `info` | Pino log level | + +--- + +## Related Documentation + +- [Backend Architecture](../backend-architecture.md) - Overall backend structure +- [Auth Module](./auth.md) - Authentication using JWT helpers +- [Middleware Module](./middleware.md) - Request middleware +- [Routes Module](./routes.md) - Route handlers using wrapAsync +- [Services Module](./services.md) - Business logic services +- [DB Config Module](./db-config.md) - Database configuration diff --git a/backend/docs/testing.md b/backend/docs/testing.md new file mode 100644 index 0000000..9a13eb8 --- /dev/null +++ b/backend/docs/testing.md @@ -0,0 +1,77 @@ +# Backend Testing + +## Overview + +Backend tests use Node.js 24's built-in test runner and TypeScript test files. +The suite is split into three layers: + +| Layer | Command | Location | Purpose | +|-------|---------|----------|---------| +| Unit | `npm run test` | `backend/tests/*.test.ts` | Pure helpers, validators, policy decisions, service contracts, and file/session utilities | +| Integration | `npm run test:integration` | `backend/tests/integration/*.test.ts` | Cross-module behavior such as DB-backed access policy and Express router factory contracts | +| E2E | `npm run test:e2e` | `backend/tests/e2e/*.test.ts` | Real HTTP request/response checks against local Express test apps | +| Full suite | `npm run test:all` | all test folders | Runs unit, integration, and e2e in sequence | +| Verification | `npm run verify` | static checks plus all test folders | Runs typecheck, lint, ESM boundary checks, and the full test suite | + +## Current Coverage + +- Request validation and `commonErrorHandler` JSON handling. +- Publish, file upload/presign, upload chunk, and tour page reorder request + validation contracts. +- File service path validation, structured file error payloads, and local + presigned URL fallback behavior. +- Environment validation for config-only runtime settings, including normalized + file-storage provider overrides. +- Circuit breaker behavior for ignored failures, recorded failures, and open + breaker rejection. +- Auth service password reset, password update rejection rules, and email + verification token behavior with real bcrypt and DB writes. +- Route ID/body ID update contract enforcement. +- Permission name mapping, own-user route bypass, runtime public read bypass, + and `AccessPolicy` public-user hardening. +- Generic DB API and entity service object-signature contracts. +- OpenAPI route coverage and internal `$ref` resolution. +- `UploadSessionManager` metadata, chunk tracking, assembly order, and cleanup behavior. +- `createEntityRouter` service/DB option propagation, query normalization, update ID mismatch handling, and CSV export behavior. +- Public runtime list blocking and response sanitization for presentation data. +- Publish service environment copy behavior for dev to stage and stage to production, + including target replacement, source preservation, audio tracks, transition + settings, and UI-control settings. +- Runtime context HTTP behavior for default, valid, and unsupported runtime headers. + +## Environment Notes + +DB-backed integration tests authenticate against the configured PostgreSQL +connection. If PostgreSQL is unavailable, those tests call `t.skip()` with the +connection error so non-DB integration tests can still run. +`npm run test:integration` runs with `--test-concurrency=1` because several +integration tests share a PostgreSQL transaction connection; keep DB-backed +tests sequential unless they use isolated connections. + +E2E tests bind a local HTTP listener on `127.0.0.1` with an ephemeral port. In +restricted sandboxes this may require elevated permission for local socket +binding. + +## Test Helpers + +`backend/tests/http-test-utils.ts` provides `startTestServer(app)`, which starts +an Express app on an ephemeral local port and returns `{ baseUrl, close }`. +Use it only for e2e-style tests that must exercise real HTTP semantics. + +Integration tests that do not need a network socket should drive Express apps in +memory with `node-mocks-http`; this keeps them usable in restricted sandboxes. + +## Adding Tests + +- Prefer unit tests for pure helpers, validators, policies, and isolated service + logic. +- Use integration tests when behavior crosses route, middleware, service, DB API, + or Sequelize transaction boundaries. +- Use e2e tests when the behavior depends on real HTTP parsing, headers, status + codes, or socket-level request handling. +- Keep new backend tests strict TypeScript compatible and avoid casts, `any`, + `eslint-disable`, and `@ts-ignore`. +- When config-backed behavior must be overridden in tests, mutate the typed + config object and restore it in `finally`; do not rely on changing + `process.env` after modules have imported `backend/src/config.ts`. +- For DB integration tests, wrap writes in a transaction and roll it back. diff --git a/documentation/access-logs-audit-trail.md b/documentation/access-logs-audit-trail.md new file mode 100644 index 0000000..2d23efd --- /dev/null +++ b/documentation/access-logs-audit-trail.md @@ -0,0 +1,708 @@ +# Access Logs & Audit Trail + +Complete documentation for the Tour Builder Platform's Access Logs and Audit Trail system including activity tracking, security auditing, and data preservation. + +## Overview + +The platform implements a comprehensive audit system with two complementary mechanisms: +- **Access Logs** - Dedicated table recording API requests with user, path, and client info +- **Audit Trail** - CreatedBy/UpdatedBy fields on all entities tracking who modified what + +``` +┌─────────────────────────────────────────────────────────────────────────────┐ +│ Access Logs & Audit Trail Architecture │ +│ │ +│ ┌────────────────────────────────────────────────────────────────────────┐ │ +│ │ Request Flow │ │ +│ │ │ │ +│ │ Client Request │ │ +│ │ │ │ │ +│ │ ▼ │ │ +│ │ ┌─────────────────────────────────────────────────────────────────┐ │ │ +│ │ │ Request Logger Middleware │ │ │ +│ │ │ │ │ │ +│ │ │ • Generate/Extract X-Request-Id │ │ │ +│ │ │ • Capture: method, path, user-agent, duration │ │ │ +│ │ │ • Log to Pino (structured logging) │ │ │ +│ │ └──────────────────────────┬──────────────────────────────────────┘ │ │ +│ │ │ │ │ +│ │ ▼ │ │ +│ │ ┌─────────────────────────────────────────────────────────────────┐ │ │ +│ │ │ JWT Authentication │ │ │ +│ │ │ │ │ │ +│ │ │ • Validate token │ │ │ +│ │ │ • Attach currentUser to request │ │ │ +│ │ └──────────────────────────┬──────────────────────────────────────┘ │ │ +│ │ │ │ │ +│ │ ▼ │ │ +│ │ ┌─────────────────────────────────────────────────────────────────┐ │ │ +│ │ │ Permission Check │ │ │ +│ │ │ │ │ │ +│ │ │ • Check READ/CREATE/UPDATE/DELETE permissions │ │ │ +│ │ │ • Role-based + custom permissions │ │ │ +│ │ └──────────────────────────┬──────────────────────────────────────┘ │ │ +│ │ │ │ │ +│ │ ▼ │ │ +│ │ ┌─────────────────────────────────────────────────────────────────┐ │ │ +│ │ │ Database Operation │ │ │ +│ │ │ │ │ │ +│ │ │ CREATE: set createdById, updatedById │ │ │ +│ │ │ UPDATE: set updatedById │ │ │ +│ │ │ DELETE: set deletedBy, mark deletedAt (soft delete) │ │ │ +│ │ └─────────────────────────────────────────────────────────────────┘ │ │ +│ └────────────────────────────────────────────────────────────────────────┘ │ +└─────────────────────────────────────────────────────────────────────────────┘ +``` + +## Data Model + +### Access Logs Schema + +**Table:** `access_logs` + +**Source:** `backend/src/db/models/access_logs.js` + +| Field | Type | Required | Default | Description | +|-------|------|----------|---------|-------------| +| id | UUID | Yes | UUIDv4 | Primary key | +| projectId | UUID (FK) | No | null | Related project (CASCADE delete) | +| userId | UUID (FK) | No | null | User who accessed (SET NULL delete) | +| environment | ENUM | Yes | - | 'admin', 'stage', 'production' | +| path | TEXT | No | null | API path accessed (max 2048 chars) | +| ip_address | TEXT | No | null | Client IP address (max 45 chars for IPv6) | +| user_agent | TEXT | No | null | HTTP User-Agent header (max 1024 chars) | +| accessed_at | TIMESTAMP | Yes | NOW | When access occurred | +| importHash | VARCHAR(255) | No | null | Unique hash for CSV imports | +| createdAt | TIMESTAMP | Yes | Auto | Record creation time | +| updatedAt | TIMESTAMP | Yes | Auto | Record update time | +| deletedAt | TIMESTAMP | No | null | Soft delete timestamp | + +### Indexes + +```javascript +indexes: [ + { fields: ['projectId'] }, // Query logs by project + { fields: ['environment'] }, // Filter by environment + { fields: ['userId'] }, // Query logs by user + { fields: ['accessed_at'] }, // Time-range queries and sorting +] +``` + +### Relationships + +``` +access_logs +├── belongsTo projects (as project) +│ ├── foreignKey: projectId +│ ├── onDelete: CASCADE ← Logs deleted when project deleted +│ └── onUpdate: CASCADE +│ +├── belongsTo users (as user) +│ ├── foreignKey: userId +│ ├── onDelete: CASCADE ← Logs deleted when user deleted +│ └── onUpdate: CASCADE +│ +├── belongsTo users (as createdBy) ─── Audit: who created this log +└── belongsTo users (as updatedBy) ─── Audit: who last updated this log +``` + +## Audit Trail Pattern + +### CreatedBy/UpdatedBy Implementation + +All entities in the system implement the audit trail pattern: + +**Source:** `backend/src/db/api/base.api.js` + +```javascript +// On CREATE +static async create({ data, currentUser = { id: null }, transaction }) { + const mappedData = this.getFieldMapping(data); + + const record = await this.MODEL.create({ + ...mappedData, + createdById: currentUser.id, // Track creator + updatedById: currentUser.id, // Initial updater = creator + }, { transaction }); +} + +// On UPDATE +static async update({ id, data, currentUser = { id: null }, transaction }) { + + await record.update({ + ...updatePayload, + updatedById: currentUser.id, // Track who modified + }, { transaction }); +} + +// On DELETE (soft delete) +static async remove({ id, currentUser = { id: null }, transaction }) { + + await record.update({ deletedBy: currentUser.id }, { transaction }); + await record.destroy({ transaction }); // Sets deletedAt +} +``` + +### Entities with Audit Trail + +All models include audit relationships: + +```javascript +// Applied to ALL models: +db.[entity].belongsTo(db.users, { as: 'createdBy' }); +db.[entity].belongsTo(db.users, { as: 'updatedBy' }); +``` + +**Audited Entities:** +- users, roles, permissions +- projects, project_memberships +- tour_pages, project_audio_tracks +- assets, asset_variants +- element_type_defaults, project_element_defaults +- publish_events, pwa_caches +- presigned_url_requests, access_logs + +**Note:** Page elements, navigation links, and transitions are stored in `tour_pages.ui_schema_json` and audited as part of the tour_pages record. + +### Timestamp Tracking + +Sequelize provides automatic timestamps: + +```javascript +{ + timestamps: true, // Enables createdAt, updatedAt + paranoid: true, // Enables deletedAt (soft delete) +} +``` + +| Field | Set When | Never Changes | +|-------|----------|---------------| +| `createdAt` | Record created | ✓ Immutable | +| `updatedAt` | Any modification | Updates each save | +| `deletedAt` | Soft delete called | - | + +## Request Logging Middleware + +### Logger Configuration + +**Source:** `backend/src/utils/logger.js` + +```javascript +const logger = pino({ + level: process.env.LOG_LEVEL || 'info', + transport: isDevelopment + ? { target: 'pino-pretty', options: { colorize: true } } + : undefined, + base: { + service: 'tour-builder-api', + env: process.env.NODE_ENV || 'development', + }, +}); +``` + +### Request Logger Middleware + +```javascript +function requestLogger(req, res, next) { + // Generate or extract request ID for tracing + const requestId = req.headers['x-request-id'] || crypto.randomUUID(); + req.log = logger.child({ requestId }); + req.requestId = requestId; + res.setHeader('X-Request-Id', requestId); + + const start = Date.now(); + + res.on('finish', () => { + const duration = Date.now() - start; + const logData = { + method: req.method, + url: req.originalUrl || req.url, + status: res.statusCode, + duration, + userAgent: req.headers['user-agent'], + }; + + // Log level based on response status + if (res.statusCode >= 500) { + req.log.error(logData, 'Request completed with server error'); + } else if (res.statusCode >= 400) { + req.log.warn(logData, 'Request completed with client error'); + } else { + req.log.info(logData, 'Request completed'); + } + }); + + next(); +} +``` + +### Applied in Application + +**Source:** `backend/src/index.ts` + +```javascript +app.enable('trust proxy'); // Extract real IP behind proxies +app.use(requestLogger); // Apply to all routes +``` + +### Log Output Format + +**Development (pino-pretty):** +``` +[10:30:45.123] INFO: Request completed + method: "GET" + url: "/api/projects" + status: 200 + duration: 45 + userAgent: "Mozilla/5.0..." + requestId: "abc123-def456" +``` + +**Production (JSON):** +```json +{ + "level": 30, + "time": 1711234567890, + "service": "tour-builder-api", + "env": "production", + "requestId": "abc123-def456", + "method": "GET", + "url": "/api/projects", + "status": 200, + "duration": 45, + "userAgent": "Mozilla/5.0..." +} +``` + +## API Reference + +### Endpoints + +**Source:** `backend/src/routes/access_logs.ts` + +| Method | Endpoint | Permission | Description | +|--------|----------|------------|-------------| +| GET | `/api/access_logs` | READ_ACCESS_LOGS | List with filtering | +| GET | `/api/access_logs/:id` | READ_ACCESS_LOGS | Get single log | +| POST | `/api/access_logs` | CREATE_ACCESS_LOGS | Create log entry | +| PUT | `/api/access_logs/:id` | UPDATE_ACCESS_LOGS | Update log | +| DELETE | `/api/access_logs/:id` | DELETE_ACCESS_LOGS | Soft delete log | + +### Query Parameters + +**Source:** `backend/src/db/api/access_logs.ts` + +| Parameter | Type | Description | +|-----------|------|-------------| +| `limit` | number | Results per page (default: 50) | +| `page` | number | Page number (0-based) | +| `field` | string | Sort field | +| `sort` | 'asc' \| 'desc' | Sort direction | +| `path` | string | Search path (ILIKE) | +| `ip_address` | string | Search IP address (ILIKE) | +| `user_agent` | string | Search user agent (ILIKE) | +| `accessed_atRange` | [start, end] | Date range filter | +| `environment` | enum | Filter by environment | +| `project` | UUID \| string | Filter by project | +| `user` | UUID \| string | Filter by user | +| `filetype` | 'csv' | Export as CSV | + +### Request/Response Examples + +**List Logs with Filters:** +```http +GET /api/access_logs?limit=50&page=1&environment=production&accessed_atRange=["2024-01-01","2024-12-31"]&sort=DESC&field=accessed_at +Authorization: Bearer +``` + +**Response:** +```json +{ + "rows": [ + { + "id": "log-uuid-1", + "projectId": "project-uuid", + "userId": "user-uuid", + "environment": "production", + "path": "/api/tour_pages", + "ip_address": "192.168.1.100", + "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)...", + "accessed_at": "2024-03-15T10:30:00Z", + "project": { + "id": "project-uuid", + "name": "My Tour" + }, + "user": { + "id": "user-uuid", + "firstName": "John", + "lastName": "Doe" + }, + "createdAt": "2024-03-15T10:30:00Z", + "updatedAt": "2024-03-15T10:30:00Z" + } + ], + "count": 1542 +} +``` + +**Create Log Entry:** +```http +POST /api/access_logs +Content-Type: application/json +Authorization: Bearer + +{ + "data": { + "project": "project-uuid", + "user": "user-uuid", + "environment": "production", + "path": "/api/tour_pages", + "ip_address": "192.168.1.100", + "user_agent": "Mozilla/5.0...", + "accessed_at": "2024-03-15T10:30:00Z" + } +} +``` + +### CSV Export + +**Fields Exported:** +- id +- environment +- path +- ip_address +- user_agent +- accessed_at +- createdAt + +```http +GET /api/access_logs?filetype=csv +Authorization: Bearer +``` + +## Frontend Implementation + +### Admin Pages + +**Location:** `frontend/src/pages/access_logs/` + +| Page | Path | Description | +|------|------|-------------| +| List | `/access_logs/access_logs-list` | Filterable table view | +| Table | `/access_logs/access_logs-table` | Data grid variant | +| New | `/access_logs/access_logs-new` | Create log entry | +| Edit | `/access_logs/access_logs-edit?id=` | Edit log | +| View | `/access_logs/access_logs-view?id=` | Read-only details | +| Dynamic | `/access_logs/[access_logsId]` | Dynamic route (ID in path) | + +**Note:** Uses `useEditPageSync` hook for edit page data synchronization. + +### List Page Filters + +**Source:** `frontend/src/pages/access_logs/access_logs-list.tsx` + +| Filter | Type | Description | +|--------|------|-------------| +| Path | Text | Search by API path | +| IP Address | Text | Search by client IP | +| User Agent | Text | Search by browser/client | +| Accessed At | Date Range | Filter by time range | +| Project | Dropdown | Filter by project | +| User | Dropdown | Filter by user | +| Environment | Enum | admin / stage / production | + +### Table Columns + +**Source:** `frontend/src/components/Access_logs/configureAccess_logsCols.tsx` + +Uses `createColumnLoader` factory from `configBuilderFactory.tsx`: + +| Column | Editable | Type | +|--------|----------|------| +| project | Yes | singleSelectRelation (projects) | +| environment | Yes | text | +| user | Yes | singleSelectRelation (users) | +| path | Yes | text | +| ip_address | Yes | text | +| user_agent | Yes | text | +| accessed_at | Yes | datetime | +| actions | - | actions (Edit / View / Delete) | + +### TypeScript Interface + +**Source:** `frontend/src/types/entities.ts` + +```typescript +export interface AccessLog extends BaseEntity { + user?: User | string | null; + project?: Project | string | null; + environment?: 'admin' | 'stage' | 'production'; + path?: string; + action?: string; + ip_address?: string; + user_agent?: string; + accessed_at?: string | Date; + metadata?: Record; +} +``` + +### Redux State + +**Source:** `frontend/src/stores/access_logs/access_logsSlice.ts` + +Uses `createEntitySlice` factory for standardized CRUD operations: + +```typescript +const { slice, actions, reducer } = createEntitySlice({ + name: 'access_logs', + endpoint: 'access_logs', + singularName: 'Access Log', +}); + +export const { + fetch, // Load logs with query + create, // Create new log + update, // Update existing log + deleteItem, // Delete single log + deleteItemsByIds,// Batch delete + uploadCsv, // Import from CSV + setRefetch, // Trigger refresh +} = actions; + +// Usage +dispatch(fetch({ query: '?environment=production&limit=100' })); +``` + +## Security & Permissions + +### Role-Permission Matrix + +**Source:** `backend/src/db/seeders/20200430130760-user-roles.js` + +| Role | CREATE | READ | UPDATE | DELETE | +|------|--------|------|--------|--------| +| PlatformOwner | ✓ | ✓ | ✓ | ✓ | +| Administrator | ✓ | ✓ | ✓ | ✓ | +| AccountManager | ✗ | ✓ | ✗ | ✗ | +| TourDesigner | ✗ | ✓ | ✗ | ✗ | +| ContentReviewer | ✗ | ✓ | ✗ | ✗ | +| AnalyticsViewer | ✗ | ✓ | ✗ | ✗ | + +### Permission Check Flow + +``` +Request → JWT Auth → checkCrudPermissions('access_logs') → Route Handler + │ + ▼ + ┌───────────────────────────────┐ + │ 1. AccessPolicy │ + │ 2. Custom user permissions │ + │ 3. Role-based permissions │ + │ 4. Public hardening │ + └───────────────────────────────┘ +``` + +### Cascade Delete Behavior + +Both user and project deletions cascade to access logs: + +```javascript +// access_logs -> users +onDelete: 'CASCADE' // Logs deleted with user + +// access_logs -> projects +onDelete: 'CASCADE' // Logs deleted with project +``` + +> **Note:** Access logs are NOT preserved when users or projects are deleted. If audit history retention is required, consider archiving logs before deletion or implementing SET NULL behavior. + +## Data Capture Scope + +### What IS Captured + +| Data Point | Location | Purpose | +|------------|----------|---------| +| User ID | access_logs.userId | Who made the request | +| Project ID | access_logs.projectId | Which project context | +| API Path | access_logs.path | What endpoint accessed | +| IP Address | access_logs.ip_address | Client location/identity | +| User Agent | access_logs.user_agent | Browser/client identification | +| Timestamp | access_logs.accessed_at | When it happened | +| Environment | access_logs.environment | admin/stage/production | + +### What IS NOT Captured + +| Data Point | Reason | +|------------|--------| +| Request body | Privacy protection | +| Response content | Privacy/performance | +| Query parameters | Only path stored | +| Authentication credentials | Security | +| Personal user data | Beyond ID reference | + +### Audit Trail vs Access Logs + +| Aspect | Audit Trail | Access Logs | +|--------|-------------|-------------| +| Scope | All entities | Dedicated table | +| Granularity | Per-record changes | Per-request | +| Data | Who modified when | Request details | +| Retention | With entity | Separate lifecycle | +| Purpose | Change tracking | Security monitoring | + +## Soft Delete & Retention + +### Paranoid Mode + +```javascript +{ + timestamps: true, + paranoid: true, // Enables soft deletes +} +``` + +### Deletion Behavior + +1. **Soft Delete**: Sets `deletedAt` timestamp +2. **Query Filtering**: Normal queries exclude deleted records +3. **Restoration**: Possible by clearing `deletedAt` +4. **Hard Delete**: Requires explicit permanent removal + +### Retention Considerations + +**Current Implementation:** +- No automatic log pruning +- Logs retained indefinitely +- Manual cleanup via DELETE endpoints + +**Recommended Practices:** +- Implement TTL-based cleanup for old logs +- Archive logs to cold storage after 90 days +- Consider GDPR right-to-erasure requirements + +## File Structure + +### Backend + +| File | Purpose | +|------|---------| +| `backend/src/db/models/access_logs.js` | Sequelize model definition | +| `backend/src/db/api/access_logs.ts` | Database API (CRUD) | +| `backend/src/db/api/base.api.js` | Audit trail implementation | +| `backend/src/routes/access_logs.ts` | REST endpoints | +| `backend/src/services/access_logs.ts` | Business logic | +| `backend/src/utils/logger.js` | Request logging middleware | +| `backend/src/middlewares/check-permissions.ts` | Permission enforcement | + +### Frontend + +| File | Purpose | +|------|---------| +| `frontend/src/pages/access_logs/access_logs-list.tsx` | List page | +| `frontend/src/pages/access_logs/access_logs-table.tsx` | Table page | +| `frontend/src/pages/access_logs/access_logs-new.tsx` | Create page | +| `frontend/src/pages/access_logs/access_logs-edit.tsx` | Edit page | +| `frontend/src/pages/access_logs/access_logs-view.tsx` | View page | +| `frontend/src/pages/access_logs/[access_logsId].tsx` | Dynamic route | +| `frontend/src/stores/access_logs/access_logsSlice.ts` | Redux state (createEntitySlice) | +| `frontend/src/components/Access_logs/TableAccess_logs.tsx` | Data grid component | +| `frontend/src/components/Access_logs/ListAccess_logs.tsx` | List view component | +| `frontend/src/components/Access_logs/CardAccess_logs.tsx` | Card view component | +| `frontend/src/components/Access_logs/configureAccess_logsCols.tsx` | Column config (createColumnLoader) | +| `frontend/src/types/entities.ts` | TypeScript interfaces | + +## Known Considerations + +### 1. Manual Log Creation + +**Issue:** Access logs are typically created manually via API, not automatically by middleware. + +**Impact:** Requires explicit logging calls or additional middleware to auto-populate. + +### 2. User Deletion Cascades + +**Issue:** `CASCADE` on userId means logs are deleted when user is deleted. + +**Impact:** Audit history lost with user deletion. Consider archiving logs before user deletion or changing to `SET NULL` for preservation. + +### 3. Project Deletion Cascades + +**Issue:** All access logs deleted when project is deleted. + +**Impact:** Audit history lost with project. Consider archiving before deletion. + +### 4. No Query Parameter Logging + +**Issue:** Only `path` is stored, not query strings. + +**Impact:** Cannot reconstruct full request URL. May miss filter/search context. + +### 5. IPv6 Support + +**Issue:** `ip_address` limited to 45 characters. + +**Impact:** Supports full IPv6 addresses (39 chars max). Adequate for current needs. + +### 6. User Agent Truncation + +**Issue:** `user_agent` limited to 1024 characters. + +**Impact:** Very long user agents may be truncated. Rare but possible. + +### 7. No Automatic Retention + +**Issue:** Logs grow indefinitely without cleanup. + +**Impact:** Database size increases over time. Recommend implementing retention policy. + +### 8. TypeScript Interface Extras + +**Issue:** Frontend `AccessLog` interface includes `action` and `metadata` fields not in database model. + +**Impact:** These fields exist only in TypeScript for potential future use. Currently not persisted. + +## Integration Examples + +### Log Access in Custom Middleware + +```javascript +import Access_logsDBApi from '../db/api/access_logs.ts'; + +async function logAccess(req, res, next) { + // Log after response completes + res.on('finish', async () => { + try { + await Access_logsDBApi.create({ + project: req.project?.id, + user: req.currentUser?.id, + environment: determineEnvironment(req), + path: req.path, + ip_address: req.ip, + user_agent: req.headers['user-agent'], + accessed_at: new Date(), + }, { currentUser: req.currentUser }); + } catch (err) { + // Don't fail request on logging error + console.error('Failed to log access:', err); + } + }); + next(); +} +``` + +### Query Logs by Time Range + +```typescript +// Frontend +dispatch(fetch({ + query: '?accessed_atRange=["2024-01-01T00:00:00Z","2024-01-31T23:59:59Z"]&environment=production&sort=DESC&field=accessed_at' +})); +``` + +### Export Logs for Compliance + +```bash +curl -H "Authorization: Bearer $TOKEN" \ + "https://api.example.com/api/access_logs?filetype=csv&accessed_atRange=[\"2024-01-01\",\"2024-03-31\"]" \ + > access_logs_q1_2024.csv +``` diff --git a/documentation/api-reference.md b/documentation/api-reference.md new file mode 100644 index 0000000..8c03521 --- /dev/null +++ b/documentation/api-reference.md @@ -0,0 +1,1700 @@ +# API Reference - E2E Documentation + +## Overview + +The Tour Builder Platform exposes a RESTful API built with Express.js. All endpoints follow consistent patterns for CRUD operations, authentication, and response formats. + +**Base URL:** `http://localhost:3000/api` for local development. + +**Standard VM:** Apache serves the public domain on port `80`, the frontend +PM2 app listens on `3001`, and the backend PM2 app listens on `3000` with +`NODE_ENV=dev_stage`. Use `http://127.0.0.1:3000/api/...` for direct VM backend +checks, or `http://tbp.flatlogic.app/api/...` through Apache/Cloudflare. See +[deployment-vm.md](deployment-vm.md). + +--- + +## Authentication + +### JWT Bearer Token + +Most endpoints require JWT authentication via the Authorization header: + +``` +Authorization: Bearer +``` + +**Token Details:** +- Expiration: 6 hours +- Payload: `{ id, email, name }` +- Obtained via `/api/auth/signin/local` or OAuth callbacks + +### Public Endpoints (No Auth Required) + +- `GET /api/health` +- `POST /api/auth/signin/local` +- `POST /api/auth/send-password-reset-email` +- `PUT /api/auth/password-reset` +- `PUT /api/auth/verify-email` +- `GET /api/auth/email-configured` +- `GET /api/auth/signin/google` +- `GET /api/auth/signin/microsoft` +- `GET /api/runtime-context` + +### Runtime Public Access + +In **production** runtime mode only, certain GET endpoints allow unauthenticated read access (stage requires authentication): +- `GET /api/projects` +- `GET /api/tour_pages` +- `GET /api/project_audio_tracks` +- `GET /api/global-ui-control-defaults` +- `GET /api/project-ui-control-settings/project/:projectId/env/production` + +**Note:** The `X-Runtime-Environment` header must be set to `production` for public access. Stage environment (`stage`) requires JWT authentication as it serves as a workspace for review. + +--- + +## Standard Response Formats + +### Success Responses + +**Single Entity:** +```json +{ + "id": "uuid-here", + "name": "Entity Name", + "createdAt": "2024-01-01T00:00:00.000Z", + "updatedAt": "2024-01-01T00:00:00.000Z" +} +``` + +**List Response:** +```json +{ + "rows": [ + { "id": "uuid-1", "name": "Entity 1" }, + { "id": "uuid-2", "name": "Entity 2" } + ], + "count": 100 +} +``` + +**Count Response:** +```json +{ + "count": 100 +} +``` + +### Error Responses + +```json +{ + "error": "Error message description" +} +``` + +**HTTP Status Codes:** +| Code | Description | +|------|-------------| +| 200 | Success | +| 201 | Created | +| 400 | Bad Request (validation errors) | +| 401 | Unauthorized (missing/invalid token) | +| 403 | Forbidden (insufficient permissions) | +| 404 | Not Found | +| 429 | Too Many Requests (rate limited) | +| 500 | Internal Server Error | + +--- + +## Rate Limiting + +The API uses rate limiting to prevent abuse. Rate limits are applied per IP address, and rate limit headers are returned with each response: + +| Header | Description | +|--------|-------------| +| `X-RateLimit-Limit` | Maximum requests allowed per window | +| `X-RateLimit-Remaining` | Remaining requests in current window | +| `X-RateLimit-Reset` | ISO timestamp when the window resets | +| `Retry-After` | Seconds until you can retry (when limited) | + +### Rate Limit Configuration + +| Limiter | Window | Max Requests | Endpoints | +|---------|--------|--------------|-----------| +| **Auth** | 15 minutes | 10 | `/api/auth/signin/*` | +| **Password Reset** | 1 hour | 5 | `/api/auth/send-password-reset-email` | +| **API** | 1 minute | 100 | General API endpoints | +| **Upload** | 1 minute | 10 | `/api/file/upload*` | +| **Download** | 1 minute | 200 | `/api/file/download`, `/api/file/presign` | +| **Search** | 1 minute | 30 | `/api/search` | + +**Note:** Rate limiting is skipped for localhost (`127.0.0.1`, `::1`) in development mode. + +--- + +## Standard Entity Endpoints + +All entity routes (projects, assets, users, etc.) support these standard endpoints: + +| Method | Endpoint | Description | +|--------|----------|-------------| +| `GET` | `/api/{entity}` | List entities (paginated) | +| `GET` | `/api/{entity}/count` | Get count only | +| `GET` | `/api/{entity}/autocomplete` | Autocomplete search | +| `GET` | `/api/{entity}/:id` | Get single entity by ID | +| `POST` | `/api/{entity}` | Create entity | +| `PUT` | `/api/{entity}/:id` | Update entity | +| `DELETE` | `/api/{entity}/:id` | Delete entity | +| `POST` | `/api/{entity}/deleteByIds` | Bulk delete | +| `POST` | `/api/{entity}/bulk-import` | CSV import | + +### GET /api/{entity}/count + +Get count of entities matching the query. + +**Query Parameters:** Same as list endpoint + +**Response:** +```json +{ + "count": 100 +} +``` + +### GET /api/{entity}/autocomplete + +Get autocomplete suggestions for a field. + +**Query Parameters:** +| Parameter | Type | Description | +|-----------|------|-------------| +| `query` | string | Search query | +| `limit` | number | Max results (default: 10) | +| `offset` | number | Pagination offset | + +**Response:** +```json +[ + { "id": "uuid-1", "label": "Option 1" }, + { "id": "uuid-2", "label": "Option 2" } +] +``` + +--- + +## Standard Query Parameters + +All list endpoints support these query parameters: + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `limit` | number | 10 | Items per page | +| `offset` | number | 0 | Pagination offset | +| `page` | number | 0 | Page number (alternative to offset) | +| `search` | string | - | Text search across searchable fields | +| `sort` | string | - | Sort field (e.g., `name` or `name:DESC`) | +| `filetype` | string | - | Export format (`csv`) | + +### Entity-Specific Filters + +Filters vary by entity based on their `SEARCHABLE_FIELDS`, `RANGE_FIELDS`, and `ENUM_FIELDS`: + +``` +# Text search +?search=keyword + +# Range filters (numeric/date) +?createdAtRange=2024-01-01,2024-12-31 +?sizeRange=0,1000000 + +# Enum/exact match filters +?type=image +?status=active +?environment=production + +# Relationship filters +?projectId=uuid +?tour_pageId=uuid +``` + +--- + +## Authentication Endpoints + +### POST /api/auth/signin/local + +Login with email and password. + +**Rate Limit:** 10 requests per 15 minutes + +**Request:** +```json +{ + "email": "user@example.com", + "password": "password123" +} +``` + +**Response:** +```json +{ + "id": "user-uuid", + "email": "user@example.com", + "firstName": "John", + "lastName": "Doe", + "token": "eyJhbGciOiJIUzI1NiIs..." +} +``` + +### Self-Registration + +Self-registration is disabled. The API does not expose +`POST /api/auth/signup`; new users are created through the authenticated Users +flow and receive an invitation/setup link. + +### GET /api/auth/me + +Get current authenticated user. + +**Auth:** Required + +**Response:** +```json +{ + "id": "user-uuid", + "email": "user@example.com", + "firstName": "John", + "lastName": "Doe", + "app_role": { + "id": "role-uuid", + "name": "Admin" + } +} +``` + +### PUT /api/auth/password-update + +Update password for authenticated user. + +**Auth:** Required + +**Request:** +```json +{ + "currentPassword": "oldpassword", + "newPassword": "newpassword123" +} +``` + +### POST /api/auth/send-password-reset-email + +Send password reset email. + +**Rate Limit:** 5 requests per hour + +**Request:** +```json +{ + "email": "user@example.com" +} +``` + +### POST /api/auth/send-email-address-verification-email + +Resend email verification email for current user. + +**Auth:** Required + +**Response:** `true` on success + +### PUT /api/auth/password-reset + +Reset password with token. + +**Request:** +```json +{ + "token": "reset-token-from-email", + "password": "newpassword123" +} +``` + +### PUT /api/auth/profile + +Update user profile. + +**Auth:** Required + +**Request:** +```json +{ + "profile": { + "firstName": "John", + "lastName": "Doe", + "phoneNumber": "+1234567890" + } +} +``` + +### OAuth Endpoints + +| Endpoint | Description | +|----------|-------------| +| `GET /api/auth/signin/google` | Initiate Google OAuth | +| `GET /api/auth/signin/google/callback` | Google OAuth callback | +| `GET /api/auth/signin/microsoft` | Initiate Microsoft OAuth | +| `GET /api/auth/signin/microsoft/callback` | Microsoft OAuth callback | + +--- + +## System Endpoints + +### GET /api/health + +Health check endpoint. + +**Auth:** Not required + +**Response:** +```json +{ + "status": "ok", + "timestamp": "2024-01-01T00:00:00.000Z", + "uptime": 3600, + "environment": "production", + "database": "connected" +} +``` + +### GET /api/runtime-context + +Get runtime context information. + +**Auth:** Not required + +**Response:** +```json +{ + "mode": "admin", + "projectSlug": null +} +``` + +**Modes:** +- `admin` - Full access (localhost or admin subdomain) +- `stage` - Stage environment (stage.{project}.domain) +- `production` - Production environment ({project}.domain) + +--- + +## File Endpoints + +### POST /api/file/upload-sessions/init + +Initialize chunked upload session. + +**Auth:** Required + +**Request:** +```json +{ + "filename": "video.mp4", + "size": 104857600, + "mimeType": "video/mp4", + "projectId": "project-uuid" +} +``` + +**Response:** +```json +{ + "sessionId": "session-uuid", + "chunkSize": 5242880, + "totalChunks": 20, + "expiresAt": "2024-01-02T00:00:00.000Z" +} +``` + +### GET /api/file/upload-sessions/:sessionId + +Get upload session status. + +**Auth:** Required + +**Response:** +```json +{ + "sessionId": "session-uuid", + "filename": "video.mp4", + "uploadedChunks": [0, 1, 2], + "totalChunks": 20, + "status": "uploading" +} +``` + +### PUT /api/file/upload-sessions/:sessionId/chunks/:chunkIndex + +Upload file chunk. + +**Auth:** Required + +**Content-Type:** `application/octet-stream` + +**Body:** Raw binary data + +**Response:** +```json +{ + "chunkIndex": 0, + "received": true +} +``` + +### POST /api/file/upload-sessions/:sessionId/finalize + +Finalize chunked upload. + +**Auth:** Required + +**Response:** +```json +{ + "success": true, + "asset": { + "id": "asset-uuid", + "url": "https://cdn.example.com/video.mp4", + "storageKey": "uploads/video.mp4" + } +} +``` + +### GET /api/file/download + +Download file. + +**Query Parameters:** +- `privateUrl` - Encoded file path +- `id` - Asset ID (alternative) + +**Response:** File stream with appropriate Content-Type + +### POST /api/file/presign + +Generate presigned URLs for batch asset downloads. For S3 storage, returns direct S3 signed URLs for client-side downloads (bypassing the backend). For other storage providers, returns backend proxy URLs. + +**Auth:** Not required (public endpoint for runtime asset preloading) + +**Request:** +```json +{ + "urls": [ + "uploads/projects/abc/image1.jpg", + "uploads/projects/abc/video.mp4", + "uploads/projects/abc/audio.mp3" + ] +} +``` + +**Limits:** +- Maximum 50 URLs per request +- All URLs must be non-empty strings + +**Response:** +```json +{ + "presignedUrls": { + "uploads/projects/abc/image1.jpg": "https://bucket.s3.region.amazonaws.com/uploads/projects/abc/image1.jpg?X-Amz-...", + "uploads/projects/abc/video.mp4": "https://bucket.s3.region.amazonaws.com/uploads/projects/abc/video.mp4?X-Amz-...", + "uploads/projects/abc/audio.mp3": "https://bucket.s3.region.amazonaws.com/uploads/projects/abc/audio.mp3?X-Amz-..." + } +} +``` + +**Presigned URL Expiry:** 1 hour (3600 seconds) + +**Use Case:** Frontend preloading uses this endpoint to get direct S3 download URLs, enabling: +- Parallel asset downloads without backend proxy overhead +- Direct storage in browser Cache API +- Blob URL creation for instant display + +### POST /api/file/upload/:table/:field + +Legacy single-file upload. + +**Auth:** Required + +**Content-Type:** `multipart/form-data` + +--- + +## Projects Endpoints + +### POST /api/projects + +Create a new project. + +**Auth:** Required + +**Request:** +```json +{ + "data": { + "name": "My Tour", + "slug": "my-tour", + "description": "A virtual tour", + "logo_url": "https://...", + "favicon_url": "https://...", + "og_image_url": "https://...", + "theme_config_json": {}, + "custom_css_json": {}, + "cdn_base_url": "https://cdn.example.com" + } +} +``` + +### GET /api/projects + +List all projects. + +**Auth:** Optional (runtime public mode) + +**Query Parameters:** +- Standard pagination/filtering + +**Response:** +```json +{ + "rows": [ + { + "id": "project-uuid", + "name": "My Tour", + "slug": "my-tour", + "tour_pages": [...], + "assets": [...] + } + ], + "count": 10 +} +``` + +### GET /api/projects/:id + +Get project by ID. + +**Auth:** Optional (runtime public mode) + +**Includes:** tour_pages, assets, project_audio_tracks, project_element_defaults + +### PUT /api/projects/:id + +Update project. + +**Auth:** Required + +**Request:** +```json +{ + "id": "project-uuid", + "data": { + "name": "Updated Tour Name" + } +} +``` + +### DELETE /api/projects/:id + +Delete project. + +**Auth:** Required + +### POST /api/projects/:id/clone + +Clone project with all related entities. + +**Auth:** Required + +**Response:** +```json +{ + "id": "new-project-uuid", + "name": "My Tour (Copy)", + "slug": "my-tour-copy" +} +``` + +### POST /api/publish/save-to-stage + +Copy all `dev` environment content to `stage` for preview. + +**Auth:** Required + +**Request:** +```json +{ + "projectId": "project-uuid" +} +``` + +**Response:** +```json +{ + "success": true, + "publishEventId": "event-uuid", + "summary": { + "pages_copied": 10, + "audios_copied": 2 + } +} +``` + +**Note:** This is part of the dev → stage → production workflow. Content is edited in `dev` (Constructor), previewed in `stage`, then published to `production`. Page elements, navigation, and transitions are stored in `tour_pages.ui_schema_json` and copied with pages. + +## Tour Pages Endpoints + +### POST /api/tour_pages + +Create a tour page. + +**Auth:** Required + +**Request:** +```json +{ + "data": { + "title": "Home Page", + "slug": "home", + "sort_order": 0, + "projectId": "project-uuid", + "environment": "dev", + "background_image_url": "https://...", + "background_video_url": "https://...", + "ui_schema_json": {} + } +} +``` + +### GET /api/tour_pages + +List tour pages. + +**Auth:** Optional (runtime public mode) + +**Query Parameters:** +- `projectId` - Filter by project +- `environment` - Filter by environment + +### GET /api/tour_pages/:id + +Get tour page by ID. + +**Includes:** project + +**Note:** Page elements are stored directly in the `ui_schema_json` field. + +### PUT /api/tour_pages/:id + +Update tour page. + +**Auth:** Required + +### POST /api/tour_pages/reorder + +Update page order within a project/environment without touching page content. + +**Auth:** Required + +**Request:** +```json +{ + "data": { + "projectId": "project-uuid", + "environment": "dev", + "orderedPageIds": ["page-uuid-1", "page-uuid-2", "page-uuid-3"] + } +} +``` + +**Behavior:** +- Reordering is allowed only in `dev`. +- `orderedPageIds` must contain every dev page in that project exactly once. +- Duplicate IDs, missing IDs, IDs from another project, and unknown IDs are + rejected. +- The backend updates only `sort_order`; page content, slugs, navigation links, + and media fields are not changed. +- Stage receives the new order after `POST /api/publish/save-to-stage`. +- Production receives the new order after `POST /api/publish`. + +**Response:** +```json +[ + { "id": "page-uuid-1", "sort_order": 1 }, + { "id": "page-uuid-2", "sort_order": 2 }, + { "id": "page-uuid-3", "sort_order": 3 } +] +``` + +**Errors:** +| Status | Reason | +|--------|--------| +| 400 | Missing `projectId`, invalid `orderedPageIds`, duplicate/missing pages, or `environment` other than `dev` | +| 403 | Authenticated user is not allowed to update pages in the project | + +### POST /api/tour_pages/:id/duplicate + +Duplicate a dev tour page into a new independent dev page. + +**Auth:** Required + +**Request:** +```json +{ + "data": { + "projectId": "project-uuid", + "environment": "dev", + "name": "Lobby Copy", + "slug": "lobby-copy" + } +} +``` + +**Behavior:** +- Only dev pages can be duplicated. Stage and production are updated through + Save to Stage and Publish. +- The duplicate receives a new page ID, unique slug, blank `source_key`, and + `sort_order = max(project dev sort_order) + 1`. +- Page media/settings fields and `ui_schema_json` are copied from the source + page. +- Inline element IDs and nested item IDs in `ui_schema_json` are regenerated so + the new page can be edited independently. +- Asset URLs, transition URLs, and navigation `targetPageSlug` values are + preserved. +- Existing reverse-video processing still runs through `TourPagesService`, so + transition reverse URLs are handled consistently with normal page save/create. + +**Response:** Created tour page record. + +**Errors:** +| Status | Reason | +|--------|--------| +| 400 | Missing source page, source page outside project, or non-dev source/target environment | +| 403 | Authenticated user is not allowed to create/update pages in the project | + +### DELETE /api/tour_pages/:id + +Delete tour page. + +**Auth:** Required + +**Constructor behavior:** the constructor uses this same endpoint for the active +dev page after showing a confirmation modal. After deletion it reloads page data +and selects the next page in presentation order when available. + +--- + +## Assets Endpoints + +### POST /api/assets + +Create an asset record. + +**Auth:** Required + +**Request:** +```json +{ + "data": { + "name": "Hero Image", + "cdn_url": "https://cdn.example.com/image.jpg", + "storage_key": "uploads/image.jpg", + "mime_type": "image/jpeg", + "checksum": "abc123...", + "width_px": 1920, + "height_px": 1080, + "size_mb": 0.5, + "duration_sec": null, + "projectId": "project-uuid" + } +} +``` + +### GET /api/assets + +List assets. + +**Auth:** Required + +**Query Parameters:** +- `projectId` - Filter by project +- `mime_type` - Filter by MIME type + +**Includes:** asset_variants, project + +### GET /api/assets/:id + +Get asset by ID. + +**Includes:** asset_variants + +### PUT /api/assets/:id + +Update asset. + +**Auth:** Required + +### DELETE /api/assets/:id + +Delete asset (and file from storage). + +**Auth:** Required + +--- + +## Asset Variants Endpoints + +### POST /api/asset_variants + +Create an asset variant. + +**Auth:** Required + +**Request:** +```json +{ + "data": { + "assetId": "asset-uuid", + "variant_type": "webp", + "cdn_url": "https://cdn.example.com/image.webp", + "storage_key": "uploads/image.webp", + "mime_type": "image/webp", + "width_px": 1920, + "height_px": 1080, + "size_bytes": 51200 + } +} +``` + +**Variant Types:** +- `thumbnail` - Small preview +- `preview` - Medium preview +- `webp` - WebP format +- `mp4_low` - Low quality video +- `mp4_high` - High quality video +- `original` - Original file + +### GET /api/asset_variants + +List asset variants. + +**Auth:** Required + +**Query Parameters:** +- `assetId` - Filter by parent asset +- `variant_type` - Filter by variant type + +--- + +## Project Audio Tracks Endpoints + +### POST /api/project_audio_tracks + +Create a project audio track. + +**Auth:** Required + +**Request:** +```json +{ + "data": { + "name": "Background Music", + "projectId": "project-uuid", + "audio_url": "https://cdn.example.com/music.mp3", + "duration_sec": 180, + "volume": 0.8, + "loop": true, + "autoplay": true + } +} +``` + +### GET /api/project_audio_tracks + +List audio tracks. + +**Auth:** Optional (runtime public mode) + +**Query Parameters:** +- `projectId` - Filter by project + +--- + +## Project Memberships Endpoints + +Manage team collaboration and per-project access control. + +### POST /api/project_memberships + +Create a project membership (invite user to project). + +**Auth:** Required + +**Request:** +```json +{ + "data": { + "projectId": "project-uuid", + "userId": "user-uuid", + "access_level": "editor", + "is_active": true, + "invited_at": "2024-01-01T00:00:00.000Z" + } +} +``` + +**Access Levels:** `owner`, `editor`, `reviewer`, `viewer` + +### GET /api/project_memberships + +List project memberships. + +**Auth:** Required + +**Query Parameters:** +- `projectId` or `project` - Filter by project +- `userId` or `user` - Filter by user +- `access_level` - Filter by access level +- `is_active` - Filter by active status + +### GET /api/project_memberships/:id + +Get project membership by ID. + +**Auth:** Required + +**Includes:** project, user + +### PUT /api/project_memberships/:id + +Update project membership. + +**Auth:** Required + +**Request:** +```json +{ + "id": "membership-uuid", + "data": { + "access_level": "viewer", + "is_active": false + } +} +``` + +### DELETE /api/project_memberships/:id + +Remove user from project. + +**Auth:** Required + +--- + +## PWA Caches Endpoints + +Track PWA cache versions and offline asset manifests per project. + +### POST /api/pwa_caches + +Create a PWA cache record. + +**Auth:** Required + +**Request:** +```json +{ + "data": { + "projectId": "project-uuid", + "environment": "production", + "cache_version": "v1704067200000", + "manifest_json": {}, + "asset_list_json": [], + "generated_at": "2024-01-01T00:00:00.000Z", + "is_active": true + } +} +``` + +**Environment Values:** `dev`, `stage`, `production` + +### GET /api/pwa_caches + +List PWA cache records. + +**Auth:** Required + +**Query Parameters:** +- `projectId` or `project` - Filter by project +- `environment` - Filter by environment +- `is_active` - Filter by active status + +### GET /api/pwa_caches/:id + +Get PWA cache by ID. + +**Auth:** Required + +**Includes:** project + +### PUT /api/pwa_caches/:id + +Update PWA cache record. + +**Auth:** Required + +### DELETE /api/pwa_caches/:id + +Delete PWA cache record. + +**Auth:** Required + +--- + +## Users Endpoints + +### POST /api/users + +Create a user. + +**Auth:** Required (Admin) + +**Request:** +```json +{ + "data": { + "email": "newuser@example.com", + "password": "password123", + "firstName": "John", + "lastName": "Doe", + "phoneNumber": "+1234567890", + "app_roleId": "role-uuid" + } +} +``` + +### GET /api/users + +List users. + +**Auth:** Required + +**Searchable Fields:** firstName, lastName, email, phoneNumber + +### GET /api/users/:id + +Get user by ID. + +**Auth:** Required + +**Includes:** app_role + +### PUT /api/users/:id + +Update user. + +**Auth:** Required + +### DELETE /api/users/:id + +Delete user. + +**Auth:** Required (Admin) + +--- + +## Roles Endpoints + +### POST /api/roles + +Create a role. + +**Auth:** Required (Admin) + +**Request:** +```json +{ + "data": { + "name": "Editor", + "globalAccess": false + } +} +``` + +### GET /api/roles + +List roles. + +**Auth:** Required + +### GET /api/roles/:id + +Get role by ID. + +**Includes:** permissions + +--- + +## Permissions Endpoints + +### POST /api/permissions + +Create a permission. + +**Auth:** Required (Admin) + +**Request:** +```json +{ + "data": { + "roleId": "role-uuid", + "entity": "projects", + "action": "read" + } +} +``` + +**Actions:** `create`, `read`, `update`, `delete` + +--- + +## Publishing Endpoints + +The platform uses a three-tier publishing workflow: + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ Constructor (dev) Stage (preview) Production (live) │ +│ dev ───► stage ───► production │ +│ "Save to Stage" "Publish" │ +└─────────────────────────────────────────────────────────────────┘ +``` + +### POST /api/publish/save-to-stage + +Copy `dev` content to `stage` for preview. See [Projects Endpoints](#post-apipublishsave-to-stage). + +### POST /api/publish (or POST /api/publish/publish) + +Publish project from `stage` to `production` environment. Both endpoints are aliases and perform the same action. + +**Auth:** Required + +**Request:** +```json +{ + "projectId": "project-uuid", + "title": "v1.0 Release", + "description": "Initial production release" +} +``` + +**Response:** +```json +{ + "success": true, + "publishEventId": "event-uuid", + "summary": { + "pages_copied": 10, + "audios_copied": 2 + } +} +``` + +**Note:** Publishes `stage` environment content to `production`. Use `save-to-stage` first to copy changes from `dev`. Page elements, navigation links, and transitions are stored within `tour_pages.ui_schema_json` and copied automatically with pages. + +--- + +## Publish Events Endpoints + +### GET /api/publish_events + +List publish events. + +**Auth:** Required + +**Query Parameters:** +- `projectId` - Filter by project + +**Response:** +```json +{ + "rows": [ + { + "id": "event-uuid", + "projectId": "project-uuid", + "title": "v1.0 Release", + "description": "Initial release", + "publishedAt": "2024-01-01T00:00:00.000Z", + "createdBy": "user-uuid" + } + ], + "count": 5 +} +``` + +--- + +## Access Logs Endpoints + +### POST /api/access_logs + +Create an access log entry. + +**Auth:** Required + +**Request:** +```json +{ + "data": { + "project": "project-uuid", + "user": "user-uuid", + "environment": "production", + "path": "/api/tour_pages", + "ip_address": "192.168.1.1", + "user_agent": "Mozilla/5.0...", + "accessed_at": "2024-01-01T00:00:00.000Z" + } +} +``` + +**Environment Values:** `admin`, `stage`, `production` + +### GET /api/access_logs + +List access logs. + +**Auth:** Required + +**Searchable Fields:** `path`, `ip_address`, `user_agent` + +**Query Parameters:** +- `project` - Filter by project ID or name +- `user` - Filter by user ID or name +- `environment` - Filter by environment (admin/stage/production) +- `accessed_atRange` - Filter by date range + +--- + +## Presigned URL Requests Endpoints + +Track and audit presigned URL generation requests. + +### POST /api/presigned_url_requests + +Create a presigned URL request record. + +**Auth:** Required + +**Request:** +```json +{ + "data": { + "requested_key": "uploads/projects/abc/image.jpg", + "mime_type": "image/jpeg", + "status": "pending", + "requested_size_mb": 2.5, + "expires_at": "2024-01-01T01:00:00.000Z" + } +} +``` + +**Status Values:** `pending`, `completed`, `expired`, `failed` + +### GET /api/presigned_url_requests + +List presigned URL requests. + +**Auth:** Required + +**Query Parameters:** +- `status` - Filter by status +- `requested_key` - Filter by storage key + +### GET /api/presigned_url_requests/:id + +Get presigned URL request by ID. + +**Auth:** Required + +### PUT /api/presigned_url_requests/:id + +Update presigned URL request. + +**Auth:** Required + +### DELETE /api/presigned_url_requests/:id + +Delete presigned URL request. + +**Auth:** Required + +--- + +## Global UI Control Defaults Endpoints + +Global UI controls are the system-owned fullscreen, sound, and offline buttons. +Runtime settings resolve through: + +``` +global_ui_control_defaults → project_ui_control_settings → tour_pages.global_ui_controls_settings_json +``` + +### GET /api/global-ui-control-defaults + +Get singleton platform defaults. + +**Auth:** Not required for runtime reads + +### PUT /api/global-ui-control-defaults/:id + +Update singleton platform defaults. + +**Auth:** Required + +### GET /api/project-ui-control-settings/project/:projectId/env/:environment + +Get project/environment overrides. `environment` is `dev`, `stage`, or +`production`. + +**Auth:** Required for `dev` and `stage`. Production is public for public +presentations and requires JWT/access grants for private production +presentations. + +### PUT /api/project-ui-control-settings/project/:projectId/env/:environment + +Upsert project/environment overrides. + +**Auth:** Required + +### DELETE /api/project-ui-control-settings/project/:projectId/env/:environment + +Delete project/environment overrides so runtime falls back to global defaults. + +**Auth:** Required + +--- + +## Element Type Defaults Endpoints + +Global platform-wide default settings for each element type. These are the source of truth that get snapshotted to `project_element_defaults` when a project is created. + +**Alias:** `/api/ui-elements` routes to the same endpoints as `/api/element-type-defaults`. + +**Note:** The Constructor page uses `project_element_defaults` (not this endpoint) to fetch element defaults for the current project. This endpoint is for platform-wide administration of default settings. + +### GET /api/element-type-defaults + +List all element type defaults. + +**Auth:** Required + +**Response:** +```json +{ + "rows": [ + { + "id": "uuid", + "element_type": "navigation_next", + "name": "Navigation: Forward", + "sort_order": 1, + "is_active": true, + "default_settings_json": { + "width": "60px", + "height": "60px", + "opacity": 1 + } + } + ], + "count": 11 +} +``` + +### GET /api/element-type-defaults/:id + +Get element type default by ID. + +**Auth:** Required + +### PUT /api/element-type-defaults/:id + +Update element type default settings. + +**Auth:** Required (Admin) + +**Request:** +```json +{ + "id": "uuid", + "data": { + "default_settings_json": { + "width": "80px", + "height": "80px" + } + } +} +``` + +--- + +## Project Element Defaults Endpoints + +Project-specific default settings for UI elements. These are automatically snapshotted from global defaults (`element_type_defaults`) when a project is created. The Constructor page uses this endpoint to fetch element defaults for the current project. + +**Three-Tier Element Defaults Hierarchy:** +``` +element_type_defaults (global) → project_element_defaults (project) → tour_pages.ui_schema_json (instance) +``` + +### GET /api/project-element-defaults + +List project element defaults. + +**Auth:** Required + +**Query Parameters:** +- `projectId` or `project` - Filter by project ID or name (recommended) +- Supports `|` pipe separator for multiple values + +**Response:** +```json +{ + "rows": [ + { + "id": "uuid", + "projectId": "project-uuid", + "element_type": "navigation_next", + "name": "Navigation: Forward", + "sort_order": 1, + "settings_json": { + "width": "60px", + "height": "60px" + }, + "source_element_id": "global-default-uuid", + "snapshot_version": 1 + } + ], + "count": 11 +} +``` + +### GET /api/project-element-defaults/:id + +Get project element default by ID. + +**Auth:** Required + +### PUT /api/project-element-defaults/:id + +Update project-specific default settings. + +**Auth:** Required + +**Request:** +```json +{ + "id": "uuid", + "data": { + "settings_json": { + "width": "100px", + "backgroundColor": "#ff0000" + } + } +} +``` + +### POST /api/project-element-defaults/:id/reset + +Reset project default to current global default. Copies settings from `element_type_defaults` and increments `snapshot_version`. + +**Auth:** Required + +**Response:** Returns the updated project element default record: +```json +{ + "id": "uuid", + "projectId": "project-uuid", + "element_type": "navigation_next", + "name": "Navigation: Forward", + "sort_order": 1, + "settings_json": { "width": "60px", "height": "60px" }, + "source_element_id": "global-default-uuid", + "snapshot_version": 2, + "createdAt": "2024-01-01T00:00:00.000Z", + "updatedAt": "2024-01-15T00:00:00.000Z" +} +``` + +**Error:** Returns 500 if global default not found for the element type. + +### GET /api/project-element-defaults/:id/diff + +Compare project default with current global default. + +**Auth:** Required + +**Response:** +```json +{ + "projectDefault": { + "id": "uuid", + "element_type": "navigation_next", + "settings_json": { "width": "100px" } + }, + "globalDefault": { + "id": "global-uuid", + "element_type": "navigation_next", + "default_settings_json": { "width": "60px" } + }, + "hasGlobalDefault": true, + "isDifferent": true, + "projectSettings": { "width": "100px" }, + "globalSettings": { "width": "60px" } +} +``` + +**Response when global default not found:** +```json +{ + "projectDefault": { ... }, + "globalDefault": null, + "hasGlobalDefault": false, + "isDifferent": true +} +``` + +--- + +## Search Endpoint + +### POST /api/search + +Search across all entities. Searches both text fields and numeric fields (cast to text for matching). + +**Auth:** Required + +**Rate Limit:** 30 requests per minute per IP + +**Request:** +```json +{ + "searchQuery": "keyword" +} +``` + +**Response:** +Returns a flat array of matching records from all searchable tables. Each record includes `tableName` and `matchAttribute` fields to identify the source and matching fields. + +```json +[ + { + "id": "uuid-1", + "name": "My Project", + "slug": "my-project", + "tableName": "projects", + "matchAttribute": ["name", "slug"] + }, + { + "id": "uuid-2", + "title": "Home Page", + "tableName": "tour_pages", + "matchAttribute": ["title"] + } +] +``` + +**Searchable Tables & Fields:** + +| Table | Text Fields | Numeric Fields | +|-------|-------------|----------------| +| `users` | firstName, lastName, phoneNumber, email | - | +| `projects` | name, slug, description, logo_url, favicon_url, og_image_url | - | +| `assets` | name, cdn_url, storage_key, mime_type, checksum | size_mb, width_px, height_px, duration_sec | +| `asset_variants` | cdn_url | width_px, height_px, size_mb | +| `presigned_url_requests` | requested_key, mime_type, status | requested_size_mb | +| `tour_pages` | source_key, name, slug, background_image_url, background_video_url, background_audio_url, ui_schema_json | sort_order | +| `project_audio_tracks` | source_key, name, slug, url | volume, sort_order | +| `publish_events` | error_message | pages_copied, transitions_copied, audios_copied | +| `pwa_caches` | cache_version, manifest_json, asset_list_json | - | +| `access_logs` | path, ip_address, user_agent | - | + +**Permission Check:** Search results are filtered based on user permissions. Only tables where user has `READ_{TABLE_NAME}` permission are searched. + +**Limit:** Maximum 50 results per table. + +--- + +## Bulk Operations + +### POST /api/{entity}/deleteByIds + +Bulk delete entities. + +**Auth:** Required + +**Request:** +```json +{ + "data": ["uuid-1", "uuid-2", "uuid-3"] +} +``` + +**Response:** +```json +{ + "deleted": 3 +} +``` + +### POST /api/{entity}/bulk-import + +Bulk import from CSV. + +**Auth:** Required + +**Content-Type:** `multipart/form-data` + +**Form Fields:** +- `file` - CSV file +- `importHash` - Unique import identifier + +**Response:** +```json +{ + "imported": 50, + "errors": [] +} +``` + +--- + +## CSV Export + +Add `?filetype=csv` to any list endpoint to export as CSV. + +**Example:** +``` +GET /api/projects?filetype=csv&limit=1000 +``` + +**Response:** CSV file download with appropriate headers + +--- + +## Swagger Documentation + +Interactive API documentation available at: + +``` +GET /api-docs +``` + +Provides Swagger UI with all endpoints, request/response schemas, and testing capabilities. diff --git a/documentation/asset-upload-variants.md b/documentation/asset-upload-variants.md new file mode 100644 index 0000000..d9bebcb --- /dev/null +++ b/documentation/asset-upload-variants.md @@ -0,0 +1,1239 @@ +# Asset Upload & Variants + +Complete documentation for the Tour Builder Platform's asset management system including chunked uploads, storage providers, and variant tracking. + +For persisted media metadata, the backend is the source of truth for video FPS. +Frontend upload probing is still used for immediate UX, but the asset +create/update path now probes the stored file with bundled `ffprobe-static` and +saves actual `frame_rate` metadata on the asset record. + +## Overview + +The platform implements a **robust asset management system** with: +- **Chunked uploads** - Large files uploaded in 5MB chunks with resumability +- **Multi-provider storage** - S3, Google Cloud Storage, or local filesystem +- **Asset variants** - Metadata tracking for different file formats/sizes +- **Media probing** - Automatic extraction of duration, dimensions + +## Architecture + +``` +┌─────────────────────────────────────────────────────────────────────────┐ +│ Upload Flow │ +│ │ +│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ +│ │ Frontend │ │ Backend │ │ Storage │ │ +│ │ Uploader │───▶│ Service │───▶│ Provider │ │ +│ └─────────────┘ └─────────────┘ └─────────────┘ │ +│ │ │ │ │ +│ │ Chunks (5MB) │ Session Mgmt │ S3/GCloud/Local │ +│ ▼ ▼ ▼ │ +│ UploadService.js file.js service AWS SDK / GCloud SDK │ +└─────────────────────────────────────────────────────────────────────────┘ + +┌─────────────────────────────────────────────────────────────────────────┐ +│ Database Structure │ +│ │ +│ assets ─────────────────────────────────────────┐ │ +│ │ │ │ +│ ▼ ▼ │ +│ asset_variants projects │ +│ (variant_type, cdn_url, dimensions) (owner) │ +└─────────────────────────────────────────────────────────────────────────┘ + +┌─────────────────────────────────────────────────────────────────────────┐ +│ Download Flow (S3 with Presigned URLs) │ +│ │ +│ ┌─────────────┐ 1. Get presigned URLs ┌─────────────┐ │ +│ │ Frontend │────────────────────────▶│ Backend │ │ +│ │ Preloader │◀────────────────────────│ /presign │ │ +│ └─────────────┘ 2. Return signed URLs └─────────────┘ │ +│ │ │ +│ │ 3. Direct download (fast) │ +│ ▼ │ +│ ┌─────────────┐ │ +│ │ S3 │ No backend proxy - assets served directly from S3 │ +│ └─────────────┘ │ +└─────────────────────────────────────────────────────────────────────────┘ +``` + +## Chunked Upload System + +### Overview + +Large files are uploaded in chunks to handle network interruptions and support resumable uploads. + +**Key Parameters:** +| Parameter | Value | Description | +|-----------|-------|-------------| +| Chunk Size | 5 MB | `5 * 1024 * 1024` bytes | +| Max Retries | 3 | Per-chunk retry limit | +| Retry Backoff | 500ms × retry | Exponential backoff | +| Session TTL | 24 hours | Expired sessions auto-cleaned | + +### Upload Session Flow + +``` +┌─────────────────────────────────────────────────────────────────────────┐ +│ Phase 1: Initialize Session │ +├─────────────────────────────────────────────────────────────────────────┤ +│ POST /file/upload-sessions/init │ +│ Body: { folder, filename, totalChunks, size, contentType } │ +│ Response: { sessionId, uploadedChunks: [], totalChunks } │ +│ │ +│ - Creates UUID session │ +│ - Stores metadata (local: JSON file, S3: S3 object) │ +│ - Triggers cleanup of expired sessions │ +└─────────────────────────────────────────────────────────────────────────┘ + ↓ +┌─────────────────────────────────────────────────────────────────────────┐ +│ Phase 2: Upload Chunks │ +├─────────────────────────────────────────────────────────────────────────┤ +│ PUT /file/upload-sessions/{sessionId}/chunks/{chunkIndex} │ +│ Headers: Content-Type: application/octet-stream │ +│ Body: │ +│ Response: { sessionId, chunkIndex, uploadedChunks, totalChunks } │ +│ │ +│ For each chunk (0 to totalChunks-1): │ +│ - Slice file: start = chunkIndex * 5MB, end = min(size, start+5MB) │ +│ - Send raw binary data │ +│ - Retry up to 3 times with exponential backoff │ +│ - Track progress via onProgress callback │ +└─────────────────────────────────────────────────────────────────────────┘ + ↓ +┌─────────────────────────────────────────────────────────────────────────┐ +│ Phase 3: Finalize Upload │ +├─────────────────────────────────────────────────────────────────────────┤ +│ POST /file/upload-sessions/{sessionId}/finalize │ +│ Response: { message, privateUrl, url } │ +│ │ +│ - Verify all chunks exist │ +│ - Assemble chunks into final file │ +│ - Validate assembled size matches declared size │ +│ - Upload to final storage location │ +│ - Cleanup session directory/objects │ +│ - Return public and private URLs │ +└─────────────────────────────────────────────────────────────────────────┘ +``` + +### Session Status Check (Resumability) + +```http +GET /file/upload-sessions/{sessionId} +``` + +**Response:** +```json +{ + "sessionId": "uuid", + "totalChunks": 10, + "uploadedChunks": [0, 1, 2, 3], + "status": "active" +} +``` + +Use this to resume interrupted uploads by checking which chunks are already uploaded. + +### Session Metadata Structure + +```javascript +{ + id: "uuid", + userId: "user-uuid", // Owner (enforced on all operations) + folder: "assets/project-id", + filename: "uuid.mp4", + totalChunks: 10, + size: 52428800, // 50MB + contentType: "video/mp4", + uploadedChunks: [0, 1, 2], // Completed chunk indices + status: "active", + createdAt: "2025-01-01T00:00:00Z", + updatedAt: "2025-01-01T00:05:00Z" +} +``` + +### Session Storage Locations + +Session management is handled by `UploadSessionManager` class (`backend/src/services/file/UploadSessionManager.ts`). + +**Local Storage:** +``` +{uploadDir}/upload_sessions/{sessionId}/ +├── meta.json # Session metadata +└── chunks/ + ├── 0.part # Chunk 0 + ├── 1.part # Chunk 1 + └── ... +``` + +**S3 Storage:** +``` +{prefix}/_upload_sessions/{sessionId}/ +├── meta.json # Session metadata +└── chunks/ + ├── 0.part # Chunk 0 + ├── 1.part # Chunk 1 + └── ... +``` + +### Chunk Assembly + +During finalization, chunks are assembled in a temp directory: + +```javascript +// Temp assembly location +const tempDir = path.join(config.uploadDir, '_temp_assembly'); +const assembledPath = path.join(tempDir, `${sessionId}.bin`); + +// For each chunk, append to assembled file +for (let chunkIndex = 0; chunkIndex < totalChunks; chunkIndex++) { + await streamAppendFile(assembledPath, chunkPath); +} + +// Validate size +if (assembledStats.size !== session.size) { + throw new Error('Assembled file size mismatch'); +} +``` + +## Storage Providers + +### Modular File Service Architecture + +The file service uses a **Strategy Pattern** with modular providers: + +``` +backend/src/services/ +├── file.js # Unified interface (provider initialization, routing) +└── file/ + ├── index.js # Module exports + ├── BaseStorageProvider.ts # Abstract base class + ├── S3StorageProvider.ts # AWS S3 implementation + ├── LocalStorageProvider.ts # Local filesystem implementation + └── UploadSessionManager.ts # Chunked upload session management +``` + +### External Storage Circuit Breaker + +The unified file service protects S3/GCloud operations used by media processing +with an in-process circuit breaker. Local filesystem storage is not breaker +limited, which keeps local development and simple VM filesystem access direct. +The values are validated in `backend/src/utils/env-validation.ts` and exposed +through `config.resilience.fileStorage.breaker`; service code does not read +these environment variables directly. + +Protected operations: + +- `downloadToBuffer` +- `downloadToTempFile` +- `uploadBuffer` +- `deleteFile` + +Configuration overrides: + +| Variable | Default | Description | +|----------|---------|-------------| +| `FILE_STORAGE_BREAKER_FAILURE_THRESHOLD` | `5` | Consecutive failures before the breaker opens | +| `FILE_STORAGE_BREAKER_COOLDOWN_MS` | `30000` | Cooldown before a half-open probe is allowed | +| `FILE_STORAGE_BREAKER_SUCCESS_THRESHOLD` | `2` | Half-open successes required to close the breaker | + +When the breaker is open, protected operations fail fast with a `503`-status +error instead of piling more requests onto the external storage provider. + +### Provider Selection Logic + +**File:** `backend/src/services/file.ts` + +```javascript +const getFileStorageProvider = () => { + // 1. Explicit override from validated backend config + const provider = config.fileStorage.provider; + if (provider === 's3' || provider === 'gcloud' || provider === 'local') { + return provider; + } + + // 2. Auto-detect S3 from credentials + const hasS3 = Boolean( + config.s3.bucket && config.s3.region && + config.s3.accessKeyId && config.s3.secretAccessKey + ); + if (hasS3) return 's3'; + + // 3. Auto-detect GCloud from credentials + const hasGCloud = Boolean( + config.gcloud.projectId && + config.gcloud.clientEmail && + config.gcloud.privateKey && + config.gcloud.bucket && + config.gcloud.hash + ); + if (hasGCloud) return 'gcloud'; + + // 4. Default to local filesystem + return 'local'; +}; +``` + +`FILE_STORAGE_PROVIDER` is trimmed/lowercased and validated in +`backend/src/utils/env-validation.ts`; runtime service code reads +`config.fileStorage.provider` only. + +### S3 Configuration + +**Environment Variables:** +```bash +FILE_STORAGE_PROVIDER=s3 +AWS_S3_BUCKET=your-bucket-name +AWS_S3_REGION=us-east-1 # Default: us-east-1 +AWS_ACCESS_KEY_ID=AKIA... +AWS_SECRET_ACCESS_KEY=secret... +AWS_S3_PREFIX=optional-prefix # Default: afeefb9d49f5b7977577876b99532ac7 +``` + +**Implementation:** +```javascript +const initS3 = () => { + const client = new S3Client({ + region: config.s3.region, + credentials: { + accessKeyId: config.s3.accessKeyId, + secretAccessKey: config.s3.secretAccessKey, + }, + }); + + return { + client, + bucket: config.s3.bucket, + region: config.s3.region, + prefix: config.s3.prefix, + }; +}; +``` + +**URL Format:** +``` +https://{bucket}.s3.{region}.amazonaws.com/{prefix}/{folder}/{filename} +``` + +**SDK:** AWS SDK v3 (`@aws-sdk/client-s3`, `@aws-sdk/s3-request-presigner`). The TypeScript provider uses official AWS SDK command/config/exception types and shared project storage contracts from `backend/src/types/file.ts`. + +### Google Cloud Storage Configuration + +**Environment Variables:** +```bash +FILE_STORAGE_PROVIDER=gcloud +GC_PROJECT_ID=your-project-id +GC_CLIENT_EMAIL=service-account@project.iam.gserviceaccount.com +GC_PRIVATE_KEY="-----BEGIN PRIVATE KEY-----\n..." +``` + +**Config (config.ts):** Currently hardcoded values: +```javascript +gcloud: { + bucket: "fldemo-files", + hash: "afeefb9d49f5b7977577876b99532ac7" +} +``` + +**Note:** Unlike S3, GCloud bucket and hash are hardcoded in `config.ts`. To use different values, modify the config file directly. + +**URL Format:** +``` +https://storage.googleapis.com/{bucket}/{hash}/{folder}/{filename} +``` + +**SDK:** `@google-cloud/storage` + +### Local Storage Configuration + +**Default behavior** when no cloud credentials are configured. + +**Upload Directory:** `os.tmpdir()` (OS temp directory) + +**URL Format:** +``` +/api/file/download?privateUrl={encodedPath} +``` + +Files are served via the backend download endpoint. Note: The download endpoint does **not** require authentication. + +## Asset Database Models + +### Assets Model + +**File:** `backend/src/db/models/assets.js` + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| id | UUID | Yes | Primary key | +| name | TEXT(255) | No | Display name | +| asset_type | ENUM | Yes | `image`, `video`, `audio`, `file` | +| type | ENUM | Yes | `icon`, `background_image`, `audio`, `video`, `transition`, `logo`, `favicon`, `document`, `general` (default) | +| cdn_url | TEXT | No | Public CDN/storage URL | +| storage_key | TEXT | No | Private storage path | +| mime_type | TEXT | No | MIME type (validated format) | +| size_mb | DECIMAL | No | File size in MB | +| width_px | INTEGER | No | Image/video width | +| height_px | INTEGER | No | Image/video height | +| duration_sec | DECIMAL | No | Video/audio duration | +| frame_rate | DECIMAL | No | Video FPS from backend `ffprobe` | +| checksum | TEXT | No | File checksum | +| is_public | BOOLEAN | Yes | Public visibility (default: false) | +| projectId | UUID | Yes | FK to projects (CASCADE) | + +**Indexes:** `projectId`, `asset_type`, `type`, `is_public`, `deletedAt` + +**Note on Environment Scoping:** Assets are scoped to projects, NOT environments. Unlike `tour_pages` which have `dev`, `stage`, and `production` versions, assets are shared across all environments within a project. When pages are published (dev → stage → production), the same asset URLs are used - only the page content (`ui_schema_json`) is copied between environments. + +### Asset Variants Model + +**File:** `backend/src/db/models/asset_variants.js` + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| id | UUID | Yes | Primary key | +| variant_type | ENUM | No | `thumbnail`, `preview`, `webp`, `mp4_low`, `mp4_high`, `original` | +| cdn_url | TEXT(2048) | No | Variant CDN URL (validated URL format) | +| width_px | INTEGER | No | Variant width | +| height_px | INTEGER | No | Variant height | +| size_mb | DECIMAL | No | Variant file size | +| assetId | UUID | Yes | FK to assets (CASCADE) | + +**Variant Types:** +| Type | Description | Use Case | +|------|-------------|----------| +| `thumbnail` | Small preview image | List views, galleries | +| `preview` | Medium quality preview | Quick previews | +| `webp` | WebP format | Web optimization | +| `mp4_low` | Low bitrate video | Mobile, slow connections | +| `mp4_high` | High bitrate video | Desktop, fast connections | +| `original` | Unmodified original | Full quality access | +| `reversed` | FFmpeg-reversed video | Back navigation transitions | + +**Note:** Most variants are **metadata records only** - the system tracks variant information but does not auto-generate them. The exception is `reversed` variants, which are auto-generated by the server when a page with transition videos is saved. + +**Critical:** The `assetId` field must be properly set when creating variant records. Without it, the variant becomes an orphaned record that cannot be found when looking up an asset's variants via the `asset_variants_asset` association. This is handled in `Asset_variantsDBApi.getFieldMapping()` which must include `assetId` in the field mapping. + +### Reversed Video Variant + +Reversed videos are a special variant type generated server-side for back navigation transitions. They use a different storage path pattern than other assets. + +**Storage Patterns:** +| Asset Type | Storage Path Pattern | Example | +|------------|---------------------|---------| +| Primary assets | `assets/{projectId}/{uuid}.ext` | `assets/abc-123/def-456.mp4` | +| Reversed videos | `assets/{assetId}/reversed.mp4` | `assets/xyz-789/reversed.mp4` | + +**Key Difference:** Reversed videos use the **asset ID** (not project ID) in their path. This is because they are tied to a specific asset (the transition video), not just the project. + +**Generation Flow:** +1. User adds transition video to navigation element +2. User saves the tour page +3. Server detects `transitionVideoUrl` in navigation elements +4. Server generates reversed video using FFmpeg +5. Reversed video uploaded to `assets/{assetId}/reversed.mp4` +6. `reverseVideoUrl` populated in `ui_schema_json` + +**Note:** Not all assets have reversed videos - only transition videos used in navigation elements. + +## Frontend Upload Implementation + +### useAssetUploader Hook + +**File:** `frontend/src/components/Assets/useAssetUploader.ts` + +```typescript +interface UseAssetUploaderOptions { + selectedProjectId: string; + onUploadComplete: () => void; +} + +interface UseAssetUploaderReturn { + uploadingSections: string[]; + uploadQueues: Record; + runBatchUpload: (section: AssetSection, files: File[]) => Promise; +} +``` + +**Key Features:** +- **Batch upload** with queue management +- **2 concurrent uploads** maximum (`maxConcurrent = 2`) +- **Per-file progress** tracking +- **Abortable** via AbortController +- **Media probing** for video/audio duration and dimensions + +**Status States:** +| Status | Description | +|--------|-------------| +| `queued` | File in queue, waiting to upload | +| `uploading` | Actively uploading chunks | +| `saving` | Finalizing upload and creating asset record | +| `success` | Upload and save completed | +| `error` | Upload failed (error message available) | + +### UploadService Class + +**File:** `frontend/src/components/Uploaders/UploadService.js` + +**Single File Upload:** +```javascript +const result = await FileUploader.upload(path, file, schema); +// Returns: { id, name, sizeInBytes, privateUrl, publicUrl, new: true } +``` + +**Chunked Upload:** +```javascript +const result = await FileUploader.uploadChunked( + 'assets/project-id', // path + file, // File object + {}, // schema (validation) + { + chunkSize: 5 * 1024 * 1024, // 5MB + maxRetries: 3, + signal: abortController.signal, + onProgress: (percent, { chunkIndex, totalChunks }) => {}, + onStatus: (status, details) => {}, + } +); +``` + +### Media Duration Probing + +**File:** `frontend/src/lib/mediaDuration.ts` + +After upload, video/audio files are probed for metadata: + +```typescript +interface MediaDurationResult { + duration: number; + width?: number; // Video only + height?: number; // Video only +} + +// Probe video +const result = await probeMediaDuration(file, 'video', 10000); +// Returns: { duration: 120.5, width: 1920, height: 1080 } + +// Probe audio +const result = await probeMediaDuration(file, 'audio', 10000); +// Returns: { duration: 180.0 } +``` + +**Implementation:** +- Creates HTML5 `