39948-vm/backend/docs/backend-architecture.md
2026-07-03 16:11:24 +02:00

682 lines
30 KiB
Markdown

# Backend Architecture Documentation
This document provides a comprehensive analysis of the Tour Builder Platform backend architecture, including design patterns, layers, and implementation details.
## Overview
- **Runtime**: Node.js 24 LTS
- **Framework**: Express.js 4.x
- **ORM**: Sequelize 6.x with PostgreSQL
- **Authentication**: Passport.js (JWT, Google OAuth, Microsoft OAuth)
- **Documentation**: Swagger/OpenAPI 3.0
- **Logging**: Pino (structured JSON logging)
- **TypeScript/ESM**: the backend package is `"type": "module"` and active
backend source is strict TypeScript ESM.
- **File Storage**: AWS S3 / Local filesystem (Strategy Pattern, provider-based)
---
## Architecture Diagram
```
┌──────────────────────────────────────────────────────────────────────────┐
│ Express Application │
│ (src/index.ts) │
└──────────────────────────────────────────────────────────────────────────┘
┌─────────────────┼─────────────────┐
▼ ▼ ▼
┌────────────────────────┐ ┌────────────────┐ ┌────────────────────────┐
│ Middleware │ │ Rate Limiter │ │ Request Logger │
│ • runtimeContext │ │ • auth (10/15m)│ │ • Pino structured │
│ • runtimePublic │ │ • api (100/1m) │ │ • Request ID tracking │
│ • checkPermissions │ │ • upload (10/m)│ │ • Duration metrics │
│ • passport JWT │ │ • download(200)│ └────────────────────────┘
└────────────────────────┘ │ • search (30/m)│
│ • AI (20/1m) │
└────────────────┘
┌──────────────────────────────────────────────────────────────────────────┐
│ Routes Layer │
│ (src/routes/*.ts) │
│ ┌────────────────────────────────────────────────────────────────────┐ │
│ │ Factory-generated routes: router.factory.js │ │
│ │ • Standard CRUD: POST /, PUT /:id, DELETE /:id, GET /, GET /:id │ │
│ │ • Extra: GET /count, GET /autocomplete, POST /deleteByIds │ │
│ │ • CSV export: GET /?filetype=csv │ │
│ └────────────────────────────────────────────────────────────────────┘ │
│ ┌────────────────────────────────────────────────────────────────────┐ │
│ │ Custom routes: auth, file, publish, search, sql, runtime │ │
│ └────────────────────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────────────────┘
┌──────────────────────────────────────────────────────────────────────────┐
│ Service Layer │
│ (src/services/*.js) │
│ ┌────────────────────────────────────────────────────────────────────┐ │
│ │ Factory-generated services: service.factory.js │ │
│ │ • Transaction-wrapped CRUD operations │ │
│ │ • Delegates to DB API layer │ │
│ └────────────────────────────────────────────────────────────────────┘ │
│ ┌────────────────────────────────────────────────────────────────────┐ │
│ │ Custom services: │ │
│ │ • auth.js - Authentication logic (signin, signup, password) │ │
│ │ • publish.ts - Publishing workflow service (dev→stage→production)│ │
│ │ • file.ts - File storage operations │ │
│ │ • email/index.js - Email sending via Nodemailer/SES │ │
│ │ • search.ts - Full-text search route │ │
│ │ • pwa_manifest.js - PWA offline manifest generation │ │
│ └────────────────────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────────────────┘
┌──────────────────────────────────────────────────────────────────────────┐
│ Database API Layer │
│ (src/db/api/*.js) │
│ ┌────────────────────────────────────────────────────────────────────┐ │
│ │ GenericDBApi (base.api.js) - Template Method Pattern │ │
│ │ • Configurable: MODEL, SEARCHABLE_FIELDS, RANGE_FIELDS, etc. │ │
│ │ • CRUD: create, update, remove, deleteByIds │ │
│ │ • Query: findBy, findAll, findAllAutocomplete │ │
│ │ • Data Transform: getFieldMapping(), JSON_FIELDS, FIELD_DEFAULTS │ │
│ └────────────────────────────────────────────────────────────────────┘ │
│ ┌────────────────────────────────────────────────────────────────────┐ │
│ │ Entity APIs extend GenericDBApi: │ │
│ │ UsersDBApi, ProjectsDBApi, AssetsDBApi, TourPagesDBApi, etc. │ │
│ └────────────────────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────────────────┘
┌──────────────────────────────────────────────────────────────────────────┐
│ Sequelize Models │
│ (src/db/models/*.ts) │
│ Entity models + file model, loaded dynamically via loader.ts │
└──────────────────────────────────────────────────────────────────────────┘
┌──────────────────────────────────────────────────────────────────────────┐
│ PostgreSQL │
└──────────────────────────────────────────────────────────────────────────┘
```
---
## Directory Structure
```
backend/src/
├── index.ts # Application entry point
├── config.ts # Environment configuration
├── helpers.ts # Utility functions (wrapAsync, JWT, UUID validation)
├── types/ # Reusable strict TypeScript contracts for migrated code
├── auth/
│ └── auth.ts # Passport strategies (JWT, Google, Microsoft)
├── middlewares/
│ ├── check-permissions.ts # RBAC permission checking
│ ├── runtime-context.ts # Runtime environment context (dev/stage/production)
│ ├── runtime-public.ts # Public runtime access control & field sanitization
│ ├── rateLimiter.ts # Rate limiting (auth, API, upload, download)
│ └── upload.ts # File upload handling (multer)
├── routes/
│ ├── auth.ts # Authentication routes (custom)
│ ├── file.ts # File upload/download routes (custom)
│ ├── publish.ts # Publishing workflow routes (custom)
│ ├── search.ts # Full-text search routes (custom)
│ ├── runtime-context.ts # Runtime context detection (custom)
│ └── [entity].ts # Entity CRUD routes
├── services/
│ ├── auth.ts # Authentication service
│ ├── publish.ts # Publishing workflow (dev→stage→production)
│ ├── file.ts # Unified file storage service
│ ├── search.ts # Full-text search service
│ ├── file/ # File storage providers (S3, Local)
│ │ ├── index.ts # Module exports & provider factory
│ │ ├── BaseStorageProvider.ts # Abstract base class
│ │ ├── S3StorageProvider.ts # AWS S3 implementation
│ │ ├── LocalStorageProvider.ts # Local filesystem implementation
│ │ └── UploadSessionManager.ts # Chunked upload session management
│ ├── email/
│ │ ├── index.ts # Email sender (Nodemailer/SES)
│ │ └── list/ # Email templates
│ ├── notifications/
│ │ ├── helpers.ts # Notification helpers
│ │ └── errors/ # Error classes (ValidationError, ForbiddenError)
│ └── [entity].ts # Entity services
├── factories/
│ ├── router.factory.ts # Route generator (createEntityRouter)
│ └── service.factory.ts # Service generator (createEntityService)
├── db/
│ ├── db-config.ts # Database configuration (env-based)
│ ├── umzug.ts # Migration and seeder runner
│ ├── models/
│ │ ├── index.ts # Model registry entrypoint
│ │ ├── loader.ts # Model loader
│ │ └── [entity].ts # Sequelize models
│ ├── api/
│ │ ├── base.api.ts # GenericDBApi base class
│ │ ├── runtime-context.ts # Runtime filtering helpers
│ │ └── [entity].ts # Entity DB APIs
│ ├── migrations/ # Applied migration history
│ └── seeders/ # Typed seed data files
└── utils/
├── index.ts # Utils barrel export
├── errors.ts # Error classes (AppError, NotFoundError, etc.)
├── logger.ts # Pino logger configuration
└── env-validation.ts # Environment variable validation
```
---
## Design Patterns
### 1. Factory Pattern
**Router Factory** (`factories/router.factory.js`)
Generates standardized CRUD routes for entities:
```javascript
const { createEntityRouter } = require('../factories/router.factory');
// Creates routes: POST /, PUT /:id, DELETE /:id, GET /, GET /:id, GET /count, GET /autocomplete
module.exports = createEntityRouter('assets', AssetsService, AssetsDBApi, {
permissionEntity: 'assets',
csvFields: ['id', 'name', 'asset_type', 'createdAt'],
});
```
**Service Factory** (`factories/service.factory.js`)
Generates transaction-wrapped service classes:
```javascript
const { createEntityService } = require('../factories/service.factory');
// Creates: create(), update(), remove(), deleteByIds(), bulkImport()
module.exports = createEntityService(AssetsDBApi, { entityName: 'Asset' });
```
### 2. Template Method Pattern
**GenericDBApi** (`db/api/base.api.js`)
Base class with configurable hooks for entity-specific behavior:
```javascript
class AssetsDBApi extends GenericDBApi {
// Required: Define the Sequelize model
static get MODEL() { return db.assets; }
// Configurable behavior via static getters
static get SEARCHABLE_FIELDS() { return ['name', 'cdn_url']; }
static get RANGE_FIELDS() { return ['size_mb', 'width_px']; }
static get ENUM_FIELDS() { return ['asset_type', 'is_public']; }
static get JSON_FIELDS() { return ['settings_json']; }
static get FIELD_DEFAULTS() { return { type: { default: 'general' } }; }
static get ASSOCIATIONS() { return [{ field: 'project', setter: 'setProject' }]; }
static get FIND_BY_INCLUDES() { return [{ association: 'project' }]; }
static get FIND_ALL_INCLUDES() { return [{ model: db.projects, as: 'project' }]; }
// Custom field transformation
static getFieldMapping(data) {
return {
name: data.name || null,
asset_type: data.asset_type || null,
type: data.type || 'general',
// ...
};
}
}
```
### 3. Strategy Pattern
**File Storage Providers** (`services/file/`)
Two concrete implementations with pluggable architecture:
```
BaseStorageProvider (abstract)
├── S3StorageProvider # AWS S3 implementation (with timeout/retry)
└── LocalStorageProvider # Local filesystem implementation
```
The storage provider base, S3 provider, and local provider are migrated TS/ESM modules. The S3 implementation uses official AWS SDK v3 types; shared provider-domain contracts are in `src/types/file.ts`.
Interface:
- `upload(key, data, options)``{ key, url }`
- `download(key)``{ body, contentType }`
- `delete(key)``void`
- `deleteMany(keys)``void`
- `exists(key)``boolean`
- `list(prefix)``string[]`
- `getSignedUrl(key, expiresIn)``string`
### 4. Middleware Chain Pattern
Request flow through middleware stack:
```
Request → requestLogger → runtimeContextMiddleware → rateLimiter
→ passport.authenticate('jwt') → checkCrudPermissions
→ Route Handler → Service → DB API → Database
→ Response
```
---
## Layers in Detail
### Entry Point (`src/index.ts`)
Application bootstrap:
1. **Security**: Helmet, CORS configuration
2. **Logging**: Request logger middleware (Pino)
3. **Authentication**: Passport JWT initialization
4. **Rate Limiting**: Per-route rate limiters
5. **Body Parsing**: JSON (1mb limit), applied after file routes
6. **Runtime Context**: Environment detection middleware
7. **Route Mounting**: Entity routes with auth/permissions
8. **Error Handling**: Generic error handler
9. **Static Files**: Public directory serving
```javascript
// Key route mounting patterns
app.use('/api/auth', authRoutes); // No JWT required
app.use('/api/users', jwtAuth, usersRoutes); // JWT required
// Runtime public routes (production content accessible without auth)
const mountRuntimeEntityRoute = (path, entityName, router) => {
app.use(path,
requireRuntimeReadOrAuth, // JWT or public production
blockNonPublicRuntimeListEndpoints, // Block non-list endpoints
sanitizePublicRuntimeListResponse(entityName), // Filter sensitive fields
router
);
};
mountRuntimeEntityRoute('/api/projects', 'projects', projectsRoutes);
mountRuntimeEntityRoute('/api/tour_pages', 'tour_pages', tour_pagesRoutes);
```
### Routes Layer
**Factory-Generated Routes** provide standard CRUD:
| Method | Path | Description |
|--------|------|-------------|
| POST | `/` | Create record |
| POST | `/bulk-import` | Bulk import from CSV |
| PUT | `/:id` | Update record |
| DELETE | `/:id` | Delete record |
| POST | `/deleteByIds` | Bulk delete |
| GET | `/` | List with pagination & filters |
| GET | `/count` | Count only |
| GET | `/autocomplete` | Autocomplete search |
| GET | `/:id` | Get single record |
**Custom Routes** (auth, file, publish, search, runtime-context):
| Route | Endpoints |
|-------|-----------|
| `/api/auth` | signin, signup, me, password-reset, verify-email, Google/Microsoft OAuth |
| `/api/file` | upload, download, presign, upload-sessions (chunked) |
| `/api/publish` | publish (stage→production), save-to-stage (dev→stage) |
| `/api/search` | Global full-text search |
| `/api/runtime-context` | Runtime environment detection |
### Service Layer
**Transaction Management**: Services wrap operations in transactions:
```javascript
static async create({ data, currentUser, transaction: externalTransaction, runtimeContext }) {
const transaction = externalTransaction || await db.sequelize.transaction();
const ownsTransaction = !externalTransaction;
try {
const record = await DBApi.create({ data, currentUser, transaction, runtimeContext });
if (ownsTransaction) await transaction.commit();
return record;
} catch (error) {
if (ownsTransaction) await transaction.rollback();
throw error;
}
}
```
**Publish Service** (`services/publish.ts`):
Implements the dev→stage→production workflow with:
- Transaction locking to prevent concurrent publishes
- Source key tracking for content lineage
- Bulk copy operations for pages and audio tracks
### Database API Layer
**Query Building** in `findAll()`:
| Filter Type | Example | SQL |
|-------------|---------|-----|
| Text search | `?name=foo` | `name ILIKE '%foo%'` |
| Range | `?size_mbRange=[0,100]` | `size_mb >= 0 AND size_mb <= 100` |
| Enum | `?asset_type=image` | `asset_type = 'image'` |
| Relation | `?project=uuid` | JOIN with projects table |
| Sort | `?field=name&sort=asc` | `ORDER BY name ASC` |
| Pagination | `?page=1&limit=10` | `OFFSET 0 LIMIT 10` |
---
## Authentication & Authorization
### Authentication Flow
```
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ POST /signin │ │ Passport JWT │ │ Protected │
│ → JWT Token │────▶│ Middleware │────▶│ Route │
└─────────────────┘ └─────────────────┘ └─────────────────┘
req.currentUser = {
id, email, app_role,
custom_permissions
}
```
### Authorization (RBAC)
**Permission Check Flow** (`middlewares/check-permissions.ts`):
1. Self-access bypass (user accessing own resource)
2. Check custom permissions (user-specific)
3. Check role permissions (from app_role)
4. Fallback to Public role for unauthenticated
**Permission Naming Convention**:
- `CREATE_<ENTITY>` - Create records
- `READ_<ENTITY>` - Read records
- `UPDATE_<ENTITY>` - Modify records
- `DELETE_<ENTITY>` - Delete records
```javascript
// Auto-generated from HTTP method
const permissionName = `${METHOD_MAP[req.method]}_${name.toUpperCase()}`;
// POST /api/assets → CREATE_ASSETS
// GET /api/assets → READ_ASSETS
```
### Runtime Public Access
For production content accessible without authentication:
```javascript
const requireRuntimeReadOrAuth = (req, res, next) => {
const isPublicEnvironment = req.runtimeContext?.headerEnvironment === 'production';
const isReadOnlyRequest = ['GET', 'OPTIONS'].includes(req.method);
if (isPublicEnvironment && isReadOnlyRequest && !hasAuthHeader) {
req.isRuntimePublicRequest = true;
return next(); // Allow without JWT
}
return jwtAuth(req, res, next); // Require JWT
};
```
---
## Rate Limiting
Pre-configured limiters (`middlewares/rateLimiter.ts`):
| Limiter | Window | Max Requests | Use Case |
|---------|--------|--------------|----------|
| `authLimiter` | 15 min | 10 | Authentication endpoints |
| `passwordResetLimiter` | 1 hour | 5 | Password reset |
| `apiLimiter` | 1 min | 100 | General API |
| `uploadLimiter` | 1 min | 10 | File uploads |
| `downloadLimiter` | 1 min | 200 | File downloads |
| `searchLimiter` | 1 min | 30 | Search queries |
Headers returned:
- `X-RateLimit-Limit`: Maximum requests
- `X-RateLimit-Remaining`: Remaining requests
- `X-RateLimit-Reset`: Reset time (ISO timestamp)
- `Retry-After`: Seconds until reset (when limited)
---
## File Storage
**Storage Provider Selection**:
```javascript
const provider = config.fileStorage.provider ||
(hasS3Credentials ? 's3' : hasGCloudCredentials ? 'gcloud' : 'local');
```
**S3 Operations**:
| Operation | Method | Description |
|-----------|--------|-------------|
| Upload | `upload(key, data, options)` | Put object with metadata |
| Download | `download(key)` | Get object stream |
| Presign | `getSignedUrl(key, expiresIn)` | Generate presigned URL |
| Delete | `delete(key)` / `deleteMany(keys)` | Remove objects |
| Check | `exists(key)` | Head object |
| List | `list(prefix)` | List objects with prefix |
**Chunked Uploads** (`UploadSessionManager`):
For large files, supports multipart upload sessions:
1. `POST /upload-sessions/init` - Create session
2. `POST /upload-sessions/:id/chunk` - Upload chunk
3. `POST /upload-sessions/:id/finalize` - Complete upload
---
## Error Handling
**Error Classes** (`utils/errors.js`):
```javascript
class AppError extends Error {
constructor(message, statusCode = 500, details = null) {
super(message);
this.statusCode = statusCode;
this.details = details;
this.isOperational = true;
}
}
class NotFoundError extends AppError { statusCode = 404 }
class ValidationError extends AppError { statusCode = 400 }
class ForbiddenError extends AppError { statusCode = 403 }
class UnauthorizedError extends AppError { statusCode = 401 }
class ConflictError extends AppError { statusCode = 409 }
```
**Async Handler** (`helpers.ts`):
```javascript
// Wraps async route handlers to catch errors
static wrapAsync(fn) {
return function(req, res, next) {
fn(req, res, next).catch(next);
};
}
```
**Common Error Handler** (`helpers.ts`):
```javascript
static commonErrorHandler(error, req, res, _next) {
const statusCode = error.code || error.status;
if ([400, 401, 403, 404, 409, 422].includes(statusCode)) {
return res.status(statusCode).send(error.message);
}
console.error(error);
return res.status(500).send('Internal server error');
}
```
---
## Logging
**Pino Logger** (`utils/logger.js`):
Logger initialization is a bootstrap exception: it may read `process.env`
directly because importing `config.ts` would create an initialization cycle.
```javascript
const logger = pino({
level: process.env.LOG_LEVEL || 'info',
transport: isDevelopment
? { target: 'pino-pretty', options: { colorize: true } }
: undefined,
base: {
service: 'tour-builder-api',
env: process.env.NODE_ENV || 'development',
},
});
```
**Request Logging**:
```javascript
function requestLogger(req, res, next) {
const requestId = req.headers['x-request-id'] || crypto.randomUUID();
req.log = logger.child({ requestId });
req.requestId = requestId;
res.setHeader('X-Request-Id', requestId);
res.on('finish', () => {
req.log.info({
method: req.method,
url: req.originalUrl,
status: res.statusCode,
duration: Date.now() - start,
}, 'Request completed');
});
}
```
---
## Configuration
**Environment Variables** (`config.ts`):
| Variable | Description | Default |
|----------|-------------|---------|
| `SECRET_KEY` | JWT signing key | UUID-based default |
| `ADMIN_EMAIL` | Admin user email | `admin@flatlogic.com` |
| `ADMIN_PASS` | Admin user password | Generated |
| `AWS_S3_BUCKET` | S3 bucket name | - |
| `AWS_S3_REGION` | S3 region | `us-east-1` |
| `AWS_ACCESS_KEY_ID` | AWS access key | - |
| `AWS_SECRET_ACCESS_KEY` | AWS secret key | - |
| `GOOGLE_CLIENT_ID` | Google OAuth client ID | - |
| `GOOGLE_CLIENT_SECRET` | Google OAuth client secret | - |
| `MS_CLIENT_ID` | Microsoft OAuth client ID | - |
| `MS_CLIENT_SECRET` | Microsoft OAuth client secret | - |
| `EMAIL_USER` | SMTP username | - |
| `EMAIL_PASS` | SMTP password | - |
| `LOG_LEVEL` | Logging level | `info` |
**Database Configuration** (`db/db-config.ts`):
| Environment | Database | Logging |
|-------------|----------|---------|
| `production` | `DB_*` env vars | Disabled |
| `development` | `db_tour_builder_platform` | Console |
| `dev_stage` | `DB_*` env vars | Console |
---
## API Documentation
Swagger/OpenAPI documentation is available at `/api-docs`.
The served document is centralized in `backend/src/openapi/document.ts`. The
OpenAPI module defines shared schemas, common parameters, reusable responses,
and generated standard CRUD paths for every `createEntityRouter` resource. This
keeps the documented factory contract aligned with the route factory endpoints:
create, bulk import, update, delete, bulk delete, list, count, autocomplete,
and get by ID.
```javascript
const specs = createOpenApiDocument({
serverUrl: config.server.swaggerServerUrl,
});
```
When adding a new route, update `backend/src/openapi/document.ts` in the same
change. For new factory-backed entities, add the entity schema and `CrudResource`
entry; for custom routes, add an explicit path item.
---
## Health Check
```javascript
GET /api/health
{
"status": "ok", // or "degraded"
"timestamp": "2026-03-29T...",
"uptime": 12345.678,
"environment": "production",
"database": "connected" // or "disconnected"
}
```
---
## Key Implementation Files
| File | Purpose |
|------|---------|
| `src/index.ts` | Application entry, middleware setup, route mounting |
| `src/config.ts` | Environment configuration |
| `src/helpers.ts` | wrapAsync, commonErrorHandler, jwtSign, isUuidV4 |
| `src/auth/auth.ts` | Passport strategies (JWT, Google, Microsoft) |
| `src/factories/router.factory.ts` | Route generator for entities |
| `src/factories/service.factory.ts` | Service generator for entities |
| `src/db/api/base.api.ts` | GenericDBApi base class |
| `src/middlewares/check-permissions.ts` | RBAC permission checking |
| `src/middlewares/rateLimiter.ts` | Rate limiting configuration |
| `src/middlewares/runtime-context.ts` | Runtime environment detection |
| `src/middlewares/runtime-public.ts` | Public runtime access control & field sanitization |
| `src/services/publish.ts` | Publishing workflow service |
| `src/services/file/S3StorageProvider.ts` | S3 storage implementation using official AWS SDK v3 types |
| `src/utils/logger.ts` | Pino logger configuration |
| `src/utils/errors.ts` | Error class definitions |
---
## Best Practices Implemented
1. **Factory Patterns** - Reduce boilerplate for CRUD operations
2. **Template Method** - Configurable base class for DB operations
3. **Strategy Pattern** - Pluggable storage providers
4. **Middleware Chain** - Composable request processing
5. **Transaction Management** - Consistent rollback on errors
6. **Rate Limiting** - Protection against abuse
7. **Structured Logging** - JSON logs with request IDs
8. **Environment-Based Config** - Secure credential handling
9. **Soft Deletes** - Paranoid models for data recovery
10. **RBAC** - Fine-grained permission control