39948-vm/documentation/access-logs-audit-trail.md
2026-07-03 16:11:24 +02:00

709 lines
24 KiB
Markdown

# Access Logs & Audit Trail
Complete documentation for the Tour Builder Platform's Access Logs and Audit Trail system including activity tracking, security auditing, and data preservation.
## Overview
The platform implements a comprehensive audit system with two complementary mechanisms:
- **Access Logs** - Dedicated table recording API requests with user, path, and client info
- **Audit Trail** - CreatedBy/UpdatedBy fields on all entities tracking who modified what
```
┌─────────────────────────────────────────────────────────────────────────────┐
│ Access Logs & Audit Trail Architecture │
│ │
│ ┌────────────────────────────────────────────────────────────────────────┐ │
│ │ Request Flow │ │
│ │ │ │
│ │ Client Request │ │
│ │ │ │ │
│ │ ▼ │ │
│ │ ┌─────────────────────────────────────────────────────────────────┐ │ │
│ │ │ Request Logger Middleware │ │ │
│ │ │ │ │ │
│ │ │ • Generate/Extract X-Request-Id │ │ │
│ │ │ • Capture: method, path, user-agent, duration │ │ │
│ │ │ • Log to Pino (structured logging) │ │ │
│ │ └──────────────────────────┬──────────────────────────────────────┘ │ │
│ │ │ │ │
│ │ ▼ │ │
│ │ ┌─────────────────────────────────────────────────────────────────┐ │ │
│ │ │ JWT Authentication │ │ │
│ │ │ │ │ │
│ │ │ • Validate token │ │ │
│ │ │ • Attach currentUser to request │ │ │
│ │ └──────────────────────────┬──────────────────────────────────────┘ │ │
│ │ │ │ │
│ │ ▼ │ │
│ │ ┌─────────────────────────────────────────────────────────────────┐ │ │
│ │ │ Permission Check │ │ │
│ │ │ │ │ │
│ │ │ • Check READ/CREATE/UPDATE/DELETE permissions │ │ │
│ │ │ • Role-based + custom permissions │ │ │
│ │ └──────────────────────────┬──────────────────────────────────────┘ │ │
│ │ │ │ │
│ │ ▼ │ │
│ │ ┌─────────────────────────────────────────────────────────────────┐ │ │
│ │ │ Database Operation │ │ │
│ │ │ │ │ │
│ │ │ CREATE: set createdById, updatedById │ │ │
│ │ │ UPDATE: set updatedById │ │ │
│ │ │ DELETE: set deletedBy, mark deletedAt (soft delete) │ │ │
│ │ └─────────────────────────────────────────────────────────────────┘ │ │
│ └────────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘
```
## Data Model
### Access Logs Schema
**Table:** `access_logs`
**Source:** `backend/src/db/models/access_logs.js`
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| id | UUID | Yes | UUIDv4 | Primary key |
| projectId | UUID (FK) | No | null | Related project (CASCADE delete) |
| userId | UUID (FK) | No | null | User who accessed (SET NULL delete) |
| environment | ENUM | Yes | - | 'admin', 'stage', 'production' |
| path | TEXT | No | null | API path accessed (max 2048 chars) |
| ip_address | TEXT | No | null | Client IP address (max 45 chars for IPv6) |
| user_agent | TEXT | No | null | HTTP User-Agent header (max 1024 chars) |
| accessed_at | TIMESTAMP | Yes | NOW | When access occurred |
| importHash | VARCHAR(255) | No | null | Unique hash for CSV imports |
| createdAt | TIMESTAMP | Yes | Auto | Record creation time |
| updatedAt | TIMESTAMP | Yes | Auto | Record update time |
| deletedAt | TIMESTAMP | No | null | Soft delete timestamp |
### Indexes
```javascript
indexes: [
{ fields: ['projectId'] }, // Query logs by project
{ fields: ['environment'] }, // Filter by environment
{ fields: ['userId'] }, // Query logs by user
{ fields: ['accessed_at'] }, // Time-range queries and sorting
]
```
### Relationships
```
access_logs
├── belongsTo projects (as project)
│ ├── foreignKey: projectId
│ ├── onDelete: CASCADE ← Logs deleted when project deleted
│ └── onUpdate: CASCADE
├── belongsTo users (as user)
│ ├── foreignKey: userId
│ ├── onDelete: CASCADE ← Logs deleted when user deleted
│ └── onUpdate: CASCADE
├── belongsTo users (as createdBy) ─── Audit: who created this log
└── belongsTo users (as updatedBy) ─── Audit: who last updated this log
```
## Audit Trail Pattern
### CreatedBy/UpdatedBy Implementation
All entities in the system implement the audit trail pattern:
**Source:** `backend/src/db/api/base.api.js`
```javascript
// On CREATE
static async create({ data, currentUser = { id: null }, transaction }) {
const mappedData = this.getFieldMapping(data);
const record = await this.MODEL.create({
...mappedData,
createdById: currentUser.id, // Track creator
updatedById: currentUser.id, // Initial updater = creator
}, { transaction });
}
// On UPDATE
static async update({ id, data, currentUser = { id: null }, transaction }) {
await record.update({
...updatePayload,
updatedById: currentUser.id, // Track who modified
}, { transaction });
}
// On DELETE (soft delete)
static async remove({ id, currentUser = { id: null }, transaction }) {
await record.update({ deletedBy: currentUser.id }, { transaction });
await record.destroy({ transaction }); // Sets deletedAt
}
```
### Entities with Audit Trail
All models include audit relationships:
```javascript
// Applied to ALL models:
db.[entity].belongsTo(db.users, { as: 'createdBy' });
db.[entity].belongsTo(db.users, { as: 'updatedBy' });
```
**Audited Entities:**
- users, roles, permissions
- projects, project_memberships
- tour_pages, project_audio_tracks
- assets, asset_variants
- element_type_defaults, project_element_defaults
- publish_events, pwa_caches
- presigned_url_requests, access_logs
**Note:** Page elements, navigation links, and transitions are stored in `tour_pages.ui_schema_json` and audited as part of the tour_pages record.
### Timestamp Tracking
Sequelize provides automatic timestamps:
```javascript
{
timestamps: true, // Enables createdAt, updatedAt
paranoid: true, // Enables deletedAt (soft delete)
}
```
| Field | Set When | Never Changes |
|-------|----------|---------------|
| `createdAt` | Record created | ✓ Immutable |
| `updatedAt` | Any modification | Updates each save |
| `deletedAt` | Soft delete called | - |
## Request Logging Middleware
### Logger Configuration
**Source:** `backend/src/utils/logger.js`
```javascript
const logger = pino({
level: process.env.LOG_LEVEL || 'info',
transport: isDevelopment
? { target: 'pino-pretty', options: { colorize: true } }
: undefined,
base: {
service: 'tour-builder-api',
env: process.env.NODE_ENV || 'development',
},
});
```
### Request Logger Middleware
```javascript
function requestLogger(req, res, next) {
// Generate or extract request ID for tracing
const requestId = req.headers['x-request-id'] || crypto.randomUUID();
req.log = logger.child({ requestId });
req.requestId = requestId;
res.setHeader('X-Request-Id', requestId);
const start = Date.now();
res.on('finish', () => {
const duration = Date.now() - start;
const logData = {
method: req.method,
url: req.originalUrl || req.url,
status: res.statusCode,
duration,
userAgent: req.headers['user-agent'],
};
// Log level based on response status
if (res.statusCode >= 500) {
req.log.error(logData, 'Request completed with server error');
} else if (res.statusCode >= 400) {
req.log.warn(logData, 'Request completed with client error');
} else {
req.log.info(logData, 'Request completed');
}
});
next();
}
```
### Applied in Application
**Source:** `backend/src/index.ts`
```javascript
app.enable('trust proxy'); // Extract real IP behind proxies
app.use(requestLogger); // Apply to all routes
```
### Log Output Format
**Development (pino-pretty):**
```
[10:30:45.123] INFO: Request completed
method: "GET"
url: "/api/projects"
status: 200
duration: 45
userAgent: "Mozilla/5.0..."
requestId: "abc123-def456"
```
**Production (JSON):**
```json
{
"level": 30,
"time": 1711234567890,
"service": "tour-builder-api",
"env": "production",
"requestId": "abc123-def456",
"method": "GET",
"url": "/api/projects",
"status": 200,
"duration": 45,
"userAgent": "Mozilla/5.0..."
}
```
## API Reference
### Endpoints
**Source:** `backend/src/routes/access_logs.ts`
| Method | Endpoint | Permission | Description |
|--------|----------|------------|-------------|
| GET | `/api/access_logs` | READ_ACCESS_LOGS | List with filtering |
| GET | `/api/access_logs/:id` | READ_ACCESS_LOGS | Get single log |
| POST | `/api/access_logs` | CREATE_ACCESS_LOGS | Create log entry |
| PUT | `/api/access_logs/:id` | UPDATE_ACCESS_LOGS | Update log |
| DELETE | `/api/access_logs/:id` | DELETE_ACCESS_LOGS | Soft delete log |
### Query Parameters
**Source:** `backend/src/db/api/access_logs.ts`
| Parameter | Type | Description |
|-----------|------|-------------|
| `limit` | number | Results per page (default: 50) |
| `page` | number | Page number (0-based) |
| `field` | string | Sort field |
| `sort` | 'asc' \| 'desc' | Sort direction |
| `path` | string | Search path (ILIKE) |
| `ip_address` | string | Search IP address (ILIKE) |
| `user_agent` | string | Search user agent (ILIKE) |
| `accessed_atRange` | [start, end] | Date range filter |
| `environment` | enum | Filter by environment |
| `project` | UUID \| string | Filter by project |
| `user` | UUID \| string | Filter by user |
| `filetype` | 'csv' | Export as CSV |
### Request/Response Examples
**List Logs with Filters:**
```http
GET /api/access_logs?limit=50&page=1&environment=production&accessed_atRange=["2024-01-01","2024-12-31"]&sort=DESC&field=accessed_at
Authorization: Bearer <jwt-token>
```
**Response:**
```json
{
"rows": [
{
"id": "log-uuid-1",
"projectId": "project-uuid",
"userId": "user-uuid",
"environment": "production",
"path": "/api/tour_pages",
"ip_address": "192.168.1.100",
"user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)...",
"accessed_at": "2024-03-15T10:30:00Z",
"project": {
"id": "project-uuid",
"name": "My Tour"
},
"user": {
"id": "user-uuid",
"firstName": "John",
"lastName": "Doe"
},
"createdAt": "2024-03-15T10:30:00Z",
"updatedAt": "2024-03-15T10:30:00Z"
}
],
"count": 1542
}
```
**Create Log Entry:**
```http
POST /api/access_logs
Content-Type: application/json
Authorization: Bearer <jwt-token>
{
"data": {
"project": "project-uuid",
"user": "user-uuid",
"environment": "production",
"path": "/api/tour_pages",
"ip_address": "192.168.1.100",
"user_agent": "Mozilla/5.0...",
"accessed_at": "2024-03-15T10:30:00Z"
}
}
```
### CSV Export
**Fields Exported:**
- id
- environment
- path
- ip_address
- user_agent
- accessed_at
- createdAt
```http
GET /api/access_logs?filetype=csv
Authorization: Bearer <jwt-token>
```
## Frontend Implementation
### Admin Pages
**Location:** `frontend/src/pages/access_logs/`
| Page | Path | Description |
|------|------|-------------|
| List | `/access_logs/access_logs-list` | Filterable table view |
| Table | `/access_logs/access_logs-table` | Data grid variant |
| New | `/access_logs/access_logs-new` | Create log entry |
| Edit | `/access_logs/access_logs-edit?id=<uuid>` | Edit log |
| View | `/access_logs/access_logs-view?id=<uuid>` | Read-only details |
| Dynamic | `/access_logs/[access_logsId]` | Dynamic route (ID in path) |
**Note:** Uses `useEditPageSync` hook for edit page data synchronization.
### List Page Filters
**Source:** `frontend/src/pages/access_logs/access_logs-list.tsx`
| Filter | Type | Description |
|--------|------|-------------|
| Path | Text | Search by API path |
| IP Address | Text | Search by client IP |
| User Agent | Text | Search by browser/client |
| Accessed At | Date Range | Filter by time range |
| Project | Dropdown | Filter by project |
| User | Dropdown | Filter by user |
| Environment | Enum | admin / stage / production |
### Table Columns
**Source:** `frontend/src/components/Access_logs/configureAccess_logsCols.tsx`
Uses `createColumnLoader` factory from `configBuilderFactory.tsx`:
| Column | Editable | Type |
|--------|----------|------|
| project | Yes | singleSelectRelation (projects) |
| environment | Yes | text |
| user | Yes | singleSelectRelation (users) |
| path | Yes | text |
| ip_address | Yes | text |
| user_agent | Yes | text |
| accessed_at | Yes | datetime |
| actions | - | actions (Edit / View / Delete) |
### TypeScript Interface
**Source:** `frontend/src/types/entities.ts`
```typescript
export interface AccessLog extends BaseEntity {
user?: User | string | null;
project?: Project | string | null;
environment?: 'admin' | 'stage' | 'production';
path?: string;
action?: string;
ip_address?: string;
user_agent?: string;
accessed_at?: string | Date;
metadata?: Record<string, unknown>;
}
```
### Redux State
**Source:** `frontend/src/stores/access_logs/access_logsSlice.ts`
Uses `createEntitySlice` factory for standardized CRUD operations:
```typescript
const { slice, actions, reducer } = createEntitySlice<AccessLog>({
name: 'access_logs',
endpoint: 'access_logs',
singularName: 'Access Log',
});
export const {
fetch, // Load logs with query
create, // Create new log
update, // Update existing log
deleteItem, // Delete single log
deleteItemsByIds,// Batch delete
uploadCsv, // Import from CSV
setRefetch, // Trigger refresh
} = actions;
// Usage
dispatch(fetch({ query: '?environment=production&limit=100' }));
```
## Security & Permissions
### Role-Permission Matrix
**Source:** `backend/src/db/seeders/20200430130760-user-roles.js`
| Role | CREATE | READ | UPDATE | DELETE |
|------|--------|------|--------|--------|
| PlatformOwner | ✓ | ✓ | ✓ | ✓ |
| Administrator | ✓ | ✓ | ✓ | ✓ |
| AccountManager | ✗ | ✓ | ✗ | ✗ |
| TourDesigner | ✗ | ✓ | ✗ | ✗ |
| ContentReviewer | ✗ | ✓ | ✗ | ✗ |
| AnalyticsViewer | ✗ | ✓ | ✗ | ✗ |
### Permission Check Flow
```
Request → JWT Auth → checkCrudPermissions('access_logs') → Route Handler
┌───────────────────────────────┐
│ 1. AccessPolicy │
│ 2. Custom user permissions │
│ 3. Role-based permissions │
│ 4. Public hardening │
└───────────────────────────────┘
```
### Cascade Delete Behavior
Both user and project deletions cascade to access logs:
```javascript
// access_logs -> users
onDelete: 'CASCADE' // Logs deleted with user
// access_logs -> projects
onDelete: 'CASCADE' // Logs deleted with project
```
> **Note:** Access logs are NOT preserved when users or projects are deleted. If audit history retention is required, consider archiving logs before deletion or implementing SET NULL behavior.
## Data Capture Scope
### What IS Captured
| Data Point | Location | Purpose |
|------------|----------|---------|
| User ID | access_logs.userId | Who made the request |
| Project ID | access_logs.projectId | Which project context |
| API Path | access_logs.path | What endpoint accessed |
| IP Address | access_logs.ip_address | Client location/identity |
| User Agent | access_logs.user_agent | Browser/client identification |
| Timestamp | access_logs.accessed_at | When it happened |
| Environment | access_logs.environment | admin/stage/production |
### What IS NOT Captured
| Data Point | Reason |
|------------|--------|
| Request body | Privacy protection |
| Response content | Privacy/performance |
| Query parameters | Only path stored |
| Authentication credentials | Security |
| Personal user data | Beyond ID reference |
### Audit Trail vs Access Logs
| Aspect | Audit Trail | Access Logs |
|--------|-------------|-------------|
| Scope | All entities | Dedicated table |
| Granularity | Per-record changes | Per-request |
| Data | Who modified when | Request details |
| Retention | With entity | Separate lifecycle |
| Purpose | Change tracking | Security monitoring |
## Soft Delete & Retention
### Paranoid Mode
```javascript
{
timestamps: true,
paranoid: true, // Enables soft deletes
}
```
### Deletion Behavior
1. **Soft Delete**: Sets `deletedAt` timestamp
2. **Query Filtering**: Normal queries exclude deleted records
3. **Restoration**: Possible by clearing `deletedAt`
4. **Hard Delete**: Requires explicit permanent removal
### Retention Considerations
**Current Implementation:**
- No automatic log pruning
- Logs retained indefinitely
- Manual cleanup via DELETE endpoints
**Recommended Practices:**
- Implement TTL-based cleanup for old logs
- Archive logs to cold storage after 90 days
- Consider GDPR right-to-erasure requirements
## File Structure
### Backend
| File | Purpose |
|------|---------|
| `backend/src/db/models/access_logs.js` | Sequelize model definition |
| `backend/src/db/api/access_logs.ts` | Database API (CRUD) |
| `backend/src/db/api/base.api.js` | Audit trail implementation |
| `backend/src/routes/access_logs.ts` | REST endpoints |
| `backend/src/services/access_logs.ts` | Business logic |
| `backend/src/utils/logger.js` | Request logging middleware |
| `backend/src/middlewares/check-permissions.ts` | Permission enforcement |
### Frontend
| File | Purpose |
|------|---------|
| `frontend/src/pages/access_logs/access_logs-list.tsx` | List page |
| `frontend/src/pages/access_logs/access_logs-table.tsx` | Table page |
| `frontend/src/pages/access_logs/access_logs-new.tsx` | Create page |
| `frontend/src/pages/access_logs/access_logs-edit.tsx` | Edit page |
| `frontend/src/pages/access_logs/access_logs-view.tsx` | View page |
| `frontend/src/pages/access_logs/[access_logsId].tsx` | Dynamic route |
| `frontend/src/stores/access_logs/access_logsSlice.ts` | Redux state (createEntitySlice) |
| `frontend/src/components/Access_logs/TableAccess_logs.tsx` | Data grid component |
| `frontend/src/components/Access_logs/ListAccess_logs.tsx` | List view component |
| `frontend/src/components/Access_logs/CardAccess_logs.tsx` | Card view component |
| `frontend/src/components/Access_logs/configureAccess_logsCols.tsx` | Column config (createColumnLoader) |
| `frontend/src/types/entities.ts` | TypeScript interfaces |
## Known Considerations
### 1. Manual Log Creation
**Issue:** Access logs are typically created manually via API, not automatically by middleware.
**Impact:** Requires explicit logging calls or additional middleware to auto-populate.
### 2. User Deletion Cascades
**Issue:** `CASCADE` on userId means logs are deleted when user is deleted.
**Impact:** Audit history lost with user deletion. Consider archiving logs before user deletion or changing to `SET NULL` for preservation.
### 3. Project Deletion Cascades
**Issue:** All access logs deleted when project is deleted.
**Impact:** Audit history lost with project. Consider archiving before deletion.
### 4. No Query Parameter Logging
**Issue:** Only `path` is stored, not query strings.
**Impact:** Cannot reconstruct full request URL. May miss filter/search context.
### 5. IPv6 Support
**Issue:** `ip_address` limited to 45 characters.
**Impact:** Supports full IPv6 addresses (39 chars max). Adequate for current needs.
### 6. User Agent Truncation
**Issue:** `user_agent` limited to 1024 characters.
**Impact:** Very long user agents may be truncated. Rare but possible.
### 7. No Automatic Retention
**Issue:** Logs grow indefinitely without cleanup.
**Impact:** Database size increases over time. Recommend implementing retention policy.
### 8. TypeScript Interface Extras
**Issue:** Frontend `AccessLog` interface includes `action` and `metadata` fields not in database model.
**Impact:** These fields exist only in TypeScript for potential future use. Currently not persisted.
## Integration Examples
### Log Access in Custom Middleware
```javascript
import Access_logsDBApi from '../db/api/access_logs.ts';
async function logAccess(req, res, next) {
// Log after response completes
res.on('finish', async () => {
try {
await Access_logsDBApi.create({
project: req.project?.id,
user: req.currentUser?.id,
environment: determineEnvironment(req),
path: req.path,
ip_address: req.ip,
user_agent: req.headers['user-agent'],
accessed_at: new Date(),
}, { currentUser: req.currentUser });
} catch (err) {
// Don't fail request on logging error
console.error('Failed to log access:', err);
}
});
next();
}
```
### Query Logs by Time Range
```typescript
// Frontend
dispatch(fetch({
query: '?accessed_atRange=["2024-01-01T00:00:00Z","2024-01-31T23:59:59Z"]&environment=production&sort=DESC&field=accessed_at'
}));
```
### Export Logs for Compliance
```bash
curl -H "Authorization: Bearer $TOKEN" \
"https://api.example.com/api/access_logs?filetype=csv&accessed_atRange=[\"2024-01-01\",\"2024-03-31\"]" \
> access_logs_q1_2024.csv
```