Dmitri 9f111d6226 added documentation

2026-07-03 16:11:24 +02:00

23 KiB

Raw Blame History

DB Migrations Module

Overview

The DB Migrations module manages database schema evolution using a typed Umzug v3 runner. It provides version-controlled, reversible database changes that run automatically on server startup.

Location: backend/src/db/migrations/

Files: 34+ migration files (as of June 2026)

Migration safety policy: Do not rewrite, rename, or reformat already applied migration files. Production databases track migration names in SequelizeMeta, and fresh databases must replay the same schema history.

Architecture

backend/
├── package.json                    # Umzug-backed migration scripts
└── src/db/
    ├── db-config.ts               # Database connection settings
    ├── umzug.ts                   # Typed Umzug runner
    └── migrations/                 # Migration files
        ├── package.json            # CommonJS boundary for legacy migrations
        ├── 20260319000001-*.js    # Foreign key constraints
        ├── 20260319000002-*.js    # Column cleanup
        ├── 20260326000001-*.js    # Table rename
        ├── 20260326000002-*.js    # ENUM to TEXT conversion
        ├── 20260326000003-*.js    # Create table
        ├── 20260326000004-*.js    # Data backfill
        ├── 20260326000005-*.js    # Environment fix
        ├── 20260326000006-*.js    # Cross-environment copy
        ├── 20260326043002-*.js    # NOT NULL enforcement
        ├── 20260326050442-*.js    # Column removal
        ├── 20260326054410-*.js    # Column removal
        ├── 20260326060000-*.js    # JSON data transformation
        ├── 20260326060001-*.js    # Table drop (page_elements)
        ├── 20260326060002-*.js    # Table drop (page_links)
        ├── 20260326060003-*.js    # Table drop (transitions)
        ├── 20260326171017-*.js    # Missing data insertion
        ├── 20260327000001-*.js    # Full data sync
        ├── 20260331024423-*.js    # Column removal (theme/css)
        ├── 20260331054340-*.js    # Duplicate cleanup
        ├── 20260331063424-*.js    # Invalid data cleanup
        ├── 20260403000001-*.js    # Background video settings
        ├── 20260409000001-*.js    # Design dimensions (projects)
        ├── 20260409111309-*.js    # Design dimensions (tour_pages)
        ├── 20260605000001-*.js    # Background audio settings
        ├── 20260626000001-*.js    # Private production presentation access
        ├── 20260626000002-*.js    # Account manager user creation permission
        ├── 20260628000001-*.js    # Global UI-control settings tables
        └── 20260628000005-*.js    # Existing-project UI-control snapshots

Configuration

Umzug Runner

backend/src/db/umzug.ts owns migration and seeder execution. It uses official Umzug types, SequelizeStorage, and the existing storage tables:

Flow	Files	Storage Table	Stored Names
Migrations	`src/db/migrations/*.js`	`SequelizeMeta`	`*.js`
Seeders	`src/db/seeders/.ts` in source, `dist/src/db/seeders/.js` in build	`SequelizeData`	stable `*.js` names

Seeder files are typed ESM source, and the runner stores stable execution names so already executed seeders are not treated as pending.

NPM Scripts

# Run pending migrations
npm run db:migrate

# Undo last migration
npm run db:migrate:undo

# Undo all migrations
npm run db:migrate:undo:all

# Check migration status
npm run db:migrate:status

# Full database reset (drop, create, migrate, seed)
npm run db:reset

# Seed data
npm run db:seed

Migration File Structure

Standard Template

'use strict';

/**
 * Migration: [Description]
 *
 * [Explanation of what this migration does and why]
 */
module.exports = {
  async up(queryInterface, Sequelize) {
    // Apply changes
  },

  async down(queryInterface, Sequelize) {
    // Revert changes
  },
};

Use one project-wide migration template for new schema changes. Do not modify already applied migration files to match newer style choices.

Naming Convention

YYYYMMDDHHMMSS-descriptive-name.js

Examples:
- 20260319000001-add-foreign-key-constraints.js
- 20260326060000-convert-targetpageid-to-slug.js
- 20260326060001-drop-page-elements-table.js
- 20260628000001-create-ui-control-settings.js
- 20260628000005-snapshot-existing-project-ui-controls.js

Migration Patterns

1. Transaction Wrapper Pattern

Purpose: Ensure atomic operations - all changes succeed or all fail.

module.exports = {
  async up(queryInterface) {
    const transaction = await queryInterface.sequelize.transaction();

    try {
      // Multiple operations...
      await queryInterface.addColumn('table', 'column', { ... }, { transaction });
      await queryInterface.addIndex('table', ['column'], { transaction });

      await transaction.commit();
    } catch (error) {
      await transaction.rollback();
      throw error;
    }
  },
};

Used in: Foreign key constraints, ENUM conversions, data transformations

2. Idempotent Check Pattern

Purpose: Safely re-run migrations without errors.

// Check if table/column/constraint exists before modifying
const [results] = await queryInterface.sequelize.query(
  `SELECT column_name FROM information_schema.columns
   WHERE table_name = 'tableName' AND column_name = 'columnName'`,
  { transaction }
);

if (results.length > 0) {
  console.log('Column already exists, skipping');
  return;
}

// Proceed with modification
await queryInterface.addColumn('tableName', 'columnName', { ... });

Used in: All migrations that add columns, constraints, or tables

3. Helper Function Pattern

Purpose: Reduce repetition for bulk operations.

module.exports = {
  async up(queryInterface) {
    const transaction = await queryInterface.sequelize.transaction();

    // Define reusable helper
    const addForeignKey = async (tableName, columnName, references, onDelete) => {
      const constraintName = `${tableName}_${columnName}_fkey`;

      // Check existence
      const [results] = await queryInterface.sequelize.query(
        `SELECT constraint_name FROM information_schema.table_constraints
         WHERE table_name = '${tableName}' AND constraint_name = '${constraintName}'`,
        { transaction }
      );

      if (results.length === 0) {
        await queryInterface.addConstraint(tableName, {
          fields: [columnName],
          type: 'foreign key',
          name: constraintName,
          references,
          onDelete,
          onUpdate: 'CASCADE',
          transaction,
        });
        console.log(`Added FK: ${constraintName}`);
      }
    };

    // Use helper multiple times
    await addForeignKey('assets', 'projectId', { table: 'projects', field: 'id' }, 'CASCADE');
    await addForeignKey('tour_pages', 'projectId', { table: 'projects', field: 'id' }, 'CASCADE');
    // ... more FKs
  },
};

Used in: Foreign key constraints, column removal, index management

4. Safe Table Drop Pattern

Purpose: Prevent accidental data loss when dropping tables.

module.exports = {
  async up(queryInterface) {
    // Verify table is empty before dropping
    const [results] = await queryInterface.sequelize.query(
      'SELECT COUNT(*) as count FROM table_name'
    );
    const count = parseInt(results[0].count, 10);

    if (count > 0) {
      throw new Error(
        `Cannot drop table_name: it contains ${count} records. ` +
        `Please migrate or delete them first.`
      );
    }

    await queryInterface.dropTable('table_name');
    console.log('Dropped table_name (was empty)');
  },

  async down(queryInterface, Sequelize) {
    // Full table recreation with all columns, indexes, and constraints
    await queryInterface.createTable('table_name', {
      id: { type: Sequelize.UUID, ... },
      // ... all columns
    });
  },
};

Used in: drop-page-elements-table, drop-page-links-table, drop-transitions-table

5. ENUM to TEXT Conversion Pattern

Purpose: Convert restrictive ENUMs to flexible TEXT while preserving data.

module.exports = {
  async up(queryInterface, Sequelize) {
    const transaction = await queryInterface.sequelize.transaction();

    try {
      // 1. Create temporary TEXT column
      await queryInterface.addColumn('table', 'column_text', {
        type: Sequelize.TEXT,
        allowNull: true,
      }, { transaction });

      // 2. Copy ENUM values to TEXT
      await queryInterface.sequelize.query(
        `UPDATE table SET column_text = column::TEXT`,
        { transaction }
      );

      // 3. Drop old ENUM column
      await queryInterface.removeColumn('table', 'column', { transaction });

      // 4. Rename TEXT column
      await queryInterface.renameColumn('table', 'column_text', 'column', { transaction });

      // 5. Add NOT NULL constraint
      await queryInterface.changeColumn('table', 'column', {
        type: Sequelize.TEXT,
        allowNull: false,
      }, { transaction });

      // 6. Drop ENUM type
      await queryInterface.sequelize.query(
        `DROP TYPE IF EXISTS "enum_table_column"`,
        { transaction }
      );

      await transaction.commit();
    } catch (error) {
      await transaction.rollback();
      throw error;
    }
  },

  async down(queryInterface, Sequelize) {
    // Recreate ENUM type
    await queryInterface.sequelize.query(`
      CREATE TYPE "enum_table_column" AS ENUM ('value1', 'value2', 'value3')
    `);
    // ... reverse the process
  },
};

Used in: convert-element-type-enum-to-text

6. Data Backfill Pattern

Purpose: Populate new tables/columns with data from existing records.

// Define default data
const DEFAULT_ELEMENT_TYPES = [
  { element_type: 'navigation_next', name: 'Forward Button', sort_order: 1, settings_json: {...} },
  { element_type: 'navigation_prev', name: 'Back Button', sort_order: 2, settings_json: {...} },
  // ... more defaults
];

module.exports = {
  async up(queryInterface, Sequelize) {
    // Get existing records
    const [projects] = await queryInterface.sequelize.query(
      `SELECT id FROM projects WHERE "deletedAt" IS NULL`,
      { type: Sequelize.QueryTypes.SELECT }
    );

    // For each project, check and insert missing records
    for (const project of projects) {
      const [existing] = await queryInterface.sequelize.query(
        `SELECT element_type FROM project_element_defaults
         WHERE "projectId" = :projectId AND "deletedAt" IS NULL`,
        { replacements: { projectId: project.id }, type: Sequelize.QueryTypes.SELECT }
      );

      const existingTypes = new Set(existing.map(d => d.element_type));

      for (const defaultType of DEFAULT_ELEMENT_TYPES) {
        if (!existingTypes.has(defaultType.element_type)) {
          await queryInterface.sequelize.query(`
            INSERT INTO project_element_defaults (...)
            VALUES (gen_random_uuid(), :element_type, :name, ...)
          `, { replacements: { ... } });
        }
      }
    }
  },

  async down(queryInterface) {
    // Delete only records created by this migration
    await queryInterface.sequelize.query(
      `DELETE FROM project_element_defaults WHERE snapshot_version = 1`
    );
  },
};

Used in: backfill-project-element-defaults, sync-all-element-type-defaults

7. Cross-Environment Data Copy Pattern

Purpose: Copy content between environments (dev → stage → production).

module.exports = {
  async up(queryInterface, Sequelize) {
    const projects = await queryInterface.sequelize.query(
      `SELECT id FROM projects WHERE "deletedAt" IS NULL`,
      { type: Sequelize.QueryTypes.SELECT }
    );

    for (const project of projects) {
      // Check if target environment already has content
      const [stageCheck] = await queryInterface.sequelize.query(
        `SELECT COUNT(*)::int as count FROM tour_pages
         WHERE "projectId" = '${project.id}' AND environment = 'stage'`
      );

      if (stageCheck?.count > 0) continue;

      // Copy with INSERT...SELECT, generating new UUIDs
      await queryInterface.sequelize.query(`
        INSERT INTO tour_pages (id, slug, name, ..., environment, source_key, ...)
        SELECT
          gen_random_uuid(),
          slug, name, ...,
          'stage',        -- New environment
          id::text,       -- Track source record for rollback
          ...
        FROM tour_pages
        WHERE "projectId" = '${project.id}' AND environment = 'dev'
      `);

      // Copy related records using source_key for ID mapping
      await queryInterface.sequelize.query(`
        INSERT INTO page_elements (id, ..., "pageId", ...)
        SELECT
          gen_random_uuid(), ...,
          stage_page.id,  -- Map to new page ID
          ...
        FROM page_elements pe
        INNER JOIN tour_pages dev_page ON pe."pageId" = dev_page.id
        INNER JOIN tour_pages stage_page ON stage_page.source_key = dev_page.id::text
        WHERE dev_page.environment = 'dev'
      `);
    }
  },

  async down(queryInterface) {
    // Delete records with source_key (created by migration)
    await queryInterface.sequelize.query(
      `DELETE FROM tour_pages WHERE environment = 'stage' AND source_key IS NOT NULL`
    );
  },
};

Used in: copy-dev-to-stage

8. JSON Field Transformation Pattern

Purpose: Transform data stored in JSON columns.

module.exports = {
  async up(queryInterface, Sequelize) {
    const transaction = await queryInterface.sequelize.transaction();

    try {
      // Get all records with JSON data
      const [records] = await queryInterface.sequelize.query(
        `SELECT id, "projectId", environment, slug, json_column
         FROM table_name WHERE json_column IS NOT NULL`,
        { transaction }
      );

      // Build lookup maps for ID → slug transformations
      const slugById = new Map();
      records.forEach(r => slugById.set(r.id, { projectId: r.projectId, slug: r.slug }));

      // Transform each record
      for (const record of records) {
        const jsonData = typeof record.json_column === 'string'
          ? JSON.parse(record.json_column)
          : record.json_column;

        let hasChanges = false;

        // Transform JSON structure
        if (jsonData.elements) {
          jsonData.elements.forEach(element => {
            if (element.targetPageId) {
              const target = slugById.get(element.targetPageId);
              if (target) {
                element.targetPageSlug = target.slug;
                delete element.targetPageId;
                hasChanges = true;
              }
            }
          });
        }

        if (hasChanges) {
          await queryInterface.sequelize.query(
            `UPDATE table_name SET json_column = :json WHERE id = :id`,
            {
              replacements: { json: JSON.stringify(jsonData), id: record.id },
              transaction
            }
          );
        }
      }

      await transaction.commit();
    } catch (error) {
      await transaction.rollback();
      throw error;
    }
  },
};

Used in: convert-targetpageid-to-slug

9. Constraint Enforcement Pattern

Purpose: Add NOT NULL constraints after fixing existing NULL values.

module.exports = {
  async up(queryInterface) {
    // First, fix any NULL values
    await queryInterface.sequelize.query(
      `UPDATE table_name SET column = 'default' WHERE column IS NULL`
    );

    // Then add NOT NULL constraint with default
    await queryInterface.sequelize.query(`
      ALTER TABLE table_name
        ALTER COLUMN column SET NOT NULL,
        ALTER COLUMN column SET DEFAULT 'default'
    `);
  },

  async down(queryInterface) {
    await queryInterface.sequelize.query(`
      ALTER TABLE table_name
        ALTER COLUMN column DROP NOT NULL,
        ALTER COLUMN column DROP DEFAULT
    `);
  },
};

Used in: enforce-environment-not-null

10. Safe Down Migration Pattern

Purpose: Handle cases where down migration isn't meaningful.

module.exports = {
  async up(queryInterface, Sequelize) {
    // Add/update missing data
    // ...
  },

  async down(_queryInterface, _Sequelize) {
    // This migration only adds missing data, not destructive
    console.log('No down migration needed - this migration only adds missing data.');
  },
};

Used in: sync-all-element-type-defaults

Migration Categories

Schema Changes

Migration	Description
`add-foreign-key-constraints`	Add FK constraints to all model associations
`create-project-element-defaults`	Create new table with indexes
`drop-page-elements-table`	Drop unused table
`drop-page-links-table`	Drop unused table
`drop-transitions-table`	Drop unused table

Column Modifications

Migration	Description
`remove-redundant-deletion-columns`	Remove `is_deleted`, `deleted_at_time`
`remove-project-phase-column`	Remove redundant `phase` column
`remove-entry-page-slug-column`	Remove unused column
`convert-element-type-enum-to-text`	ENUM → TEXT for flexibility
`enforce-environment-not-null`	Add NOT NULL constraint
`remove-unused-theme-columns-from-projects`	Remove `theme_config_json`, `custom_css_json`, `cdn_base_url`
`add-background-video-settings`	Add video playback settings (autoplay, loop, muted, start/end time) to tour_pages
`add-design-dimensions-to-projects`	Add `design_width`, `design_height` to projects table
`add-design-dimensions-to-tour-pages`	Add `design_width`, `design_height` to tour_pages table

Table Renames

Migration	Description
`rename-ui-elements-to-element-type-defaults`	Rename for clarity

Data Migrations

Migration	Description
`backfill-project-element-defaults`	Populate new table for existing projects
`copy-dev-to-stage`	Initialize stage environment
`convert-targetpageid-to-slug`	Transform JSON navigation references
`fix-project-audio-tracks-environment`	Fix environment values
`add-missing-element-type-defaults`	Insert missing default rows
`sync-all-element-type-defaults`	Full sync of all 11 element types
`remove-duplicate-element-type-defaults`	Remove duplicate records created during earlier migrations
`cleanup-invalid-element-type-defaults`	Clean up invalid entries and ensure data integrity

Foreign Key Strategies

Strategy	When to Use	Example
`CASCADE`	Delete child when parent deleted	`assets.projectId → projects.id`
`SET NULL`	Preserve record, nullify FK	`publish_events.userId → users.id` (audit trail)
`SET NULL` + `allowNull: true`	Optional FK	`users.app_roleId → roles.id`

// CASCADE - delete assets when project is deleted
await addForeignKey('assets', 'projectId', { table: 'projects', field: 'id' }, 'CASCADE');

// SET NULL - preserve audit log when user is deleted
await addForeignKey('access_logs', 'userId', { table: 'users', field: 'id' }, 'SET NULL');

Best Practices

1. Always Use Transactions

const transaction = await queryInterface.sequelize.transaction();
try {
  // ... operations
  await transaction.commit();
} catch (error) {
  await transaction.rollback();
  throw error;
}

2. Check Before Modify

// Always check existence before adding/removing
const tableExists = await queryInterface.sequelize.query(
  `SELECT EXISTS (SELECT FROM information_schema.tables WHERE table_name = 'name')`
);

3. Log Progress

console.log(`Migrating project ${projectId}: ${addedCount} records added`);
console.log('Migration complete: All foreign keys added');

4. Safe Drops

// Never drop non-empty tables silently
if (count > 0) {
  throw new Error(`Cannot drop table: contains ${count} records`);
}

5. Reversible Operations

// Down migration should restore previous state
async down(queryInterface, Sequelize) {
  // Recreate everything that was dropped
  await queryInterface.createTable('table_name', { ... });
  await queryInterface.addIndex('table_name', [...]);
}

6. Use Parameterized Queries

// Good - prevents SQL injection
await queryInterface.sequelize.query(
  `UPDATE table SET column = :value WHERE id = :id`,
  { replacements: { value: 'safe', id: record.id } }
);

// Avoid - SQL injection risk
await queryInterface.sequelize.query(
  `UPDATE table SET column = '${unsafeValue}' WHERE id = '${unsafeId}'`
);

Running Migrations

Development

cd backend
npm run db:migrate

Server Startup

Migrations run automatically via npm start:

{
  "scripts": {
    "start": "npm run db:migrate && npm run db:seed && npm run watch"
  }
}

Migration Status

npm run db:migrate:status

Undo Migrations

# Undo last migration
npm run db:migrate:undo

# Undo all migrations (dangerous!)
npm run db:migrate:undo:all

Create New Migration

Create new migration files manually under backend/src/db/migrations/ only when a schema change is required. Keep every migration reversible or document an explicit rollback/backup plan.

Current Migration Inventory

#	Timestamp	Name	Type
1	20260319000001	add-foreign-key-constraints	Schema
2	20260319000002	remove-redundant-deletion-columns	Column
3	20260326000001	rename-ui-elements-to-element-type-defaults	Rename
4	20260326000002	convert-element-type-enum-to-text	Column
5	20260326000003	create-project-element-defaults	Schema
6	20260326000004	backfill-project-element-defaults	Data
7	20260326000005	fix-project-audio-tracks-environment	Data
8	20260326000006	copy-dev-to-stage	Data
9	20260326043002	enforce-environment-not-null	Column
10	20260326050442	remove-project-phase-column	Column
11	20260326054410	remove-entry-page-slug-column	Column
12	20260326060000	convert-targetpageid-to-slug	Data
13	20260326060001	drop-page-elements-table	Schema
14	20260326060002	drop-page-links-table	Schema
15	20260326060003	drop-transitions-table	Schema
16	20260326171017	add-missing-element-type-defaults	Data
17	20260327000001	sync-all-element-type-defaults	Data
18	20260331024423	remove-unused-theme-columns-from-projects	Column
19	20260331054340	remove-duplicate-element-type-defaults	Data
20	20260331063424	cleanup-invalid-element-type-defaults	Data
21	20260403000001	add-background-video-settings	Column
22	20260409000001	add-design-dimensions-to-projects	Column
23	20260409111309	add-design-dimensions-to-tour-pages	Column
24	20260605000001	add-background-audio-settings	Column

DB Models - Sequelize model definitions
DB API - Database access layer
Database Schema - Complete schema reference

23 KiB Raw Blame History