# DB Migrations Module

## Overview

The DB Migrations module manages database schema evolution using a typed
**Umzug v3** runner. It provides version-controlled, reversible database
changes that run automatically on server startup.

**Location:** `backend/src/db/migrations/`

**Files:** 34+ migration files (as of June 2026)

**Migration safety policy:** Do not rewrite, rename, or reformat already
applied migration files. Production databases track migration names in
`SequelizeMeta`, and fresh databases must replay the same schema history.

---

## Architecture

```
backend/
├── package.json                    # Umzug-backed migration scripts
└── src/db/
    ├── db-config.ts               # Database connection settings
    ├── umzug.ts                   # Typed Umzug runner
    └── migrations/                 # Migration files
        ├── package.json            # CommonJS boundary for legacy migrations
        ├── 20260319000001-*.js    # Foreign key constraints
        ├── 20260319000002-*.js    # Column cleanup
        ├── 20260326000001-*.js    # Table rename
        ├── 20260326000002-*.js    # ENUM to TEXT conversion
        ├── 20260326000003-*.js    # Create table
        ├── 20260326000004-*.js    # Data backfill
        ├── 20260326000005-*.js    # Environment fix
        ├── 20260326000006-*.js    # Cross-environment copy
        ├── 20260326043002-*.js    # NOT NULL enforcement
        ├── 20260326050442-*.js    # Column removal
        ├── 20260326054410-*.js    # Column removal
        ├── 20260326060000-*.js    # JSON data transformation
        ├── 20260326060001-*.js    # Table drop (page_elements)
        ├── 20260326060002-*.js    # Table drop (page_links)
        ├── 20260326060003-*.js    # Table drop (transitions)
        ├── 20260326171017-*.js    # Missing data insertion
        ├── 20260327000001-*.js    # Full data sync
        ├── 20260331024423-*.js    # Column removal (theme/css)
        ├── 20260331054340-*.js    # Duplicate cleanup
        ├── 20260331063424-*.js    # Invalid data cleanup
        ├── 20260403000001-*.js    # Background video settings
        ├── 20260409000001-*.js    # Design dimensions (projects)
        ├── 20260409111309-*.js    # Design dimensions (tour_pages)
        ├── 20260605000001-*.js    # Background audio settings
        ├── 20260626000001-*.js    # Private production presentation access
        ├── 20260626000002-*.js    # Account manager user creation permission
        ├── 20260628000001-*.js    # Global UI-control settings tables
        └── 20260628000005-*.js    # Existing-project UI-control snapshots
```

---

## Configuration

### Umzug Runner

`backend/src/db/umzug.ts` owns migration and seeder execution. It uses official
Umzug types, `SequelizeStorage`, and the existing storage tables:

| Flow | Files | Storage Table | Stored Names |
|------|-------|---------------|--------------|
| Migrations | `src/db/migrations/*.js` | `SequelizeMeta` | `*.js` |
| Seeders | `src/db/seeders/*.ts` in source, `dist/src/db/seeders/*.js` in build | `SequelizeData` | stable `*.js` names |

Seeder files are typed ESM source, and the runner stores stable execution names
so already executed seeders are not treated as pending.

### NPM Scripts
```bash
# Run pending migrations
npm run db:migrate

# Undo last migration
npm run db:migrate:undo

# Undo all migrations
npm run db:migrate:undo:all

# Check migration status
npm run db:migrate:status

# Full database reset (drop, create, migrate, seed)
npm run db:reset

# Seed data
npm run db:seed
```

---

## Migration File Structure

### Standard Template
```javascript
'use strict';

/**
 * Migration: [Description]
 *
 * [Explanation of what this migration does and why]
 */
module.exports = {
  async up(queryInterface, Sequelize) {
    // Apply changes
  },

  async down(queryInterface, Sequelize) {
    // Revert changes
  },
};
```

Use one project-wide migration template for new schema changes. Do not modify
already applied migration files to match newer style choices.

### Naming Convention
```
YYYYMMDDHHMMSS-descriptive-name.js

Examples:
- 20260319000001-add-foreign-key-constraints.js
- 20260326060000-convert-targetpageid-to-slug.js
- 20260326060001-drop-page-elements-table.js
- 20260628000001-create-ui-control-settings.js
- 20260628000005-snapshot-existing-project-ui-controls.js
```

---

## Migration Patterns

### 1. Transaction Wrapper Pattern
**Purpose:** Ensure atomic operations - all changes succeed or all fail.

```javascript
module.exports = {
  async up(queryInterface) {
    const transaction = await queryInterface.sequelize.transaction();

    try {
      // Multiple operations...
      await queryInterface.addColumn('table', 'column', { ... }, { transaction });
      await queryInterface.addIndex('table', ['column'], { transaction });

      await transaction.commit();
    } catch (error) {
      await transaction.rollback();
      throw error;
    }
  },
};
```

**Used in:** Foreign key constraints, ENUM conversions, data transformations

---

### 2. Idempotent Check Pattern
**Purpose:** Safely re-run migrations without errors.

```javascript
// Check if table/column/constraint exists before modifying
const [results] = await queryInterface.sequelize.query(
  `SELECT column_name FROM information_schema.columns
   WHERE table_name = 'tableName' AND column_name = 'columnName'`,
  { transaction }
);

if (results.length > 0) {
  console.log('Column already exists, skipping');
  return;
}

// Proceed with modification
await queryInterface.addColumn('tableName', 'columnName', { ... });
```

**Used in:** All migrations that add columns, constraints, or tables

---

### 3. Helper Function Pattern
**Purpose:** Reduce repetition for bulk operations.

```javascript
module.exports = {
  async up(queryInterface) {
    const transaction = await queryInterface.sequelize.transaction();

    // Define reusable helper
    const addForeignKey = async (tableName, columnName, references, onDelete) => {
      const constraintName = `${tableName}_${columnName}_fkey`;

      // Check existence
      const [results] = await queryInterface.sequelize.query(
        `SELECT constraint_name FROM information_schema.table_constraints
         WHERE table_name = '${tableName}' AND constraint_name = '${constraintName}'`,
        { transaction }
      );

      if (results.length === 0) {
        await queryInterface.addConstraint(tableName, {
          fields: [columnName],
          type: 'foreign key',
          name: constraintName,
          references,
          onDelete,
          onUpdate: 'CASCADE',
          transaction,
        });
        console.log(`Added FK: ${constraintName}`);
      }
    };

    // Use helper multiple times
    await addForeignKey('assets', 'projectId', { table: 'projects', field: 'id' }, 'CASCADE');
    await addForeignKey('tour_pages', 'projectId', { table: 'projects', field: 'id' }, 'CASCADE');
    // ... more FKs
  },
};
```

**Used in:** Foreign key constraints, column removal, index management

---

### 4. Safe Table Drop Pattern
**Purpose:** Prevent accidental data loss when dropping tables.

```javascript
module.exports = {
  async up(queryInterface) {
    // Verify table is empty before dropping
    const [results] = await queryInterface.sequelize.query(
      'SELECT COUNT(*) as count FROM table_name'
    );
    const count = parseInt(results[0].count, 10);

    if (count > 0) {
      throw new Error(
        `Cannot drop table_name: it contains ${count} records. ` +
        `Please migrate or delete them first.`
      );
    }

    await queryInterface.dropTable('table_name');
    console.log('Dropped table_name (was empty)');
  },

  async down(queryInterface, Sequelize) {
    // Full table recreation with all columns, indexes, and constraints
    await queryInterface.createTable('table_name', {
      id: { type: Sequelize.UUID, ... },
      // ... all columns
    });
  },
};
```

**Used in:** `drop-page-elements-table`, `drop-page-links-table`, `drop-transitions-table`

---

### 5. ENUM to TEXT Conversion Pattern
**Purpose:** Convert restrictive ENUMs to flexible TEXT while preserving data.

```javascript
module.exports = {
  async up(queryInterface, Sequelize) {
    const transaction = await queryInterface.sequelize.transaction();

    try {
      // 1. Create temporary TEXT column
      await queryInterface.addColumn('table', 'column_text', {
        type: Sequelize.TEXT,
        allowNull: true,
      }, { transaction });

      // 2. Copy ENUM values to TEXT
      await queryInterface.sequelize.query(
        `UPDATE table SET column_text = column::TEXT`,
        { transaction }
      );

      // 3. Drop old ENUM column
      await queryInterface.removeColumn('table', 'column', { transaction });

      // 4. Rename TEXT column
      await queryInterface.renameColumn('table', 'column_text', 'column', { transaction });

      // 5. Add NOT NULL constraint
      await queryInterface.changeColumn('table', 'column', {
        type: Sequelize.TEXT,
        allowNull: false,
      }, { transaction });

      // 6. Drop ENUM type
      await queryInterface.sequelize.query(
        `DROP TYPE IF EXISTS "enum_table_column"`,
        { transaction }
      );

      await transaction.commit();
    } catch (error) {
      await transaction.rollback();
      throw error;
    }
  },

  async down(queryInterface, Sequelize) {
    // Recreate ENUM type
    await queryInterface.sequelize.query(`
      CREATE TYPE "enum_table_column" AS ENUM ('value1', 'value2', 'value3')
    `);
    // ... reverse the process
  },
};
```

**Used in:** `convert-element-type-enum-to-text`

---

### 6. Data Backfill Pattern
**Purpose:** Populate new tables/columns with data from existing records.

```javascript
// Define default data
const DEFAULT_ELEMENT_TYPES = [
  { element_type: 'navigation_next', name: 'Forward Button', sort_order: 1, settings_json: {...} },
  { element_type: 'navigation_prev', name: 'Back Button', sort_order: 2, settings_json: {...} },
  // ... more defaults
];

module.exports = {
  async up(queryInterface, Sequelize) {
    // Get existing records
    const [projects] = await queryInterface.sequelize.query(
      `SELECT id FROM projects WHERE "deletedAt" IS NULL`,
      { type: Sequelize.QueryTypes.SELECT }
    );

    // For each project, check and insert missing records
    for (const project of projects) {
      const [existing] = await queryInterface.sequelize.query(
        `SELECT element_type FROM project_element_defaults
         WHERE "projectId" = :projectId AND "deletedAt" IS NULL`,
        { replacements: { projectId: project.id }, type: Sequelize.QueryTypes.SELECT }
      );

      const existingTypes = new Set(existing.map(d => d.element_type));

      for (const defaultType of DEFAULT_ELEMENT_TYPES) {
        if (!existingTypes.has(defaultType.element_type)) {
          await queryInterface.sequelize.query(`
            INSERT INTO project_element_defaults (...)
            VALUES (gen_random_uuid(), :element_type, :name, ...)
          `, { replacements: { ... } });
        }
      }
    }
  },

  async down(queryInterface) {
    // Delete only records created by this migration
    await queryInterface.sequelize.query(
      `DELETE FROM project_element_defaults WHERE snapshot_version = 1`
    );
  },
};
```

**Used in:** `backfill-project-element-defaults`, `sync-all-element-type-defaults`

---

### 7. Cross-Environment Data Copy Pattern
**Purpose:** Copy content between environments (dev → stage → production).

```javascript
module.exports = {
  async up(queryInterface, Sequelize) {
    const projects = await queryInterface.sequelize.query(
      `SELECT id FROM projects WHERE "deletedAt" IS NULL`,
      { type: Sequelize.QueryTypes.SELECT }
    );

    for (const project of projects) {
      // Check if target environment already has content
      const [stageCheck] = await queryInterface.sequelize.query(
        `SELECT COUNT(*)::int as count FROM tour_pages
         WHERE "projectId" = '${project.id}' AND environment = 'stage'`
      );

      if (stageCheck?.count > 0) continue;

      // Copy with INSERT...SELECT, generating new UUIDs
      await queryInterface.sequelize.query(`
        INSERT INTO tour_pages (id, slug, name, ..., environment, source_key, ...)
        SELECT
          gen_random_uuid(),
          slug, name, ...,
          'stage',        -- New environment
          id::text,       -- Track source record for rollback
          ...
        FROM tour_pages
        WHERE "projectId" = '${project.id}' AND environment = 'dev'
      `);

      // Copy related records using source_key for ID mapping
      await queryInterface.sequelize.query(`
        INSERT INTO page_elements (id, ..., "pageId", ...)
        SELECT
          gen_random_uuid(), ...,
          stage_page.id,  -- Map to new page ID
          ...
        FROM page_elements pe
        INNER JOIN tour_pages dev_page ON pe."pageId" = dev_page.id
        INNER JOIN tour_pages stage_page ON stage_page.source_key = dev_page.id::text
        WHERE dev_page.environment = 'dev'
      `);
    }
  },

  async down(queryInterface) {
    // Delete records with source_key (created by migration)
    await queryInterface.sequelize.query(
      `DELETE FROM tour_pages WHERE environment = 'stage' AND source_key IS NOT NULL`
    );
  },
};
```

**Used in:** `copy-dev-to-stage`

---

### 8. JSON Field Transformation Pattern
**Purpose:** Transform data stored in JSON columns.

```javascript
module.exports = {
  async up(queryInterface, Sequelize) {
    const transaction = await queryInterface.sequelize.transaction();

    try {
      // Get all records with JSON data
      const [records] = await queryInterface.sequelize.query(
        `SELECT id, "projectId", environment, slug, json_column
         FROM table_name WHERE json_column IS NOT NULL`,
        { transaction }
      );

      // Build lookup maps for ID → slug transformations
      const slugById = new Map();
      records.forEach(r => slugById.set(r.id, { projectId: r.projectId, slug: r.slug }));

      // Transform each record
      for (const record of records) {
        const jsonData = typeof record.json_column === 'string'
          ? JSON.parse(record.json_column)
          : record.json_column;

        let hasChanges = false;

        // Transform JSON structure
        if (jsonData.elements) {
          jsonData.elements.forEach(element => {
            if (element.targetPageId) {
              const target = slugById.get(element.targetPageId);
              if (target) {
                element.targetPageSlug = target.slug;
                delete element.targetPageId;
                hasChanges = true;
              }
            }
          });
        }

        if (hasChanges) {
          await queryInterface.sequelize.query(
            `UPDATE table_name SET json_column = :json WHERE id = :id`,
            {
              replacements: { json: JSON.stringify(jsonData), id: record.id },
              transaction
            }
          );
        }
      }

      await transaction.commit();
    } catch (error) {
      await transaction.rollback();
      throw error;
    }
  },
};
```

**Used in:** `convert-targetpageid-to-slug`

---

### 9. Constraint Enforcement Pattern
**Purpose:** Add NOT NULL constraints after fixing existing NULL values.

```javascript
module.exports = {
  async up(queryInterface) {
    // First, fix any NULL values
    await queryInterface.sequelize.query(
      `UPDATE table_name SET column = 'default' WHERE column IS NULL`
    );

    // Then add NOT NULL constraint with default
    await queryInterface.sequelize.query(`
      ALTER TABLE table_name
        ALTER COLUMN column SET NOT NULL,
        ALTER COLUMN column SET DEFAULT 'default'
    `);
  },

  async down(queryInterface) {
    await queryInterface.sequelize.query(`
      ALTER TABLE table_name
        ALTER COLUMN column DROP NOT NULL,
        ALTER COLUMN column DROP DEFAULT
    `);
  },
};
```

**Used in:** `enforce-environment-not-null`

---

### 10. Safe Down Migration Pattern
**Purpose:** Handle cases where down migration isn't meaningful.

```javascript
module.exports = {
  async up(queryInterface, Sequelize) {
    // Add/update missing data
    // ...
  },

  async down(_queryInterface, _Sequelize) {
    // This migration only adds missing data, not destructive
    console.log('No down migration needed - this migration only adds missing data.');
  },
};
```

**Used in:** `sync-all-element-type-defaults`

---

## Migration Categories

### Schema Changes
| Migration | Description |
|-----------|-------------|
| `add-foreign-key-constraints` | Add FK constraints to all model associations |
| `create-project-element-defaults` | Create new table with indexes |
| `drop-page-elements-table` | Drop unused table |
| `drop-page-links-table` | Drop unused table |
| `drop-transitions-table` | Drop unused table |

### Column Modifications
| Migration | Description |
|-----------|-------------|
| `remove-redundant-deletion-columns` | Remove `is_deleted`, `deleted_at_time` |
| `remove-project-phase-column` | Remove redundant `phase` column |
| `remove-entry-page-slug-column` | Remove unused column |
| `convert-element-type-enum-to-text` | ENUM → TEXT for flexibility |
| `enforce-environment-not-null` | Add NOT NULL constraint |
| `remove-unused-theme-columns-from-projects` | Remove `theme_config_json`, `custom_css_json`, `cdn_base_url` |
| `add-background-video-settings` | Add video playback settings (autoplay, loop, muted, start/end time) to tour_pages |
| `add-design-dimensions-to-projects` | Add `design_width`, `design_height` to projects table |
| `add-design-dimensions-to-tour-pages` | Add `design_width`, `design_height` to tour_pages table |

### Table Renames
| Migration | Description |
|-----------|-------------|
| `rename-ui-elements-to-element-type-defaults` | Rename for clarity |

### Data Migrations
| Migration | Description |
|-----------|-------------|
| `backfill-project-element-defaults` | Populate new table for existing projects |
| `copy-dev-to-stage` | Initialize stage environment |
| `convert-targetpageid-to-slug` | Transform JSON navigation references |
| `fix-project-audio-tracks-environment` | Fix environment values |
| `add-missing-element-type-defaults` | Insert missing default rows |
| `sync-all-element-type-defaults` | Full sync of all 11 element types |
| `remove-duplicate-element-type-defaults` | Remove duplicate records created during earlier migrations |
| `cleanup-invalid-element-type-defaults` | Clean up invalid entries and ensure data integrity |

---

## Foreign Key Strategies

| Strategy | When to Use | Example |
|----------|-------------|---------|
| `CASCADE` | Delete child when parent deleted | `assets.projectId → projects.id` |
| `SET NULL` | Preserve record, nullify FK | `publish_events.userId → users.id` (audit trail) |
| `SET NULL` + `allowNull: true` | Optional FK | `users.app_roleId → roles.id` |

```javascript
// CASCADE - delete assets when project is deleted
await addForeignKey('assets', 'projectId', { table: 'projects', field: 'id' }, 'CASCADE');

// SET NULL - preserve audit log when user is deleted
await addForeignKey('access_logs', 'userId', { table: 'users', field: 'id' }, 'SET NULL');
```

---

## Best Practices

### 1. Always Use Transactions
```javascript
const transaction = await queryInterface.sequelize.transaction();
try {
  // ... operations
  await transaction.commit();
} catch (error) {
  await transaction.rollback();
  throw error;
}
```

### 2. Check Before Modify
```javascript
// Always check existence before adding/removing
const tableExists = await queryInterface.sequelize.query(
  `SELECT EXISTS (SELECT FROM information_schema.tables WHERE table_name = 'name')`
);
```

### 3. Log Progress
```javascript
console.log(`Migrating project ${projectId}: ${addedCount} records added`);
console.log('Migration complete: All foreign keys added');
```

### 4. Safe Drops
```javascript
// Never drop non-empty tables silently
if (count > 0) {
  throw new Error(`Cannot drop table: contains ${count} records`);
}
```

### 5. Reversible Operations
```javascript
// Down migration should restore previous state
async down(queryInterface, Sequelize) {
  // Recreate everything that was dropped
  await queryInterface.createTable('table_name', { ... });
  await queryInterface.addIndex('table_name', [...]);
}
```

### 6. Use Parameterized Queries
```javascript
// Good - prevents SQL injection
await queryInterface.sequelize.query(
  `UPDATE table SET column = :value WHERE id = :id`,
  { replacements: { value: 'safe', id: record.id } }
);

// Avoid - SQL injection risk
await queryInterface.sequelize.query(
  `UPDATE table SET column = '${unsafeValue}' WHERE id = '${unsafeId}'`
);
```

---

## Running Migrations

### Development
```bash
cd backend
npm run db:migrate
```

### Server Startup
Migrations run automatically via `npm start`:
```json
{
  "scripts": {
    "start": "npm run db:migrate && npm run db:seed && npm run watch"
  }
}
```

### Migration Status
```bash
npm run db:migrate:status
```

### Undo Migrations
```bash
# Undo last migration
npm run db:migrate:undo

# Undo all migrations (dangerous!)
npm run db:migrate:undo:all
```

### Create New Migration

Create new migration files manually under `backend/src/db/migrations/` only
when a schema change is required. Keep every migration reversible or document an
explicit rollback/backup plan.

---

## Current Migration Inventory

| # | Timestamp | Name | Type |
|---|-----------|------|------|
| 1 | 20260319000001 | add-foreign-key-constraints | Schema |
| 2 | 20260319000002 | remove-redundant-deletion-columns | Column |
| 3 | 20260326000001 | rename-ui-elements-to-element-type-defaults | Rename |
| 4 | 20260326000002 | convert-element-type-enum-to-text | Column |
| 5 | 20260326000003 | create-project-element-defaults | Schema |
| 6 | 20260326000004 | backfill-project-element-defaults | Data |
| 7 | 20260326000005 | fix-project-audio-tracks-environment | Data |
| 8 | 20260326000006 | copy-dev-to-stage | Data |
| 9 | 20260326043002 | enforce-environment-not-null | Column |
| 10 | 20260326050442 | remove-project-phase-column | Column |
| 11 | 20260326054410 | remove-entry-page-slug-column | Column |
| 12 | 20260326060000 | convert-targetpageid-to-slug | Data |
| 13 | 20260326060001 | drop-page-elements-table | Schema |
| 14 | 20260326060002 | drop-page-links-table | Schema |
| 15 | 20260326060003 | drop-transitions-table | Schema |
| 16 | 20260326171017 | add-missing-element-type-defaults | Data |
| 17 | 20260327000001 | sync-all-element-type-defaults | Data |
| 18 | 20260331024423 | remove-unused-theme-columns-from-projects | Column |
| 19 | 20260331054340 | remove-duplicate-element-type-defaults | Data |
| 20 | 20260331063424 | cleanup-invalid-element-type-defaults | Data |
| 21 | 20260403000001 | add-background-video-settings | Column |
| 22 | 20260409000001 | add-design-dimensions-to-projects | Column |
| 23 | 20260409111309 | add-design-dimensions-to-tour-pages | Column |
| 24 | 20260605000001 | add-background-audio-settings | Column |

---

## Related Documentation

- [DB Models](./db-models.md) - Sequelize model definitions
- [DB API](./db-api.md) - Database access layer
- [Database Schema](../database-schema.md) - Complete schema reference