300 lines
9.1 KiB
Markdown
300 lines
9.1 KiB
Markdown
# VM Deployment Runbook
|
|
|
|
Operational notes for the standard Flatlogic VM deployment used by this
|
|
project. This document describes the VM runtime layout, health checks, and the
|
|
June 2026 `503 Service Unavailable` recovery path.
|
|
|
|
## Runtime Topology
|
|
|
|
The standard VM runs the app behind Apache and Cloudflare:
|
|
|
|
```
|
|
Cloudflare
|
|
-> Apache :80
|
|
-> Frontend Next.js production server :3001
|
|
-> Backend API :3000
|
|
```
|
|
|
|
Do not assume older local development ports on the VM. The standard port split
|
|
is frontend `3001` and backend `3000`:
|
|
|
|
| Component | VM process | Port | Notes |
|
|
|-----------|------------|------|-------|
|
|
| Apache | `apache2` | 80 | Public entrypoint, reverse proxy |
|
|
| Frontend | `frontend-dev` | 3001 | `npm run build`, then `npm run start` |
|
|
| Backend | `backend-dev` | 3000 | `NODE_ENV=dev_stage npm run start` |
|
|
| Telemetry | `fl-telemetry` | 4317/4318 | Executor telemetry daemon |
|
|
| Executor | `fl-executor` | n/a | VM command/executor bridge |
|
|
|
|
The backend returns `401 Unauthorized` for protected API endpoints without a
|
|
JWT. A `401` from `http://127.0.0.1:3000/api/...` means the backend is alive.
|
|
The backend default is port `3000` for `dev_stage`; an explicit `PORT` env var
|
|
overrides that when needed.
|
|
|
|
## Process Manager
|
|
|
|
PM2 is managed by systemd:
|
|
|
|
```bash
|
|
sudo systemctl status pm2-ubuntu --no-pager
|
|
pm2 status
|
|
```
|
|
|
|
Expected PM2 apps:
|
|
|
|
| Name | Purpose |
|
|
|------|---------|
|
|
| `frontend-dev` | Next.js frontend production server |
|
|
| `backend-dev` | Express API, migrations, seed, watcher |
|
|
| `fl-telemetry` | Local telemetry daemon |
|
|
| `fl-executor` | Standard VM executor bridge |
|
|
|
|
The frontend PM2 app name may remain `frontend-dev` for compatibility with the
|
|
standard VM image, but the process should run the production script. Build the
|
|
VM frontend with:
|
|
|
|
```bash
|
|
cd /home/ubuntu/executor/workspace/frontend
|
|
npm run build
|
|
```
|
|
|
|
Start it with:
|
|
|
|
```bash
|
|
FRONT_PORT=3001 npm run start
|
|
```
|
|
|
|
The production frontend is a Next.js server build served by `next start`.
|
|
Do not run the VM frontend with `next dev`; the dev server displays the Next.js
|
|
dev indicator in presentations.
|
|
|
|
## Frontend Release Deploys
|
|
|
|
Automatic VM pulls should deploy the frontend as immutable releases instead of
|
|
rebuilding in the live workspace. The executor VCS layer builds a fresh copy
|
|
under:
|
|
|
|
```bash
|
|
/home/ubuntu/executor/frontend-releases/<timestamp>-<git-sha>/frontend
|
|
```
|
|
|
|
The deploy order is:
|
|
|
|
1. Pull the requested branch into `/home/ubuntu/executor/workspace`.
|
|
2. Archive `HEAD` into a new release directory.
|
|
3. Copy frontend env files from the live workspace when present:
|
|
`.env`, `.env.local`, `.env.production`, `.env.production.local`.
|
|
4. Run `npm ci`.
|
|
5. Run `npm run build`.
|
|
6. Remove non-runtime build caches from the new release:
|
|
`.next`, `.turbo`, `build/cache`. Production runtime assets stay in
|
|
`build`; local `next dev --turbopack` uses `.next` to avoid conflicts with
|
|
production build manifests.
|
|
7. Switch `frontend-dev` to the new release with
|
|
`FRONT_PORT=3001 pm2 start npm --name frontend-dev -- run start`.
|
|
8. Save PM2 and remove old frontend releases.
|
|
|
|
The active frontend release is the PM2 `frontend-dev` working directory. Check
|
|
it with:
|
|
|
|
```bash
|
|
pm2 jlist | jq '.[] | select(.name=="frontend-dev") | {
|
|
cwd:.pm2_env.pm_cwd,
|
|
script:.pm2_env.pm_exec_path,
|
|
args:.pm2_env.args,
|
|
env:{FRONT_PORT:.pm2_env.FRONT_PORT}
|
|
}'
|
|
```
|
|
|
|
Retention defaults to the latest 2 release directories. Override it by setting
|
|
`FRONTEND_RELEASES_KEEP` for the executor process before deploy. Do not delete
|
|
the active release directory; `next start` serves production assets from its
|
|
`build` directory.
|
|
|
|
Manual rollback is possible by starting `frontend-dev` from an older retained
|
|
release:
|
|
|
|
```bash
|
|
cd /home/ubuntu/executor/frontend-releases/<release-id>/frontend
|
|
pm2 delete frontend-dev
|
|
FRONT_PORT=3001 pm2 start npm --name frontend-dev -- run start
|
|
pm2 save --force
|
|
```
|
|
|
|
The PM2 dump is stored at:
|
|
|
|
```bash
|
|
~/.pm2/dump.pm2
|
|
```
|
|
|
|
This file contains environment variables and may contain secrets. Do not paste
|
|
it into public tools or tickets without redacting tokens, DB passwords, SMTP
|
|
credentials, API keys, and tunnel credentials.
|
|
|
|
## Health Checks
|
|
|
|
Use these checks after a deploy or incident:
|
|
|
|
```bash
|
|
df -h
|
|
df -ih
|
|
free -h
|
|
sudo ss -ltnp | grep -E ':80|:3001|:3000|:4317|:4318'
|
|
curl -I http://127.0.0.1:3001
|
|
curl -I http://127.0.0.1:3000/api/auth/me
|
|
curl -I http://tbp.flatlogic.app
|
|
pm2 status
|
|
```
|
|
|
|
Expected healthy responses:
|
|
|
|
- `http://127.0.0.1:3001` returns `200 OK`.
|
|
- `http://127.0.0.1:3000/api/auth/me` returns `401 Unauthorized` without JWT.
|
|
- `http://tbp.flatlogic.app` returns `200 OK`.
|
|
- PM2 shows all four apps `online`.
|
|
|
|
## Recovering From Apache `503 Service Unavailable`
|
|
|
|
If Apache returns:
|
|
|
|
```text
|
|
Service Unavailable
|
|
Apache/2.4.x Server at tbp.flatlogic.app Port 80
|
|
```
|
|
|
|
first check whether upstream app processes are listening:
|
|
|
|
```bash
|
|
sudo ss -ltnp | grep -E ':80|:3001|:3000'
|
|
curl -I http://127.0.0.1:3001
|
|
curl -I http://127.0.0.1:3000/api/auth/me
|
|
sudo systemctl status pm2-ubuntu --no-pager
|
|
```
|
|
|
|
If Apache is listening but `3001` and `3000` are not, PM2 did not restore or was
|
|
stopped. Restart it:
|
|
|
|
```bash
|
|
sudo systemctl reset-failed pm2-ubuntu
|
|
sudo systemctl restart pm2-ubuntu
|
|
pm2 status
|
|
```
|
|
|
|
Then re-run the health checks.
|
|
|
|
## OOM-Kill Diagnosis
|
|
|
|
A VM can have enough disk and still fail if the kernel kills PM2 or a child
|
|
process because memory spikes. Check kernel logs:
|
|
|
|
```bash
|
|
journalctl -k --since "YYYY-MM-DD HH:MM" --until "YYYY-MM-DD HH:MM" \
|
|
| grep -Ei 'oom|killed process|out of memory'
|
|
```
|
|
|
|
Known June 2026 incident:
|
|
|
|
- `pm2-ubuntu.service` failed with `Result: oom-kill`.
|
|
- Kernel killed `ffmpeg`.
|
|
- `ffmpeg` used about 3.3 GiB RSS on a 3.8 GiB RAM VM.
|
|
- PM2 then stopped `frontend-dev`, `backend-dev`, `fl-telemetry`, and
|
|
`fl-executor`.
|
|
|
|
This points to reversed video generation rather than Apache, disk space, or
|
|
frontend routing.
|
|
|
|
## FFmpeg and Reverse Video Generation
|
|
|
|
The backend uses bundled `ffmpeg-static`/`ffprobe-static` via
|
|
`backend/src/services/videoProcessing.ts`; manual OS-level FFmpeg installation
|
|
is not required for this project.
|
|
|
|
Reverse video generation can be memory-heavy for large videos. Operational
|
|
guardrails:
|
|
|
|
- FFmpeg reversal is serialized by `videoProcessing.reverseVideo()`: only one
|
|
FFmpeg process runs at a time in the backend process, and additional reverse
|
|
generation requests wait in an in-process queue.
|
|
- FFmpeg reversal uses `-threads 1`.
|
|
- FFmpeg reversal has a hard timeout (`FFMPEG_REVERSE_TIMEOUT_MS`, default
|
|
`600000`, exposed as `config.resilience.ffmpeg.reverseTimeoutMs`) and kills
|
|
the child process if it exceeds the limit.
|
|
- FFmpeg reversal is protected by an in-process circuit breaker
|
|
(`FFMPEG_BREAKER_FAILURE_THRESHOLD`, `FFMPEG_BREAKER_COOLDOWN_MS`,
|
|
`FFMPEG_BREAKER_SUCCESS_THRESHOLD`, exposed under
|
|
`config.resilience.ffmpeg.breaker`) so repeated media failures stop launching
|
|
new heavy jobs during the cooldown window.
|
|
- FFprobe metadata extraction has a timeout (`FFPROBE_TIMEOUT_MS`, default
|
|
`30000`, exposed as `config.resilience.ffmpeg.ffprobeTimeoutMs`).
|
|
- `TourPagesService` deduplicates reverse generation for the same source video
|
|
storage key.
|
|
- Treat large source videos as risky on small VMs.
|
|
- Check backend PM2 logs for `ffmpeg` or publish/save background errors.
|
|
- If the VM OOMs, inspect kernel logs before changing Apache or database config.
|
|
|
|
Remaining hardening work and follow-up:
|
|
|
|
- Add input duration/resolution/size checks before reversal.
|
|
- Structured logs now include reverse-video input/output size and probed media
|
|
metadata. Continue tuning rejection thresholds as real VM media patterns are
|
|
observed.
|
|
- Consider running media processing in a separate worker with memory limits.
|
|
|
|
## Logs
|
|
|
|
Useful log commands:
|
|
|
|
```bash
|
|
sudo journalctl -u pm2-ubuntu -n 200 --no-pager
|
|
pm2 logs frontend-dev --lines 100
|
|
pm2 logs backend-dev --lines 100
|
|
pm2 logs fl-executor --lines 100
|
|
pm2 logs fl-telemetry --lines 100
|
|
sudo tail -n 100 /var/log/apache2/error.log
|
|
```
|
|
|
|
`pm2 logs` tails by default. Press `Ctrl-C` before running the next command.
|
|
|
|
## Executor Notes
|
|
|
|
The standard VM `executor.js` in `~/executor` is not the web app startup script.
|
|
It handles VM commands, VCS operations, AI runner prompts, screenshots, and
|
|
telemetry. Starting it manually does not start the frontend/backend app.
|
|
|
|
Executor workspace path:
|
|
|
|
```bash
|
|
/home/ubuntu/executor/workspace
|
|
```
|
|
|
|
The executor can perform git operations when commanded, including reset/clean
|
|
workflows through VCS commands. Do not run executor commands blindly when the
|
|
goal is only to restore the web app. Use PM2/systemd for process recovery.
|
|
|
|
## Node Version
|
|
|
|
The project requirement is Node.js 20.x LTS. Some standard VMs may report
|
|
`/usr/bin/node` as Node 22 in PM2. If startup fails after a system update,
|
|
verify:
|
|
|
|
```bash
|
|
node -v
|
|
which node
|
|
pm2 describe backend-dev
|
|
pm2 describe frontend-dev
|
|
```
|
|
|
|
Changing the VM Node version should be coordinated with PM2 startup paths and a
|
|
full frontend/backend build check.
|
|
|
|
## Persistence
|
|
|
|
After changing PM2 process definitions, save the process list:
|
|
|
|
```bash
|
|
pm2 save
|
|
```
|
|
|
|
For an incident-only restart where the process definitions were unchanged,
|
|
`pm2 save` is still safe and keeps the current expected app list for reboot.
|