# 👻 Ghost Node — Progress Log

> **Read this alongside CLAUDE.md at the start of every new session.**
> Full session history (all sessions, including Q&A decisions) → **ARCHIVE.md** — do NOT load at session start.
>
> ### ⚠️ MD Update Rule — DYNAMIC, NOT STATIC
> Only update the MD file whose content was actually touched this session. Not all files — only the relevant ones.
> - **PROGRESS.md** → update when any code file changed. Add a brief session entry here, move it to ARCHIVE.md when it's no longer active context.
> - **CLAUDE.md** → update only if API endpoints, DB schema, config keys, or architecture changed.
> - **MEMORY.md** → update only if a new reusable gotcha, pattern, or architecture decision was discovered.
> - **ARCHIVE.md** → move old PROGRESS.md session entries here. Also log Q&A sessions that produced lasting decisions.
> - If nothing changed in a session (pure Q&A, no files edited, no decisions made) → update nothing.

---

## Current State

- **Version:** v2.7
- **Status:** ✅ Fully operational — 21/21 tests passing
- **Last session:** 52 (2026-03-19) — wording normalization pass for mechanics docs (canonical phrasing alignment).
- **Previous session:** 44 (2026-03-19) — clean-room marketing landing rebuild (new `LandingPageV3` + OG image).
- **Previous session:** 39 (2026-03-19) — added `/api/listings/refresh-status` contract test (Option A baseline).
- **Previous session:** 38 (2026-03-19) — added backend contract tests for `/api/stats` and `/api/listings/countdown-sync`.
- **Previous session:** 34 (2026-03-18) — added `docs/MD_UPDATE_PLAYBOOK.md` playbook for deep semantic Markdown audits + minimal targeted MD edits workflow.
- **Previous session:** 33 (2026-03-18) — per-site visible browser override + keyword batching progress tracking + hourly retry warnings. See details below.
- **Previous session:** 32 (2026-03-18) — Redis cache layer, Docker Compose, lot description extraction (N18). See details below.
- **Previous session:** 31 (2026-03-13) — Fixed Framer Motion SSR opacity:0 bug across all components (LandingPage hero, dashboard, StatsGrid, RecentListings, EngineConsole, ActivityLog, ListingRow, ListingsTable, ListingDetailPanel, 4 page routes). All above-the-fold elements now render visible in static HTML (opacity:1, only transforms animate). Below-fold scroll-triggered sections intentionally keep opacity:0 for scroll animations. 21/21 tests passing.
- **Previous session:** 30 (2026-03-13) — Full marketing landing page built from AI_WEB_BRIEF.md (Hero, Platform Strip, 6-feature grid, How It Works 4-step flow, Who It's For personas, FAQ accordion, Footer CTA + site footer). Light/dark theme toggle added (ThemeToggle.tsx, localStorage-persistent, vivid lavender-ivory light palette via data-theme attribute). globals.css extended with complete [data-theme="light"] overrides for all g-* classes. 21/21 tests passing.
- **Previous session:** 29 (2026-03-13) — Master dashboard "Mission Control" rebuilt: RecentListings (polls /api/listings every 10s), EngineConsole (status orb + controls + site health, polls /api/sites every 25s), enhanced ActivityLog (tracks engine state transitions), StatsGrid updated (keywords prop replaces engine card), utils.ts extended (formatUptime, formatMins, formatPrice, timeAgo). 21/21 tests passing.
- **worker.py:** 5,000+ lines (definitive — ahead of all exports)
- **frontend/:** Next.js 16 + React 19 + TypeScript + Tailwind v4 — static build in `frontend/out/` (1.8MB). `node_modules/` is 553MB build tooling — safe to delete, run `npm install` to restore.
- **models.py:** ~420 lines
- **Serving:** `http://localhost:7000` (React UI) | `http://localhost:7000/legacy` (HTML fallback)

---

## 🟢 Session 52 — 2026-03-19

**Canonical wording normalization (Markdown only)**
- Ran a minimal wording-only pass on already-touched docs so mechanics phrasing is consistent across files.
- Standardized the same canonical statements for:
  - `show_browser=true` precedence vs per-site `custom_visible_browser`
  - `keyword_batch_enabled` + `scrape_rounds`/`scrape_round_items` statuses (`pending`, `in_progress`, `done`, `failed`) and 4-hour window
  - Dashboard "Keyword Retry Tracking" source `GET /api/scrape/progress` with hourly `warn_due` from `last_hour_warn_at` fallback logic

## 🟢 Session 51 — 2026-03-19

**Deep Markdown mechanics audit and alignment**
- Audited all repo `*.md` files and applied minimal edits only where mechanics were missing/unclear.
- Updated docs to consistently reflect:
  - `custom_visible_browser` behavior and `show_browser=true` precedence.
  - `keyword_batch_enabled` + `scrape_rounds`/`scrape_round_items` status flow (`pending`, `in_progress`, `done`, `failed`).
  - 4-hour retry window, hourly warning bookkeeping (`last_hour_warn_at`), and `warn_due`.
  - Dashboard "Keyword Retry Tracking" source endpoint: `GET /api/scrape/progress`.

## 🟢 Session 42 — 2026-03-19

:**Premium UX/UI: Landing page rewrite**
- Added `frontend/components/landing/LandingPageV2.tsx` with a richer premium layout (neon particle canvas, interactive console preview, improved Features / How-it-works / Developer / Pricing / FAQ sections).
- Switched `frontend/app/page.tsx` to render `LandingPageV2`.
- Verified: `cd frontend && npx vitest run` and `npm run build --prefix frontend`.

## 🟢 Session 43 — 2026-03-19

:**Marketing UX cleanup: remove old landing**
- Deleted `frontend/components/landing/LandingPage.tsx` (unused now) so the marketing site is not based on the previous landing design.
- Re-ran `cd frontend && npx vitest run` and `npm run build` to confirm everything still passes.

## 🟢 Session 44 — 2026-03-19

:**Clean-room marketing rebuild (LandingPageV3 + OG)**
- Deleted `frontend/out/` and regenerated the Next.js static export.
- Added `frontend/components/landing/LandingPageV3.tsx` (shadcn + motion + lucide) and switched `frontend/app/page.tsx` to render it.
- Added missing landing shadcn primitives (`button`, `card`, `accordion`, `badge`, `separator`).
- Added `frontend/app/opengraph-image.tsx` (Satori via `next/og`).
- Verified: `http://localhost:7000/` and `http://localhost:7000/legacy` return `200`.

## 🟢 Session 45 — 2026-03-19

**Full-app shell redesign pass (all routes)**
- Rebuilt the shared top chrome so every route has a clearly new look without changing API wiring:
  - `frontend/components/layout/Header.tsx`
  - `frontend/components/layout/Nav.tsx`
  - `frontend/components/layout/StatusBar.tsx`
- New shell direction: thicker glass layers, stronger typography hierarchy, capsule nav tabs, route-wide control rail styling, and cardized telemetry status tiles.
- Kept all engine controls and route paths intact (`pause`, `resume`, `restart`, `kill`, and all nav route links).
- Verified: `cd frontend && npx vitest run` and `npm run build` both pass (same known static-export/metadata warnings only).

## 🟢 Session 46 — 2026-03-19

**Total visual wipe phase (hard reset baseline)**
- Replaced `frontend/app/globals.css` with a minimal baseline (`tailwind` imports + basic box-model/body reset), removing all prior custom visual system layers (`g-*`, glass, gradients, neon, theme-specific utility classes).
- Flattened visual-heavy components to plain functional markup:
  - `frontend/components/dashboard/StatsGrid.tsx`
  - `frontend/components/listings/ListingsTable.tsx`
  - `frontend/components/keywords/KeywordRow.tsx`
- Kept core behavior and wiring intact (data hooks, actions, filters, exports, delete flows, detail panel opening).
- Verified: `cd frontend && npm run build` passes. Lint has one non-blocking warning in `KeywordRow.tsx` from `dnd-kit` inline transform style usage.

## 🟢 Session 47 — 2026-03-19

**Total visual wipe phase (remaining shared/UI surfaces)**
- Flattened remaining visual-heavy components to plain functional markup:
  - `frontend/components/sites/SiteCard.tsx`
  - `frontend/components/sites/SitesTable.tsx`
  - `frontend/components/listings/ListingCard.tsx`
  - `frontend/components/listings/ListingDetailPanel.tsx`
  - `frontend/components/ai-log/AILogCard.tsx`
  - `frontend/components/ai-log/AILogFeed.tsx`
  - `frontend/components/layout/Header.tsx`
  - `frontend/components/layout/Nav.tsx`
  - `frontend/components/layout/StatusBar.tsx`
- Removed old class-driven visual styling from these files while preserving core interactions (drag reorder, toggles, adapt trigger, engine actions, AI log polling/filtering, listing detail open/close).
- Verified: `cd frontend && npx vitest run && npm run build` passes (same known Next.js warnings only). One non-blocking linter warning remains from required `dnd-kit` inline transform style in `SiteCard.tsx`.

## 🟢 Session 48 — 2026-03-19

**Final `g-*` cleanup sweep (active app surfaces)**
- Replaced remaining active visual-heavy files with plain functional markup and removed `g-*` usage from active routes/components:
  - `frontend/components/dashboard/EngineConsole.tsx`
  - `frontend/components/dashboard/RecentListings.tsx`
  - `frontend/components/dashboard/ActivityLog.tsx`
  - `frontend/components/keywords/KeywordCard.tsx`
  - `frontend/components/keywords/KeywordsTable.tsx`
  - `frontend/components/keywords/ScoringRulesPanel.tsx`
  - `frontend/components/listings/ImageGallery.tsx`
  - `frontend/components/layout/ThemeToggle.tsx`
  - `frontend/app/settings/page.tsx`
- Preserved key behavior (polling, engine controls, retry tracking, drag reorder, scoring CRUD, backup/restore, AI log/config interactions).
- Updated minimal test expectations indirectly through component text compatibility (no test file edits in this pass).
- Verified: `cd frontend && npx vitest run && npm run build` passes (same known Next.js static-export warnings only).

## 🟢 Session 49 — 2026-03-19

**Lint cleanup finalization (ImageGallery)**
- Removed inline-style usage from `frontend/components/listings/ImageGallery.tsx` and converted to class-based styling (`fixed` overlay/panel/image sizing + strip layout), including replacing runtime display toggle with class toggling.
- Resolved final class lint recommendation (`z-[100]` → `z-100`).
- Verified: `cd frontend && npx vitest run && npm run build` passes; `ImageGallery.tsx` lints clean.

## 🟢 Session 50 — 2026-03-19

**100% wipe completion pass**
- Neutralized final leftover old-visual components with plain implementations:
  - `frontend/components/sites/SiteRow.tsx`
  - `frontend/components/listings/ListingRow.tsx`
  - `frontend/components/layout/AmbientBackground.tsx`
- Confirmed no `g-*` class tokens remain in active frontend codebase outside non-app hidden tooling (`frontend/.cursor/...`).
- Final verification passed: `cd frontend && npx vitest run && npm run build`.

## 🟢 Session 35 — 2026-03-19

**Option A baseline: pinned retry tracking contract tests**
- Added `frontend/__tests__/EngineConsole.test.tsx` to validate the pinned “Keyword Retry Tracking” panel contract via mocked `/api/sites` + `/api/scrape/progress`.
- Updated `frontend/vitest.config.ts` so Vitest runs only `frontend/__tests__` (avoids picking up Cursor harness `.cursor/tests`).
- Verified: `cd frontend && npx vitest run` + `npm run build --prefix frontend`.

## 🟢 Session 36 — 2026-03-19

**Option A baseline: scrape-progress endpoint contract tests**
- Added `backend_tests/test_scrape_progress.py` to seed a temp SQLite DB and verify `GET /api/scrape/progress` output (`keyword_batch_enabled`, `active_round`, `pending_items`, and `warn_due` logic).
- Verified: `python -m unittest discover -s backend_tests -p 'test_*.py'`.
- Re-verified: `cd frontend && npx vitest run` remained green.

## 🟢 Session 37 — 2026-03-19

**Option A baseline: UI limiting behavior**
- Updated `frontend/__tests__/EngineConsole.test.tsx` with a new case that mocks 10 pending items and asserts:
  - only the first 8 keyword terms render
  - the UI shows `Showing first 8 of 10`.
- Verified: `cd frontend && npx vitest run`.

## 🟢 Session 38 — 2026-03-19

**Option A baseline: API contract tests for dashboard**
- Extended `backend_tests/test_scrape_progress.py` with:
  - `/api/stats` response shape assertions
  - `/api/listings/countdown-sync` values + ISO timestamp fields based on seeded `Listing` rows.
- Verified: `python -m unittest discover -s backend_tests -p 'test_*.py'` and frontend `vitest` remained green.

## 🟢 Session 39 — 2026-03-19

**Option A baseline: `/api/listings/refresh-status` contract test**
- Extended `backend_tests/test_scrape_progress.py` with assertions for:
  - `GET /api/listings/refresh-status` returns `last_price_update` ISO string and `listing_count`.
- Verified: `python -m unittest discover -s backend_tests -p 'test_*.py'` + frontend `vitest`.

## 🟢 Session 40 — 2026-03-19

**Option A baseline: `/api/sites` numeric flags contract**
- Added backend contract assertions in `backend_tests/test_scrape_progress.py` verifying `enabled`, `custom_visible_browser`, `requires_login`, `login_enabled` are returned as integer `0/1` (not JSON booleans).
- Fixed `models.py` `TargetSite.to_dict()` to emit `0/1` integers for those flag fields.
- Verified: backend unittests + frontend `vitest` remain green.

## 🟢 Session 41 — 2026-03-19

**Option A baseline: `/api/config` contract tests (Settings page)**
- Added backend contract assertions in `backend_tests/test_scrape_progress.py` for:
  - `GET /api/config` returning a flat `{key: value}` object with string values
  - `POST /api/config` upserting values as strings and returning `{status, keys}`
- Verified: backend unittests and frontend `vitest` remained green.

## 🟢 Session 34 — 2026-03-18

**MD Update Playbook (repeatable doc job)**
- Added `docs/MD_UPDATE_PLAYBOOK.md` with:
  - deep semantic audit workflow for all `*.md` files
  - exact mechanics checklist for per-site visibility override and keyword batching + retry tracking
  - reusable prompt (TEXT + JSON) with approval gate + verification checklist

## 🟢 Session 33 — 2026-03-18

**Per-site Visible/Custom Browser Override**
- Added `TargetSite.custom_visible_browser` (0/1).
- Global `show_browser=true` forces visible mode for all sites.
- When `show_browser=false`, a site becomes visible only if `custom_visible_browser=1`.

**Keyword Batching + Persistent Progress Tracking**
- Added config `keyword_batch_enabled` (default `false`).
- Introduced `scrape_rounds` + `scrape_round_items` to track per-(site, keyword) status across cycles.
- Engine batches at most `max_tabs_per_site` keyword attempts per site per cycle/round.
- Failures mark items `pending` and keep retrying until the round’s 4-hour retry window expires.

**Hourly Retry Warnings + Live Dashboard**
- Hourly bookkeeping updates `scrape_round_items.last_hour_warn_at`.
- Added API: `GET /api/scrape/progress` for the dashboard to render pinned “Keyword Retry Tracking”.

## 🟢 Session 32 — 2026-03-18

**Redis Cache Layer (Priority 1 — DONE)**
- Added `_init_redis()`, `_redis_set_stats()`, `_redis_publish()`, `_redis_cache_set/get()` helpers in worker.py
- Graceful write-through: app runs unchanged when `REDIS_URL` not set
- Stats synced to Redis on cycle complete, engine pause/resume, engine running
- `new_listing` pub/sub event published on every alert
- New endpoint: `GET /api/redis/status` — connectivity check + cached stats
- Redis channel: `ghostnode:events`, stats key: `ghostnode:stats`

**Docker Compose (Priority 2 — DONE)**
- `Dockerfile` — Python 3.11-slim + Playwright Chromium + all deps, serves on port 8000
- `docker-compose.yml` — 3 services: ghostnode, postgres:16-alpine, redis:7-alpine
- Auto health checks on Postgres + Redis before ghostnode starts
- One-command: `docker compose up --build`
- Volumes: `postgres_data`, `redis_data`, `ghostnode_sessions`

**Lot Description Extraction / N18 (Priority 3 — DONE)**
- `JS_DETAIL_TEXT` — 4-layer JS extractor: JSON-LD → OG meta → known selectors → largest `<p>` block
- `description (Text)` column added to `Listing` ORM + migration in `_migrate_schema()` for SQLite + PostgreSQL
- `_fetch_listing_images_batch()` now evaluates `JS_DETAIL_TEXT` on every detail page visit (zero extra network cost — same page already open for images)
- `_build_ai_prompt()` and `_ai_analyze()` updated to accept optional `description` param
- Re-analysis: listings that passed AI on title alone (`ai_match=1`) are re-evaluated with description after detail fetch
- `to_dict()` on `Listing` now includes `description` field

## 🟡 Pending Features — Priority Queue

### Priority 1 — Frontend Polish
Badge system refinement, animation tuning, show `description` in ListingDetailPanel.

### Priority 2 — Docker Testing
Test the docker-compose stack end-to-end. Verify PostgreSQL migration, Redis pub/sub, Playwright in container.

### Priority 3 — vLLM Production Inference
Replace Groq rate-limited free tier with self-hosted vLLM on rented GPU for production scale.

### Priority 4 — Frontend Visual Polish
Session 27: Full premium redesign — ambient animated gradient background (3 floating orbs + dot grid), glassmorphism shell (backdrop-blur + saturate), gradient glow cards (`g-card-glow` with animated border), Framer Motion staggered card/row/page animations, gradient text headings, badge system (green/amber/red/blue/purple/neutral), gradient underline active nav with glow, status pill with live pulse dot + aura shadow. New CSS prefix: `g-*` (e.g. `bg-g-base`, `text-g-green`). All 21 tests + build passing.

Session 28: Added a dedicated marketing landing page (`frontend/components/landing/LandingPage.tsx`) and switched `frontend/app/page.tsx` from `redirect('/dashboard')` to render the landing component. Root route now serves landing content in fresh static export.

---

## 🔵 Known Improvements (Lower Priority)

| Item | Description |
|---|---|
| Edit Site modal | No way to edit login fields on existing sites after creation |
| Cloudflare Turnstile | Current solver handles reCAPTCHA/hCaptcha only — Turnstile needs `cf-clearance-scraper` |
| Mobile dashboard | Desktop-only — needs responsive grid breakpoints |
| Sort by USD price | Listings table sorts by raw price — should optionally sort by `price_usd` |
| N18: Site Discovery | Skipped — AI-powered discovery of new auction sites from web |

*Last updated: 2026-03-13 — Session 28*