39246-vm/docs/PROGRESS.md

👻 Ghost Node — Progress Log

Read this alongside CLAUDE.md at the start of every new session. Full session history (all sessions, including Q&A decisions) → ARCHIVE.md — do NOT load at session start.

⚠️ MD Update Rule — DYNAMIC, NOT STATIC

Only update the MD file whose content was actually touched this session. Not all files — only the relevant ones.

• PROGRESS.md → update when any code file changed. Add a brief session entry here, move it to ARCHIVE.md when it's no longer active context.
• CLAUDE.md → update only if API endpoints, DB schema, config keys, or architecture changed.
• MEMORY.md → update only if a new reusable gotcha, pattern, or architecture decision was discovered.
• ARCHIVE.md → move old PROGRESS.md session entries here. Also log Q&A sessions that produced lasting decisions.
• If nothing changed in a session (pure Q&A, no files edited, no decisions made) → update nothing.

---

Current State

• Version: v2.7
• Status: ✅ Fully operational — 21/21 tests passing
• Last session: 52 (2026-03-19) — wording normalization pass for mechanics docs (canonical phrasing alignment).
• Previous session: 44 (2026-03-19) — clean-room marketing landing rebuild (new LandingPageV3 + OG image).
• Previous session: 39 (2026-03-19) — added /api/listings/refresh-status contract test (Option A baseline).
• Previous session: 38 (2026-03-19) — added backend contract tests for /api/stats and /api/listings/countdown-sync.
• Previous session: 34 (2026-03-18) — added docs/MD_UPDATE_PLAYBOOK.md playbook for deep semantic Markdown audits + minimal targeted MD edits workflow.
• Previous session: 33 (2026-03-18) — per-site visible browser override + keyword batching progress tracking + hourly retry warnings. See details below.
• Previous session: 32 (2026-03-18) — Redis cache layer, Docker Compose, lot description extraction (N18). See details below.
• Previous session: 31 (2026-03-13) — Fixed Framer Motion SSR opacity:0 bug across all components (LandingPage hero, dashboard, StatsGrid, RecentListings, EngineConsole, ActivityLog, ListingRow, ListingsTable, ListingDetailPanel, 4 page routes). All above-the-fold elements now render visible in static HTML (opacity:1, only transforms animate). Below-fold scroll-triggered sections intentionally keep opacity:0 for scroll animations. 21/21 tests passing.
• Previous session: 30 (2026-03-13) — Full marketing landing page built from AI_WEB_BRIEF.md (Hero, Platform Strip, 6-feature grid, How It Works 4-step flow, Who It's For personas, FAQ accordion, Footer CTA + site footer). Light/dark theme toggle added (ThemeToggle.tsx, localStorage-persistent, vivid lavender-ivory light palette via data-theme attribute). globals.css extended with complete [data-theme="light"] overrides for all g-* classes. 21/21 tests passing.
• Previous session: 29 (2026-03-13) — Master dashboard "Mission Control" rebuilt: RecentListings (polls /api/listings every 10s), EngineConsole (status orb + controls + site health, polls /api/sites every 25s), enhanced ActivityLog (tracks engine state transitions), StatsGrid updated (keywords prop replaces engine card), utils.ts extended (formatUptime, formatMins, formatPrice, timeAgo). 21/21 tests passing.
• worker.py: 5,000+ lines (definitive — ahead of all exports)
• frontend/: Next.js 16 + React 19 + TypeScript + Tailwind v4 — static build in frontend/out/ (1.8MB). node_modules/ is 553MB build tooling — safe to delete, run npm install to restore.
• models.py: ~420 lines
• Serving: http://localhost:7000 (React UI) | http://localhost:7000/legacy (HTML fallback)

---

🟢 Session 52 — 2026-03-19

Canonical wording normalization (Markdown only)

• Ran a minimal wording-only pass on already-touched docs so mechanics phrasing is consistent across files.
• Standardized the same canonical statements for:
  • show_browser=true precedence vs per-site custom_visible_browser
  • keyword_batch_enabled + scrape_rounds/scrape_round_items statuses (pending, in_progress, done, failed) and 4-hour window
  • Dashboard "Keyword Retry Tracking" source GET /api/scrape/progress with hourly warn_due from last_hour_warn_at fallback logic

🟢 Session 51 — 2026-03-19

Deep Markdown mechanics audit and alignment

• Audited all repo *.md files and applied minimal edits only where mechanics were missing/unclear.
• Updated docs to consistently reflect:
  • custom_visible_browser behavior and show_browser=true precedence.
  • keyword_batch_enabled + scrape_rounds/scrape_round_items status flow (pending, in_progress, done, failed).
  • 4-hour retry window, hourly warning bookkeeping (last_hour_warn_at), and warn_due.
  • Dashboard "Keyword Retry Tracking" source endpoint: GET /api/scrape/progress.

🟢 Session 42 — 2026-03-19

:Premium UX/UI: Landing page rewrite

• Added frontend/components/landing/LandingPageV2.tsx with a richer premium layout (neon particle canvas, interactive console preview, improved Features / How-it-works / Developer / Pricing / FAQ sections).
• Switched frontend/app/page.tsx to render LandingPageV2.
• Verified: cd frontend && npx vitest run and npm run build --prefix frontend.

🟢 Session 43 — 2026-03-19

:Marketing UX cleanup: remove old landing

• Deleted frontend/components/landing/LandingPage.tsx (unused now) so the marketing site is not based on the previous landing design.
• Re-ran cd frontend && npx vitest run and npm run build to confirm everything still passes.

🟢 Session 44 — 2026-03-19

:Clean-room marketing rebuild (LandingPageV3 + OG)

• Deleted frontend/out/ and regenerated the Next.js static export.
• Added frontend/components/landing/LandingPageV3.tsx (shadcn + motion + lucide) and switched frontend/app/page.tsx to render it.
• Added missing landing shadcn primitives (button, card, accordion, badge, separator).
• Added frontend/app/opengraph-image.tsx (Satori via next/og).
• Verified: http://localhost:7000/ and http://localhost:7000/legacy return 200.

🟢 Session 45 — 2026-03-19

Full-app shell redesign pass (all routes)

• Rebuilt the shared top chrome so every route has a clearly new look without changing API wiring:
  • frontend/components/layout/Header.tsx
  • frontend/components/layout/Nav.tsx
  • frontend/components/layout/StatusBar.tsx
• New shell direction: thicker glass layers, stronger typography hierarchy, capsule nav tabs, route-wide control rail styling, and cardized telemetry status tiles.
• Kept all engine controls and route paths intact (pause, resume, restart, kill, and all nav route links).
• Verified: cd frontend && npx vitest run and npm run build both pass (same known static-export/metadata warnings only).

🟢 Session 46 — 2026-03-19

Total visual wipe phase (hard reset baseline)

• Replaced frontend/app/globals.css with a minimal baseline (tailwind imports + basic box-model/body reset), removing all prior custom visual system layers (g-*, glass, gradients, neon, theme-specific utility classes).
• Flattened visual-heavy components to plain functional markup:
  • frontend/components/dashboard/StatsGrid.tsx
  • frontend/components/listings/ListingsTable.tsx
  • frontend/components/keywords/KeywordRow.tsx
• Kept core behavior and wiring intact (data hooks, actions, filters, exports, delete flows, detail panel opening).
• Verified: cd frontend && npm run build passes. Lint has one non-blocking warning in KeywordRow.tsx from dnd-kit inline transform style usage.

🟢 Session 47 — 2026-03-19

Total visual wipe phase (remaining shared/UI surfaces)

• Flattened remaining visual-heavy components to plain functional markup:
  • frontend/components/sites/SiteCard.tsx
  • frontend/components/sites/SitesTable.tsx
  • frontend/components/listings/ListingCard.tsx
  • frontend/components/listings/ListingDetailPanel.tsx
  • frontend/components/ai-log/AILogCard.tsx
  • frontend/components/ai-log/AILogFeed.tsx
  • frontend/components/layout/Header.tsx
  • frontend/components/layout/Nav.tsx
  • frontend/components/layout/StatusBar.tsx
• Removed old class-driven visual styling from these files while preserving core interactions (drag reorder, toggles, adapt trigger, engine actions, AI log polling/filtering, listing detail open/close).
• Verified: cd frontend && npx vitest run && npm run build passes (same known Next.js warnings only). One non-blocking linter warning remains from required dnd-kit inline transform style in SiteCard.tsx.

🟢 Session 48 — 2026-03-19

Final g-* cleanup sweep (active app surfaces)

• Replaced remaining active visual-heavy files with plain functional markup and removed g-* usage from active routes/components:
  • frontend/components/dashboard/EngineConsole.tsx
  • frontend/components/dashboard/RecentListings.tsx
  • frontend/components/dashboard/ActivityLog.tsx
  • frontend/components/keywords/KeywordCard.tsx
  • frontend/components/keywords/KeywordsTable.tsx
  • frontend/components/keywords/ScoringRulesPanel.tsx
  • frontend/components/listings/ImageGallery.tsx
  • frontend/components/layout/ThemeToggle.tsx
  • frontend/app/settings/page.tsx
• Preserved key behavior (polling, engine controls, retry tracking, drag reorder, scoring CRUD, backup/restore, AI log/config interactions).
• Updated minimal test expectations indirectly through component text compatibility (no test file edits in this pass).
• Verified: cd frontend && npx vitest run && npm run build passes (same known Next.js static-export warnings only).

🟢 Session 49 — 2026-03-19

Lint cleanup finalization (ImageGallery)

• Removed inline-style usage from frontend/components/listings/ImageGallery.tsx and converted to class-based styling (fixed overlay/panel/image sizing + strip layout), including replacing runtime display toggle with class toggling.
• Resolved final class lint recommendation (z-[100] → z-100).
• Verified: cd frontend && npx vitest run && npm run build passes; ImageGallery.tsx lints clean.

🟢 Session 50 — 2026-03-19

100% wipe completion pass

• Neutralized final leftover old-visual components with plain implementations:
  • frontend/components/sites/SiteRow.tsx
  • frontend/components/listings/ListingRow.tsx
  • frontend/components/layout/AmbientBackground.tsx
• Confirmed no g-* class tokens remain in active frontend codebase outside non-app hidden tooling (frontend/.cursor/...).
• Final verification passed: cd frontend && npx vitest run && npm run build.

🟢 Session 35 — 2026-03-19

Option A baseline: pinned retry tracking contract tests

• Added frontend/__tests__/EngineConsole.test.tsx to validate the pinned “Keyword Retry Tracking” panel contract via mocked /api/sites + /api/scrape/progress.
• Updated frontend/vitest.config.ts so Vitest runs only frontend/__tests__ (avoids picking up Cursor harness .cursor/tests).
• Verified: cd frontend && npx vitest run + npm run build --prefix frontend.

🟢 Session 36 — 2026-03-19

Option A baseline: scrape-progress endpoint contract tests

• Added backend_tests/test_scrape_progress.py to seed a temp SQLite DB and verify GET /api/scrape/progress output (keyword_batch_enabled, active_round, pending_items, and warn_due logic).
• Verified: python -m unittest discover -s backend_tests -p 'test_*.py'.
• Re-verified: cd frontend && npx vitest run remained green.

🟢 Session 37 — 2026-03-19

Option A baseline: UI limiting behavior

• Updated frontend/__tests__/EngineConsole.test.tsx with a new case that mocks 10 pending items and asserts:
  • only the first 8 keyword terms render
  • the UI shows Showing first 8 of 10.
• Verified: cd frontend && npx vitest run.

🟢 Session 38 — 2026-03-19

Option A baseline: API contract tests for dashboard

• Extended backend_tests/test_scrape_progress.py with:
  • /api/stats response shape assertions
  • /api/listings/countdown-sync values + ISO timestamp fields based on seeded Listing rows.
• Verified: python -m unittest discover -s backend_tests -p 'test_*.py' and frontend vitest remained green.

🟢 Session 39 — 2026-03-19

Option A baseline: /api/listings/refresh-status contract test

• Extended backend_tests/test_scrape_progress.py with assertions for:
  • GET /api/listings/refresh-status returns last_price_update ISO string and listing_count.
• Verified: python -m unittest discover -s backend_tests -p 'test_*.py' + frontend vitest.

🟢 Session 40 — 2026-03-19

Option A baseline: /api/sites numeric flags contract

• Added backend contract assertions in backend_tests/test_scrape_progress.py verifying enabled, custom_visible_browser, requires_login, login_enabled are returned as integer 0/1 (not JSON booleans).
• Fixed models.py TargetSite.to_dict() to emit 0/1 integers for those flag fields.
• Verified: backend unittests + frontend vitest remain green.

🟢 Session 41 — 2026-03-19

Option A baseline: /api/config contract tests (Settings page)

• Added backend contract assertions in backend_tests/test_scrape_progress.py for:
  • GET /api/config returning a flat {key: value} object with string values
  • POST /api/config upserting values as strings and returning {status, keys}
• Verified: backend unittests and frontend vitest remained green.

🟢 Session 34 — 2026-03-18

MD Update Playbook (repeatable doc job)

• Added docs/MD_UPDATE_PLAYBOOK.md with:
  • deep semantic audit workflow for all *.md files
  • exact mechanics checklist for per-site visibility override and keyword batching + retry tracking
  • reusable prompt (TEXT + JSON) with approval gate + verification checklist

🟢 Session 33 — 2026-03-18

Per-site Visible/Custom Browser Override

• Added TargetSite.custom_visible_browser (0/1).
• Global show_browser=true forces visible mode for all sites.
• When show_browser=false, a site becomes visible only if custom_visible_browser=1.

Keyword Batching + Persistent Progress Tracking

• Added config keyword_batch_enabled (default false).
• Introduced scrape_rounds + scrape_round_items to track per-(site, keyword) status across cycles.
• Engine batches at most max_tabs_per_site keyword attempts per site per cycle/round.
• Failures mark items pending and keep retrying until the round’s 4-hour retry window expires.

Hourly Retry Warnings + Live Dashboard

• Hourly bookkeeping updates scrape_round_items.last_hour_warn_at.
• Added API: GET /api/scrape/progress for the dashboard to render pinned “Keyword Retry Tracking”.

🟢 Session 32 — 2026-03-18

Redis Cache Layer (Priority 1 — DONE)

• Added _init_redis(), _redis_set_stats(), _redis_publish(), _redis_cache_set/get() helpers in worker.py
• Graceful write-through: app runs unchanged when REDIS_URL not set
• Stats synced to Redis on cycle complete, engine pause/resume, engine running
• new_listing pub/sub event published on every alert
• New endpoint: GET /api/redis/status — connectivity check + cached stats
• Redis channel: ghostnode:events, stats key: ghostnode:stats

Docker Compose (Priority 2 — DONE)

• Dockerfile — Python 3.11-slim + Playwright Chromium + all deps, serves on port 8000
• docker-compose.yml — 3 services: ghostnode, postgres:16-alpine, redis:7-alpine
• Auto health checks on Postgres + Redis before ghostnode starts
• One-command: docker compose up --build
• Volumes: postgres_data, redis_data, ghostnode_sessions

Lot Description Extraction / N18 (Priority 3 — DONE)

• JS_DETAIL_TEXT — 4-layer JS extractor: JSON-LD → OG meta → known selectors → largest <p> block
• description (Text) column added to Listing ORM + migration in _migrate_schema() for SQLite + PostgreSQL
• _fetch_listing_images_batch() now evaluates JS_DETAIL_TEXT on every detail page visit (zero extra network cost — same page already open for images)
• _build_ai_prompt() and _ai_analyze() updated to accept optional description param
• Re-analysis: listings that passed AI on title alone (ai_match=1) are re-evaluated with description after detail fetch
• to_dict() on Listing now includes description field

🟡 Pending Features — Priority Queue

Priority 1 — Frontend Polish

Badge system refinement, animation tuning, show description in ListingDetailPanel.

Priority 2 — Docker Testing

Test the docker-compose stack end-to-end. Verify PostgreSQL migration, Redis pub/sub, Playwright in container.

Priority 3 — vLLM Production Inference

Replace Groq rate-limited free tier with self-hosted vLLM on rented GPU for production scale.

Priority 4 — Frontend Visual Polish

Session 27: Full premium redesign — ambient animated gradient background (3 floating orbs + dot grid), glassmorphism shell (backdrop-blur + saturate), gradient glow cards (g-card-glow with animated border), Framer Motion staggered card/row/page animations, gradient text headings, badge system (green/amber/red/blue/purple/neutral), gradient underline active nav with glow, status pill with live pulse dot + aura shadow. New CSS prefix: g-* (e.g. bg-g-base, text-g-green). All 21 tests + build passing.

Session 28: Added a dedicated marketing landing page (frontend/components/landing/LandingPage.tsx) and switched frontend/app/page.tsx from redirect('/dashboard') to render the landing component. Root route now serves landing content in fresh static export.

---

🔵 Known Improvements (Lower Priority)

Item	Description
Edit Site modal	No way to edit login fields on existing sites after creation
Cloudflare Turnstile	Current solver handles reCAPTCHA/hCaptcha only — Turnstile needs cf-clearance-scraper
Mobile dashboard	Desktop-only — needs responsive grid breakpoints
Sort by USD price	Listings table sorts by raw price — should optionally sort by price_usd
N18: Site Discovery	Skipped — AI-powered discovery of new auction sites from web

Last updated: 2026-03-13 — Session 28