282 lines
18 KiB
Markdown
282 lines
18 KiB
Markdown
# π» Ghost Node β Progress Log
|
||
|
||
> **Read this alongside CLAUDE.md at the start of every new session.**
|
||
> Full session history (all sessions, including Q&A decisions) β **ARCHIVE.md** β do NOT load at session start.
|
||
>
|
||
> ### β οΈ MD Update Rule β DYNAMIC, NOT STATIC
|
||
> Only update the MD file whose content was actually touched this session. Not all files β only the relevant ones.
|
||
> - **PROGRESS.md** β update when any code file changed. Add a brief session entry here, move it to ARCHIVE.md when it's no longer active context.
|
||
> - **CLAUDE.md** β update only if API endpoints, DB schema, config keys, or architecture changed.
|
||
> - **MEMORY.md** β update only if a new reusable gotcha, pattern, or architecture decision was discovered.
|
||
> - **ARCHIVE.md** β move old PROGRESS.md session entries here. Also log Q&A sessions that produced lasting decisions.
|
||
> - If nothing changed in a session (pure Q&A, no files edited, no decisions made) β update nothing.
|
||
|
||
---
|
||
|
||
## Current State
|
||
|
||
- **Version:** v2.7
|
||
- **Status:** β
Fully operational β 21/21 tests passing
|
||
- **Last session:** 52 (2026-03-19) β wording normalization pass for mechanics docs (canonical phrasing alignment).
|
||
- **Previous session:** 44 (2026-03-19) β clean-room marketing landing rebuild (new `LandingPageV3` + OG image).
|
||
- **Previous session:** 39 (2026-03-19) β added `/api/listings/refresh-status` contract test (Option A baseline).
|
||
- **Previous session:** 38 (2026-03-19) β added backend contract tests for `/api/stats` and `/api/listings/countdown-sync`.
|
||
- **Previous session:** 34 (2026-03-18) β added `docs/MD_UPDATE_PLAYBOOK.md` playbook for deep semantic Markdown audits + minimal targeted MD edits workflow.
|
||
- **Previous session:** 33 (2026-03-18) β per-site visible browser override + keyword batching progress tracking + hourly retry warnings. See details below.
|
||
- **Previous session:** 32 (2026-03-18) β Redis cache layer, Docker Compose, lot description extraction (N18). See details below.
|
||
- **Previous session:** 31 (2026-03-13) β Fixed Framer Motion SSR opacity:0 bug across all components (LandingPage hero, dashboard, StatsGrid, RecentListings, EngineConsole, ActivityLog, ListingRow, ListingsTable, ListingDetailPanel, 4 page routes). All above-the-fold elements now render visible in static HTML (opacity:1, only transforms animate). Below-fold scroll-triggered sections intentionally keep opacity:0 for scroll animations. 21/21 tests passing.
|
||
- **Previous session:** 30 (2026-03-13) β Full marketing landing page built from AI_WEB_BRIEF.md (Hero, Platform Strip, 6-feature grid, How It Works 4-step flow, Who It's For personas, FAQ accordion, Footer CTA + site footer). Light/dark theme toggle added (ThemeToggle.tsx, localStorage-persistent, vivid lavender-ivory light palette via data-theme attribute). globals.css extended with complete [data-theme="light"] overrides for all g-* classes. 21/21 tests passing.
|
||
- **Previous session:** 29 (2026-03-13) β Master dashboard "Mission Control" rebuilt: RecentListings (polls /api/listings every 10s), EngineConsole (status orb + controls + site health, polls /api/sites every 25s), enhanced ActivityLog (tracks engine state transitions), StatsGrid updated (keywords prop replaces engine card), utils.ts extended (formatUptime, formatMins, formatPrice, timeAgo). 21/21 tests passing.
|
||
- **worker.py:** 5,000+ lines (definitive β ahead of all exports)
|
||
- **frontend/:** Next.js 16 + React 19 + TypeScript + Tailwind v4 β static build in `frontend/out/` (1.8MB). `node_modules/` is 553MB build tooling β safe to delete, run `npm install` to restore.
|
||
- **models.py:** ~420 lines
|
||
- **Serving:** `http://localhost:7000` (React UI) | `http://localhost:7000/legacy` (HTML fallback)
|
||
|
||
---
|
||
|
||
## π’ Session 52 β 2026-03-19
|
||
|
||
**Canonical wording normalization (Markdown only)**
|
||
- Ran a minimal wording-only pass on already-touched docs so mechanics phrasing is consistent across files.
|
||
- Standardized the same canonical statements for:
|
||
- `show_browser=true` precedence vs per-site `custom_visible_browser`
|
||
- `keyword_batch_enabled` + `scrape_rounds`/`scrape_round_items` statuses (`pending`, `in_progress`, `done`, `failed`) and 4-hour window
|
||
- Dashboard "Keyword Retry Tracking" source `GET /api/scrape/progress` with hourly `warn_due` from `last_hour_warn_at` fallback logic
|
||
|
||
## π’ Session 51 β 2026-03-19
|
||
|
||
**Deep Markdown mechanics audit and alignment**
|
||
- Audited all repo `*.md` files and applied minimal edits only where mechanics were missing/unclear.
|
||
- Updated docs to consistently reflect:
|
||
- `custom_visible_browser` behavior and `show_browser=true` precedence.
|
||
- `keyword_batch_enabled` + `scrape_rounds`/`scrape_round_items` status flow (`pending`, `in_progress`, `done`, `failed`).
|
||
- 4-hour retry window, hourly warning bookkeeping (`last_hour_warn_at`), and `warn_due`.
|
||
- Dashboard "Keyword Retry Tracking" source endpoint: `GET /api/scrape/progress`.
|
||
|
||
## π’ Session 42 β 2026-03-19
|
||
|
||
:**Premium UX/UI: Landing page rewrite**
|
||
- Added `frontend/components/landing/LandingPageV2.tsx` with a richer premium layout (neon particle canvas, interactive console preview, improved Features / How-it-works / Developer / Pricing / FAQ sections).
|
||
- Switched `frontend/app/page.tsx` to render `LandingPageV2`.
|
||
- Verified: `cd frontend && npx vitest run` and `npm run build --prefix frontend`.
|
||
|
||
## π’ Session 43 β 2026-03-19
|
||
|
||
:**Marketing UX cleanup: remove old landing**
|
||
- Deleted `frontend/components/landing/LandingPage.tsx` (unused now) so the marketing site is not based on the previous landing design.
|
||
- Re-ran `cd frontend && npx vitest run` and `npm run build` to confirm everything still passes.
|
||
|
||
## π’ Session 44 β 2026-03-19
|
||
|
||
:**Clean-room marketing rebuild (LandingPageV3 + OG)**
|
||
- Deleted `frontend/out/` and regenerated the Next.js static export.
|
||
- Added `frontend/components/landing/LandingPageV3.tsx` (shadcn + motion + lucide) and switched `frontend/app/page.tsx` to render it.
|
||
- Added missing landing shadcn primitives (`button`, `card`, `accordion`, `badge`, `separator`).
|
||
- Added `frontend/app/opengraph-image.tsx` (Satori via `next/og`).
|
||
- Verified: `http://localhost:7000/` and `http://localhost:7000/legacy` return `200`.
|
||
|
||
## π’ Session 45 β 2026-03-19
|
||
|
||
**Full-app shell redesign pass (all routes)**
|
||
- Rebuilt the shared top chrome so every route has a clearly new look without changing API wiring:
|
||
- `frontend/components/layout/Header.tsx`
|
||
- `frontend/components/layout/Nav.tsx`
|
||
- `frontend/components/layout/StatusBar.tsx`
|
||
- New shell direction: thicker glass layers, stronger typography hierarchy, capsule nav tabs, route-wide control rail styling, and cardized telemetry status tiles.
|
||
- Kept all engine controls and route paths intact (`pause`, `resume`, `restart`, `kill`, and all nav route links).
|
||
- Verified: `cd frontend && npx vitest run` and `npm run build` both pass (same known static-export/metadata warnings only).
|
||
|
||
## π’ Session 46 β 2026-03-19
|
||
|
||
**Total visual wipe phase (hard reset baseline)**
|
||
- Replaced `frontend/app/globals.css` with a minimal baseline (`tailwind` imports + basic box-model/body reset), removing all prior custom visual system layers (`g-*`, glass, gradients, neon, theme-specific utility classes).
|
||
- Flattened visual-heavy components to plain functional markup:
|
||
- `frontend/components/dashboard/StatsGrid.tsx`
|
||
- `frontend/components/listings/ListingsTable.tsx`
|
||
- `frontend/components/keywords/KeywordRow.tsx`
|
||
- Kept core behavior and wiring intact (data hooks, actions, filters, exports, delete flows, detail panel opening).
|
||
- Verified: `cd frontend && npm run build` passes. Lint has one non-blocking warning in `KeywordRow.tsx` from `dnd-kit` inline transform style usage.
|
||
|
||
## π’ Session 47 β 2026-03-19
|
||
|
||
**Total visual wipe phase (remaining shared/UI surfaces)**
|
||
- Flattened remaining visual-heavy components to plain functional markup:
|
||
- `frontend/components/sites/SiteCard.tsx`
|
||
- `frontend/components/sites/SitesTable.tsx`
|
||
- `frontend/components/listings/ListingCard.tsx`
|
||
- `frontend/components/listings/ListingDetailPanel.tsx`
|
||
- `frontend/components/ai-log/AILogCard.tsx`
|
||
- `frontend/components/ai-log/AILogFeed.tsx`
|
||
- `frontend/components/layout/Header.tsx`
|
||
- `frontend/components/layout/Nav.tsx`
|
||
- `frontend/components/layout/StatusBar.tsx`
|
||
- Removed old class-driven visual styling from these files while preserving core interactions (drag reorder, toggles, adapt trigger, engine actions, AI log polling/filtering, listing detail open/close).
|
||
- Verified: `cd frontend && npx vitest run && npm run build` passes (same known Next.js warnings only). One non-blocking linter warning remains from required `dnd-kit` inline transform style in `SiteCard.tsx`.
|
||
|
||
## π’ Session 48 β 2026-03-19
|
||
|
||
**Final `g-*` cleanup sweep (active app surfaces)**
|
||
- Replaced remaining active visual-heavy files with plain functional markup and removed `g-*` usage from active routes/components:
|
||
- `frontend/components/dashboard/EngineConsole.tsx`
|
||
- `frontend/components/dashboard/RecentListings.tsx`
|
||
- `frontend/components/dashboard/ActivityLog.tsx`
|
||
- `frontend/components/keywords/KeywordCard.tsx`
|
||
- `frontend/components/keywords/KeywordsTable.tsx`
|
||
- `frontend/components/keywords/ScoringRulesPanel.tsx`
|
||
- `frontend/components/listings/ImageGallery.tsx`
|
||
- `frontend/components/layout/ThemeToggle.tsx`
|
||
- `frontend/app/settings/page.tsx`
|
||
- Preserved key behavior (polling, engine controls, retry tracking, drag reorder, scoring CRUD, backup/restore, AI log/config interactions).
|
||
- Updated minimal test expectations indirectly through component text compatibility (no test file edits in this pass).
|
||
- Verified: `cd frontend && npx vitest run && npm run build` passes (same known Next.js static-export warnings only).
|
||
|
||
## π’ Session 49 β 2026-03-19
|
||
|
||
**Lint cleanup finalization (ImageGallery)**
|
||
- Removed inline-style usage from `frontend/components/listings/ImageGallery.tsx` and converted to class-based styling (`fixed` overlay/panel/image sizing + strip layout), including replacing runtime display toggle with class toggling.
|
||
- Resolved final class lint recommendation (`z-[100]` β `z-100`).
|
||
- Verified: `cd frontend && npx vitest run && npm run build` passes; `ImageGallery.tsx` lints clean.
|
||
|
||
## π’ Session 50 β 2026-03-19
|
||
|
||
**100% wipe completion pass**
|
||
- Neutralized final leftover old-visual components with plain implementations:
|
||
- `frontend/components/sites/SiteRow.tsx`
|
||
- `frontend/components/listings/ListingRow.tsx`
|
||
- `frontend/components/layout/AmbientBackground.tsx`
|
||
- Confirmed no `g-*` class tokens remain in active frontend codebase outside non-app hidden tooling (`frontend/.cursor/...`).
|
||
- Final verification passed: `cd frontend && npx vitest run && npm run build`.
|
||
|
||
## π’ Session 35 β 2026-03-19
|
||
|
||
**Option A baseline: pinned retry tracking contract tests**
|
||
- Added `frontend/__tests__/EngineConsole.test.tsx` to validate the pinned βKeyword Retry Trackingβ panel contract via mocked `/api/sites` + `/api/scrape/progress`.
|
||
- Updated `frontend/vitest.config.ts` so Vitest runs only `frontend/__tests__` (avoids picking up Cursor harness `.cursor/tests`).
|
||
- Verified: `cd frontend && npx vitest run` + `npm run build --prefix frontend`.
|
||
|
||
## π’ Session 36 β 2026-03-19
|
||
|
||
**Option A baseline: scrape-progress endpoint contract tests**
|
||
- Added `backend_tests/test_scrape_progress.py` to seed a temp SQLite DB and verify `GET /api/scrape/progress` output (`keyword_batch_enabled`, `active_round`, `pending_items`, and `warn_due` logic).
|
||
- Verified: `python -m unittest discover -s backend_tests -p 'test_*.py'`.
|
||
- Re-verified: `cd frontend && npx vitest run` remained green.
|
||
|
||
## π’ Session 37 β 2026-03-19
|
||
|
||
**Option A baseline: UI limiting behavior**
|
||
- Updated `frontend/__tests__/EngineConsole.test.tsx` with a new case that mocks 10 pending items and asserts:
|
||
- only the first 8 keyword terms render
|
||
- the UI shows `Showing first 8 of 10`.
|
||
- Verified: `cd frontend && npx vitest run`.
|
||
|
||
## π’ Session 38 β 2026-03-19
|
||
|
||
**Option A baseline: API contract tests for dashboard**
|
||
- Extended `backend_tests/test_scrape_progress.py` with:
|
||
- `/api/stats` response shape assertions
|
||
- `/api/listings/countdown-sync` values + ISO timestamp fields based on seeded `Listing` rows.
|
||
- Verified: `python -m unittest discover -s backend_tests -p 'test_*.py'` and frontend `vitest` remained green.
|
||
|
||
## π’ Session 39 β 2026-03-19
|
||
|
||
**Option A baseline: `/api/listings/refresh-status` contract test**
|
||
- Extended `backend_tests/test_scrape_progress.py` with assertions for:
|
||
- `GET /api/listings/refresh-status` returns `last_price_update` ISO string and `listing_count`.
|
||
- Verified: `python -m unittest discover -s backend_tests -p 'test_*.py'` + frontend `vitest`.
|
||
|
||
## π’ Session 40 β 2026-03-19
|
||
|
||
**Option A baseline: `/api/sites` numeric flags contract**
|
||
- Added backend contract assertions in `backend_tests/test_scrape_progress.py` verifying `enabled`, `custom_visible_browser`, `requires_login`, `login_enabled` are returned as integer `0/1` (not JSON booleans).
|
||
- Fixed `models.py` `TargetSite.to_dict()` to emit `0/1` integers for those flag fields.
|
||
- Verified: backend unittests + frontend `vitest` remain green.
|
||
|
||
## π’ Session 41 β 2026-03-19
|
||
|
||
**Option A baseline: `/api/config` contract tests (Settings page)**
|
||
- Added backend contract assertions in `backend_tests/test_scrape_progress.py` for:
|
||
- `GET /api/config` returning a flat `{key: value}` object with string values
|
||
- `POST /api/config` upserting values as strings and returning `{status, keys}`
|
||
- Verified: backend unittests and frontend `vitest` remained green.
|
||
|
||
## π’ Session 34 β 2026-03-18
|
||
|
||
**MD Update Playbook (repeatable doc job)**
|
||
- Added `docs/MD_UPDATE_PLAYBOOK.md` with:
|
||
- deep semantic audit workflow for all `*.md` files
|
||
- exact mechanics checklist for per-site visibility override and keyword batching + retry tracking
|
||
- reusable prompt (TEXT + JSON) with approval gate + verification checklist
|
||
|
||
## π’ Session 33 β 2026-03-18
|
||
|
||
**Per-site Visible/Custom Browser Override**
|
||
- Added `TargetSite.custom_visible_browser` (0/1).
|
||
- Global `show_browser=true` forces visible mode for all sites.
|
||
- When `show_browser=false`, a site becomes visible only if `custom_visible_browser=1`.
|
||
|
||
**Keyword Batching + Persistent Progress Tracking**
|
||
- Added config `keyword_batch_enabled` (default `false`).
|
||
- Introduced `scrape_rounds` + `scrape_round_items` to track per-(site, keyword) status across cycles.
|
||
- Engine batches at most `max_tabs_per_site` keyword attempts per site per cycle/round.
|
||
- Failures mark items `pending` and keep retrying until the roundβs 4-hour retry window expires.
|
||
|
||
**Hourly Retry Warnings + Live Dashboard**
|
||
- Hourly bookkeeping updates `scrape_round_items.last_hour_warn_at`.
|
||
- Added API: `GET /api/scrape/progress` for the dashboard to render pinned βKeyword Retry Trackingβ.
|
||
|
||
## π’ Session 32 β 2026-03-18
|
||
|
||
**Redis Cache Layer (Priority 1 β DONE)**
|
||
- Added `_init_redis()`, `_redis_set_stats()`, `_redis_publish()`, `_redis_cache_set/get()` helpers in worker.py
|
||
- Graceful write-through: app runs unchanged when `REDIS_URL` not set
|
||
- Stats synced to Redis on cycle complete, engine pause/resume, engine running
|
||
- `new_listing` pub/sub event published on every alert
|
||
- New endpoint: `GET /api/redis/status` β connectivity check + cached stats
|
||
- Redis channel: `ghostnode:events`, stats key: `ghostnode:stats`
|
||
|
||
**Docker Compose (Priority 2 β DONE)**
|
||
- `Dockerfile` β Python 3.11-slim + Playwright Chromium + all deps, serves on port 8000
|
||
- `docker-compose.yml` β 3 services: ghostnode, postgres:16-alpine, redis:7-alpine
|
||
- Auto health checks on Postgres + Redis before ghostnode starts
|
||
- One-command: `docker compose up --build`
|
||
- Volumes: `postgres_data`, `redis_data`, `ghostnode_sessions`
|
||
|
||
**Lot Description Extraction / N18 (Priority 3 β DONE)**
|
||
- `JS_DETAIL_TEXT` β 4-layer JS extractor: JSON-LD β OG meta β known selectors β largest `<p>` block
|
||
- `description (Text)` column added to `Listing` ORM + migration in `_migrate_schema()` for SQLite + PostgreSQL
|
||
- `_fetch_listing_images_batch()` now evaluates `JS_DETAIL_TEXT` on every detail page visit (zero extra network cost β same page already open for images)
|
||
- `_build_ai_prompt()` and `_ai_analyze()` updated to accept optional `description` param
|
||
- Re-analysis: listings that passed AI on title alone (`ai_match=1`) are re-evaluated with description after detail fetch
|
||
- `to_dict()` on `Listing` now includes `description` field
|
||
|
||
## π‘ Pending Features β Priority Queue
|
||
|
||
### Priority 1 β Frontend Polish
|
||
Badge system refinement, animation tuning, show `description` in ListingDetailPanel.
|
||
|
||
### Priority 2 β Docker Testing
|
||
Test the docker-compose stack end-to-end. Verify PostgreSQL migration, Redis pub/sub, Playwright in container.
|
||
|
||
### Priority 3 β vLLM Production Inference
|
||
Replace Groq rate-limited free tier with self-hosted vLLM on rented GPU for production scale.
|
||
|
||
### Priority 4 β Frontend Visual Polish
|
||
Session 27: Full premium redesign β ambient animated gradient background (3 floating orbs + dot grid), glassmorphism shell (backdrop-blur + saturate), gradient glow cards (`g-card-glow` with animated border), Framer Motion staggered card/row/page animations, gradient text headings, badge system (green/amber/red/blue/purple/neutral), gradient underline active nav with glow, status pill with live pulse dot + aura shadow. New CSS prefix: `g-*` (e.g. `bg-g-base`, `text-g-green`). All 21 tests + build passing.
|
||
|
||
Session 28: Added a dedicated marketing landing page (`frontend/components/landing/LandingPage.tsx`) and switched `frontend/app/page.tsx` from `redirect('/dashboard')` to render the landing component. Root route now serves landing content in fresh static export.
|
||
|
||
---
|
||
|
||
## π΅ Known Improvements (Lower Priority)
|
||
|
||
| Item | Description |
|
||
|---|---|
|
||
| Edit Site modal | No way to edit login fields on existing sites after creation |
|
||
| Cloudflare Turnstile | Current solver handles reCAPTCHA/hCaptcha only β Turnstile needs `cf-clearance-scraper` |
|
||
| Mobile dashboard | Desktop-only β needs responsive grid breakpoints |
|
||
| Sort by USD price | Listings table sorts by raw price β should optionally sort by `price_usd` |
|
||
| N18: Site Discovery | Skipped β AI-powered discovery of new auction sites from web |
|
||
|
||
*Last updated: 2026-03-13 β Session 28*
|
||
|