166 lines
9.0 KiB
Markdown
166 lines
9.0 KiB
Markdown
# Ghost Node — Session Memory
|
|
|
|
> **What belongs here:** Current architecture state, key gotchas, reusable patterns, dev paths.
|
|
> **What does NOT belong here:** Session history (→ PROGRESS.md or ARCHIVE.md).
|
|
>
|
|
> ### ⚠️ MD Update Rule — DYNAMIC, NOT STATIC
|
|
> Only update the MD file(s) whose content was actually touched this session.
|
|
> Do NOT update all files after every task. Each file gets only what is relevant to it.
|
|
> - Update **this file** only when a new reusable gotcha, pattern, or architecture decision was made.
|
|
> - Update **PROGRESS.md** whenever any code file was changed.
|
|
> - Update **CLAUDE.md** only if API endpoints, DB schema, config keys, or architecture changed.
|
|
> - Move old PROGRESS.md session entries to **ARCHIVE.md** when they're no longer active context.
|
|
|
|
---
|
|
|
|
## Project Identity
|
|
- Ghost Node International Auction Sniper v2.7
|
|
- Owner: Abbas, Baghdad, Iraq
|
|
- Path: `C:\Users\Abbas\Documents\Downloads\ClaudeAuction2\`
|
|
- Backend: Python FastAPI (worker.py, 5000+ lines), port 7000
|
|
- NOT a git repository
|
|
|
|
---
|
|
|
|
## Frontend Stack (Current — Session 16, fully migrated)
|
|
|
|
- Location: `frontend/` directory
|
|
- Next.js 16.1.6, React 19, TypeScript, Tailwind CSS v4 (**NOT v3**)
|
|
- **Tailwind v4**: CSS variables via `@theme {}` in `globals.css` — NO `tailwind.config.ts`
|
|
- **Session 27 redesign**: CSS prefix changed from `ghost-*` to `g-*`. Colors: `--color-g-base` (#050510), `--color-g-green` (#00e87b), `--color-g-cyan` (#06b6d4), etc. Classes: `bg-g-base`, `text-g-green`. Components: `.g-card`, `.g-card-glow`, `.g-btn`, `.g-btn-primary`, `.g-badge`, `.g-table`, `.glass`, `.glass-strong`, `.gradient-text`, `.gradient-accent`. Backward compat aliases (`.card-panel`, `.btn-ghost`, `.btn-danger`, `.page-title`, `.page-subtitle`) still work.
|
|
- `output: 'export'` in next.config.ts → static build at `frontend/out/`
|
|
- Node.js: default installer `C:\Program Files\nodejs\` (on PATH). Alternative: portable at `C:\Users\Abbas\AppData\Local\nodejs-portable\node-v22.14.0-win-x64\`
|
|
|
|
### Key Architecture Decisions
|
|
- SSE route disabled for static build (`app/api/stream/route.ts.disabled`)
|
|
- All API files use `http://localhost:7000` as BASE (full URL, not relative)
|
|
- FastAPI serves Next.js static build when `frontend/out/` exists
|
|
- `if _frontend_out.exists():` guard is safe — `dashboard.html` still works without out/ dir
|
|
- **SPA routing**: `app.mount("/_next", ...)` for assets only + `@app.get("/{full_path:path}")` catch-all for SPA routing. Never use `app.mount("/", StaticFiles(html=True))` — it shadows all explicit routes
|
|
|
|
### File Structure
|
|
```
|
|
frontend/
|
|
├── app/
|
|
│ ├── ai-log/page.tsx
|
|
│ ├── dashboard/page.tsx
|
|
│ ├── keywords/page.tsx
|
|
│ ├── listings/page.tsx
|
|
│ ├── settings/page.tsx
|
|
│ ├── sites/page.tsx
|
|
│ ├── globals.css (Tailwind v4 @theme, cyberpunk palette, btn-ghost, btn-danger)
|
|
│ ├── layout.tsx
|
|
│ ├── page.tsx (redirects to /dashboard)
|
|
│ └── providers.tsx ('use client' QueryClientProvider)
|
|
├── components/
|
|
│ ├── ai-log/ (AILogCard, AILogFeed)
|
|
│ ├── dashboard/ (StatsGrid, ActivityLog)
|
|
│ ├── keywords/ (KeywordRow, KeywordsTable — dnd-kit, ScoringRulesPanel)
|
|
│ ├── layout/ (Header, Nav, StatusBar)
|
|
│ ├── listings/ (ListingRow, ListingsTable, ListingDetailPanel, ImageGallery)
|
|
│ └── sites/ (SiteRow, SitesTable — dnd-kit)
|
|
├── hooks/ (useSSE, useListings, useCountdown, useKeywords, useSites)
|
|
├── lib/
|
|
│ ├── api/ (listings, keywords, sites, config, engine, system, ai, scoring-rules)
|
|
│ ├── types.ts
|
|
│ └── utils.ts (cn, formatUptime)
|
|
├── store/ (engineStore, settingsStore)
|
|
├── out/ (static build — served by FastAPI)
|
|
└── vitest.config.ts (has @/ alias resolution)
|
|
```
|
|
|
|
### Tests (21 passing)
|
|
- theme.test.ts (2), types.test.ts (4), engineStore.test.ts (3)
|
|
- listings-api.test.ts (3), useCountdown.test.ts (2)
|
|
- StatsGrid.test.tsx (1), StatusBar.test.tsx (3), ListingRow.test.tsx (3)
|
|
|
|
---
|
|
|
|
## Dev Paths & Commands
|
|
|
|
### Node.js
|
|
- **Default installer** (`C:\Program Files\nodejs\`): usually on system PATH — open new terminal and run `node --version`; no manual PATH needed.
|
|
- **Portable** (if used): set PATH in each shell before `npx`/`npm`:
|
|
- PowerShell: `$env:PATH = "C:\Users\Abbas\AppData\Local\nodejs-portable\node-v22.14.0-win-x64;$env:PATH"`
|
|
- Bash: `export PATH="/c/Users/Abbas/AppData/Local/nodejs-portable/node-v22.14.0-win-x64:$PATH"`
|
|
|
|
### Common Commands
|
|
```bash
|
|
# Start backend
|
|
python worker.py # port 7000
|
|
|
|
# Frontend build (from project root)
|
|
export PATH="..." && npm run build --prefix frontend
|
|
|
|
# Run tests (from project root)
|
|
export PATH="..." && cd frontend && npx vitest run
|
|
|
|
# Combined build + test
|
|
export PATH="..." && cd frontend && npx vitest run && npm run build
|
|
```
|
|
|
|
---
|
|
|
|
## Key Gotchas (Never Forget These)
|
|
|
|
### Python / Backend
|
|
- **`threading.Event` not `asyncio.Event`** for cross-thread signalling — asyncio.Event is not thread-safe
|
|
- **SQLite `enabled` field**: Store as `int` (1/0), never `bool` — `filter(enabled==1)` breaks on Python `True`
|
|
- **JS in f-strings**: Use backtick template literals, never apostrophes — breaks Python f-string parsing
|
|
- **`db.flush()` before `db.commit()`**: SQLite WAL locking requires this
|
|
- **`calculate_attribute_score()` opens own DB session** — don't call inside an already-open session loop
|
|
- **Keyword batching retry warnings**: the dashboard pins only retry candidates with `attempt_count > 0` via `/api/scrape/progress` (queued-but-never-attempted items stay hidden until first failure).
|
|
- **Per-site visible override precedence**: `show_browser=true` forces visible mode for all sites and ignores per-site `custom_visible_browser`; `show_browser=false` applies per-site visibility.
|
|
- **Retry-tracking state model**: `scrape_round_items` moves through `pending | in_progress | done | failed`; each active round has a 4-hour retry window, and `/api/scrape/progress` computes hourly `warn_due` from `last_hour_warn_at` fallback logic (`first_pending_at`, then round start).
|
|
- **`closing_alerts_sent`**: JSON list — always `json.loads` + append, never overwrite
|
|
|
|
### FastAPI Routing
|
|
- **NEVER `app.mount("/", StaticFiles(html=True))`** with explicit routes you want to preserve — the root mount captures ALL paths. SPA fallback returns 200+index.html for unknown paths, shadowing `@app.get("/legacy")` etc.
|
|
- **Fix**: `app.mount("/_next", ...)` for assets + `@app.get("/{full_path:path}")` explicit catch-all
|
|
|
|
### API Format
|
|
- **`/api/config` GET** returns flat `{key: value}` dict, NOT `Config[]` array
|
|
- **`/api/config` POST** expects flat dict — not `[{key, value}]` array
|
|
- Frontend `fetchConfig()` must call `.then(data => setConfig(data))` directly — no `.map()`
|
|
|
|
### Frontend
|
|
- **SSE disabled in static export** — `EventSource('/api/stream')` always 404s. Use HTTP polling instead
|
|
- **`createPortal` needs SSR guard** — `document.body` unavailable during Next.js SSR. Use `useEffect(() => setMounted(true), [])` before rendering portal
|
|
|
|
### HiBid
|
|
- **Apollo cache**: Search results pages have 0 Lot entries — images only from detail pages
|
|
- **Image dedup**: Use full URL including query string (HiBid CDN uses same path `img.axd` for all images, differentiated only by query params)
|
|
|
|
---
|
|
|
|
## Redis Layer (added Session 32)
|
|
|
|
- Enabled via `REDIS_URL` env var — app runs unchanged without it (graceful fallback)
|
|
- `_redis_publish("new_listing", {...})` fires on every alert; channel: `ghostnode:events`
|
|
- `_redis_set_stats(_stats)` — call after every meaningful `_stats` write (cycle end, pause, resume, running)
|
|
- `GET /api/redis/status` — connectivity check + cached stats hash
|
|
- Install locally: `pip install redis` (auto-installed in Docker)
|
|
|
|
## N18 — Lot Description (added Session 32)
|
|
|
|
- `JS_DETAIL_TEXT` runs on the already-open detail page (same visit as images — zero extra HTTP cost)
|
|
- `description` column: `Text`, max 1500 chars, nullable — in `Listing` ORM + `_migrate_schema()` for both SQLite and PostgreSQL
|
|
- `_build_ai_prompt(title, ai_target, description="")` — description appended as `Lot description:` block when non-empty
|
|
- `_ai_analyze(title, ai_target, description="")` — forwards to prompt builder
|
|
- Re-analysis: after detail fetch, `ai_match=1` listings with `ai_target` are re-evaluated with description; verdict updated in DB
|
|
|
|
## Docker (added Session 32)
|
|
|
|
- `Dockerfile` — Python 3.11-slim + Playwright Chromium; serves port 7000
|
|
- `docker-compose.yml` — ghostnode + postgres:16-alpine + redis:7-alpine; health checks before app starts
|
|
- `DATABASE_URL` + `REDIS_URL` wired in compose env
|
|
- `shm_size: 512mb` on ghostnode service — Chromium requires shared memory
|
|
|
|
## Pending Work (Priority Order)
|
|
|
|
1. **Docker Testing** — run `docker compose up --build`; verify PostgreSQL migration, Redis pub/sub, Playwright headless in container
|
|
2. **Frontend: show description** — add `description` field to `ListingDetailPanel` (already in `to_dict()`, just needs UI)
|
|
3. **vLLM production inference** — replace Groq free tier for production scale
|
|
|
|
|