# 👻 Ghost Node — Session History Archive

> **DO NOT load this at the start of a session — it is archaeology, not active reference.**
> Open it only if asked about historical context or old decisions.
> Current work → PROGRESS.md | Architecture → CLAUDE.md | Gotchas → MEMORY.md
>
> **No session limit.** Every completed session gets a full entry here permanently.
> When a session in PROGRESS.md is old enough to no longer be active context, move its full entry here.
>
> ### What gets logged here
> - **Code sessions** — any session where files were edited. Full detail: what changed, why, how.
> - **Q&A sessions that produced lasting decisions** — tech stack choices, architectural decisions, strategic direction, important "why we do it this way" answers. Log with a `[Q&A]` tag.
> - **Skip** — pure troubleshooting Q&A where nothing was decided and no files changed.

---

## Sessions 1–8 — Foundation Build (2026-03-06 – 2026-03-09)

### Session 1 — 2026-03-06 — Foundation
- Initial 6-file build: `database.py`, `models.py`, `worker.py`, `dashboard.html`, `requirements.txt`, `README.md`
- Three-thread architecture (A: FastAPI, B: Scraper, C: Telegram C2)
- Basic Playwright scraper, Telegram alerts, seeded DB (keywords, sites, config)

### Session 2 — 2026-03-08 Morning — Navigation & Bug Fixes
- Dual-mode navigation: Mode A (direct URL) + Mode B (homepage + search discovery)
- Semantic search box discovery (ARIA, placeholder, label, ID heuristics)
- Browser auto-detection: Edge → Yandex → Chrome → Brave → Chromium
- UI fixes and stability improvements

### Session 3 — 2026-03-08 Afternoon — Controls & UI
- Browser selector UI in dashboard Settings tab
- Incognito mode toggle, show/hide browser toggle
- Restart button, Kill button
- Four delay controls: launch / site-open / post-search / page-hold

### Session 4 — 2026-03-08 Evening — Extraction System
- Background-window protection (3 layers preventing Chromium throttle)
- Link extraction fix
- Price/currency extraction system supporting 20 currencies
- Time-left extraction with `time_left_mins` as Float (seconds precision)
- New DB columns: `price_raw`, `currency`, `time_left`, `time_left_mins`, `price_updated_at`

### Session 5 — 2026-03-08 Late — Thread D & Sortable UI
- **Thread D** — Price refresh loop (isolated event loop, every 5 min)
- Card-anchored DOM extraction for complex sites (HiBid)
- Sortable Price + Time Left columns in listings table
- Urgency colour coding for closing-soon lots

### Session 6 — 2026-03-08 Night — Anti-Bot Humanisation
- 30+ stealth script patches (WebGL, canvas fingerprint noise, audio fingerprint, battery API, network info, media devices, iframe patching)
- Humanize level system: `raw` / `low` / `medium` / `heavy`
- Colour-coded humanize buttons in UI

### Session 7 — 2026-03-09 Morning — Live Countdown
- Live countdown: 1-second ticker + 60-second sync endpoint
- `time_left_mins` precision fix (seconds-level Float)
- JS syntax bug fix (apostrophe in `_humanizeDescs` broke all UI buttons — use backtick JS literals always)

### Session 8 — 2026-03-09 — Major Feature Pack (N2/N3/N5/N9/N10/N12/N13/N14/N15)
- **N2** — CAPTCHA solver: 2captcha + CapSolver wired into navigation
- **N3** — Block/rate-limit detection + 30min site cooldown + health tracking
- **N5** — Pagination: `max_pages` per site, universal next-page detection
- **N9** — Thread E closing-soon alerts (user-controlled threshold, toggle, only lots with captured time data, 7-day staleness guard)
- **N10** — Multi-channel alerts: Telegram + Discord webhook + SMTP email (simultaneous or single)
- **N12** — PostgreSQL support via `DATABASE_URL` env var (auto-detected in database.py)
- **N13** — Site health dashboard column (cooldown timer, error count, last error preview)
- **N14** — Login session support (persistent browser profile, 🔑 Login button, pre-scrape session check)
- **N15** — Export: CSV, JSON, HTML cyberpunk report + Database Backup & Restore (download/upload `.db`)

### URL Validation Fix — 2026-03-07
- Fixed validator rejecting homepage URLs without `{keyword}` placeholder
- New rule: block only if *neither* `{keyword}` in URL *nor* `search_selector` provided

---

## Sessions 9–15 — Feature Expansion (2026-03-10 – 2026-03-11)

### Session 9 — 2026-03-10 — N16: AI Smart Filter
- **N16** — AI-powered lot analysis using Groq (free cloud) + Ollama (local, unlimited)
- `_ai_analyze()` dispatcher, `_build_ai_prompt()`, `_ai_call_groq()`, `_ai_call_ollama()`
- `PUT /api/keywords/{id}` — update `weight` and/or `ai_target`
- `POST /api/ai/test` — test AI verdict in dashboard
- `Keyword.ai_target` — new column (Text — natural-language filter description)
- `Listing.ai_match` (Integer: 1/0/NULL) + `Listing.ai_reason` (String 200)
- Schema migration: auto-adds `ai_match`, `ai_reason` to listings; `ai_target` to keywords
- New config keys: `ai_filter_enabled`, `ai_provider`, `ai_model`, `ai_api_key`, `ai_base_url`
- Pipeline: if keyword has ai_target + AI enabled → AI is judge (score still calculated for display)

### Session 10 — 2026-03-10 — N17: Auto-Adapter (AI Selector Generator)
- **N17** — AI-powered CSS selector generator using Groq + Ollama
- `_clean_html_for_ai(raw_html, max_chars=14000)` — strips scripts/styles/SVGs, isolates main content
- `_build_selector_prompt(cleaned_html, site_name)` — token-efficient prompt for 6-selector JSON
- `_generate_selectors_ai(cleaned_html, site_name)` — calls Groq or Ollama, extracts JSON
- `_validate_selectors(page, sel_dict)` — live-tests selectors, computes confidence score 0–100
- `_extract_with_selectors(page, ss)` — uses stored SiteSelectors rows to extract listings
- `adapt_site_now(site_id)` — full pipeline: browser → navigate → clean HTML → AI → validate → persist
- `SiteSelectors` ORM model added: `site_id`, selector fields, `confidence`, rates, `stale`, `provider`
- 3 new API endpoints: `POST/GET/DELETE /api/sites/{id}/adapt|selectors`
- Dashboard: confidence badges (≥70%=green, ≥40%=orange, <40%=red), 🤖 Adapt / ↺ Re-adapt / × Clear

### Session 11 — 2026-03-10 — Major Feature Pack (N4/N7/N1/Gmail/Multi-Closing/C2/AI-Debug)
- **N4 — Currency Display**: Optional per-session display currency (ISO code). Rates via `frankfurter.app` cached 6h. `price_usd` stored on every listing
- **N7 — Price Filters**: `min_price` / `max_price` on Keyword model. Compared in USD via `_convert_price()`. 💰 button per keyword
- **N7 — Location Capture**: `location` column on Listing. Extracted from lot cards via `LOC_SELS` in JS_EXTRACT
- **N1 — Proxy Rotation**: `_RoundRobin` class, `proxy_enabled` / `proxy_list` config keys. Passed to Playwright browser launch
- **Gmail**: Replaced 6-field SMTP with 3 fields: `gmail_address` + `gmail_app_password` + `email_to`. Hardcoded `smtp.gmail.com:587`
- **Multi-Interval Closing Alerts**: `closing_alert_schedule` (comma-separated, e.g. `60,30,10,5`). `0` = no countdown. `closing_alerts_sent` JSON list tracks which thresholds fired
- **Telegram C2 Commands**: `/top5`, `/sites`, `/keywords`, `/alert on|off <kw>`, `/help`
- **AI Debug Log**: In-memory circular buffer (`deque(maxlen=300)`). `GET /api/ai/debug/log` (with `since_id`) + `DELETE`. Dashboard: "🧠 AI Log" tab, filter buttons, colour-coded cards

### Session 12 — 2026-03-10 — UI/UX Overhaul + DB Cleanup
- Inline keyword + site editing (click to rename/edit values)
- Drag-and-drop reordering (⋮⋮ handles) — `sort_order` column on Keyword + TargetSite. `POST /api/keywords/reorder` + `POST /api/sites/reorder`
- Activity Log: search/filter box, scroll-to-bottom, clear. Limit raised 80 → 200 lines
- AI Debug Log: search/filter across all cards, manual refresh
- Edit Site modal: full ✏ Edit button exposing all fields including login fields
- Listing detail view: click any listing title → panel with all captured data. Toggle: `listing_detail_enabled`
- Batch keyword import: textarea, one keyword per line, optional `keyword:weight` suffix
- Currency picker: searchable dropdown, 22 currencies + IQD, ✕ Raw to clear
- **DELETED** `closing_alert_sent` boolean from ORM — `closing_alerts_sent` JSON list is the sole tracker
- **DELETED** dead SMTP seed keys: `email_smtp_host/port/user/pass`, `email_from`, `closing_alert_mins`

### Session 13 — 2026-03-10 — Lot Image Extraction + Display
- `JS_EXTRACT` module-level constant replaces old inline `page.evaluate("""...""")`. Adds `extractImages(root)` — tries `data-src`, `data-lazy-src`, `data-original`, `data-lazy`, then `img.src`. Skips tiny icons (<40px). Returns up to 5 URLs per card
- Pagination fix: `JS_EXTRACT` was undefined when pagination ran — both page-1 and paginated pages now use same constant
- `images` column on Listing (TEXT, JSON array, 0–10 URLs, NULL if none)
- All 4 `Listing(...)` constructors updated to pass `images=json.dumps(images_list[:5])`
- Thumbnail in listings table (48×48px, flexbox left of title, `onerror` hide fallback)
- Gallery in Listing Detail View (140×110px clickable thumbnails, flex-wrap, "LOT IMAGES" header)

### Session 13b — 2026-03-10 — Image Dedup Fix
- **Root cause found**: HiBid CDN uses same path `img.axd` for ALL images, differentiated only by query params. Old dedup stripped query strings → all 5 images collapsed to 1
- **Fix**: `addUrl()` now deduplicates by full URL including query string
- Apollo cache polling added: `wait_for_function(JS_APOLLO_WAIT, timeout=8000)` before extraction

### Session 14 — 2026-03-10 — Detail-Page Image Fetch
- **Root cause**: HiBid search results pages have 0 `Lot:*` entries in Apollo cache — all images only available on detail pages
- `JS_DETAIL_IMAGES` — 5-layer image extractor: (1) Apollo cache, (2) JSON-LD, (3) OG meta, (4) DOM img tags, (5) srcset. Reusable module-level constant
- `JS_APOLLO_WAIT` — reusable Apollo cache readiness poller
- `_fetch_listing_images_batch()` — visits each new listing's detail page immediately after scraping. Saves full-size gallery
- `_price_refresh_pass` refactored to use shared `JS_DETAIL_IMAGES`/`JS_APOLLO_WAIT` constants

### Session 14b — 2026-03-10 — Image Count Edge Cases
- `[:5]` → `[:10]` cap in all 4 Listing constructors (raised to match JS extractor)
- Removed `> len(existing)` guard — detail page result is always authoritative (full-size, not thumbnail)
- `_price_refresh_pass`: `img_urls != existing_imgs` comparison replaces `len > len` — catches quality upgrades

### Session 15 — 2026-03-11 — Tech Stack Research + Strategic Direction
- Researched tech stacks used by OpenRouter, Kling AI, SimilarWeb, Accio
- Ghost Node finalized stack: Python (FastAPI) backend, Next.js + React + TypeScript + Tailwind frontend, PostgreSQL (not MongoDB), Redis (cache/queue)
- Key decisions: no Rust/Go (network I/O bound), no PyTorch/TF yet (inference only), vLLM for production
- Ghost Node vision: auction intelligence layer as SaaS — no competitor offers this
- Created HANDOFF.md (later merged into CLAUDE.md in Session 20b restructure)

---

## Sessions 16–20b — Frontend Migration & Polish (2026-03-11 – 2026-03-12)

### Session 16 — 2026-03-11 — Next.js Frontend Migration (COMPLETE)
All 13 tasks complete:
- Node.js v22.14.0 installed portable at `%LOCALAPPDATA%\nodejs-portable\`
- Scaffolded `frontend/` with Next.js 16.1.6, React 19, TypeScript, Tailwind CSS v4
- **Tailwind v4 discovery**: CSS variable-based config via `@theme {}` in `globals.css` — NO `tailwind.config.ts`
- Phase 0: Scaffold (next.config.ts, Tailwind v4 theme, types.ts, engineStore, useSSE, layout shell)
- Phase 1: Dashboard tab (StatsGrid, ActivityLog)
- Phase 2: Listings tab (ListingsTable, ListingRow, ListingDetailPanel, ImageGallery, useCountdown)
- Phase 3: Keywords tab (drag-drop via @dnd-kit, inline edit, batch import)
- Phase 4: Sites tab (health badges, AI confidence badge, adapt button, drag-drop)
- Phase 5: Settings tab (all 30+ config keys, backup/restore, Telegram test)
- Phase 6: AI Log tab (live polling, filter buttons, search, AILogCard)
- Phase 7: `output: 'export'` → `frontend/out/` → FastAPI StaticFiles mount in worker.py
- **21/21 tests passing** (Vitest + React Testing Library)
- Backend: `sys.stdout.reconfigure(encoding='utf-8')`, `→` → `->` in database.py, `\d`/`\/` escapes fixed

### Session 17 — 2026-03-11 — N6/N8/N11 Feature Pack
- **N6 — Editable Scoring Rules**: `ScoringRule(id, signal, delta, category, notes)` ORM model. Seeded from old hardcoded lists. `calculate_attribute_score()` queries DB. 4 new endpoints: `GET/POST/PUT/DELETE /api/scoring-rules`. `ScoringRulesPanel` React component: add/edit/delete inline, colour-coded boosts/penalties, TanStack Query sync
- **N8 — Scrape Window + Boost Mode**: Engine checks time-of-day before each cycle. Config: `scrape_window_enabled`, `scrape_start_hour`, `scrape_end_hour`. Overnight windows supported (start > end triggers overnight logic). Boost: `boost_interval_mins` replaces `timer` when any lot has `time_left_mins ≤ 30`
- **N11 — Cross-site Dedup (eBay only)**: `difflib.SequenceMatcher > 0.85` before saving any new eBay listing. 24h window, different `site_name`. Only fires when site name/URL contains "ebay"

### Session 17b — 2026-03-11 — AI-First Architecture + Scoring Toggle
- **`scoring_enabled` config key**: Default `true`. When `false`, score gate skipped entirely — AI sole judge. Three-way filter: (1) AI target + ai_filter_enabled → AI; (2) scoring_enabled=true → score gate; (3) scoring_enabled=false → all lots pass
- **ScoringRulesPanel toggle**: `● ON / ○ OFF` button. When OFF: rules dim, green "⚡ AI-FIRST MODE ACTIVE" banner
- **Keywords → Targets relabelling**: Header "TARGETS", column "TARGET LABEL", AI column "AI DESCRIPTION ★". Guidance text explains search term vs AI description roles
- **proxy.ts migration**: `middleware.ts` → `proxy.ts`, `middleware()` → `proxy()` (Next.js 16 requirement)
- 5 new config keys added to CLAUDE.md; ScoringRule added to DB schema

### Session 18 — 2026-03-11 — Live Reload + AI Log Fix + Auto-Adapt 4 Fixes
- **Live reload**: `_cycle_now = threading.Event()` (thread-safe). All 7 write endpoints call `_cycle_now.set()`. Scraper polls every 5s instead of single blocking `asyncio.sleep(timer)` — wakes immediately on change
- **AI debug log fix**: `AILogFeed` unwraps `data.entries` (was storing whole envelope). `AILogCard` completely rewritten with correct `RawEntry` interface — `call_type`, `direction`, `content`, `tokens_prompt`, `tokens_completion`, `verdict`, `status_code`
- **Auto-adapt gate fix**: Removed `auto_adapt_enabled` check from manual ADAPT endpoint — toggle only gates auto-trigger on new site creation, not manual clicks
- **Auto-adapt max_tokens 300 → 500**: 300 was insufficient for 6-selector JSON (~400–500 tokens needed)
- **Auto-adapt JSON extraction**: `_extract_json()` 4-strategy helper: (1) direct `json.loads`, (2) strip markdown fences, (3) brace-depth counter, (4) regex fallback
- **ADAPT button UX**: `adapting` local state + 45s timer in `SiteRow.tsx`. Shows `⏳ ADAPTING…` (gold, disabled). Auto-refetches selectors via `qc.invalidateQueries` on completion

### Session 18b — 2026-03-11 — Auto-Adapt: Popup Dismiss + CF Bypass
- **`_auto_dismiss_popups(page)`**: 17 known consent-framework selectors (OneTrust, Cookiebot, etc.) + text fallback (Accept, I Agree, OK, etc.). Called twice: after initial `goto()` and after Mode B search navigation
- **Saved profile reuse**: `adapt_site_now()` checks `.browser_profiles/<site_slug>/`. If exists and non-empty → `launch_persistent_context()`. CF cookies/login sessions carry over automatically
- **CF non-headless fallback**: CF detected in headless → auto-retry with `headless=False`. Covers ~95% of CF cases with zero user action. Browser briefly appears then auto-closes
- **Turnstile CAPTCHA solver**: Non-headless also hits CF Turnstile + solver configured → extracts `data-sitekey`, calls 2captcha/CapSolver Turnstile endpoint, injects token, waits for CF redirect

### Session 19 — 2026-03-12 — Image Lightbox + Thumbnail Arrow Navigation
- **Lightbox**: `createPortal(…, document.body)` to escape panel `z-50` stacking context. Same size/position as detail panel (`w-96 h-full fixed right-0 top-0`). ✕ CLOSE, `‹ PREV / NEXT ›`, dot indicators, backdrop-click-to-close, keyboard (←/→/Esc)
- **Thumbnail strip arrows**: `‹` / `›` buttons cycle `activeIdx` without opening lightbox
- **`activeIdx` sync**: Strip arrows and lightbox arrows share same state — highlighted thumbnail always reflects lightbox view
- **`mounted` SSR guard**: `useEffect(() => setMounted(true), [])` — portal only renders after hydration (`document.body` unavailable during Next.js SSR)

### Session 20 — 2026-03-12 — Bug Fixes Batch
- **Fix: ENGINE OFFLINE**: `useSSE.ts` used `EventSource('/api/stream')` (disabled in static build → instant 404 → `onerror` → permanent offline). Replaced with 5s `setInterval` polling of `/api/stats`
- **Fix: Settings stuck "Loading settings…"**: (1) `fetchConfig()` called `.map()` on dict → silent TypeError → `loaded` never `true`. (2) `saveConfig()` sent `[{key,value}]` array but backend expects flat dict. Both fixed. Catch calls `setConfig({})` so form renders on error
- **Fix: Header version**: "v2.5" → "v2.7" in `Header.tsx`
- **Fix: `serve_dashboard()` always serving old HTML**: Explicit `@app.get("/")` was beating StaticFiles mount but returning dashboard.html. Now checks `frontend/out/index.html` first
- **`/legacy` route added**: `GET /legacy` in worker.py always serves `dashboard.html`

### Session 20b — 2026-03-12 — Fix: /legacy Route + MD File Restructure
- **Root cause**: `app.mount("/", StaticFiles(html=True))` registers as Starlette sub-application that captures ALL paths. SPA fallback returns `index.html` with 200 for unknown paths, shadowing `@app.get("/legacy")`
- **Fix**: Replaced with `app.mount("/_next", StaticFiles(...))` (assets only) + `@app.get("/{full_path:path}")` 3-step catch-all: (a) exact file, (b) `.html` match, (c) `index.html` SPA fallback
- **Why this works**: FastAPI routes always take priority over parameterised `{path}` catch-all. `/legacy` hits `serve_legacy()` first
- **MD restructure**: HANDOFF.md deleted (content merged into CLAUDE.md). ARCHIVE.md created. PROGRESS.md and MEMORY.md stripped of session history. Dynamic MD update policy embedded in all files

### Session 21 — 2026-03-12 — MD Restructure Completion + Q&A Decisions [Q&A + Config]

**Code changes:**
- CLAUDE.md: Added "Why These Decisions" section under Tech Stack (PostgreSQL, Redis, no Rust/Go, no PyTorch yet, vLLM rationale, Ghost Node unique value). Added tech stack reasoning from deleted HANDOFF.md.
- PROGRESS.md: Stripped all session history (now in ARCHIVE.md). Now contains only: Current State, Pending Features, Known Improvements. Added Priority 4 (Frontend Visual Polish).
- ARCHIVE.md: Added Q&A logging rule. Added this session entry. All session history (1–21) now lives here permanently.
- HANDOFF.md: Confirmed deleted (was already removed in 20b).

**Q&A decisions made this session:**
- **node_modules 553MB** → Normal for Next.js. `out/` is only 1.8MB. Safe to delete `node_modules` — run `npm install` to restore before next build.
- **Q&A in ARCHIVE.md** → Yes: log Q&A sessions that produced lasting decisions. Skip pure troubleshooting Q&A with no lasting output. Tag with `[Q&A]`.
- **MD update policy finalised** → Dynamic, not static. Each MD file only updated when session touched content relevant to it. Reminder embedded in all MD files.
- **Frontend "looks beginner noob"** → Real concern. UI structure is solid (Next.js + Tailwind v4 + components). Needs visual design pass: gradients, shadows, typography, spacing, transitions. Added as Priority 4 in PROGRESS.md.

---

## Full Session Log (All Sessions — Updated Permanently)

| Date | Session | Key Deliverable |
|---|---|---|
| 2026-03-06 | 1 | Foundation — full 3-thread architecture |
| 2026-03-07 | fix | URL validation for homepage Mode B |
| 2026-03-08 AM | 2 | Dual-mode navigation, browser auto-detect |
| 2026-03-08 PM | 3 | Controls UI, delay system |
| 2026-03-08 Eve | 4 | Price/time extraction, 20 currencies |
| 2026-03-08 Late | 5 | Thread D, sortable listings |
| 2026-03-08 Night | 6 | 30+ stealth patches, humanize levels |
| 2026-03-09 AM | 7 | Live countdown, JS bug fix |
| 2026-03-09 | 8 | N2+N3+N5+N9+N10+N12+N13+N14+N15 |
| 2026-03-10 | 9 | N16: AI Smart Filter — Groq + Ollama |
| 2026-03-10 | 10 | N17: Auto-Adapter — AI CSS selector generator |
| 2026-03-10 | 11 | N4+N7+N1+Gmail+Multi-Interval Closing+Telegram C2+AI Debug |
| 2026-03-10 | 12 | Inline edit, drag-drop, log controls, listing detail, batch import, DB cleanup |
| 2026-03-10 | 13 | Lot image extraction, thumbnail, gallery |
| 2026-03-10 | 13b | HiBid image dedup fix (full URL), Apollo cache polling |
| 2026-03-10 | 14 | Detail-page image fetch (`_fetch_listing_images_batch`) |
| 2026-03-10 | 14b | Image count edge-cases fixed |
| 2026-03-11 | 15 | Tech stack research, strategic direction, HANDOFF.md |
| 2026-03-11 | 16 | Next.js frontend migration complete (all 13 tasks) |
| 2026-03-11 | 17 | N6/N8/N11: scoring rules, scrape window, eBay dedup |
| 2026-03-11 | 17b | AI-first mode, scoring toggle, Keywords→Targets relabel |
| 2026-03-11 | 18 | Live reload, AI log fix, auto-adapt 4 fixes, ADAPT UX |
| 2026-03-11 | 18b | Auto-adapt: popup dismiss, CF bypass, Turnstile solver |
| 2026-03-12 | 19 | Image lightbox + thumbnail arrow navigation |
| 2026-03-12 | 20 | ENGINE OFFLINE fix, Settings fix, /legacy route |
| 2026-03-12 | 20b | /legacy routing fix (StaticFiles refactor), MD restructure |
| 2026-03-12 | 21 [Q&A] | MD restructure completion, tech stack reasoning added, Q&A policy set |