Compare commits

...

9 Commits

Author SHA1 Message Date
Konrad du Plessis
8f495064c3 docs(perf): fix CLAUDE.md runbook step 3 causal chain
Final whole-impl review catch on the Perf Quick-Wins Pass. Step 3
said "the mtime of the collected copy under staticfiles/ stays the
same, so the token doesn't bump." That's backwards — the token is
read from static/css/custom.css (the SOURCE file), so editing the
source DOES bump the token and Cloudflare correctly misses on the
next request. What actually breaks is the VM's response to the miss:
Apache serves stale bytes from staticfiles/ because collectstatic
hasn't refreshed the collected copy. New URL, old bytes behind it.

Rewording makes the causal chain correct so future Gemini/Claude
debugging "CSS change deployed but old file still shows" reaches
the right conclusion (run collectstatic on the VM) via the right
reasoning.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 01:40:43 +02:00
Konrad du Plessis
167c8216fe fix(perf): Coalesce project FK in adjustment aggregates (dedupe)
Spec-review catch on 61c485f: the batched GROUP BY aggregates for
unpaid-per-project and paid-per-project x month were running TWO
filtered queries and summing them in Python. Any adjustment with
BOTH project FK AND work_log.project set was double-counted.

Every Overtime adjustment fits that shape (price_overtime sets
both). So every unpaid Overtime was silently inflating the
outstanding-costs dashboard by its own amount, and every paid
Overtime inflated the Per-project-monthly-payroll stacked chart.

Fix: annotate Coalesce('project_id', 'work_log__project_id') so
each adjustment contributes to exactly one project (matches the
original Q(...) | Q(...) OR-filter semantics).

New regression test locks in the "count once" behaviour with an
Overtime adjustment that has both FKs set. Previously there was no
test covering the sum correctness of outstanding-costs - only
context-key presence.

Tests: 69/69. Query counts per tab: pending 24q / history 24q /
loans 25q / adjustments 32q (2 fewer per tab than 61c485f because
Coalesce folded two filtered queries into one).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 01:30:25 +02:00
Konrad du Plessis
61c485ffcf perf(payroll): batch project-loop N+1s + quick-wins pass closing summary
Profiled /payroll/ under Django Debug Toolbar and confirmed heavy N+1
patterns in the shared payroll_dashboard() code path (shared by all four
tabs). Main wins:

1. outstanding_project_costs loop + project_chart_data loop previously
   fired one PayrollAdjustment SELECT per project (outstanding) and per
   (project x 6 months) (chart) — ~42+7 = 49 round-trips on a 7-project
   dataset. Replaced with 4 GROUP BY aggregate queries keyed by
   project_id / (project_id, month), merged in Python.

2. Per-worker Loan.exists() and get_worker_active_team() checks inside
   the workers_data loop — pre-computed into a set + dict once, up-front.

3. team_workers_map loop used `team.workers.filter(active=True)` which
   bypasses the prefetch cache; switched to a Prefetch(to_attr=) that
   returns already-filtered active workers, dropping 6 duplicate SELECTs.

4. Adjustments tab: reused `paginator.count` for the "Total" stat card
   (was firing a second identical COUNT(*)) and reused existing
   all_workers / all_teams querysets instead of re-querying for the
   filter popovers.

5. Hoisted shared lookups (all_workers, active_projects_list, chart
   date-window) so duplicate ordering-identical SELECTs from multiple
   call sites collapse into a single evaluated queryset.

===== Quick-Wins Pass A - before/after query counts =====
  /                            15q, no duplicates (healthy, no fix)
  /payroll/?status=pending     157q (before) -> 26q (after), 0 dupes
  /payroll/?status=history     157q          -> 26q,         0 dupes
  /payroll/?status=loans       158q          -> 27q,         0 dupes
  /payroll/?status=adjustments 168q          -> 34q,         0 dupes

CSS cache-bust token (0c42cde) is still expected to be the biggest
user-felt improvement of this pass — custom.css now holds at
Cloudflare's edge for its full 4h TTL instead of being re-fetched
from the VM on every page load. The payroll-dashboard query-count
cut (~131 SQL round-trips trimmed per render) is a meaningful
admin-UX latency win on top of that, especially under MySQL over
the Flatlogic network.

WeasyPrint confirmed still lazy-imported.
Test suite: 68/68.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 01:16:37 +02:00
Konrad du Plessis
2731ac9ffd fix(dev): simplify Debug Toolbar wiring (review followups)
Three followups on 7075269:

- config/urls.py: drop dead try/except ImportError fallback.
  The settings.py gate already guarantees debug_toolbar is
  importable before we reach this line, so the except branch
  was unreachable and the re-import of include/path was
  redundant (both imported at top of file).

- config/settings.py: SHOW_TOOLBAR_CALLBACK now returns True
  unconditionally. The triple gate passed at settings-load time,
  so re-checking DEBUG and _IS_DEV inside the lambda was
  redundant. Comment corrected — the callback has nothing to
  do with "stale cached pages".

- requirements.txt: inline comment noting django-debug-toolbar
  is dev-only and gated.

No behavioural change. Tests: 68/68.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 00:55:44 +02:00
Konrad du Plessis
7075269a07 chore(dev): add Django Debug Toolbar (dev-only, DEBUG+USE_SQLITE gated)
Double-gated install: only loads when DEBUG=true AND USE_SQLITE=true,
never in prod. Lets us profile SQL query counts on the dashboard and
payroll pages before attacking N+1 hotspots.

requirements.txt adds django-debug-toolbar==6.0.0
config/settings.py conditionally appends to INSTALLED_APPS + MIDDLEWARE
config/urls.py conditionally includes __debug__ route

No behavioural change to production.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 00:47:19 +02:00
Konrad du Plessis
0c42cde4ff fix(perf): CLAUDE.md runbook + drop dead var in cache-bust test
Code-review followups on 16d4399:

- CLAUDE.md's "When CSS changes don't appear" diagnostic steps
  were written for the old per-request token. Under mtime-based
  caching, a stable ?v= number is the healthy expected state,
  not a broken one. Rewrote steps 1 + 3 so someone debugging
  a real production CSS issue gets the right advice.

- Dropped unused `original = cp._compute_cache_bust_token` line
  in test_token_falls_back_if_file_missing - it misled readers
  into thinking the function itself was patched. Added a one-
  line comment clarifying the monkey-patch is path-only.

Tests: still 68/68.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 00:38:52 +02:00
Konrad du Plessis
16d4399c28 perf(cache): mtime-based CSS cache-bust token
deployment_timestamp was int(time.time()) per-request, giving every
page load a new ?v=... query string on custom.css. Cloudflare treats
each unique URL as a new resource, so the CSS was fetched from the VM
on every page load — 64 KB over the wire per navigation.

Token now tied to static/css/custom.css mtime. The URL only changes
when the CSS actually changes, so Cloudflare can hold the file for
its full 4h TTL. Degraded-mode fallback preserves today's behaviour
if the file isn't on disk.

3 new CacheBustTokenTests; all 68 tests pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 00:31:05 +02:00
Konrad du Plessis
bcd0112687 docs(perf): task-by-task plan for Quick-Wins Pass A
Four tasks: mtime cache-bust token + tests; install & gate Django
Debug Toolbar dev-only; profile + fix N+1 on /; profile + fix N+1 on
/payroll/ (all four tabs) with before/after summary in the final
commit message.

Execute via subagent-driven-development. Auto mode — no mid-execution
checkpoints.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 00:26:42 +02:00
Konrad du Plessis
d1490c4639 docs(perf): design for Quick-Wins Pass A
Short design covering four changes: mtime-based CSS cache-bust token,
Django Debug Toolbar (dev-only) for profiling, N+1 fixes on Dashboard
and Payroll pages, and a before/after measurement in the commit message.
Scope is deliberately tight — plan B (template splitting) and plan C
(full audit) are deferred until plan A evidence lands.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 00:23:48 +02:00
9 changed files with 1177 additions and 83 deletions

View File

@ -242,16 +242,18 @@ change when the file does.
<link rel="stylesheet" href="{% static 'css/custom.css' %}?v={{ deployment_timestamp|default:'1' }}">
```
`deployment_timestamp` comes from `core/context_processors.py::project_context` as
`int(time.time())` — meaning every Django request generates a new query-string value.
Cloudflare treats each new `?v=...` value as a new URL → `cf-cache-status: MISS`
fresh fetch from the VM. Users always see the latest CSS as soon as the Django
process restarts.
**Trade-off**: because the timestamp changes every second, CDN cache-hit rate on
CSS is effectively zero. For a low-traffic app this is fine. If traffic grows,
consider switching to a file-mtime-based token so the URL only changes when the
CSS actually changes.
`deployment_timestamp` comes from `core/context_processors.py::project_context`
via `_compute_cache_bust_token()` — returns the mtime of
`static/css/custom.css` as an integer. The token only changes when the
CSS file is modified, so Cloudflare's edge cache holds each version for
its full 4h TTL and repeat page loads in a session hit the browser
cache (304 Not Modified). Deploys that include a CSS change bump the
mtime → new token → cache busts. Pre-24-Apr-2026 this was
`int(time.time())` per-request, which defeated the CDN cache entirely
(effectively 0% hit rate on CSS). Degraded-mode fallback: if
`custom.css` isn't on disk (e.g., fresh container before
`collectstatic`), the function falls back to the old per-request
timestamp rather than crashing.
### The pitfall this replaced
Pre-Apr 2026, the template used `{{ request.timestamp|default:'1.0' }}`. But
@ -263,9 +265,10 @@ that only a hard refresh in incognito temporarily fixed. Never use `request.time
in templates — it doesn't exist.
### When CSS changes don't appear on production
1. Confirm Django is rendering the new URL: `curl -s https://foxlog.flatlogic.app/ | grep -oE 'custom\.css\?v=[^"]+'`the `v=` number should change per request (or at least per restart)
1. Confirm Django is rendering a stable URL: `curl -s https://foxlog.flatlogic.app/ | grep -oE 'custom\.css\?v=[^"]+'`run it twice; the `v=` number must be IDENTICAL across requests. Under the mtime-based token (see previous subsection), the number only changes after `static/css/custom.css` is edited. If it DOES change every request, the fallback branch of `_compute_cache_bust_token()` is active (the CSS file couldn't be stat'd) — check the file exists and is readable.
2. Confirm the CDN honours it: `curl -sI "https://foxlog.flatlogic.app/static/css/custom.css?cb=test$(date +%s)" | grep -i cache` — expect `cf-cache-status: MISS` then `HIT` on repeat
3. If the Django URL still looks like `?v=1.0` (constant), `deployment_timestamp` isn't being injected — check that `core.context_processors.project_context` is listed in `TEMPLATES[0]['OPTIONS']['context_processors']` in `config/settings.py`
3. If you've just deployed a CSS change and the old file is still showing: confirm `collectstatic` ran on the VM after pull (Flatlogic doesn't auto-run it). The token is read from `static/css/custom.css` (the SOURCE file) — so editing the source and pushing DOES bump the token, and Cloudflare correctly misses its edge cache on the next request. What breaks is the VM's response to that miss: Apache serves the OLD bytes from `staticfiles/` because `collectstatic` hasn't refreshed the collected copy. You'll see a URL with a new `?v=...` value (confirming the token bumped) but stale bytes behind it. Fix: `python3 manage.py collectstatic --noinput`, then restart the service.
4. If `deployment_timestamp` isn't being injected at all (the `?v=` query string is missing from the rendered URL): check that `core.context_processors.project_context` is listed in `TEMPLATES[0]['OPTIONS']['context_processors']` in `config/settings.py`.
### `collectstatic` is required after CSS/JS changes on production
Flatlogic's rebuild does NOT automatically run `collectstatic`. If new CSS is on

View File

@ -300,3 +300,43 @@ if os.getenv('USE_SQLITE', 'false').lower() == 'true':
CSRF_COOKIE_SECURE = False
SESSION_COOKIE_SAMESITE = 'Lax'
CSRF_COOKIE_SAMESITE = 'Lax'
# === DEV-ONLY: Django Debug Toolbar ===
# Loaded ONLY when BOTH DEBUG=true AND USE_SQLITE=true AND we're not
# running tests. This is a deliberately strict gate — the toolbar
# exposes query internals, settings, and request state that should
# never appear in production. The USE_SQLITE half of the check acts
# as a belt-and-suspenders guard against accidentally enabling DEBUG
# on production (which would be its own serious problem, but at least
# wouldn't leak toolbar data).
#
# Test-run skip: Django forces DEBUG=False during `manage.py test`,
# which makes the toolbar emit its own E001 system-check error AND
# leaves template tags referencing the unregistered `djdt` URL
# namespace — both fatal to the test suite. Detecting the test
# command up-front and skipping the install entirely is cleaner than
# trying to work around both symptoms.
import sys as _sys
_IS_RUNNING_TESTS = 'test' in _sys.argv
if DEBUG and _IS_DEV and not _IS_RUNNING_TESTS:
try:
import debug_toolbar # noqa: F401 — probe for installed package
except ImportError:
pass
else:
INSTALLED_APPS += ['debug_toolbar']
# Insert the middleware as early as possible in the chain so it
# captures every request, but AFTER SecurityMiddleware (standard
# recommendation in the toolbar's install docs).
MIDDLEWARE.insert(1, 'debug_toolbar.middleware.DebugToolbarMiddleware')
INTERNAL_IPS = ['127.0.0.1', 'localhost']
DEBUG_TOOLBAR_CONFIG = {
# Don't auto-collapse the SQL panel — the SQL count is the
# main thing we check on every page.
'SHOW_COLLAPSED': False,
# This callback is invoked per-request by the toolbar's middleware.
# Since we only REACH this config block when the triple gate
# (DEBUG + _IS_DEV + not running tests) already passed at settings
# load time, there's nothing to re-check here — just return True.
'SHOW_TOOLBAR_CALLBACK': lambda request: True,
}

View File

@ -11,4 +11,14 @@ urlpatterns = [
if settings.DEBUG:
urlpatterns += static(settings.STATIC_URL, document_root=settings.STATIC_ROOT)
urlpatterns += static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)
urlpatterns += static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)
# === DEV-ONLY: Django Debug Toolbar URL include ===
# Matches the conditional load in settings.py. No-op in prod.
# The settings.py gate already guarantees debug_toolbar is importable
# by the time we get here, so no try/except is needed. If we ever
# downgrade to a toolbar version older than v4.0 (which introduced
# debug_toolbar_urls), swap this for `path('__debug__/', include(debug_toolbar.urls))`.
if 'debug_toolbar' in settings.INSTALLED_APPS:
from debug_toolbar.toolbar import debug_toolbar_urls
urlpatterns += debug_toolbar_urls()

View File

@ -1,13 +1,44 @@
# === core/context_processors.py ===
# Globals injected into every template render.
#
# `deployment_timestamp` is a cache-bust token on our CSS URL. Historically
# it was `int(time.time())` — a new value every second — which defeated
# Cloudflare's edge cache because every page load saw a different `?v=...`
# string. We now tie the token to `custom.css`'s mtime, so the URL
# changes ONLY when the CSS actually changes. Cloudflare can hold the
# file for its full 4h TTL, and users on repeat visits hit the browser
# cache (304 Not Modified).
import os
import time
from pathlib import Path
from django.conf import settings
# Path to the file whose mtime drives the cache-bust token. Module-level
# constant so tests can monkey-patch it to simulate "file missing".
_CSS_PATH_FOR_TOKEN = Path(settings.BASE_DIR) / 'static' / 'css' / 'custom.css'
def _compute_cache_bust_token():
"""Return an integer cache-bust token.
Normal path: returns the CSS file's mtime (an integer).
Fallback: if the file can't be stat'd (doesn't exist / permission
error / disk issue), returns the current wall-clock time in seconds
that degrades to the PRE-FIX behaviour (new token per request)
rather than crashing the whole request cycle.
"""
try:
return int(os.path.getmtime(_CSS_PATH_FOR_TOKEN))
except (OSError, FileNotFoundError):
return int(time.time())
def project_context(request):
"""
Adds project-specific environment variables to the template context globally.
"""
"""Adds project-specific environment variables to the template context globally."""
return {
"project_description": os.getenv("PROJECT_DESCRIPTION", ""),
"project_image_url": os.getenv("PROJECT_IMAGE_URL", ""),
# Used for cache-busting static assets
"deployment_timestamp": int(time.time()),
# Cache-busts static assets — see _compute_cache_bust_token().
"deployment_timestamp": _compute_cache_bust_token(),
}

View File

@ -5,6 +5,7 @@
import datetime
from decimal import Decimal
from django.conf import settings
from django.contrib.auth.models import User
from django.test import TestCase
from django.urls import reverse
@ -1237,3 +1238,144 @@ class AdjustmentsTabTests(TestCase):
self.assertIn((self.team.id, self.w2.id), pair_set)
self.assertIn((self.team2.id, self.w1.id), pair_set)
self.assertIn((self.team2.id, self.w2.id), pair_set)
# === Cache-bust token tests ===
# The deployment_timestamp context variable controls the ?v=... query
# string on our CSS URL. It MUST only change when custom.css changes
# — otherwise Cloudflare cache-HIT rate on the CSS drops to zero and
# every page reload re-fetches 64 KB from the VM.
class CacheBustTokenTests(TestCase):
"""Regression tests for the mtime-based cache-bust token."""
def test_token_is_an_integer(self):
"""Token must be int (templates cast to str; float would show a dot)."""
from core.context_processors import project_context
ctx = project_context(request=None)
self.assertIsInstance(ctx['deployment_timestamp'], int)
def test_token_is_stable_across_two_calls(self):
"""Critical property: two back-to-back calls return the same
token because custom.css hasn't changed between them. This
is the entire point of the mtime-based approach."""
from core.context_processors import project_context
t1 = project_context(request=None)['deployment_timestamp']
t2 = project_context(request=None)['deployment_timestamp']
self.assertEqual(t1, t2)
def test_token_falls_back_if_file_missing(self):
"""If static/css/custom.css is somehow missing (fresh
container pre-collectstatic), we must NOT crash. We fall back
to int(time.time()) so every page still renders."""
import core.context_processors as cp
try:
# Monkey-patch so the function sees a guaranteed-missing path.
# (The function itself is not patched — only the path constant.)
cp._CSS_PATH_FOR_TOKEN = cp.Path('/definitely/does/not/exist.css')
# Should return an int and NOT raise.
token = cp._compute_cache_bust_token()
self.assertIsInstance(token, int)
finally:
# Reset the module-level path constant so other tests (or reruns)
# get the real CSS file back.
cp._CSS_PATH_FOR_TOKEN = cp.Path(settings.BASE_DIR) / 'static' / 'css' / 'custom.css'
# === REGRESSION: adjustment project-attribution double-count ===
# Ticket summary: commit 61c485f replaced a per-project loop using SQL
# OR (`Q(project=P) | Q(work_log__project=P)`) with two SEPARATE
# filter+GROUP BY queries that were summed in Python. Any adjustment
# with BOTH `project_id` AND `work_log.project_id` set matched both
# queries and got counted TWICE.
#
# Every Overtime adjustment fits that shape — `price_overtime()` in
# views.py sets both FKs to the same project. So every unpaid
# Overtime silently inflated the outstanding-costs dashboard by its
# own amount. This test locks in the "count once" behaviour.
class PayrollDashboardAdjustmentAggregationTests(TestCase):
"""Regression tests for the per-project adjustment aggregation used by
the payroll dashboard. An adjustment with both `project` and
`work_log.project` set must contribute its amount ONCE to that
project, not twice."""
def setUp(self):
# Admin so the payroll_dashboard view accepts us (is_admin helper
# returns True for is_staff OR is_superuser).
self.admin = User.objects.create_user(
username='double-count-admin',
password='pw',
is_staff=True,
is_superuser=True,
)
self.client.force_login(self.admin)
# One active project, one worker, one fresh work log.
self.project = Project.objects.create(
name='Double-Count Test Site',
start_date=datetime.date(2026, 4, 1),
active=True,
)
self.worker = Worker.objects.create(
name='DC Worker',
id_number='DC1',
monthly_salary=Decimal('10000'),
)
self.work_log = WorkLog.objects.create(
date=datetime.date(2026, 4, 23),
project=self.project,
supervisor=self.admin,
)
# NOTE: deliberately no workers on the log — we do NOT want
# unpaid-log wage cost to pollute this test; we only want to
# measure the adjustment contribution.
# THE KEY SHAPE: one unpaid adjustment with BOTH FKs set to the
# same project. This mirrors how price_overtime() creates every
# Overtime adjustment (adj.project = worklog.project, and
# adj.work_log = worklog).
PayrollAdjustment.objects.create(
worker=self.worker,
type='Overtime', # additive → contributes positively
amount=Decimal('500'),
project=self.project,
work_log=self.work_log,
date=datetime.date(2026, 4, 23),
description='Regression-test Overtime',
)
def test_overtime_with_both_project_and_work_log_counts_once(self):
"""An unpaid Overtime with BOTH project + work_log.project set
must contribute its R500 amount once to outstanding-costs
not R1000.
Before the Coalesce fix, the batched aggregates summed the
direct-project group + the work_log-project group separately
in Python, so an Overtime adjustment landed in both and got
double-counted. After the fix (Coalesce('project_id',
'work_log__project_id')) each adjustment row is attributed to
exactly one effective project.
"""
response = self.client.get(reverse('payroll_dashboard'))
self.assertEqual(response.status_code, 200)
outstanding = response.context['outstanding_project_costs']
# Find the entry for our test project by name (the shape is
# {'name': str, 'cost': Decimal} — no 'id' key).
ours = next(
(p for p in outstanding if p['name'] == self.project.name),
None,
)
self.assertIsNotNone(
ours,
"Test project should appear in outstanding_project_costs "
"(its unpaid Overtime is non-zero).",
)
self.assertEqual(
ours['cost'],
Decimal('500'),
"Overtime adjustment with both project + work_log.project "
"FKs set must count ONCE (R500), not twice (R1000). If "
"this fails with R1000 the project-attribution double-"
"count bug has reappeared.",
)

View File

@ -12,7 +12,7 @@ from django.shortcuts import render, redirect, get_object_or_404
from django.utils import timezone
from django.db import transaction
from django.db.models import Sum, Count, Q, F, Prefetch, Max, Min
from django.db.models.functions import TruncMonth
from django.db.models.functions import Coalesce, TruncMonth
from django.contrib import messages
from django.contrib.auth.decorators import login_required
from django.http import JsonResponse, HttpResponseForbidden, HttpResponse
@ -2617,6 +2617,30 @@ def payroll_dashboard(request):
pending_adj_sub_total = Decimal('0.00') # Unpaid deductive adjustments
all_ot_data = [] # For the Price Overtime modal
# === PRE-COMPUTED LOOKUPS — avoid per-worker SELECTs in the loop below ===
# Previously the loop fired:
# - one `Loan.objects.filter(worker=w, active=True).exists()` per worker
# - one `worker.teams.filter(active=True).first()` per worker (via
# get_worker_active_team) — which fires a fresh SELECT even though
# active_workers was prefetched, because `.filter()` bypasses the
# prefetch cache.
# We batch both into dict lookups keyed by worker_id.
workers_with_active_loan = set(
Loan.objects.filter(active=True).values_list('worker_id', flat=True).distinct()
)
# Map worker_id → first active Team instance (mirrors get_worker_active_team).
# We load every active team once, then walk the through-table to find the
# first active team per worker.
active_team_by_id = {t.id: t for t in Team.objects.filter(active=True)}
worker_active_team = {}
for membership in Team.workers.through.objects.filter(
team_id__in=active_team_by_id.keys()
).values('team_id', 'worker_id'):
wid = membership['worker_id']
if wid in worker_active_team:
continue
worker_active_team[wid] = active_team_by_id[membership['team_id']]
for worker in active_workers:
# Find unpaid work logs for this worker.
# A log is "unpaid for this worker" if no PayrollRecord links
@ -2668,7 +2692,8 @@ def payroll_dashboard(request):
# --- Overdue detection ---
# A worker is "overdue" if they have unpaid work from a completed pay period.
# Uses their team's pay schedule to determine the cutoff date.
team = get_worker_active_team(worker)
# PERF: team lookup via pre-computed dict (no per-worker SELECT).
team = worker_active_team.get(worker.id)
team_name = team.name if team else ''
earliest_unpaid = min((l.date for l in unpaid_logs), default=None) if unpaid_logs else None
is_overdue = False
@ -2678,7 +2703,8 @@ def payroll_dashboard(request):
cutoff = period_start - datetime.timedelta(days=1)
is_overdue = earliest_unpaid <= cutoff
has_loan = Loan.objects.filter(worker=worker, active=True).exists()
# PERF: loan membership via pre-computed set (no per-worker SELECT).
has_loan = worker.id in workers_with_active_loan
# Most recent project — used by the "Adjust" button to pre-select project
last_project_id = unpaid_logs[-1].project_id if unpaid_logs else None
@ -2718,31 +2744,16 @@ def payroll_dashboard(request):
# --- Outstanding cost per project ---
# Check per-worker: a WorkLog is "unpaid for worker X" if no PayrollRecord
# links BOTH that log AND that worker. This handles partially-paid logs.
outstanding_project_costs = []
for project in Project.objects.filter(active=True):
project_outstanding = Decimal('0.00')
# Unpaid work log costs — check each worker individually
for log in project.work_logs.prefetch_related('payroll_records', 'workers').all():
paid_worker_ids = {pr.worker_id for pr in log.payroll_records.all()}
for w in log.workers.all():
if w.id not in paid_worker_ids:
project_outstanding += w.daily_rate
# Unpaid adjustments for this project
unpaid_adjs = PayrollAdjustment.objects.filter(
payroll_record__isnull=True
).filter(Q(project=project) | Q(work_log__project=project))
for adj in unpaid_adjs:
if adj.type in ADDITIVE_TYPES:
project_outstanding += adj.amount
elif adj.type in DEDUCTIVE_TYPES:
project_outstanding -= adj.amount
if project_outstanding != 0:
outstanding_project_costs.append({
'name': project.name,
'cost': project_outstanding,
})
#
# PERF: materialise the active-project list once and reuse it for both
# the outstanding-costs loop and the chart-data loop below. Previously
# each loop re-queried `Project.objects.filter(active=True)`, firing the
# same SELECT twice per dashboard render.
active_projects_list = list(Project.objects.filter(active=True))
active_project_ids = [p.id for p in active_projects_list]
# --- Chart data: last 6 months ---
# === CHART DATE-WINDOW SETUP (moved up so the batched queries below can
# also use it) ===
today = timezone.now().date()
chart_months = []
for i in range(5, -1, -1):
@ -2756,6 +2767,81 @@ def payroll_dashboard(request):
chart_labels = [
datetime.date(y, m, 1).strftime('%b %Y') for y, m in chart_months
]
six_months_ago_date = datetime.date(chart_months[0][0], chart_months[0][1], 1)
# === BATCHED AGGREGATES: one SQL query per concept instead of per-project ===
# Previously we looped over each active project and issued:
# - 1 SELECT of WorkLog (with workers prefetch) per project
# - 1 SELECT of PayrollAdjustment (unpaid) per project
# - 1 SELECT of WorkLog (workers prefetch) per project × 6 months
# - 1 SELECT of PayrollAdjustment (paid) per project × 6 months
# On a ~7-project dataset that's ~7+7+42+42 ≈ 98 SQL round-trips.
# The rewrite replaces those with 4 GROUP-BY queries that return
# project_id (and month, where relevant) → total, plus one query for
# per-log paid-worker sets.
# --- 1. Unpaid-work-log cost per project ---
# We can't do pure SQL aggregation for this because a WorkLog can be
# partially paid (one worker of two). We still need per-log inspection,
# BUT we can load all unpaid-or-partially-paid logs + their workers +
# payroll_records in a bounded set of queries using prefetch_related
# rather than looping one project at a time.
project_outstanding_map = {pid: Decimal('0.00') for pid in active_project_ids}
all_project_logs = WorkLog.objects.filter(
project_id__in=active_project_ids
).prefetch_related('payroll_records', 'workers')
for log in all_project_logs:
paid_worker_ids = {pr.worker_id for pr in log.payroll_records.all()}
for w in log.workers.all():
if w.id not in paid_worker_ids:
project_outstanding_map[log.project_id] += w.daily_rate
# --- 2. Unpaid-adjustment net per project (ONE GROUP BY via Coalesce) ---
# Each unpaid adjustment is attributed to exactly ONE project: its
# direct FK (`project_id`) if set, otherwise its work_log's project.
# This mirrors the original OR-filter semantics
# (`Q(project=P) | Q(work_log__project=P)`) which naturally dedupes:
# a row with BOTH FKs pointing at the same project matches ONCE.
#
# Why Coalesce matters: every Overtime adjustment is created by
# price_overtime() with BOTH adj.project AND adj.work_log.project set
# to the same project. A naive "filter by direct FK + filter by
# work_log FK + sum both maps in Python" approach double-counts
# every Overtime row. Coalesce picks ONE effective project per row
# so each amount contributes to the outstanding-costs card once.
unpaid_adj_rows = (
PayrollAdjustment.objects
.filter(payroll_record__isnull=True)
.filter(
Q(project_id__in=active_project_ids)
| Q(work_log__project_id__in=active_project_ids)
)
.annotate(
effective_project_id=Coalesce('project_id', 'work_log__project_id')
)
.values('effective_project_id', 'type')
.annotate(total=Sum('amount'))
)
for row in unpaid_adj_rows:
pid = row['effective_project_id']
# Only contribute to active projects we're tracking.
if pid not in project_outstanding_map:
continue
total = row['total'] or Decimal('0.00')
if row['type'] in ADDITIVE_TYPES:
project_outstanding_map[pid] += total
elif row['type'] in DEDUCTIVE_TYPES:
project_outstanding_map[pid] -= total
outstanding_project_costs = []
for project in active_projects_list:
cost = project_outstanding_map[project.id]
if cost != 0:
outstanding_project_costs.append({
'name': project.name,
'cost': cost,
})
# Monthly payroll totals
paid_by_month_qs = PayrollRecord.objects.annotate(
@ -2767,28 +2853,72 @@ def payroll_dashboard(request):
}
chart_totals = [paid_by_month.get((y, m), 0) for y, m in chart_months]
# Per-project monthly costs (for stacked bar chart)
# --- 3. Per-project × per-month work-log cost (for stacked bar chart) ---
# Aggregate worker×log rows directly in SQL: one GROUP BY
# (project_id, year, month) returns all we need.
project_month_wage = {
(pid, y, m): Decimal('0.00')
for pid in active_project_ids for y, m in chart_months
}
wage_rows = WorkLog.objects.filter(
project_id__in=active_project_ids,
date__gte=six_months_ago_date,
).annotate(month=TruncMonth('date')).values(
'project_id', 'month', 'workers__monthly_salary'
).annotate(worker_count=Count('workers'))
# Each row = one (project, month, distinct salary) with how many workers
# at that salary were logged. Multiply by daily_rate (salary / 20) × count.
for row in wage_rows:
salary = row['workers__monthly_salary']
if salary is None:
continue
key = (row['project_id'], row['month'].year, row['month'].month)
if key not in project_month_wage:
continue
daily = Decimal(salary) / Decimal('20.00')
project_month_wage[key] += daily * row['worker_count']
# --- 4. Per-project × per-month paid-adjustment net (ONE GROUP BY) ---
# Same Coalesce trick as site #2: pick ONE effective project per
# adjustment row so we don't double-count the stacked-chart bars
# for Overtime (which always has both FKs pointing at the same
# project — see note on the unpaid block above).
paid_adj_rows = (
PayrollAdjustment.objects
.filter(payroll_record__isnull=False, date__gte=six_months_ago_date)
.filter(
Q(project_id__in=active_project_ids)
| Q(work_log__project_id__in=active_project_ids)
)
.annotate(
effective_project_id=Coalesce('project_id', 'work_log__project_id'),
month=TruncMonth('date'),
)
.values('effective_project_id', 'month', 'type')
.annotate(total=Sum('amount'))
)
# Accumulate add/sub per (project, year, month) keys in Python.
paid_adj_add = {}
paid_adj_sub = {}
for row in paid_adj_rows:
pid = row['effective_project_id']
if pid not in project_outstanding_map: # only active projects
continue
key = (pid, row['month'].year, row['month'].month)
total = row['total'] or Decimal('0.00')
if row['type'] in ADDITIVE_TYPES:
paid_adj_add[key] = paid_adj_add.get(key, Decimal('0.00')) + total
elif row['type'] in DEDUCTIVE_TYPES:
paid_adj_sub[key] = paid_adj_sub.get(key, Decimal('0.00')) + total
project_chart_data = []
for project in Project.objects.filter(active=True):
for project in active_projects_list:
monthly_data = []
for y, m in chart_months:
month_cost = Decimal('0.00')
month_logs = project.work_logs.filter(
date__year=y, date__month=m
).prefetch_related('workers')
for log in month_logs:
for w in log.workers.all():
month_cost += w.daily_rate
# Include paid adjustments for this project in this month
paid_adjs = PayrollAdjustment.objects.filter(
payroll_record__isnull=False,
date__year=y, date__month=m,
).filter(Q(project=project) | Q(work_log__project=project))
for adj in paid_adjs:
if adj.type in ADDITIVE_TYPES:
month_cost += adj.amount
elif adj.type in DEDUCTIVE_TYPES:
month_cost -= adj.amount
key = (project.id, y, m)
month_cost = project_month_wage.get(key, Decimal('0.00'))
month_cost += paid_adj_add.get(key, Decimal('0.00'))
month_cost -= paid_adj_sub.get(key, Decimal('0.00'))
monthly_data.append(float(month_cost))
if any(v > 0 for v in monthly_data):
project_chart_data.append({
@ -2801,9 +2931,9 @@ def payroll_dashboard(request):
# This powers the "By Worker" toggle on the Monthly Payroll Totals chart.
# Only ~14 workers x 6 months = tiny dataset, so we embed it all as JSON
# and switching between workers is instant (no server round-trips).
# Starting date for the 6-month window (first day of the oldest chart month)
six_months_ago_date = datetime.date(chart_months[0][0], chart_months[0][1], 1)
#
# `six_months_ago_date` is already defined above (hoisted next to the
# date-window setup) and reused here.
# Query 1: Total amount paid per worker per month.
# Uses database-level grouping — one query for ALL workers at once.
@ -2848,8 +2978,13 @@ def payroll_dashboard(request):
# Base pay is reverse-engineered from the net total:
# amount_paid = base + overtime + bonus + new_loan - deduction - loan_repayment - advance
# So: base = amount_paid - overtime - bonus - new_loan + deduction + loan_repayment + advance
#
# PERF: reuse `active_workers` (already loaded + cached at the top of the
# function) instead of re-querying Worker.objects.filter(active=True).
# Same ordered row-set; saves an SQL round-trip. The unused prefetches
# on `active_workers` are already materialised so they cost nothing extra.
worker_chart_data = {}
for worker in Worker.objects.filter(active=True).order_by('name'):
for worker in active_workers:
months_data = []
has_any_data = False
@ -2903,16 +3038,25 @@ def payroll_dashboard(request):
)['total'] or Decimal('0.00')
# --- Active projects and workers for modal dropdowns ---
# `active_workers` is reused (already loaded + evaluated by the workers_data
# loop). For the modal-dropdown context key we alias it as `all_workers`
# so the template name stays descriptive.
all_workers = active_workers
active_projects = Project.objects.filter(active=True).order_by('name')
all_workers = Worker.objects.filter(active=True).order_by('name')
all_teams = Team.objects.filter(active=True).prefetch_related('workers').order_by('name')
all_teams = Team.objects.filter(active=True).prefetch_related(
# PERF: prefetch only the active workers so the template's
# `team.workers.all` (and our map below) already filters to active
# without re-querying. Using `.filter()` on the plain `workers`
# accessor bypasses Django's prefetch cache and fires one SELECT
# per team — an N+1 we need to avoid.
Prefetch('workers', queryset=Worker.objects.filter(active=True), to_attr='active_workers_cached')
).order_by('name')
# Team-workers map for auto-selecting workers when a team is picked
# Team-workers map for auto-selecting workers when a team is picked.
# Uses the prefetched `active_workers_cached` list — no extra queries.
team_workers_map = {}
for team in all_teams:
team_workers_map[str(team.id)] = list(
team.workers.filter(active=True).values_list('id', flat=True)
)
team_workers_map[str(team.id)] = [w.id for w in team.active_workers_cached]
# NOTE: Pass raw Python objects here, NOT json.dumps() strings.
# The template uses Django's |json_script filter which handles
@ -3026,10 +3170,17 @@ def payroll_dashboard(request):
# main sort key has ties (e.g. two adjustments on the same date).
adjustments = adjustments.order_by(sort_field, '-id')
# --- Stats cards (all computed BEFORE pagination) ---
# --- Pagination: 50 rows per page (flat view only) ---
# PERF: build the paginator first so we can reuse its cached `count`
# for the "Total adjustments" stat card below — avoids a duplicate
# `SELECT COUNT(*) FROM core_payrolladjustment`.
paginator = Paginator(adjustments, 50)
adj_page = paginator.get_page(request.GET.get('page', 1))
# --- Stats cards (all computed BEFORE pagination cuts the rows) ---
# These numbers always reflect what the current filter produces,
# not just what fits on the current page.
adj_total_count = adjustments.count()
adj_total_count = paginator.count
unpaid_qs = adjustments.filter(payroll_record__isnull=True)
adj_unpaid_count = unpaid_qs.count()
adj_unpaid_sum = unpaid_qs.aggregate(
@ -3054,10 +3205,6 @@ def payroll_dashboard(request):
if group_by in ('type', 'worker'):
adj_groups = _group_adjustments(list(adjustments), group_by)
# --- Pagination: 50 rows per page (flat view only) ---
paginator = Paginator(adjustments, 50)
adj_page = paginator.get_page(request.GET.get('page', 1))
# --- Everything the Adjustments tab template will need ---
context.update({
'adj_page': adj_page,
@ -3083,8 +3230,11 @@ def payroll_dashboard(request):
# 'adjustment_types' context var (which is TYPE_CHOICES tuples
# used by the Add/Edit adjustment modals).
'adj_type_choices': list(ADDITIVE_TYPES) + list(DEDUCTIVE_TYPES),
'all_workers_for_filter': Worker.objects.filter(active=True).order_by('name'),
'all_teams_for_filter': Team.objects.filter(active=True).order_by('name'),
# PERF: reuse `all_workers`/`all_teams` (already cached above for
# the Add-Adjustment modal) — same row-set, same ordering, so no
# need to re-query the database for the filter popovers.
'all_workers_for_filter': all_workers,
'all_teams_for_filter': all_teams,
# Task 4 will use this to decide +/- signs on each row.
'additive_types': list(ADDITIVE_TYPES),
# === CROSS-FILTER SOURCE: (team_id, worker_id) PAIRS ===

View File

@ -0,0 +1,175 @@
# Perf Quick-Wins Pass — Design (24 Apr 2026)
## Origin
Konrad, after a long stretch of feature work (Inline Filters + Adjustments
tab + filter-bar v2):
> _"the app feel a bit sluggish especially changing between main spaces.
> Go through the app systematically and look for bugs and un optimized
> code. systematically go through the code and expertly and thoroughly
> review and fix it."_
Presented three scopes (A quick-wins / B focused dashboard pass / C full
systematic audit). Konrad picked **A — quick-wins first**, on the
principle that perf work is notorious for "big rewrite that didn't help."
If A moves the needle, we can stop; if not, we escalate with evidence.
## Goal
Make navigation between main spaces (Dashboard ↔ Payroll ↔ Workers ↔
Report ↔ Teams ↔ Projects) feel snappier. Ship in 1-3 commits. No
architecture changes. Every change individually revertible.
## Who it's for
Everyone who uses the app — most immediately Konrad, who navigates
between Dashboard and Payroll dozens of times a day.
## What we already know (pre-measurement)
- `payroll_dashboard.html` is 213 KB / 4,147 lines — all 4 tabs rendered
server-side even when only one is visible. Addressed in plan B, not A.
- `deployment_timestamp` context var is `int(time.time())` per-request
`custom.css?v=<timestamp>` is a new URL every second → Cloudflare
edge-cache HIT rate on CSS is effectively 0 → every page load fetches
64 KB of CSS from the VM. Documented as a trade-off in CLAUDE.md.
This is almost certainly the biggest single contributor to the
"heavy navigation" feel.
- 49 `select_related`/`prefetch_related` calls vs 91
`.all()/.first()/.count()` calls in `views.py`. Not damning but worth
pointing at hot-path views.
## Scope — 4 changes, in order
### 1. Fix `deployment_timestamp` to bust cache only on real deploys
**File:** `core/context_processors.py`
Today:
```python
'deployment_timestamp': int(time.time()),
```
After:
```python
# Cache-bust token tied to the CSS file's mtime — only changes when
# custom.css actually changes. Falls back to int(time.time()) if the
# file isn't on disk yet (fresh container, pre-collectstatic).
try:
_css_path = settings.BASE_DIR / 'static' / 'css' / 'custom.css'
_token = int(os.path.getmtime(_css_path))
except (OSError, FileNotFoundError):
_token = int(time.time())
```
Effect: the `?v=...` query string stays constant across requests until
`custom.css` is modified. Cloudflare can finally hold the file at its
edge for its full 4h TTL. Repeat navigation within a session drops from
"fetch 64 KB from VM" to "304 Not Modified" from the browser cache,
after the first hit in a 4h window.
**Degraded-mode guarantee:** if the file is missing (shouldn't happen in
normal dev or prod, but could in a fresh container), we degrade to
today's behaviour (per-request timestamp) rather than crash.
### 2. Profile + fix N+1 on the two busiest pages
**Pages:** `/` (dashboard) and `/payroll/` (payroll dashboard — all 4
tabs — Pending, History, Loans, Adjustments).
**Tool:** Django Debug Toolbar, added to `requirements.txt` as a
dev-only dependency. Gated in `config/settings.py` so it only
initialises when `DJANGO_DEBUG=true` AND `USE_SQLITE=true` (never
loads in prod).
**Process:**
1. Install toolbar, confirm the SQL panel loads on `/`.
2. Navigate to `/`, read the SQL tab: flag any query count > ~20,
any row with `+N duplicate queries`, any view of the queryset
that could be answered with `select_related`/`prefetch_related`/
`annotate(Count/Sum)`.
3. Fix each flag with the minimal ORM change. One fix = one commit.
4. Re-run, confirm query count dropped, confirm no test regressions.
5. Repeat for `/payroll/?status=pending`, `/payroll/?status=history`,
`/payroll/?status=loans`, `/payroll/?status=adjustments`.
**Likely suspects** (prediction — to be confirmed by toolbar):
- **Dashboard cert-expiry card** — aggregates expired/expiring certs
across active workers. If it loops in Python instead of `annotate`-
plus-filter, that's an N+1.
- **Pending payments table** — worker + team + overdue calc per row.
The overdue check calls `get_pay_period(team)` per worker; if teams
aren't prefetched we're firing one SELECT per row.
- **Adjustments tab groupings** — we fixed `worker.teams.first()`
`worker.teams.all()` once already (commit `06b3315`); worth
double-checking grouped view for similar patterns.
**Out of scope for this step:** any fix that requires a template
rewrite. If something needs more than a `.select_related()` /
`.prefetch_related()` / `.annotate()` tweak, it goes on the plan-B list.
### 3. Double-check WeasyPrint is not eager-imported anywhere
**File:** `core/utils.py`, `core/views.py`.
We already lazy-import WeasyPrint in `render_to_pdf()` (per CLAUDE.md).
I'll grep to confirm nothing else on the app-boot path imports
`weasyprint` at module level. If anything does, move it into a function
body. 10 minutes, zero-risk.
### 4. Commit message includes before/after measurement
The final commit's message records:
- Page size bytes (DOM serialized) for `/` and `/payroll/` before & after
- Network request count on a cold cache hit
- SQL query count on both pages
If the numbers don't materially improve after steps 1-3, I stop. We
don't press on to plan B without evidence that plan A helped (or at
least surfaced what's actually slow).
## What I will NOT touch in this pass
- Splitting `payroll_dashboard.html` into tab partials
- Any refactoring of `views.py` or extraction of helpers
- Any visual / UX change
- Tests — query-count changes don't break the existing tests (they
assert URL contract + output shape, not query plans). If a test
genuinely needs updating because I materially changed a view's
behaviour, I'll note why in the commit
## Risks + rollback
All four changes are individually revertible. Biggest risks:
- **mtime-based token misfires on fresh containers** — mitigated by
try/except fallback to today's behaviour
- **A `select_related` fix changes query semantics** — e.g., eager
loading a nullable FK that used to be accessed lazily-with-None. Low
risk on Django's ORM, but the test suite (65 tests, all passing at
HEAD `503eff6`) will catch any behavioural regression
- **Django Debug Toolbar pulled in in prod** — mitigated by double-gate
(DEBUG=true AND USE_SQLITE=true) in `config/settings.py`
Rollback: `git revert <sha>` on the offending commit. No data, schema,
or URL-contract impact.
## Out of scope (explicit non-goals)
- Plan B / C items (template splitting, written baseline doc, whole-app
measurement)
- Moving CDN assets to local / self-hosted
- Changing Flatlogic's `runserver` → gunicorn
- Turning on HTTP/2 push, service workers, or other frontend perf tooling
- Any refactor that requires a migration
## Next step
Generate an implementation plan via the writing-plans skill
(task-by-task, bite-sized steps) and then execute via
subagent-driven-development. Auto mode is active — proceed
continuously, no mid-execution checkpoints (plan A is 4 small
mechanical changes; a checkpoint adds overhead without value).
Ship alongside current `ai-dev` HEAD (`503eff6`) in the same branch.

View File

@ -0,0 +1,542 @@
# Perf Quick-Wins Pass — Implementation Plan
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development to implement this plan task-by-task. Auto mode is active — execute continuously, no mid-execution checkpoints.
**Goal:** Make main-space navigation feel snappier via targeted ORM + static-asset fixes. No architecture change.
**Architecture:** Four small mechanical changes: (1) mtime-based CSS cache-bust token so Cloudflare can actually cache `custom.css`; (2) install & gate Django Debug Toolbar as a dev-only profiling tool; (3) fix any N+1 hotspots the toolbar surfaces on `/` and `/payroll/` via targeted `select_related`/`prefetch_related`/`annotate`; (4) record before/after numbers in the final commit message.
**Tech Stack:** Django 5.2.7 ORM; `django-debug-toolbar` (new dev-only dep); `os.path.getmtime` for static-asset cache-busting.
**Design doc:** `docs/plans/2026-04-24-perf-quick-wins-design.md` (committed as `d1490c4`).
**Starting HEAD:** `d1490c4` on branch `ai-dev` (clean working tree).
**Expected net change:** ~100-200 lines across 5-6 files + 1 requirements bump.
---
## Task 1: mtime-based cache-bust token + regression test
**Goal:** The `deployment_timestamp` context var changes only when `static/css/custom.css` actually changes, not on every HTTP request.
**Files:**
- Modify: `core/context_processors.py` (replace `int(time.time())` with `_compute_cache_bust_token()`)
- Test: `core/tests.py` (add `CacheBustTokenTests` class)
**Step 1: Write the failing tests**
Append to `core/tests.py` (find a spot after existing context-processor / URL-contract tests; conventional place is at the bottom of the file):
```python
# === Cache-bust token tests ===
# The deployment_timestamp context variable controls the ?v=... query
# string on our CSS URL. It MUST only change when custom.css changes
# — otherwise Cloudflare cache-HIT rate on the CSS drops to zero and
# every page reload re-fetches 64 KB from the VM.
class CacheBustTokenTests(TestCase):
"""Regression tests for the mtime-based cache-bust token."""
def test_token_is_an_integer(self):
"""Token must be int (templates cast to str; float would show a dot)."""
from core.context_processors import project_context
ctx = project_context(request=None)
self.assertIsInstance(ctx['deployment_timestamp'], int)
def test_token_is_stable_across_two_calls(self):
"""Critical property: two back-to-back calls return the same
token — because custom.css hasn't changed between them. This
is the entire point of the mtime-based approach."""
from core.context_processors import project_context
t1 = project_context(request=None)['deployment_timestamp']
t2 = project_context(request=None)['deployment_timestamp']
self.assertEqual(t1, t2)
def test_token_falls_back_if_file_missing(self):
"""If static/css/custom.css is somehow missing (fresh
container pre-collectstatic), we must NOT crash. We fall back
to int(time.time()) so every page still renders."""
import core.context_processors as cp
original = cp._compute_cache_bust_token
try:
# Monkey-patch so the function sees a guaranteed-missing path.
cp._CSS_PATH_FOR_TOKEN = cp.Path('/definitely/does/not/exist.css')
# Should return an int and NOT raise.
token = cp._compute_cache_bust_token()
self.assertIsInstance(token, int)
finally:
# Reset the module-level cache so other tests get the real value.
cp._CSS_PATH_FOR_TOKEN = cp.Path(settings.BASE_DIR) / 'static' / 'css' / 'custom.css'
```
Add the `from django.conf import settings` import at the top of `core/tests.py` if not already present.
**Step 2: Run the tests to verify they fail**
```bash
USE_SQLITE=true DJANGO_DEBUG=true python manage.py test core.tests.CacheBustTokenTests -v 2
```
Expected: 3 failures (function `_compute_cache_bust_token` doesn't exist).
**Step 3: Implement the token function**
Replace the entire contents of `core/context_processors.py` with:
```python
# === core/context_processors.py ===
# Globals injected into every template render.
#
# `deployment_timestamp` is a cache-bust token on our CSS URL. Historically
# it was `int(time.time())` — a new value every second — which defeated
# Cloudflare's edge cache because every page load saw a different `?v=...`
# string. We now tie the token to `custom.css`'s mtime, so the URL
# changes ONLY when the CSS actually changes. Cloudflare can hold the
# file for its full 4h TTL, and users on repeat visits hit the browser
# cache (304 Not Modified).
import os
import time
from pathlib import Path
from django.conf import settings
# Path to the file whose mtime drives the cache-bust token. Module-level
# constant so tests can monkey-patch it to simulate "file missing".
_CSS_PATH_FOR_TOKEN = Path(settings.BASE_DIR) / 'static' / 'css' / 'custom.css'
def _compute_cache_bust_token():
"""Return an integer cache-bust token.
Normal path: returns the CSS file's mtime (an integer).
Fallback: if the file can't be stat'd (doesn't exist / permission
error / disk issue), returns the current wall-clock time in seconds
— that degrades to the PRE-FIX behaviour (new token per request)
rather than crashing the whole request cycle.
"""
try:
return int(os.path.getmtime(_CSS_PATH_FOR_TOKEN))
except (OSError, FileNotFoundError):
return int(time.time())
def project_context(request):
"""Adds project-specific environment variables to the template context globally."""
return {
"project_description": os.getenv("PROJECT_DESCRIPTION", ""),
"project_image_url": os.getenv("PROJECT_IMAGE_URL", ""),
# Cache-busts static assets — see _compute_cache_bust_token().
"deployment_timestamp": _compute_cache_bust_token(),
}
```
**Step 4: Run the tests to verify they pass**
```bash
USE_SQLITE=true DJANGO_DEBUG=true python manage.py test core.tests.CacheBustTokenTests -v 2
```
Expected: 3 passes.
**Step 5: Run the full test suite to catch any regression**
```bash
USE_SQLITE=true DJANGO_DEBUG=true python manage.py test core.tests -v 2
```
Expected: 68 passes (65 pre-existing + 3 new). If anything else fails, stop and investigate.
**Step 6: Update CLAUDE.md's "Static Assets & Cache-Busting" note**
In `CLAUDE.md`, find the section "How cache-busting works now" — specifically the paragraph beginning "`deployment_timestamp` comes from `core/context_processors.py::project_context` as `int(time.time())`..." and replace with:
```markdown
`deployment_timestamp` comes from `core/context_processors.py::project_context`
via `_compute_cache_bust_token()` — returns the mtime of
`static/css/custom.css` as an integer. The token only changes when the
CSS file is modified, so Cloudflare's edge cache holds each version for
its full 4h TTL and repeat page loads in a session hit the browser
cache (304 Not Modified). Deploys that include a CSS change bump the
mtime → new token → cache busts. Pre-24-Apr-2026 this was
`int(time.time())` per-request, which defeated the CDN cache entirely
(effectively 0% hit rate on CSS). Degraded-mode fallback: if
`custom.css` isn't on disk (e.g., fresh container before
`collectstatic`), the function falls back to the old per-request
timestamp rather than crashing.
```
**Step 7: Commit**
```bash
git add core/context_processors.py core/tests.py CLAUDE.md
git commit -m "$(cat <<'EOF'
perf(cache): mtime-based CSS cache-bust token
deployment_timestamp was int(time.time()) per-request, giving every
page load a new ?v=... query string on custom.css. Cloudflare treats
each unique URL as a new resource, so the CSS was fetched from the VM
on every page load — 64 KB over the wire per navigation.
Token now tied to static/css/custom.css mtime. The URL only changes
when the CSS actually changes, so Cloudflare can hold the file for
its full 4h TTL. Degraded-mode fallback preserves today's behaviour
if the file isn't on disk.
3 new CacheBustTokenTests; all 68 tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
EOF
)"
```
---
## Task 2: Install & gate Django Debug Toolbar (dev-only)
**Goal:** Be able to see SQL query count, duplicate-query warnings, and execution times on any page — but ONLY when DEBUG + USE_SQLITE, never in prod.
**Files:**
- Modify: `requirements.txt` (add one line)
- Modify: `config/settings.py` (conditional INSTALLED_APPS, MIDDLEWARE, INTERNAL_IPS, DEBUG_TOOLBAR_CONFIG)
- Modify: `config/urls.py` (conditional `debug_toolbar.urls` include)
**Step 1: Add the dependency**
Append to `requirements.txt`:
```
django-debug-toolbar==6.0.0
```
Then install:
```bash
pip install django-debug-toolbar==6.0.0
```
Expected: clean install, one new package.
**Step 2: Gate it in `config/settings.py`**
At the very bottom of `config/settings.py` (after the SQLite override block), append:
```python
# === DEV-ONLY: Django Debug Toolbar ===
# Loaded ONLY when BOTH DEBUG=true AND USE_SQLITE=true. This is a
# deliberately strict gate — the toolbar exposes query internals,
# settings, and request state that should never appear in production.
# The USE_SQLITE half of the check acts as a belt-and-suspenders guard
# against accidentally enabling DEBUG on production (which would be its
# own serious problem, but at least wouldn't leak toolbar data).
if DEBUG and _IS_DEV:
try:
import debug_toolbar # noqa: F401 — probe for installed package
except ImportError:
pass
else:
INSTALLED_APPS += ['debug_toolbar']
# Insert the middleware as early as possible in the chain so it
# captures every request, but AFTER SecurityMiddleware (standard
# recommendation in the toolbar's install docs).
MIDDLEWARE.insert(1, 'debug_toolbar.middleware.DebugToolbarMiddleware')
INTERNAL_IPS = ['127.0.0.1', 'localhost']
DEBUG_TOOLBAR_CONFIG = {
# Don't auto-collapse the SQL panel — the SQL count is the
# main thing we check on every page.
'SHOW_COLLAPSED': False,
# Explicit check so the toolbar ONLY renders when the hosting
# flags are still set (guards against stale cached pages).
'SHOW_TOOLBAR_CALLBACK': lambda request: DEBUG and _IS_DEV,
}
```
**Step 3: Gate it in `config/urls.py`**
Read `config/urls.py` first to see the current shape. Then at the bottom (after `urlpatterns` is defined), append:
```python
# === DEV-ONLY: Django Debug Toolbar URL include ===
# Matches the conditional load in settings.py. No-op in prod.
from django.conf import settings
if settings.DEBUG and getattr(settings, '_IS_DEV_DEBUG_TOOLBAR_ENABLED', False) is False:
# (settings.py sets no such flag; gate on INSTALLED_APPS membership.)
pass
if 'debug_toolbar' in settings.INSTALLED_APPS:
from debug_toolbar.toolbar import debug_toolbar_urls
urlpatterns += debug_toolbar_urls()
```
Note: `debug_toolbar_urls()` is the v6.x API (replaces the older pattern-include). If that import fails in v6, fall back to:
```python
if 'debug_toolbar' in settings.INSTALLED_APPS:
import debug_toolbar
urlpatterns += [path('__debug__/', include(debug_toolbar.urls))]
```
The implementer should try `debug_toolbar_urls()` first; if it errors, switch to the include form and move on.
**Step 4: Manual verification**
Start the dev server, load `/`, confirm the toolbar appears. If the page 500s or the toolbar doesn't render, stop and investigate before profiling.
```bash
run_dev.bat # or manually: set USE_SQLITE=true && set DJANGO_DEBUG=true && python manage.py runserver 0.0.0.0:8000
```
Then open `http://127.0.0.1:8000/` in a browser and confirm the toolbar tab appears on the right edge. Click SQL panel, note the query count.
Run Django's system check too, to catch any app-config issue:
```bash
USE_SQLITE=true DJANGO_DEBUG=true python manage.py check
```
Expected: `System check identified no issues (0 silenced).`
**Step 5: Run the full test suite**
```bash
USE_SQLITE=true DJANGO_DEBUG=true python manage.py test core.tests -v 2
```
Expected: still 68 passes. Toolbar shouldn't affect tests.
**Step 6: Commit**
```bash
git add requirements.txt config/settings.py config/urls.py
git commit -m "$(cat <<'EOF'
chore(dev): add Django Debug Toolbar (dev-only, DEBUG+USE_SQLITE gated)
Double-gated install: only loads when DEBUG=true AND USE_SQLITE=true,
never in prod. Lets us profile SQL query counts on the dashboard and
payroll pages before attacking N+1 hotspots.
requirements.txt adds django-debug-toolbar==6.0.0
config/settings.py conditionally appends to INSTALLED_APPS + MIDDLEWARE
config/urls.py conditionally includes __debug__ route
No behavioural change to production.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
EOF
)"
```
---
## Task 3: Profile & fix N+1 on `/` (dashboard)
**Goal:** Confirm the dashboard's SQL query count and fix any N+1 the toolbar surfaces.
**Files:**
- Modify: `core/views.py` (the `index()` view — starts line ~353, `certs_expired/expiring` aggregation around lines ~406-413)
- Possibly modify: `core/templates/core/index.html` (only if a template access is firing queries)
**Step 1: Baseline measurement**
Start the dev server. Log in as an admin user. Open `/`. In the Debug Toolbar panel:
- Note **total SQL query count** on the page
- Note **total SQL time**
- Expand the SQL tab, scan for any group marked "N+1" or "N duplicate queries"
- Note **DOM size / page weight** from DevTools Network tab (the HTML document itself, not total transfer)
Record these four numbers. They go into the Task 4 commit message.
**Step 2: Identify each problem**
The design doc predicts three suspects:
1. **Cert-expiry card** — two separate COUNT queries at views.py:406 and views.py:409. Could become one `.aggregate()` call.
2. **Stat cards** — if any stat card loops over a queryset in the template instead of using `.count()` on the queryset, that's an N+1.
3. **Pending payments table** (if it renders on this page for admins) — worker/team/log per row.
For each "N duplicate queries" group the toolbar shows, inspect the Python side:
- Find the view function (use the traceback in the toolbar if available)
- Decide whether the fix is `select_related` (FK), `prefetch_related` (M2M / reverse FK), or `annotate` (aggregate)
- Apply the minimal fix
**Step 3: Apply fixes**
For each confirmed N+1, edit `core/views.py` with the minimal change. Example shape (only as an illustration — apply whatever the toolbar surfaces):
```python
# BEFORE
workers = Worker.objects.filter(active=True)
for w in workers:
for cert in w.workercertificate_set.all(): # N+1 — one query per worker
...
# AFTER
workers = Worker.objects.filter(active=True).prefetch_related('workercertificate_set')
for w in workers:
for cert in w.workercertificate_set.all(): # reuses prefetched cache
...
```
For the cert counts specifically (views.py:406-413), the current code is:
```python
certs_expired_count = WorkerCertificate.objects.filter(
worker__is_active=True, valid_until__lt=today
).count()
certs_expiring_count = WorkerCertificate.objects.filter(
worker__is_active=True, valid_until__gte=today, valid_until__lte=today + timedelta(days=30)
).count()
certs_alert_total = certs_expired_count + certs_expiring_count
```
This is TWO queries, not an N+1. If the toolbar shows them as hot, consolidate into one `.aggregate()`:
```python
from django.db.models import Count, Q
_cert_counts = WorkerCertificate.objects.filter(
worker__is_active=True
).aggregate(
expired=Count('id', filter=Q(valid_until__lt=today)),
expiring=Count('id', filter=Q(valid_until__gte=today, valid_until__lte=today + timedelta(days=30))),
)
certs_expired_count = _cert_counts['expired']
certs_expiring_count = _cert_counts['expiring']
certs_alert_total = certs_expired_count + certs_expiring_count
```
But ONLY apply this if the toolbar flags it. Per YAGNI, don't pre-optimise. Two simple COUNTs with an index on `valid_until` are plenty fast.
**Step 4: Re-measure**
Reload `/`. The SQL query count should drop. If it didn't, your fix didn't hit the real hot path — revert that edit and look again at the toolbar output.
**Step 5: Run the full test suite**
```bash
USE_SQLITE=true DJANGO_DEBUG=true python manage.py test core.tests -v 2
```
Expected: 68 passes. A change in the ORM query plan does NOT typically move test counts, but a bug in a `prefetch_related` pattern can. If any test fails, revert and rethink.
**Step 6: Commit**
Only commit if ≥1 N+1 was actually found and fixed. If the toolbar showed `/` at ~20 queries with no N+1s flagged, skip Task 3 entirely and note that in the final commit message.
```bash
git add core/views.py [any template files touched]
git commit -m "$(cat <<'EOF'
perf(dashboard): fix N+1 on [specific problem identified]
Debug Toolbar showed [before-count] queries on /, including [N
duplicates of query X]. Added [select_related/prefetch_related/annotate]
to [view function] to fold those into one.
After: [after-count] queries. Test suite unaffected (68/68 pass).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
EOF
)"
```
---
## Task 4: Profile & fix N+1 on `/payroll/` (all 4 tabs) + before/after summary
**Goal:** Same as Task 3, applied to the payroll dashboard's four tabs, plus a final commit that captures the full before/after picture.
**Files:**
- Modify: `core/views.py` (the `payroll_dashboard()` view — starts line ~2593; each status branch: `pending`, `history`, `loans`, `adjustments`)
- Possibly: `core/templates/core/payroll_dashboard.html` (only if a template access fires queries)
**Step 1: Baseline measurement — all 4 tabs**
For each of the four URLs, record query count + SQL time:
- `/payroll/` (defaults to pending)
- `/payroll/?status=history`
- `/payroll/?status=loans`
- `/payroll/?status=adjustments`
**Step 2: Identify N+1s per tab**
Expected suspects (design doc predictions):
- **Pending tab** — for every worker in the outstanding list, `get_pay_period(team)` is called per worker. If `team` isn't prefetched via `select_related('team')` or `prefetch_related('teams')` somewhere upstream, each call hits the DB. Look at views.py around line 2676 (the pending-payments loop where `get_pay_period` is called).
- **History tab** — PayrollRecord list view; each row needs worker + work_logs + adjustments. Check for `prefetch_related('work_logs', 'payrolladjustment_set')` on the base queryset.
- **Loans tab** — Loan list with repayment adjustments. Check for `prefetch_related('payrolladjustment_set')` on active loans.
- **Adjustments tab** — we already fixed `worker.teams.first()``worker.teams.all()` in commit `06b3315`; confirm no new N+1 introduced by the sort / group work done in later Adjustments commits.
**Step 3: Apply fixes**
Same pattern as Task 3. One fix = one toolbar re-measure. Don't batch multiple fixes into one change — it's much harder to attribute the win (or the regression).
**Step 4: Confirm WeasyPrint stays lazy-imported**
One-line grep-and-document check (design doc Task 3):
```bash
grep -rn "^import weasyprint\|^from weasyprint" core/ config/
```
Expected: no matches. WeasyPrint should only be imported inside the body of `render_to_pdf()` in `core/utils.py`. If any module-level import exists, move it inside the function.
**Step 5: Run the full test suite once more**
```bash
USE_SQLITE=true DJANGO_DEBUG=true python manage.py test core.tests -v 2
```
Expected: 68 passes.
**Step 6: Final commit with before/after summary**
Even if Task 4 found no N+1s (unlikely but possible), this commit records the measurement. If Task 3 or 4 DID find real fixes, they're each in their own commit already; this is the "closing the pass" commit.
```bash
# (Only include files in git add IF you actually changed them in this task.)
git add core/views.py [any templates touched]
git commit -m "$(cat <<'EOF'
perf(payroll): [specific fix] + quick-wins pass summary
Fixed [N+1 description] on the [pending|history|loans|adjustments] tab
by [select_related / prefetch_related / annotate added to queryset X].
Perf Quick-Wins Pass A — before/after:
/ N queries, Tms SQL → M queries, Sms SQL
/payroll/ N queries, Tms SQL → M queries, Sms SQL (pending)
/payroll/?status=history N → M queries
/payroll/?status=loans N → M queries
/payroll/?status=adjustments N → M queries
Cache-bust token (d1490c4): CSS is now browser-cached across sessions
and Cloudflare holds it at the edge for 4h — biggest felt improvement
expected from this pass.
WeasyPrint confirmed still lazy-imported (module-level grep clean).
Test suite: 68/68.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
EOF
)"
```
---
## Final acceptance checklist
Before declaring the pass complete, the controller (this session) verifies:
- [ ] All commits on `ai-dev` from `d1490c4` → HEAD show `perf(...)` or `chore(dev):` prefix
- [ ] `USE_SQLITE=true DJANGO_DEBUG=true python manage.py test core.tests -v 2` reports 68/68 passing
- [ ] Django Debug Toolbar renders on `/` with USE_SQLITE=true DEBUG=true — and does NOT render when DEBUG=false (quick toggle test)
- [ ] The final commit message contains concrete before/after query counts (not placeholders)
- [ ] Working tree clean, branch ready to push
---
## What's NOT in this plan (explicit non-goals)
- Splitting `payroll_dashboard.html` into tab partials — plan B territory
- Refactoring any view's body structure
- Any visual/UX change
- Adding a perf regression test harness — if we want that, it's a separate design
Rollback: `git revert <sha>` on any individual commit. No data, schema, or URL-contract impact in any task.

View File

@ -2,4 +2,5 @@ Django==5.2.7
mysqlclient==2.2.7
python-dotenv==1.1.1
pillow==12.1.1
weasyprint==68.1
weasyprint==68.1
django-debug-toolbar==6.0.0 # dev-only — gated in config/settings.py, never active in prod