40227-vm/docs/full-integration-refactor-plan.md

Full Integration Refactor Plan

Purpose

This document is the active cross-application backlog for integrating frontend/, backend/, and the PostgreSQL database.

It must track only unfinished integration work and hard implementation rules. Completed migration history belongs in the feature-specific docs under backend/docs/, frontend/docs/, and supporting audit docs.

When an item is completed, remove it from this plan and update the owning feature documentation instead of adding a completed checklist entry here.

Current Application Shape

• frontend/ is the only product frontend.
• backend/ is the source of truth for auth, tenant ownership, roles, permissions, users, staff profiles, campuses, persisted workflows, content catalogs, email, files, and database access.
• ref-frontend/ is a temporary generated reference. It is not runtime code and must be deleted after all needed endpoint-contract reference value is exhausted.

Non-Negotiable Rules

• No frontend persisted workflow may use mock, sample, seed, or fallback records.
• No frontend runtime code may import backend seed files.
• Frontend runtime constants may contain only UI config, route/module metadata, query keys, timing values, style tokens, static UI labels, and intentionally static product copy.
• Backend seeds are for database seeding only.
• Frontend seed data is allowed only in tests or frontend/src/test-seeds/.
• Backend owns tenant scoping and permissions. Frontend route hiding is UX only.
• All frontend backend calls go through typed modules in frontend/src/shared/api/.
• New frontend workflows must follow View -> Business Logic -> API/Data Access -> Backend.
• View components must not import frontend/src/shared/api/.
• Business logic must not import view components.
• API modules must not import business or view code.
• All imports use the @ alias.
• No any, unsafe casts, disabled TypeScript rules, disabled ESLint rules, compatibility bypasses, silent failures, or legacy re-export surfaces.
• Frontend auth must use backend-owned HttpOnly cookies only. Access and refresh tokens must never be stored in frontend browser storage.
• Secrets live only in backend .env or deployment environment variables. Frontend env values must be public browser-safe values only.
• New backend modules must include migration/model/service/route, tenant and role enforcement, docs, and focused verification.
• Every changed workflow must update the relevant docs before it is considered complete.

Current Verification Baseline

Frontend current baseline:

• npm run typecheck passes.
• npm run lint passes.
• npm run test passes with 51 files and 198 tests.
• npm run build passes.
• npm run test:e2e passes with 4 backend-free Playwright smoke tests.
• npm run test:e2e:content exists for backend-seeded content catalog integration tests. It requires backend migrations, backend seeders, and a running backend server.

Backend current baseline:

• Product module files have been syntax-checked with node -c.
• npm audit --audit-level=low reports 0 vulnerabilities.
• npm outdated reports only json2csv@6.0.0-alpha.2 above the installed stable 5.0.7; prerelease packages are not part of the stable dependency baseline.
• npm run lint still fails on existing generated/template lint debt.
• npm run db:migrate still needs a local database verification run with the configured environment.

Active Workstreams

1. Backend Migration And Runtime Verification

Status: open.

Problem:

Backend product modules exist, but the configured local database migration/seed run has not been verified as a passing gate in this plan.

Required work:

1. Run npm run db:migrate against the configured local database.
2. Run npm run db:seed.
3. Start backend with the documented env file.
4. Verify public content catalog routes, auth routes, and product module routes respond.
5. Record exact failing migration/seed/runtime errors if any.

Acceptance criteria:

• npm run db:migrate passes.
• npm run db:seed passes.
• Backend starts without generated default secrets.
• GET /api/public/content-catalog/:contentType works for the required seeded content catalog types.
• Any remaining backend runtime blocker is captured as a specific follow-up with file/error references.

2. Backend Lint Debt Cleanup

Status: open.

Problem:

Backend ESLint 10 is configured, but npm run lint fails on existing generated/template lint debt.

Required work:

1. Fix unused imports, unused variables, useless assignments, and generated-template lint failures.
2. Do not add broad ignores to hide debt.
3. Do not disable rules inline unless a documented, reviewed exception is unavoidable.
4. Keep generated-code behavior unchanged while cleaning lint.

Acceptance criteria:

• npm run lint passes from backend/.
• No broad lint-disable blocks are added.
• No generated route/service behavior changes unless required to remove a real defect.

3. Tenant Boundary Audit And Tests

Status: open and high priority.

Problem:

Generated backend code has partial tenant scoping. Multi-tenant correctness cannot rely on frontend filtering.

Required work:

1. Audit all tenant-owned generated and product routes for organization scoping.
2. Resolve inconsistent organizationsId versus organizationId usage where it affects tenant-owned data.
3. Ensure create/update/delete paths cannot accept another tenant's organization or campus.
4. Ensure list/count/autocomplete endpoints are tenant-scoped.
5. Add backend tests proving cross-tenant records are not visible or mutable.

Acceptance criteria:

• Tenant isolation tests cover users, campuses, staff, students, attendance, messages, documents, and product module tables.
• A non-global user cannot read or mutate another tenant's data.
• Campus-scoped users cannot mutate another campus unless their backend permission explicitly allows it.

4. Role Model Decision

Status: open.

Problem:

Backend currently maps generated roles to product roles. It is not yet a final product decision whether this mapping is the supported model or product roles should become first-class persisted backend roles/permissions.

Required decision:

• Option A: keep generated-role to product-role mapping as the supported model.
• Option B: create first-class persisted product roles: Teacher, Para, Office Manager, Director, Superintendent.

Required work after decision:

1. Document the chosen role model.
2. Align /api/auth/me permissions and product role payload with the chosen model.
3. Add backend role/permission tests for staff, director, and superintendent paths.
4. Keep frontend navigation driven by backend role/permissions.

Acceptance criteria:

• Role model is documented in backend docs.
• Backend tests prove role access for product modules.
• No frontend role grants capability that backend does not enforce.

5. Product Onboarding Contract

Status: blocked by customer decision.

Problem:

Generated auth signup/profile endpoints exist, but they do not define the product workflow for company creation, campus creation, user creation, staff profile creation, role assignment, campus assignment, or profile updates.

No-go rules:

• Do not implement frontend registration.
• Do not implement profile creation/editing UI.
• Do not treat generated signup/profile endpoints as the product onboarding contract.
• Do not add temporary compatibility paths.

Required customer decisions:

1. Who creates a company.
2. Who creates campuses.
3. Who invites or creates users.
4. How staff profiles are created and linked to users.
5. How roles and campuses are assigned.
6. Which profile fields users may update themselves.
7. Which profile fields require director/superintendent/admin permissions.

Required work after decision:

1. Add backend onboarding contract and docs.
2. Add tenant-scoped backend workflows.
3. Add frontend typed API modules.
4. Add frontend business hooks.
5. Add UI only after backend contract exists.
6. Add authenticated Playwright workflows using backend-seeded fixtures.

Acceptance criteria:

• Onboarding workflow is customer-approved and documented.
• Backend owns all creation, assignment, and permission checks.
• Frontend only exposes flows backed by typed backend contracts.

6. Refresh Token Maintenance

Status: open.

Problem:

Cookie auth and refresh rotation exist, but scheduled cleanup for expired/revoked refresh-token rows remains unresolved.

Required work:

1. Define refresh-token retention period.
2. Add cleanup job or operational command.
3. Ensure cleanup is observable.
4. Document operational usage.

Acceptance criteria:

• Expired and revoked refresh-token rows are cleaned after the approved retention window.
• Cleanup failures are visible and not silent.
• Auth behavior remains unchanged for valid sessions.

7. API Documentation Hardening

Status: open.

Problem:

Markdown docs exist for migrated modules, but Swagger/OpenAPI coverage for product-specific and cookie-session endpoints is incomplete.

Required work:

1. Document /api/auth/signin/local.
2. Document /api/auth/refresh.
3. Document /api/auth/signout.
4. Document /api/auth/me.
5. Document product module endpoints that are not covered by generated Swagger output.
6. Document response and error shapes per endpoint.

Acceptance criteria:

• API docs match actual route payloads and response shapes.
• Cookie auth behavior is explicit.
• Frontend API contract tests remain aligned with docs.

8. Policy And Safety Acknowledgment Persistence

Status: open pending product contract.

Problem:

Policy content is document-backed, but policy/protocol acknowledgments are not yet persisted.

Required decisions:

1. Which policies/protocols require acknowledgment.
2. Which roles must acknowledge.
3. Whether acknowledgments are per document version.
4. Who can report acknowledgment status.

Required work after decision:

1. Add acknowledgment backend model/migration/service/route.
2. Enforce tenant and role scope.
3. Add frontend typed API and business workflow.
4. Add report views only if required.

Acceptance criteria:

• Acknowledgments survive reload.
• Acknowledgment status is tenant-scoped.
• Unauthorized roles cannot view individual acknowledgment records.

9. Attendance Source Contracts

Status: partially open.

Current state:

• Campus attendance daily aggregate summaries are implemented for the current UI.
• Staff attendance snapshot/reporting is read-only.
• Student/class attendance source-of-truth workflow is not defined.

Open decisions:

1. Whether campus attendance aggregates are manually entered, imported, or derived from student attendance records.
2. Which external or internal source owns staff attendance writes/imports.
3. Whether student-level attendance UI is required.

Required work after decision:

1. Add write/import endpoints only after source contract exists.
2. Keep derived summaries server-side if summaries are derived.
3. Add backend tests for source-of-truth calculations.
4. Add frontend workflows only after backend contracts exist.

Acceptance criteria:

• Attendance source of truth is documented.
• UI values can be traced to backend records or server-side derivation.
• No frontend-only attendance source remains in persisted workflows.

10. Generated Audio Provider Contract

Status: open pending provider decision.

Problem:

Classroom timer uses built-in Web Audio sounds. AI-generated audio UI is intentionally not exposed because no backend audio provider contract exists.

No-go rules:

• Do not add generated audio UI.
• Do not call external audio providers from frontend runtime.
• Do not add frontend API keys or provider secrets.

Required work after decision:

1. Define backend provider contract.
2. Keep provider secrets in backend env.
3. Add backend service with native provider errors where required.
4. Add typed frontend API and explicit loading/error states.

Acceptance criteria:

• Generated audio is backend-mediated.
• No provider secret reaches the browser.
• Provider failures remain visible.

11. File Upload And Download Permissions

Status: open.

Problem:

Backend file and document routes exist. Product file workflows need explicit permission verification before new upload/download UI is added.

Required work:

1. Audit upload permissions.
2. Audit download permissions.
3. Verify tenant and document ownership checks.
4. Add typed frontend upload client only for approved workflows.

Acceptance criteria:

• Unauthorized users cannot access another tenant's files.
• Upload/download behavior is documented and tested before new UI is added.

12. Public Backend Route Audit

Status: open.

Problem:

Public routes must be intentionally public. This includes public content catalog routes and any generated/template public integrations.

Required work:

1. List all routes without auth middleware.
2. Mark each as public-by-design or requiring auth.
3. Add auth where required.
4. Document intentionally public routes.

Acceptance criteria:

• Every unauthenticated backend route is documented.
• No tenant-owned data is exposed through accidental public routes.

13. Backend-Seeded Authenticated E2E

Status: blocked by onboarding/profile fixtures.

Problem:

Current Playwright coverage includes backend-free smoke tests and backend-seeded content catalog tests. Authenticated persisted workflows need backend-seeded users, roles, campuses, staff profiles, and known credentials.

Required prerequisites:

1. Product onboarding/profile contract.
2. Backend-seeded auth fixtures.
3. Tenant-scoped campus/staff fixtures.
4. Stable test credentials stored only in ignored local env or dedicated test seed config.

Required workflows after prerequisites:

1. Login to dashboard.
2. Director creates or edits a FRAME entry and sees it after reload.
3. Staff completes QBS quiz and director/superintendent sees compliance.
4. Office enters campus attendance and superintendent sees aggregate.
5. Director submits walkthrough and sees summary update.
6. Staff marks a sign learned and progress persists after reload.

Acceptance criteria:

• Authenticated e2e tests use backend seeds, not frontend mock data.
• Tests do not require production secrets.
• Tests are documented and repeatable.

14. Accessibility Test Coverage

Status: open.

Required work:

1. Add axe/Playwright accessibility checks for login.
2. Add axe/Playwright checks for dashboard.
3. Add checks for sidebar navigation.
4. Add checks for modal dialogs.
5. Add checks for forms with validation.
6. Add checks for tables/reports.

Acceptance criteria:

• Accessibility tests run in a documented command.
• Critical violations block completion of the refactor.

15. ref-frontend/ Removal

Status: open.

Required work:

1. Confirm no active docs require ref-frontend/ for endpoint contract reference.
2. Confirm no scripts import or run ref-frontend/.
3. Delete ref-frontend/.
4. Update root docs and any setup docs.

Acceptance criteria:

• Only frontend/ and backend/ remain as active application code.
• No docs describe ref-frontend/ as needed for normal development.

16. OAuth Provider Strategy Modernization

Status: open. Deferred — run after the backend TypeScript/ESM migration (see backend/docs/typescript-esm-migration-plan.md, decision 0.7).

Problem:

Social login uses passport-google-oauth2 (0.2.0, last published 2022) and passport-microsoft (2.1.0). The Google strategy is low-maintenance and should be modernized, ideally consolidating both providers on a single maintained OAuth/OIDC library.

Required work:

1. Choose target: minimal swap to passport-google-oauth20, or consolidate both providers on openid-client.
2. Replace the Google (and optionally Microsoft) passport strategy in backend/src/auth/auth.ts.
3. Keep the existing cookie-based session and JWT issuance unchanged.
4. Verify callback URLs, scopes, and the social-signup user flow end to end.
5. Update auth docs (backend/docs/auth-profile.md, cookie-auth.md).

Acceptance criteria:

• Google and Microsoft sign-in work end to end with the chosen library.
• No deprecated/unmaintained OAuth strategy remains in dependencies.
• Cookie/JWT auth behavior is unchanged for existing sessions.
• This change is isolated from the language/module migration (separate PR/task).

Strict Implementation Sequence

Use this order unless the user explicitly reprioritizes:

1. Backend migration and seed verification.
2. Backend lint debt cleanup.
3. Tenant boundary audit and tests.
4. Role model decision and enforcement tests.
5. Public route audit.
6. API documentation hardening.
7. Product onboarding after customer decision.
8. Authenticated backend-seeded e2e after onboarding/profile fixtures exist.
9. Remaining optional product contracts: acknowledgments, attendance imports, generated audio, file upload/download UI.
10. Accessibility test coverage.
11. Remove ref-frontend/.
12. OAuth provider strategy modernization (after backend TS/ESM migration).

Definition Of Done

The integration refactor is complete only when all of the following are true:

• Backend migrations and seeders pass on the configured local database.
• Backend lint passes without broad ignores.
• Backend tenant isolation tests prove cross-tenant data is not visible or mutable.
• Backend role/permission tests cover product staff, director, and superintendent paths.
• Every unauthenticated backend route is documented as public-by-design.
• Product onboarding contract is customer-approved or explicitly excluded from release scope.
• Every visible frontend workflow is backed by a typed backend API contract.
• No frontend runtime workflow depends on mock, sample, seed, or fallback records for persisted behavior.
• Frontend typecheck, lint, test, build, test:e2e, and documented seeded e2e suites pass in their required environments.
• Backend auth/API docs match actual route behavior.
• Required accessibility checks pass.
• ref-frontend/ is deleted or a dated, explicit exception explains why it is still needed.