Jarvis E2E & Screenshot Testing

Audience: developers and AI agents. This file is the single source of truth for the isolated end-to-end (E2E) test + documentation-screenshot pipeline. Read it fully before touching anything under frontend/e2e/, compose.e2e.yml, Containerfile.e2e, or scripts/e2e-run.sh.

TL;DR

bash

make e2e                              # functional tests, ALL 3 auth modes in sequence (CI runs the same modes as parallel jobs)
make e2e-mode MODE=oidc               # functional tests, ONE mode
make e2e-screenshots                  # regenerate ALL doc screenshots (all modes)
make e2e-screenshot NAME=feature-card-view    # regenerate ONE screenshot (MODE=none default)
make e2e-screenshot NAME=auth-setup MODE=internal
make e2e-down                         # force-clean the stack if something is stuck

Everything runs in a fully isolated Podman/Docker stack with its own network (jarvis_e2e) and ephemeral state (tmpfs). It never touches the dev stack, your real Alertmanager, or any persistent database.

Why a dedicated stack?

The old setup screenshotted the Vite dev server with mocked API responses. That was brittle and untestable. The new pipeline runs the real production binary (frontend embedded) against a real Alertmanager, polling real alerts — so functional tests and screenshots exercise the actual system.

Architecture

E2E stack topology

(source: docs/diagrams/e2e-stack.mmd, re-render via make diagrams)

Service	Image	Purpose
`e2e-jarvis`	built from `Containerfile.e2e`	Production binary + e2e-only seed/reset endpoints. Host port 8085.
`e2e-alertmanager`	`prom/alertmanager`	Real Alertmanager. Fixtures are fired via its API v2.
`e2e-mock-oidc`	`ghcr.io/navikt/mock-oauth2-server`	Mock OIDC IdP (only started for `MODE=oidc`). Host port 8086.
`e2e-playwright`	`mcr.microsoft.com/playwright`	Runs Playwright specs on the `jarvis_e2e` network.

The e2e build tag

Containerfile.e2e compiles with -tags "prod e2e". The e2e tag enables test-only endpoints that do not exist in production builds:

Endpoint	Effect
`POST /api/v1/test/reset`	Truncate all history tables + clear in-memory store.
`POST /api/v1/test/poll`	Force an immediate poll (`jarvis.poll()`). The real `POST /api/v1/poll` needs a login from `write_protect` upward and has a 5 s minimum interval, neither of which a spec that polls after every seed can live with.
`POST /api/v1/test/seed`	Insert resolved-alert lifecycles directly into the DB (`resolved` body key), or backfill multi-cycle firing history for the heatmap (`heatmapHistory` body key — see `jarvis.seedHeatmapHistory()`).
`POST /api/v1/test/claim`	Set a claim on an alert (bypasses auth). Used by `jarvis.setClaim()`.
`POST /api/v1/test/comment`	Add a comment to an alert (bypasses auth). Note: does not broadcast a WS event — use the production endpoint `/api/v1/alerts/:fingerprint/comments` when testing WebSocket behaviour.

Implementation is split by build tag so production can never expose them:

backend/internal/api/testing_routes_e2e.go (//go:build e2e)
backend/internal/api/testing_routes_noe2e.go (//go:build !e2e, no-op)
backend/internal/history/testing_e2e.go (//go:build e2e, ResetForTesting / SeedResolvedForTesting)

⚠️ Never deploy jarvis-e2e:local / the e2e build outside the test stack.

The three auth modes

The stack is brought up once per mode. scripts/e2e-run.sh sets the env and selects the matching spec folder.

Mode	`JARVIS_AUTH_PROVIDER` / `AUTH_MODE`	Flow exercised	Spec folders
`none`	`none` / `none`	No login. NoAuthNotice modal shown.	`e2e/**/none/`
`internal`	`internal` / `write_protect`	First-run setup wizard + username/password login.	`e2e/**/internal/`
`oidc`	`oidc` / `write_protect`	Full Authorization-Code-with-PKCE flow against the mock IdP; `groups=[Administrator]` → admin role.	`e2e/**/oidc/`

Mock OIDC details (gotchas baked in)

Config is mounted as a file (scripts/mock-oidc-config.json → JSON_CONFIG_PATH). Passing it via the JSON_CONFIG env through podman-compose silently drops the value — do not switch back to inline JSON.
The claim-injecting requestMapping matches on grant_type=authorization_code, not scope or client_id. Reason: Jarvis is a confidential client and sends client_id/secret via the HTTP Basic header, and scope is only sent to /authorize (not /token). grant_type is the only param reliably present on the token request.
interactiveLogin: false → the authorize endpoint auto-approves, so the browser flow needs no manual login page interaction.
The OIDC issuer is the internal hostname http://e2e-mock-oidc:8080/default. Both Jarvis (server-side discovery) and the browser (authorize redirect) reach it under that name, so the iss claim stays consistent. Host port 8086 exists only for the readiness probe.

Directory layout

frontend/
  playwright.e2e.config.ts              # functional config; testDir = $E2E_TEST_DIR
  playwright.screenshots.e2e.config.ts  # screenshot config; testDir = $E2E_SCREENSHOT_DIR
  e2e/
    support/
      alertmanager.ts   # AM API v2 client: fire() / resolve() / clearAll()
      jarvis.ts         # Jarvis client: poll() / reset() / seedResolved() / seedHeatmapHistory()
      auth.ts           # dismissNoAuthNotice, ensureInternalAdmin, loginInternal, loginOIDC
      fixtures.ts       # test.extend (auto reset+clear per test), freezeClock, waitForActiveAlerts, waitForResolvedBuffer
      heatmapHistory.ts # fireWithHeatmapHistory() — screenshot-only, see below
      screenshotData.ts # polished alert fixture + label hiding shared by card-view / home-tour shots
    fixtures/
      alerts.ts         # kubernetesAlerts (4), manyAlerts (~14, for populated screenshots)
    functional/
      none/             # card-view, no-auth-notice
      internal/         # setup + login, no Account entry for local accounts
      oidc/             # oidc login + admin-group mapping, Account panel (groups claim)
    screenshots/
      none/             # feature-*, auth-noauth-notice, screenshot, social-templates (Open Graph 1200×630, square 1080×1080, slide 1920×1080 → docs/assets/social-*.png, from the design tokens + logo)
      internal/         # auth-setup, auth-login-internal, auth-user-menu, auth-admin-panel, auth-login-page
      oidc/             # oidc-authenticated, auth-login-oidc, screenshot (README hero)
    video/              # demo videos: recorder.ts, charts.ts (before/after and line-chart slides), build-video.mjs, fonts.conf, release + intro storyboard templates
    _video/             # gitignored: <project>.video.ts, <project>.narration.json, <project>/ (frames, audio)
  playwright.video.config.ts            # release-video config; viewport per $VIDEO_FORMAT

The release demo video reuses this stack: scripts/e2e-run.sh video none runs the storyboard once per format (VIDEO_AUTH_MODE=oidc runs it against the OIDC stack instead, for a storyboard that shows the login prompt). It is produced only on request — workflow, storyboard rules and hand-over in .agents/skills/release-video/SKILL.md (make release-video VERSION=X.Y.Z [PROJECT=release|intro]). The committed release storyboard template also defines the standard cover subtitle; the product-intro template defines the full headline and subtitle used on its cover.

Conventions

One screenshot = one named test. The test name is the PNG basename and the -g selector for single regeneration. Output goes to docs/assets/<name>.png. Exception: a test may write a tightly-coupled series sharing one expensive setup — detail-tabs writes all five feature-detail-tab-<tab>.png images (one per detail-panel tab) from a single fire+seed; regenerating the series is still one -g run (make e2e-screenshot NAME=detail-tabs).
The Settings documentation uses both feature-settings-panel (the complete two-column sheet) and feature-settings-labels (the Labels section embedded in docs/features.md). The OIDC screenshot README hero persists team as hidden so the “+N” hidden-labels chip is visible.
Fixtures are created per test via real APIs (AM for active alerts, Jarvis test endpoints for resolved/history). The page fixture auto-runs am.clearAll() + jarvis.reset() before every test → clean slate, and also forces prefers-reduced-motion: reduce (page.emulateMedia) so the empty-state Owl mesh backdrop (components/common/OwlMeshBackdrop.tsx) paints a single static frame instead of animating — otherwise a screenshot taken on an empty alerts/silences view would be flaky.
Screenshots freeze the clock (freezeClock, Playwright page.clock) and pre-dismiss the NoAuthNotice (except the one screenshot that documents it) so output is deterministic.
Use stable data-testid selectors for assertions (alert-card, login-button, user-menu). Add new ones as needed rather than relying on text/CSS.
Populate screenshots with manyAlerts so they don't look empty.
Element screenshots for a single UI area, not the whole page: when a doc image only needs to show one component (e.g. the heatmap), locate it via data-testid and call locator.screenshot(...) instead of page.screenshot({ fullPage: true }). See feature-heatmap-detail / feature-heatmap-card for the pattern.
Heatmap/sparkline data needs backfilled history, not just a live fire. Freshly fired alerts have no past firing events, so every screenshot showing alert cards or the detail panel (essentially anything that fires manyAlerts/kubernetesAlerts and doesn't immediately switch to List View) uses fireWithHeatmapHistory() (support/heatmapHistory.ts) instead of am.fire() + waitForActiveAlerts() — an empty sparkline row reads as a rendering bug, not "no data" (the sparkline is always rendered, see AlertCard.tsx). It fires normally through the real poll path, then backfills a firing→resolved history per alert spread over the last ~29 days (so 24h/7d/30d ranges all show something) directly via jarvis.seedHeatmapHistory() — no DB reset in between, since the direct insert path doesn't need a clean fingerprint row and a reset would wipe users, breaking specs that log in first. History is deterministically varied per alert (seeded by fingerprint, see the file's PRNG comment) so cards don't all render an identical pattern, and some cells land multiple hits for varying intensity — same alert reproduces the same pattern on every regeneration. Only ~55% of alerts get a history at all (HISTORY_PROBABILITY) — a real fleet is a mix of chronic recurring alerts and ones firing for the first time, so giving every alert a rich history reads as fake. Because of that mix, a screenshot meant to demo the heatmap specifically (not just look populated) can't blindly grab alerts[0] — it might land on a "no history" alert. Use pickAlertWithHistory() (same file) to pick one that actually got backfilled; see feature-heatmap-detail / feature-heatmap-card for the pattern. Do call any claim/comment/silence setup for that alert afterfireWithHeatmapHistory, not before.

When to run what

Situation	Command	Notes
Before pushing a UI/API change	`make e2e`	All 3 modes, in sequence. Same tests as CI (which runs them as parallel jobs). ~few min.
Iterating on one mode	`make e2e-mode MODE=internal`	Fast feedback.
Fast local run of one mode	`make e2e-fast [MODE=none]`	Local opt-in (`E2E_FAST=1`, CI never sets it). Skips the image build when `backend/`, `frontend/` (without `e2e/` specs) and `Containerfile.e2e` are unchanged (hash in `tmp/e2e-image.hash`), and mounts a persistent pnpm store plus corepack cache (podman volume `jarvis_e2e_pnpm_store`) into the Playwright container. Measured on the `none` suite (223 tests): 5:31 min with `make e2e-mode MODE=none` vs. 2:26 min on a warm second run. Reset the caches with `rm tmp/e2e-image.hash` and `podman volume rm jarvis_e2e_pnpm_store`.
Whole `none` suite locally	`make e2e-mode MODE=none` or `make e2e-fast`	Do not loop over `E2E_SHARD=1/4 ... 4/4` locally: every shard boots its own stack, so that is four boots. Shards exist to spread the suite across CI hosts; one unsharded run is a single boot.
Reproducing one CI shard	`E2E_SHARD=2/4 make e2e-mode MODE=none`	Runs only that Playwright shard (`--shard=2/4`) of the mode's suite. Never run two shards at the same time in the same checkout (fixed container and network names and ports, `down -v` at start).
You changed a screen and a doc image is stale	`make e2e-screenshot NAME=<id> [MODE=<m>]`	Regenerate just that PNG, commit it.
Refreshing all docs images	`make e2e-screenshots`	Cycles all modes.
Stack stuck / port in use	`make e2e-down`	Force `down -v`.

What runs in CI

The functional suite (all modes) runs in .github/workflows/e2e.yml on every PR and push to main, using COMPOSE_CMD="docker compose". The modes run as parallel jobs on separate runners, each with its own stack: none in four Playwright shards (E2E_SHARD=i/4), internal and oidc one job each, all through make e2e-mode. One stack per mode is shared and reset before every test, so the suite parallelises across stacks, never across workers in one stack. The aggregator job Functional E2E (all auth modes) is the required status check and fails if any shard fails. make e2e remains the local equivalent (all modes in sequence).
Screenshots are NOT run in CI. They are a documentation artifact; binary PNGs would create noisy diffs and pixel-flake. Regenerate them locally and commit the PNGs when the UI changes. Every screenshot is 1440×900 at device scale 2 (playwright.screenshots.e2e.config.ts), dark by default; only the hero and overview images also exist as a light pair. The social images use CSS-pixel size (scale: 'css').

Adding a new test / screenshot

Pick the auth mode → the matching functional/<mode>/ or screenshots/<mode>/ folder.
Import from ../../support/fixtures (gives you test, expect, am, jarvis, freezeClock, waitForActiveAlerts, waitForResolvedBuffer) and ../../support/auth for login helpers.
Fire fixtures → drive the UI → assert via data-testid. For screenshots, freezeClock, wait for the expected state, then page.screenshot(...).
If you need a new resolved/history scenario, extend the seed payload (jarvis.seedResolved([...])) — backed by POST /api/v1/test/seed.
Run it: make e2e-mode MODE=<m> or make e2e-screenshot NAME=<id> MODE=<m>.

Key environment variables

Var	Set by	Meaning
`E2E_FAST`	you (`make e2e-fast`)	`1` = skip the image build if unchanged and cache the pnpm store; default unset (CI behaviour).
`E2E_SHARD`	you / CI	`i/n`: run only that Playwright shard.
`COMPOSE_CMD`	you / CI	`podman compose` (default) or `docker compose`.
`E2E_BASE_URL`	compose	Jarvis URL inside the network (`http://e2e-jarvis:8080`).
`E2E_ALERTMANAGER_URL`	compose	Alertmanager URL (`http://e2e-alertmanager:9093`).
`E2E_TEST_DIR` / `E2E_SCREENSHOT_DIR`	`e2e-run.sh`	Spec folder for the current mode.
`E2E_AUTH_PROVIDER` / `E2E_AUTH_MODE`	`e2e-run.sh`	Jarvis auth config per mode.
`E2E_OIDC_*`	`e2e-run.sh`	OIDC issuer/client/redirect/groups-claim/admin-group (oidc mode).
`SCREENSHOTS_DIR`	compose	Where PNGs are written (`../docs/assets`).

Troubleshooting

jarvis did not become ready — check podman compose -f compose.e2e.yml logs e2e-jarvis. In oidc mode this usually means discovery failed (mock not up first); the script starts e2e-mock-oidc and waits before booting Jarvis.
OIDC username empty / role not admin — the token didn't get the injected claims. Verify scripts/mock-oidc-config.json still matches on grant_type.
Build error ERR_PNPM_ABORTED_REMOVE_MODULES_DIR_NO_TTY — host node_modules leaked into the build context. .containerignore must exclude **/node_modules.
missing services [e2e-playwright] under podman-compose — don't put the playwright service behind a compose profile; podman-compose run can't see profiled services.

Current test inventory

Quick reference: which spec file covers what. Use this to find the right place for a new test or to understand what already exists.

Mode: `none`

Spec file	Groups	What it covers
`a11y.spec.ts`	A13–A16	axe scan (WCAG 2 A/AA + 2.1/2.2 AA, critical and serious) of the Alerts and Silences pages in dark and light, with no rule excluded; the alert card opens from the keyboard through its named "Open details" button rather than a `role="button"` wrapper (A13); the Fast-Silence popover is a named `group` of plain buttons rather than an ARIA menu (A14) and is operable from the keyboard — focus alone does not open it, Enter does, Tab reaches its options, Escape returns focus to the trigger (A15); in the list view Enter on a button inside a row acts on that button only, not on the row behind it (A16); and the reduced-motion rule (decorative animations stop, spinners keep turning)
`app-shell.spec.ts`	A1–A12	Nav-tabs, theme toggle, mobile hamburger, WS indicator, manual refresh, cluster status in header, info popover, keyboard operation of the cluster/user/refresh header popovers (Enter, Escape, focus-out), owl mark before the tabs (also at 375 px), keyboard-reachable info hints (Escape closes the hint, not the sheet)
`card-view.spec.ts`	B1	Card view renders polled alerts (smoke test)
`alerts-views.spec.ts`	B2–B6, B9	List↔card toggle, severity ordering, card pagination, fullscreen, resolved view including right-aligned top/footer page navigation
`resolved-fetch.spec.ts`	—	Resolved history is fetched only in resolved mode; initial spinner, error/retry and mode-exit cancellation
`resolved-regex-hint.spec.ts`	—	The server-side regex (i) on the Resolved view appears only while a regex filter is set, never on Active
`resolved-pagination.spec.ts`	—	Bounded server pages, no legacy full-history fetch, visible stale-page transition and off-page navigation inputs
`alerts-views-extended.spec.ts`	B7–B8, B10	Responsive column binning, empty state, suppressed/silenced view
`alerts-overview.spec.ts`	—	Alert label breakdown/filtering plus shared modal accessibility: name, focus containment, Escape and focus restoration
`filters.spec.ts`	C1, C10, C10b, C11–C13	Exact matcher + `?filter=` URL (Alertmanager matcher syntax), state restore from URL, legacy `?matchers=` JSON link restored and rewritten to `?filter=` (C10b), `?q=` search, combined search+chips
`filters-extended.spec.ts`	C2–C8 (C9 removed)	`!=`/`=~`/`!~` operators, regex multi-value, label/value suggestions, label chip → filter, AND matchers, draft→promotion, remove-all
`copy-alert-link.spec.ts`	C1–C2	Detail-panel "Copy link": minimal link (no search/filter) opens the same alert in a fresh session; works without the Clipboard API (plain http)
`detail-panel.spec.ts`	D1–D2, D5–D11, G2	Open/close/URL param, labels/annotations, stats & timeline, claim set/release, comments add/delete, claim note edit, AI prompt, section collapse, extend menu (expiring and long-running silence, configured `silenceDurations`)
`detail-panel-extended.spec.ts`	D4, D12–D14	Runbook/URL links, AI prompt tab copy, section collapse/expand, silence from detail panel
`cluster-scoping.spec.ts`	X1–X3	Cross-cluster isolation for identical fingerprint: stats/history, comments, claims
`silences-page.spec.ts`	E1–E7, E4b, G1, G3	List view persist, grouping, show/hide expired, sort (expires/created + asc/desc direction), "By:" creator filter, matcher filter, expiry status, re-create, expire single/group
`silences-form-extended.spec.ts`	F3–F16	Operator switch, regex tags+escaping, Values/Regex mode toggle (auto-switch on regex syntax), live match count, overlap warning, zero-match warning, duration presets, spinner normalisation, inline calendar, Now/Reset, end-after-start validation, author editability, reason required, preview summary, results step
`silences-form-templates.spec.ts`	F1–F2, F14–F15, F17, G4–G8	Form open/close (Cancel/ESC/backdrop), cluster guard, templates CRUD (create/apply/edit/delete)
`silence-matching-semantics.spec.ts`	—	Differential tests against the real Alertmanager instance: form preview's affected-alerts count and actual post-submit suppression must agree — anchored `=~`/`!~` regex (not substring), regex-OR matcher escaping with metacharacter label values
`settings.spec.ts`	H1–H4, H6, H7, H9–H15 (H5, H8 removed), H1b	All settings: timeFormat, defaultViewMode, resolvedPageSize, silenceDuration, defaultCreatorName, claimAnimation, reset defaults, persistence; brand footer (logo + version); `settings=open` survives reload and is removed on close without dropping other URL params (H12); Fast-Silence duration editor — edit/reset/validate and instance defaults from a mocked `GET /api/v1/settings` `global` (H13–H15)
`label-display.spec.ts`	L1–L14	Settings → Labels (issue #189): hide removes chips in card + list view but never from the detail panel (invariant #19), pin + drag reorder, pin/hide mutual exclusion, search, alphabetical position of hidden rows, "+N" reveal chip, reset all / "Reset labels" scope, palette color apply/remove, config editable with no firing alert, "Hide all"/"Show all" (unpinned only, search-scoped)
`saved-filters.spec.ts`	K1–K13	Saved label filters (replaces "Default Filter"): save from current chips, apply replaces matchers but leaves search untouched, "modified" state (base name + unsaved dot on the closed button, "Save changes to …"), rename with live duplicate validation + Enter/Esc, delete needs a second click, default saved filter applied only with no alert-view URL params, default chips are ordinary/removable, migration of a legacy `defaultFilters` blob into a default saved filter named "Default", save-row hint text, "Reset all settings" clears saved filters, unsaved dot for a filter built from scratch, "modified" survives reload + save-as-new via Enter, two-click overwrite without a base
`no-auth-notice.spec.ts`	I1	NoAuth notice appears and dismiss persists
`websocket.spec.ts`	J1–J4	Reconnect indicator (force-close via patched WebSocket), `alerts_update` / `claim_set` / `claim_released` / `comment_added` live events
`ws-reconnect.spec.ts`	—	Reconnect jitter (P7): initial connect has no delay; a real disconnect reconnects within the 3-6s jitter window (real wall-clock bounds, not exact-ms fake-clock assertions — those interact unreliably with real WebSocket events); a reconnect triggers exactly one alerts refetch; a stale (superseded) socket's late close event schedules no extra reconnect attempt
`data-status.spec.ts`	—	Honest data state: a failed first load shows an error + Retry instead of the empty state; a stale cluster (mocked `/api/v1/clusters`) gets a banner with the data age; a failed refresh keeps the data and shows a banner; a WebSocket down for 10s shows the interrupted notice; the heartbeat watchdog replaces a silent socket after ~114s (fake WebSocket + `page.clock`, no real socket I/O)

Mode: `internal`

Spec file	Groups	What it covers
`login.spec.ts`	I2, I4, I6	First-run setup + login happy path, write_protect login modal on write attempt, retry flow after modal login
`session-resume.spec.ts`	R1–R2	Login (up front or after a mid-task session expiry → `401`) opens over the page and the interrupted silence Create completes; Preview stays enabled logged out
`admin.spec.ts`	I10–I15	Administration panel user list, add-user password validation, role change, delete confirm flow, self-row guards; Global Settings tab empty state (no section registered yet)
`setup-token.spec.ts`	—	The first-run setup page asks for the setup token only when `/auth/info` says it is required and reports a wrong one (`/auth/info` and `POST /setup` stubbed)
`account-menu.spec.ts`	U4	A local account gets no Account entry in the user menu

Mode: `oidc`

Spec file	Groups	What it covers
`login.spec.ts`	I3, I8–I9	Full PKCE flow against mock IdP, admin-claim mapping, write_protect SSO modal on write attempt
`account.spec.ts`	U1–U3	`/auth/me` reports the groups from `JARVIS_OIDC_GROUPS_CLAIM`; the user-menu name opens the Account sheet (user, e-mail, Admin badge, groups); Settings no longer carries account details
`sso-resume.spec.ts`	S1–S3	SSO in a popup keeps the silence form and completes the Create; `return_to` returns a full-page login to its page and never leaves the origin

Known gaps (candidates for future cycles)

Functional cases from the original test catalog that are not covered yet:

E8 — Silences page live update: silences_update WS push (not covered by websocket.spec.ts, which only asserts alerts_update / claim_set/claim_released/comment_added) and the 60s FALLBACK_REFETCH_INTERVAL_MS safety-net poll.
F18 — Precise/Broader/Pattern silence-matcher presets (remove label / = → =~).
I5 — full_protect LoginPage gating the whole app (only a screenshot spec exists, no functional assertion).

Partially covered / known caveats:

D3 — annotation link interaction tested; full dynamic link-button matrix not exhaustive.
D10 — own-comment delete green; two role-based delete cases are skipped.
I6/I7 — setup+login happy path green; setup validation details and logout assertion not asserted.
J3 — was flaky (WS event lost when a broadcast raced the client registration in the hub loop); fixed by registering clients synchronously in ServeWS — see .agents/lessons.md.

Jarvis E2E & Screenshot Testing ​

TL;DR ​

Why a dedicated stack? ​

Architecture ​

The e2e build tag ​

The three auth modes ​

Mock OIDC details (gotchas baked in) ​

Directory layout ​

Conventions ​

When to run what ​

What runs in CI ​

Adding a new test / screenshot ​

Key environment variables ​

Troubleshooting ​

Current test inventory ​

Mode: none ​

Mode: internal ​

Mode: oidc ​

Known gaps (candidates for future cycles) ​