Monitoring and metrics

Jarvis exposes a Prometheus-compatible /metrics endpoint so the alerting stack it fronts can also monitor Jarvis itself. The endpoint is public (like /health, /health/live and /health/ready) — it bypasses JARVIS_AUTH_MODE=full_protect — and exposes only aggregate counts, configured cluster names and the host:port of each Alertmanager member (the member label), never alert names, labels, or annotations. To restrict it, set JARVIS_METRICS_TOKEN (Protecting the endpoint).

Monitoring Jarvis closes an easy blind spot: the UI can remain reachable while an Alertmanager member is no longer being polled, a PostgreSQL follower has a stale snapshot, or retention sweeps have stopped. The metrics below cover HTTP requests, WebSockets, upstream polling, alert lifecycle events, leader election, snapshot freshness, and retention. There are currently no dedicated database, authentication, fanout, or history-write-error metrics.

Scrape configuration

Plain prometheus.yml configuration:

scrape_configs:
  - job_name: jarvis
    static_configs:
      - targets: ["jarvis:8080"]

Protecting the endpoint

With JARVIS_METRICS_TOKEN set, GET /metrics answers 401 unless the request carries Authorization: Bearer <token>; every other route is unaffected. Without it the endpoint stays open, so existing scrape configurations keep working. Prometheus sends the token with authorization:

scrape_configs:
  - job_name: jarvis
    authorization:
      credentials_file: /etc/prometheus/secrets/jarvis-metrics-token
    static_configs:
      - targets: ["jarvis:8080"]

The Helm chart takes metrics.token (or metrics.existingSecret) and configures the ServiceMonitor to send it; annotation-based scraping (metrics.podAnnotations) cannot, so use a ServiceMonitor or your own scrape config when the token is set.

For Kubernetes, the Helm chart supports Prometheus Operator discovery or pod annotations:

# Prometheus Operator (requires monitoring.coreos.com/v1 CRDs)
metrics:
  serviceMonitor:
    enabled: true
    labels:
      release: kube-prometheus-stack

# Alternatively, annotation-based scraping
metrics:
  podAnnotations: true

See the Helm values reference for all related values.

Scrape-time gauges

Computed from the in-memory alert store, the WebSocket hub, and the poller's own state at every scrape — never from a counter, so they can't drift out of sync with reality.

Metric	Labels	Meaning
jarvis_build_info	version	Always 1; join on version to track which build is running
jarvis_alerts	cluster, state	Current alert count by state (active, suppressed, unprocessed, resolved)
jarvis_alerts_by_severity	cluster, severity	Current alert count by severity label (none when unset)
jarvis_alertmanager_up	cluster, member	1 if the last poll of that Alertmanager HA member succeeded, else 0. Single-member clusters emit their one member; a cluster stays healthy overall as long as at least one member is up
jarvis_ws_clients	—	Number of currently connected WebSocket clients
jarvis_clusters_configured	—	Number of configured Alertmanager clusters
jarvis_leader	—	1 if this pod currently holds Alertmanager-polling/history-write leadership, else 0. Always 1 on SQLite (single replica by design)
jarvis_snapshot_stale	—	1 if any cluster's last successful fetch is older than max(3× JARVIS_POLL_INTERVAL, 60 s) — Alertmanager unreachable, or on a follower the leader's snapshots not arriving — else 0
jarvis_cluster_last_success_timestamp_seconds	cluster	Unix time of the last successful alert fetch of that cluster (as recorded by the polling leader). Absent until a fetch has succeeded

Event counters

Metric	Labels	Incremented when
jarvis_poll_cycles_total	cluster	Every poll attempt of a cluster
jarvis_poll_errors_total	cluster, endpoint (alerts/silences)	A poll of a cluster's alerts or silences endpoint fails
jarvis_poll_duration_seconds	—	Histogram of the full poll cycle across all clusters, including DB persistence — use this to see whether Jarvis overall is keeping up with its poll interval
jarvis_cluster_fetch_duration_seconds	cluster, member	Histogram of a single Alertmanager HA member's response time (alerts or silences) — use this to find which member is slow
jarvis_alert_events_total	cluster, status (firing/suppressed/expired/resolved)	A genuine alert lifecycle transition is recorded
jarvis_ws_broadcasts_total	type	A WebSocket event is broadcast to clients
jarvis_ws_rejected_total	—	A WebSocket connection is refused because JARVIS_WS_MAX_CONNECTIONS is reached. Anything above 0 in normal operation means the cap is too low or something is opening connections it does not close
jarvis_http_requests_total	method, path, status	Every HTTP request (labeled by route pattern, not raw URL)
jarvis_http_request_duration_seconds	method, path, status	Histogram of HTTP request duration
jarvis_retention_sweeps_total	—	A data-retention sweep completes (see docs/retention.md). Only increments while retention is enabled — stays 0 forever on a default install
jarvis_retention_deleted_rows_total	table	Rows deleted by the retention sweeper, by table (alert_events, alert_claims, alert_comments, silence_events, alert_fingerprints)
jarvis_retention_sweep_duration_seconds	—	Histogram of a full sweep's duration across all tables

Runtime metrics (go_*, process_*) are included via the standard Prometheus Go/process collectors.

Example PromQL

# Jarvis lost contact to a cluster
jarvis_alertmanager_up == 0

# Poll error rate
rate(jarvis_poll_errors_total[5m]) > 0

# Alert volume by cluster and state
sum by (cluster, state) (jarvis_alerts)

# Which Alertmanager member is slow
histogram_quantile(0.95, sum by (le, cluster, member) (rate(jarvis_cluster_fetch_duration_seconds_bucket[5m])))

# API latency p95
histogram_quantile(0.95, sum by (le, path) (rate(jarvis_http_request_duration_seconds_bucket[5m])))

Prometheus alert rules

The following starting point covers the signals Jarvis actually exports. Adjust job, durations, and thresholds to match your scrape configuration and poll interval. The leader rules intentionally aggregate across pods; checking jarvis_leader == 1 per series cannot detect zero leaders or split brain.

groups:
  - name: jarvis
    rules:
      - alert: JarvisNoLeader
        expr: sum(jarvis_leader{job="jarvis"}) == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: No Jarvis pod is the polling leader

      - alert: JarvisMultipleLeaders
        expr: sum(jarvis_leader{job="jarvis"}) > 1
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: More than one Jarvis pod reports leadership

      - alert: JarvisAlertmanagerClusterDown
        expr: max by (cluster) (jarvis_alertmanager_up{job="jarvis"}) == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: Jarvis cannot reach any member of Alertmanager cluster {{ $labels.cluster }}

      - alert: JarvisSnapshotStale
        expr: max by (instance) (jarvis_snapshot_stale{job="jarvis"}) == 1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: Jarvis {{ $labels.instance }} shows stale alert data

      - alert: JarvisClusterDataOld
        expr: time() - jarvis_cluster_last_success_timestamp_seconds{job="jarvis"} > 600
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: Jarvis last fetched Alertmanager cluster {{ $labels.cluster }} more than 10 minutes ago

      - alert: JarvisPollErrors
        expr: sum by (cluster) (rate(jarvis_poll_errors_total{job="jarvis"}[5m])) > 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: Jarvis polling cluster {{ $labels.cluster }} is failing

      # Enable only when retention is configured. Set the window comfortably
      # above JARVIS_RETENTION_SWEEP_INTERVAL (12h by default).
      - alert: JarvisRetentionSweepMissing
        expr: sum(increase(jarvis_retention_sweeps_total{job="jarvis"}[25h])) < 1
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: Jarvis retention has not completed a sweep in 25 hours

jarvis_alertmanager_up is per member, so max by (cluster) alerts only when every member of a cluster is down. JarvisRetentionSweepMissing cannot infer whether retention is enabled; omit that rule on installations where it is disabled. Prometheus should also alert on the Jarvis scrape target itself being absent, because an absent target produces no jarvis_* series for these rules to evaluate.