Alerting, SLOs and Paging That Engineers Trust

Abstract. This guide is for senior SRE, DevOps and platform engineers at startups on AWS, and especially for teams running crypto infrastructure: relayers, signers, hot wallets, indexers, RPC fleets and settlement pipelines. At those companies an alert that stays silent can mean funds stuck on-chain, and an alert that cries wolf teaches a six-person on-call rotation to stop reading Slack. It describes an alerting system built as code: Grafana-managed rules provisioned through Terraform, multi-window multi-burn-rate SLO alerts, recording rules, a three-tier severity taxonomy routed to PagerDuty, and runbook links that CI checks. The longest section is a catalogue of the ways such a system lies to its operators. Most entries are not exotic. They are ordinary PromQL, LogQL and Grafana defaults that look right in review and then return the wrong answer in production, often for weeks before anyone notices. Each one comes with the mechanism behind it, the fix, and the review question that would have caught it.

---

1. The problem space

1.1 What “trust” means for an alert

An alerting system earns trust when two things hold at once:

1. When it pages, something is actually wrong and a human can act on it. The Google SRE book puts the bar at “an otherwise undetected condition that is urgent, actionable, and actively or imminently user-visible” [2]. Every page that misses that bar trains the on-call engineer to snooze the next one.
2. When it is quiet, nothing that matters is broken. This half is harder, and most teams never check it. A rule that cannot fire is far more dangerous than a rule that fires too often, because nothing points at it. The AWS Well-Architected Reliability Pillar calls the silent failure “the worst failure mode”: “the functionality is no longer working, but there is no way to detect it except indirectly. Your customers know before you do.” [9]

Most writing on alerting is about the first property, alert fatigue. This guide puts equal weight on the second, because in the field the expensive incidents are rarely “we got paged too much”. They are “this broke at 100% for thirteen days and the alert that was supposed to cover it read green the whole time”.

1.2 Forces at a startup

• Small rotation, high stakes. A typical Series A infrastructure team has 6 to 10 engineers on a 1-in-N rotation with 24-hour shifts. Each engineer carries the pager roughly 20% of days. That budget cannot absorb noise. The SRE book’s ceiling of two incidents per 12-hour shift [14] is a reasonable upper bound, and a small team should aim well below it.
• Speed of change. Alert rules change weekly. Services are renamed, histograms re-bucketed and labels re-keyed. Every one of those changes can quietly invalidate a rule, and no compiler will notice.
• Partner and auditor pressure. Exchanges, custodians and enterprise integrators increasingly ask for an SLO, an incident process, and evidence that funds-at-risk conditions are monitored. “We have Grafana” is not evidence. A reviewed, versioned rule set with a runbook per alert is.
• Crypto-specific failure surfaces. Beyond the four golden signals [2] you need to watch:
  • Wallet balances. A relayer or fee-payer wallet that runs dry halts settlement silently.
  • Nonce lanes. One stuck transaction wedges every transaction queued behind it.
  • Third-party RPC quotas. A provider’s monthly plan can run out at 3 a.m.
  • Indexer freshness. An indexer that falls behind keeps serving stale state with HTTP 200s.
  • Testnet traffic in production. It can outnumber mainnet traffic and dilute or impersonate real incidents.
• Cost. Self-hosted Grafana, VictoriaMetrics/Prometheus, Loki and Tempo on EKS is cheap in dollars, but every component in the telemetry path is something that can fail and blind the alerting.

1.3 Design goals

Goal	Concrete test
Every page is actionable	Each sev1 rule has a runbook anchor with a Respond section, and pages are reviewed weekly
Silence means healthy	A dead-man’s-switch chain covers the pipeline (ingest → evaluate → notify), and an out-of-band heartbeat covers the evaluator itself
Rules are reviewable	Rules are Terraform in git, plan runs on every PR, and nothing hand-edited in the UI survives the next apply
Rules stay correct as code changes	CI checks runbook anchors, enforces doc co-changes and runs rule-shape lints; defects are written down in an authoring guide
Severity has one meaning	sev1 pages a human now; sev2 is working-hours; sev3 is ambient feed. Nothing else pages

---

2. Reference design

2.1 Topology

Three properties matter more than the specific vendors:

• Alert evaluation is centralised in Grafana (unified alerting), not split across Prometheus rule files, Loki ruler and CloudWatch alarms. Grafana can join metrics, logs and CloudWatch in one rule. One evaluator also means one routing tree and one place to search. The tradeoff is that Grafana becomes a critical-path component, and §2.8 deals with that.
• The pipeline is monitored from the inside (canaries) and from the outside (heartbeat). A canary inside Grafana can detect the metrics store going blind. It cannot detect Grafana going blind.
• Recording rules write back to the TSDB. SLO dashboards, burn tiles and alerts then all read the same precomputed sli:* series, which keeps the dashboard and the alert from disagreeing.

2.2 Alerts as code: layout and provisioning

Grafana’s alerting resources (rule groups, contact points, notification policies, mute timings) are first-class in the Grafana Terraform provider [5][6]. A layout that has scaled to ~200 alerting rules and ~100 recording rules:

terraform/grafana/
  folders.tf
  datasources.tf
  contact_points.tf          # slack, slack_error_feed, slack_and_pagerduty["sev1"], specialised channels
  notification_policy.tf     # the severity routing tree (single resource; it is a singleton)
  locals_testnets.tf         # chain IDs treated as testnet — shared by every chain-scoped rule
  alert_rules_slo.tf         # for_each over a map of SLIs → fast/slow burn rules
  alert_rules_slo_recording.tf
  alert_rules_telemetry_canary.tf
  alert_rules_watchdog.tf
  alert_rules_signer_balance.tf
  alert_rules_relayer_runway.tf
  alert_rules_<domain>.tf    # one file (and one rule group) per domain
  dashboards/*.json
terraform/pagerduty/
  services.tf  escalation_policies.tf  event_orchestration.tf  slack.tf

Conventions worth enforcing from day one:

• One rule group per domain, one file per group. In one repo we reviewed, a 4,500-line “core” file had been imported wholesale from the UI. Nobody was willing to split it, because the split needed a live apply to verify. Write small groups from the start.
• disable_provenance = true keeps the resources editable in the UI for emergencies. The cost is that UI edits are silently reverted on the next apply. Treat the UI as a scratchpad and git as the source of truth, and say so in the folder description.
• Stable, short UIDs. Grafana rejects alert-rule UIDs longer than 40 characters, and the provider only finds out from the API, which means mid-apply on prod. The groups written before the failure stay, and the rest are missing. Add a plan-time guard:

lifecycle {
  precondition {
    condition     = length("${each.value.uid_prefix}-p999-latency") <= 40
    error_message = "uid_prefix '${each.value.uid_prefix}' exceeds Grafana's 40-char rule UID limit."
  }
}

• Group renames are destructive. grafana_rule_group.name forces a new resource. Renaming a group replaces it, the nested rule UIDs change, and firing history and silences reset. That is sometimes correct (the SLI now measures something else), but it should be a decision, not a surprise in a plan diff.
• The Grafana root needs its own apply pipeline, and operators must know which one it is. A common trap is a general terraform apply workflow whose environment matrix is an allowlist that leaves out the Grafana root. A change under grafana/ then triggers the workflow and skips every job, which looks exactly like a successful apply. The real apply runs later from a separate workflow. Verify a rule change through the provisioning API (GET /api/v1/provisioning/alert-rules/<uid>), not from a green merge.

2.3 Severity taxonomy and routing

Three severities, each with one meaning:

Severity	Meaning	Destination	Example
sev1	A human must act now; customer impact or funds-at-risk is happening or imminent	Slack + PagerDuty (pages)	Fast SLO burn; settlement wallet empty; telemetry pipeline dead; RBAC denial on the ingress controller
sev2	Real problem, working-hours triage is enough	Slack #prod-alerts	Slow SLO burn; signer balance below floor; nonce lane wedged on one chain
sev3	Ambient signal, cause-level, testnets	Slack feed channel	Error-log rate; p99.9 latency leading indicator; testnet versions of sev2 rules

The routing tree in Terraform:

resource "grafana_notification_policy" "root" {
  contact_point   = grafana_contact_point.slack.name   # unlabelled → visible, never pages
  group_by        = ["grafana_folder", "alertname"]
  group_wait      = "30s"
  group_interval  = "5m"
  repeat_interval = "4h"

  policy {
    matcher { label = "severity" match = "=" value = "sev1" }
    contact_point = grafana_contact_point.slack_and_pagerduty["sev1"].name
  }
  policy {
    matcher { label = "severity" match = "=" value = "sev2" }
    contact_point = grafana_contact_point.slack.name
  }
  policy {
    matcher { label = "severity" match = "=" value = "sev3" }
    contact_point = grafana_contact_point.slack_error_feed.name
  }
}

Design notes:

• Unlabelled alerts default to Slack, never to the pager. If something deserves a 3 a.m. page, the author has to write severity = "sev1" on purpose.
• sev1 versus sev2 is about urgency of response, not cadence. One team ran a sev2 PagerDuty service with slower escalation. They found it meant “wake someone up, but slowly”, which pleased nobody, and removed it. A low-urgency PagerDuty tier is defensible if your team will honour it. Most small teams will not.
• A per-rule override is the escape hatch, not the norm. A rule can set notification_settings { contact_point = ... } for a specialised channel, such as a high-value-deposit pager or a vendor-specific room. Overrides bypass the tree, so audit them periodically. A contact point that nobody routes to is dead config.
• PagerDuty priority is set on the PagerDuty side. The Events API v2 payload carries severity but no incident priority [16]. If you want SEV1 pages to count as business-service impact, add a catch-all Event Orchestration rule on the SEV1 service that stamps P1. Everything routed there is SEV1 by construction.
• Escalation for a small team: primary 5 min → secondary 10 min → a named L3 at 15 min, looping twice. Run two parallel rotations, offset so nobody holds both seats on the same day. Manage schedules in the PagerDuty UI (overrides and vacation swaps should not need PRs) and reference them from Terraform by name with a data source. Renaming a schedule in the UI breaks terraform plan, so write that in the README.
• Send deploys as PagerDuty change events. Post one from the CD pipeline on every prod deploy. The incident’s “Recent Changes” tab then answers “what just shipped?” without anyone leaving the page.

2.4 SLOs and multi-window multi-burn-rate alerts

Use the SRE Workbook’s construction [1] directly. For a 30-day SLO with error budget B = 1 − target:

Tier	Burn rate	Long window	Short window	Budget consumed when it fires	Action
Fast	14.4×	1h	5m	2%	Page (sev1)
Slow	6×	6h	30m	5%	Slack (sev2)
(Optional)	1×	3d	6h	10%	Ticket

Both windows must exceed the threshold. The short window, about 1/12 of the long [1], is what makes the alert resolve quickly once the burn stops.

Express every SLI as a row in a map and generate the rules with for_each. Adding an SLI should be one map entry, not 250 lines of copied HCL:

locals {
  sev1_burn = 14.4
  sev2_burn = 6.0

  slos = {
    sli_quotes = {
      uid_prefix         = "slo-quotes"
      slo_target         = 0.97
      enable_fast_burn   = true            # ~650 req/h: enough events per 5m
      count_1h_gate      = 500             # min events in the long window
      fast_burn_severity = "sev1"
      slow_burn_severity = "sev2"
      good_query  = "sum(rate(http_server_request_duration_seconds_bucket{service_name=\"api\",http_route=\"/quotes\",http_response_status_code!~\"5..\",le=\"5\"}[__WINDOW__]))"
      total_query = "sum(rate(http_server_request_duration_seconds_count{service_name=\"api\",http_route=\"/quotes\"}[__WINDOW__]))"
      count_query = "sum(increase(http_server_request_duration_seconds_count{service_name=\"api\",http_route=\"/quotes\"}[__WINDOW__]))"
    }
    sli_deposit_credited = {
      uid_prefix         = "slo-deposit"
      slo_target         = 0.95
      enable_fast_burn   = false           # ~18 events/h: a 5m window is 1–2 events
      count_1h_gate      = 7
      fast_burn_severity = "sev1"
      slow_burn_severity = "sev2"
      good_query  = "sum(rate(deposit_outcome_total{outcome=\"succeeded\"}[__WINDOW__]))"
      total_query = "sum(rate(deposit_outcome_total[__WINDOW__]))"
      count_query = "sum(increase(deposit_outcome_total[__WINDOW__]))"
    }
  }

  slo_derived = { for k, s in local.slos : k => {
    sev1_threshold = local.sev1_burn * (1 - s.slo_target)
    sev2_threshold = local.sev2_burn * (1 - s.slo_target)
  } }
}

The fast-burn condition, as a Grafana math expression over five instant queries (good_5m, total_5m, good_1h, total_1h, count_1h) and two intermediate math nodes:

# error_5m = 1 - (${good_5m} / ${total_5m}) ; error_1h likewise
expression = join(" && ", [
  "!is_nan($${error_5m})",           # NaN guard — see §4.6
  "!is_nan($${error_1h})",
  "($${error_5m} > ${local.slo_derived[each.key].sev1_threshold})",
  "($${error_1h} > ${local.slo_derived[each.key].sev1_threshold})",
  "($${count_1h} > ${each.value.count_1h_gate})",   # volume gate
])

with no_data_state = "OK" and exec_err_state = "Error".

Four rules for choosing SLIs and targets:

1. Pick a target the current baseline already meets, then ratchet upward. The Workbook warns against anchoring permanently on current performance [3]. For alerting, though, a target below baseline means the burn alerts fire on regressions (“something changed”), not on the gap between what you have and what you wish you had. Aspirational targets go on dashboards. Write the ratchet policy into the file header.
2. Fast-burn only where volume supports it. At ~18 events/hour, a 5-minute window holds one or two events, so a single failure reads as a 50–100% error rate. One team measured 25–35 Alerting transitions per day on low-volume fast-burn rules against 1–2 for slow-burn on the same SLIs, and turned fast-burn off for those rows. Low-volume SLIs rely on slow-burn plus cause-level sev3 rules for faster attribution.
3. Gate on a minimum count in the long window, calibrated from each SLI’s 7-day traffic distribution (for example the p10 of hourly volume). A quiet hour with one error must not page.
4. Separate scopes that are not the same customer promise. If production serves both mainnet and testnet traffic, and testnet is often the majority of volume, split the SLI into two rows with the same target. Run the testnet row one severity notch down (sev2/sev3) so it can never page. Keep the chain-ID list in one shared local (locals_testnets.tf) so every chain-scoped rule uses the same definition. Without the split, one team paged sev1 on a “completion SLO” because a testnet faucet wallet ran dry while mainnet was healthy the whole time.

Latency SLIs as “non-5xx and under N seconds” need a histogram bucket at N. OpenTelemetry’s semantic conventions recommend http.server.request.duration boundaries of [0.005 … 2.5, 5, 7.5, 10] seconds [17]. Choose N from that list and see §4.7 for what happens when the buckets change under you.

Add a p99.9 leading-indicator tier (sev3) only for endpoint-shaped SLIs with enough volume. Set it just above the observed 7-day p99.9 so it fires on tail growth, not on the current baseline. Coarse top buckets saturate p99.9 at the highest finite bucket, so skip it where they do.

2.5 Recording rules

Recording rules give dashboards and alerts identical numbers, and they make the 30-day budget panels cheap. Follow the Prometheus level:metric:operations naming convention [8] loosely, as sli:<numerator>_rate:<window> with an sli label, and generate them from the same local.slos map so the alert and the recording cannot drift:

locals {
  slo_windows = ["5m", "30m", "1h", "6h", "30d"]   # add a window only when a panel consumes it
  slo_recording_rules = merge([
    for k, s in local.slos : {
      for combo in flatten([for w in local.slo_windows : [
        { kind = "good",  q = replace(s.good_query,  "__WINDOW__", w), w = w },
        { kind = "total", q = replace(s.total_query, "__WINDOW__", w), w = w },
      ]]) : "${k}-${combo.kind}-${combo.w}" => merge(combo, { sli = k })
    }
  ]...)
}

resource "grafana_rule_group" "slo_recording" {
  # ...
  dynamic "rule" {
    for_each = local.slo_recording_rules
    content {
      name = "sli:${rule.value.kind}_rate:${rule.value.w}"
      record {
        metric                = "sli:${rule.value.kind}_rate:${rule.value.w}"
        from                  = "A"
        target_datasource_uid = local.metrics_ds_uid   # MANDATORY — see §4.8
      }
      labels = { sli = rule.value.sli }
      # data { ref_id = "A" ... expr = rule.value.q }
    }
  }
}

Aggregate the numerator and the denominator separately and divide at query time. Never average ratios [8].

2.6 Runbook linking, enforced

Every rule carries a runbook_url annotation that deep-links to an anchor, not to a bare runbook.md:

annotations = {
  summary     = "Fee-payer balance low: {{ $value }} USD (wallet {{ $labels.address }})"
  description = "[prod] The fee-payer pays for every transaction on this chain; at zero, settlement halts. Top up the wallet in the `address` label."
  runbook_url = "https://github.com/example-org/infra/blob/main/docs/runbook.md#3111-fee-payer-balance-low"
  view_logs   = "https://grafana.example.internal/explore?...%7C%3D..."   # pipes URL-encoded, see §4.12
}

Two CI checks make this hold over time:

1. Co-change check. A PR that touches alert_rules*.tf must also touch the runbook. A path filter with a hard failure works, plus an escape hatch: a genuine no-op edit adds a one-line runbook note saying why. It is crude, and it works. Annotation text and runbook text drift apart within weeks without it.
2. Anchor-resolution check. Parse every runbook_url (including ones inside dynamic "rule" blocks, unquoted keys and local.* references), compute GitHub-style anchors from the runbook headings, and fail on any link that does not resolve:

#!/usr/bin/env python3
import re, sys, pathlib
heads = pathlib.Path("docs/runbook.md").read_text().splitlines()
def anchor(h):  # GitHub slug: lowercase, drop punctuation except - and space, spaces→-
    s = re.sub(r"[^\w\- ]", "", h.strip("# ").lower())
    return s.replace(" ", "-")
anchors = {anchor(h) for h in heads if h.startswith("#")}
bad = []
for f in pathlib.Path("terraform/grafana").glob("alert_rules*.tf"):
    for m in re.finditer(r'runbook\.md#([\w\-]+)', f.read_text()):
        if m.group(1) not in anchors:
            bad.append(f"{f}: #{m.group(1)}")
print("\n".join(bad)); sys.exit(1 if bad else 0)

Anchors include the section number (### 3.11.1 Fee-payer balance low → #3111-fee-payer-balance-low), so renumbering breaks links. When two sections collide on a number, move the one with the fewest inbound alert links rather than renumbering everything after it. The runbook should stay one file even when it grows large (hundreds of sections). Hundreds of alert annotations, the PR template and the README deep-link into it. Give people a script that prints a single section (runbook.sh 3.25, runbook.sh --list redis) so nobody has to read it whole.

A runbook section for an alert should carry: alert UID, severity and route, for, threshold, what it fires on, why it matters, Identify (copy-paste queries), Respond, known false-positive modes. The metadata lines (severity, for, threshold) are the ones that drift. Re-verify them against the Terraform on a schedule, or generate them.

2.7 NoData, Error and the telemetry canary

Grafana gives each rule two independent state mappings [4][6]:

• no_data_state: OK | NoData | KeepLast | Alerting, used when the query succeeds but returns nothing.
• exec_err_state: OK | Error | KeepLast | Alerting, used when the query fails.

When a rule enters NoData, Grafana creates a separate DatasourceNoData alert instance whose labels may not match your routing or silences [4]. That surprises people during a storm.

The default for most rules should be no_data_state = "OK". Most metrics are legitimately absent when nothing is happening, such as an error counter that only exists after the first error. The consequence is serious: a total telemetry outage evaluates the whole estate to OK. In one estate, roughly 110 rules carried no_data = OK. A collector or TSDB outage would have read as a perfectly healthy system.

The fix is a small set of canary rules that alert on absence, anchored on series that exist continuously regardless of product traffic:

locals {
  telemetry_canaries = {
    scrape = {
      # count of the SERIES, not its value: a failed scrape writes up=0 and keeps the
      # series alive. What trips this is nobody writing samples at all.
      expr      = "count(up{job=\"kube-state-metrics\",env=\"prod\"}) or vector(0)"
      threshold = 1
    }
    otlp = {
      # distinct services pushing OTLP HTTP metrics; flat baseline 13 → floor 8
      expr      = "count(count by (service_name) (http_server_request_duration_seconds_count{env=\"prod\"})) or vector(0)"
      threshold = 8
    }
  }
}
# rule: condition "value < threshold", for = "10m", severity = "sev1",
#       no_data_state = "Alerting", exec_err_state = "Alerting"

or vector(0) is load-bearing. It turns “the series is gone” into an ordinary threshold breach, so no_data_state keeps its real meaning: Grafana could not evaluate the rule at all. The two canaries cover different paths. The scrape path can be healthy while the OTLP receiver rejects every push, for example when the collector’s memory limiter is shedding load. If only one fires, it tells you which half is broken.

Make the canaries prod-only. The metrics store is usually shared across environments, so if it stops ingesting, the prod canary catches it.

2.8 The watchdog: who watches Grafana?

A canary inside Grafana cannot detect Grafana being down. That covers Grafana’s database, the cluster it runs on, its namespace, a bad config rollout, or an expired datasource credential. If Grafana stops evaluating, every page stops silently. An independent review of one otherwise mature platform rated this the top observability gap.

The standard pattern is an always-firing Watchdog alert [12] whose absence pages, through a service that lives outside your infrastructure:

resource "grafana_contact_point" "heartbeat" {
  name = "external-heartbeat"
  webhook {
    url                     = var.heartbeat_ping_url      # PagerDuty heartbeat / Healthchecks.io
    http_method             = "POST"
    disable_resolve_message = true
  }
}

resource "grafana_rule_group" "watchdog" {
  name             = "System: watchdog"
  folder_uid       = grafana_folder.system.uid
  interval_seconds = 60
  rule {
    uid            = "watchdog-heartbeat"
    name           = "Watchdog"
    condition      = "always"
    for            = "0s"
    no_data_state  = "Alerting"
    exec_err_state = "Alerting"
    labels         = { severity = "heartbeat" }
    annotations    = { runbook_url = "https://github.com/example-org/infra/blob/main/docs/runbook.md#393-watchdog-heartbeat-missing" }
    notification_settings {
      contact_point   = grafana_contact_point.heartbeat.name
      group_interval  = "1m"
      repeat_interval = "1m"     # re-notify every minute = the heartbeat ping
    }
    # data { ref_id = "always"  type = "math"  expression = "1 == 1" }
  }
}

On the receiving side, configure a period of 1 minute and a grace of 5–10 minutes [13]. Point the heartbeat’s own alert at a PagerDuty integration (or SMS) that does not depend on Grafana. Two things to check:

• Make the watchdog query the real datasource (for example vector(1) against the metrics store) if you also want it to prove the datasource path. A pure math expression proves only that the evaluator and notifier are alive. Either is defensible. Document which one you chose.
• Test it. Scale Grafana to zero in a game-day window and time how long the external page takes to arrive.

Together, the canaries (ingest), the watchdog (evaluate and notify) and an HA httpcheck collector (external probes, 2 replicas with TSDB-side dedup) remove the single points of failure that would otherwise silence everything at once.

2.9 Freshness gauges for batch, backup and indexer jobs

Batch work fails by not happening. The pattern is a “last success” gauge plus a freshness rule:

# Gauge freshness, scraped continuously: use the SAMPLE's timestamp, not eval time
timestamp(indexer_sync_block_timestamp) - indexer_sync_block_timestamp > 120

# Job that pushes only on success (CronJob, Lambda, Step Function):
time() - max(last_over_time(backup_last_success_timestamp_seconds{store="postgres"}[2d])) > 26*3600

For stores you already back up, the review question is “which store has no ‘no successful X in N days’ rule?” In practice the gaps are the metrics store’s own backup (often an echo to stderr on failure), message-broker definition exports, snapshot lifecycle policies, and restore-check freshness. A restore drill that silently stopped running is worse than none, because it still looks green on the dashboard.

Where possible, prefer native AWS metrics that exist the moment the resource exists. AWS/States ExecutionsFailed on a restore-check Step Function needs no instrumentation and has none of the “is the custom metric actually flowing yet?” risk. Use it as the immediate backstop, and add per-store rules on the custom metrics once you have confirmed they are flowing (§4.9). Remember that ExecutionsFailed with no_data = OK only covers failures. It cannot see a schedule that stopped firing, so pair it with a freshness rule on ExecutionsSucceeded.

2.10 On-chain alerts

Wallet balance: floor plus runway. A fixed USD floor means something different on every chain. On a busy testnet burning about $1,700/hour, a $10 floor is 20 seconds of runway. On a quiet mainnet chain it is weeks. Pair the floor with a rate rule:

# fires when the linear fit over 1h says the wallet hits zero within 4h
predict_linear(
  max by (chain_id) (relayer_account_balance_usd{env="prod"})[1h:1m], 4*3600
) < 0

The for = 15m and keep_firing_for = 10m settings have been calibrated in the field. In one week across 22 chains this rule fired exactly twice. Both were genuine near-total drains, with lead times of 1h45m and 2h30m, and there were no false positives. Its limits, which belong in the runbook:

• Don’t widen the horizon. At 24h the same rule fired at about a quarter of all sample points. The horizon behaves like a cliff, not a dial. The 4h horizon also neutralises token price moves: a balance has to shed more than 25% per hour to fire.
• Blind for about an hour after a top-up. The refill jump inside the fit window throws the slope wildly. A partial top-up that doesn’t cover the burn rate hides the problem until the window clears.
• Late when the burn accelerates. The fit is linear. The floor stays as the backstop.
• predict_linear needs two samples. NoData means the signer isn’t reporting, which is the worker-liveness rule’s job. Set no_data_state = OK here so a dead process isn’t double-reported.
• Use min/max by (address) when several pods report the same shared wallet. That collapses the duplicates and keeps the address label, so the alert names the wallet to fund.

Nonce lanes. Size of the backlog is a bad signal, because it scales with throughput: a busy chain healthily holds 6–17 in-flight transactions. The right signal is work outstanding and no confirmations. Nonce lanes are strictly sequential, so if the head transaction isn’t landing, nothing confirms:

(relayer_nonce_pending - relayer_nonce_latest) > 5
and on (chain_id)
changes(relayer_nonce_latest[10m]) == 0

This form has no “just busy” false-positive mode.

RPC provider quotas and rate limits. Watch for:

• Per-upstream error ratios from an RPC proxy can exceed 1.0. The error counter increments per retry and hedge, and across a different label cardinality from the request counter. Measured values reached 3.6×. A 50% threshold on that “ratio” flaps on bursts the proxy already absorbed by failover. Alert on network-level failure share (failed/received > 0.2 for 5m as sev2, > 0.9 as sev1) and keep per-upstream health on a dashboard.
• Use failed/received, not successful/received. Over a churning pod set, sum(rate(...)) doesn’t keep two counters in step, and the success form false-paged on near-zero “success” values while the failure counter sat flat at zero. Any deploy that rolls the proxy creates this churn.
• Gate on volume (for example at least 15 requests per 5 minutes per chain) so idle chains don’t fire.
• Watch the vendor’s quota signal directly (a “max usage reached” log line or a 429 counter), and make the runbook say what it means given your failover. “Primary out of quota, failover absorbing” is working-hours triage. “Both providers failing” turns transient rate limits into terminal user-facing failures. One team saw a transient 429 on account-creation lookups permanently fail on-chain-confirmed deposits because the error code was classified terminal.
• A rate limiter that fails open looks healthy. A “proof of life” counter that exists at 0 proves nothing. Alert on it not increasing.

Security signals. KMS key-policy changes, secret deletion, RBAC denials from tier-0 components, GuardDuty findings and CloudTrail tampering are all alertable through CloudWatch or Loki. Put the principal and the key or secret in the alert’s grouping labels, and include an Explore link. A sev1 security page that says only “a KMS policy changed” costs the on-call 20 minutes to work out who changed what.

---

3. Decisions and tradeoffs

3.1 Grafana-managed rules vs Prometheus/Loki rulers vs CloudWatch alarms

Option	Upsides	Downsides	Revisit when
Grafana unified alerting (recommended)	One evaluator across metrics, logs and CloudWatch; one routing tree; Terraform provider covers everything [5][6]; good annotation templating	Grafana becomes critical path (needs HA plus external heartbeat); math-expression semantics differ from PromQL (NaN, reducers); provenance and UI-edit friction	Grafana evaluation latency or reliability becomes a problem at thousands of rules
Prometheus/VM ruler + Alertmanager	Battle-tested, PromQL-native, evaluator sits next to the data; Watchdog idiom is standard [12]	Separate Loki ruler, separate CloudWatch path; three routing configs	You outgrow a single Grafana or need evaluator-per-cluster isolation
CloudWatch alarms	Zero infrastructure; native for AWS resources; survives your cluster dying	Poor at app metrics and cross-source joins; cost per alarm and metric; clunky in Terraform at scale	Use as a complement for “the observability cluster itself is down” signals

Pragmatic hybrid: Grafana for everything, plus a handful of CloudWatch alarms or an external heartbeat for the observability stack itself. That follows the Well-Architected guidance that “alerting should be decoupled from your systems as much as possible” [9].

3.2 Symptom vs cause alerts

Page on symptoms (SLO burn, funds-at-risk), and route causes (error-log rate, pod restarts, CPU) to sev3 [2][7]. The exceptions are precursors with a known short fuse. A tier-0 component logging RBAC denials can turn into a total ingress outage on the next pod roll. In one post-incident review, the denial had been logged continuously for over three hours before the outage, the correct fix had been identified more than an hour and a half before impact, and nobody was paged. Those precursors deserve sev1.

3.3 no_data_state: OK vs Alerting vs NoData

Choice	Use for	Risk
OK	Metrics legitimately absent when idle; rate rules whose absence another rule covers	Blind to telemetry loss, so you must have canaries
Alerting	Canaries and watchdogs; rules where absence is the failure	One missing scrape job is an instant false page (§4.9)
NoData	Rules where absence is a distinct, interesting condition (a balance gauge that skips observation when the price feed fails)	Spawns DatasourceNoData instances that can bypass routing [4]
KeepLast	Flaky datasources during short gaps	Can hold a firing alert open through a real recovery

For up-based rules, prefer no_data_state = OK. up == 0 is a real outage. An absent up means only that the scrape job is missing, and total pipeline loss is already sev1 through the canary.

3.4 Global vs per-dimension aggregation

This is the most consequential decision in rule authoring, and §4.1 is the field lesson behind it. Rule of thumb: a global aggregate is safe only when (a) the metric has no independently breakable dimension, or (b) the condition is > 0 on a count. A ratio or a count threshold above what one slice can produce on its own needs by (dimension) or min() over the slices. The cost is alert cardinality and noisier grouping, which is cheap next to a two-week silent outage.

3.5 Severity granularity

Three tiers is the right number for teams under about 30 engineers. More tiers produce arguments about the tier instead of action. Revisit when you have multiple on-call rotations (per product or per domain). Then add a team label and route on team × severity, still with three severities.

3.6 Who writes and reviews rules

At seed stage, the person who builds the feature writes the rule. From Series A onward, every rule PR gets reviewed against the authoring guide (§5.1), and a bot or linter checks the mechanical items. The authoring guide is the cheapest high-leverage artefact in the system. Write each entry after a defect reaches production, as a rule rather than a style preference.

3.7 SLO targets: baseline-anchored vs contractual

Anchor alerting targets on current baseline and ratchet them [3]. Keep contractual SLAs separate, lower, and on a dashboard. When a partner asks for 99.9%, the alerting target stays where it is. The SLA report gets its own query over a calendar month.

3.8 Changing a target touches more than the rule

Budget 1 − target tends to get duplicated: hard-coded divisors in budget-viz panels, threshold lines, stat-panel colour thresholds, perf dashboards, runbook tables. Either template dashboards from the same Terraform map, or record the budget as a series (sli:budget per sli) so panels read it. Until then, keep a written checklist of the five or more places a target change touches.

---

4. Failure modes and field lessons

Every entry below happened in production on a real platform. Each has a mechanism, a fix and a review question.

4.1 The aggregate that hides a failing shard

What happened, four times on one platform:

• Pinned label. A gas top-up job for one wallet profile failed 100% for four days. The alert was pinned to a different profile.
• De-pinned, but blind to a slice. After the first fix de-pinned the rule to by (profile), another profile failed 100% for thirteen days. That profile wrote to a different metric family, so the de-pinned rule never saw it.
• Ratio diluted by healthy slices. One newly added chain ran at about 50% failure for a day behind a global sum() ratio, diluted by every healthy chain.
• Count threshold out of reach. A 5xx rule summed across 27 routes. A fully broken low-traffic route could return 100% 5xx indefinitely without producing enough absolute errors to reach the threshold.

Fix.

• Aggregate by (dimension), or use min() over a balance so the worst slice drives the rule.
• Before writing the rule:
  1. List every value the aggregated-away label actually takes in prod.
  2. Verify that every value emits the metric family the rule reads.
• Watch what de-pinning gives up. A single pinned series gave you NoData on total absence for free. A by (x) aggregate stays non-empty while any slice lives, so a fully dead slice goes stale and resolves.

Don’t mitigate with or (0 * last_over_time(m[24h])). Once a slice has been dead longer than 24h, that term goes empty and the blind spot comes back. Use a keep-alive that doesn’t depend on the dead metric’s history:

sum by (profile) (rate(leg_failed_total[30m]))
  or (0 * label_replace(vector(1), "profile", "gas-topup",  "", ""))
  or (0 * label_replace(vector(1), "profile", "sponsor",    "", ""))
# … one term per value in a hand-maintained list

Review question: “What is every value of the label you summed away, and does each one write this metric?”

4.2 Freshness with time() - m

time() is rule-evaluation time, so time() - last_update also includes the age of the last scrape. That adds a 0-to-scrape-interval sawtooth unrelated to real lag. With a 60s evaluation and a 30s scrape, the aliasing can hold the stale phase across consecutive evaluations and satisfy for: on a healthy target. One team measured 25.9s of “lag” on a market-data feed whose own health endpoint reported 0ms. Use timestamp(m) - m [11]. It keeps the subtraction inside the app’s clock and allows thresholds near the scrape interval.

4.3 The rule merged before its scrape job

The alert root applies on a fast dedicated workflow. The scrape job lives in an app root, and its NetworkPolicy allow lives in a GitOps repo. Merge them together and the rule goes live before its metric exists. With no_data_state = Alerting, that is an instant false page. In one rollout, sev1 paged for nine minutes with an empty label set. A [no value] in a summary template is the NoData tell. Order: (1) merge the scrape job and network allow; (2) wait for both the apply and the GitOps sync, then verify up{job="x"} returns one series per target, all 1; (3) merge the rules. More generally, a clean plan and an empty manifest diff describe the end state, not the transition. Ordering bugs are invisible to both.

4.4 Loki count alerts with a sum reducer

In a Grafana rule, a Loki count_over_time(...[5m]) range query returns one point per step [18]. A sum reduce then sums the rolling count across every step, giving a value proportional to the step count (around 277 in one case). The rule therefore fires on a single error regardless of its threshold. Canonical form:

sum(count_over_time({app="payments-api", env="prod"} |= "level=error" [5m])) or vector(0)

with a reduce of last, then the threshold.

4.5 The alert that matches its own query log

An RBAC-denial rule matched the phrase is forbidden. Loki and Grafana both log the text of every query they execute, including this rule’s own query, from their own pods. Every evaluation therefore produced a matching log line, an infinite false-positive loop with zero real denials. Fix: anchor the match on text that appears only in a genuine event and never in the query source, for example is forbidden: User "system:serviceaccount:. Alternatively, exclude the observability namespace explicitly. Any log-pattern alert over a stream that includes the observability stack’s own logs is exposed to this.

4.6 NaN in Grafana math

On a quiet short window, good/total = 0/0 = NaN. Grafana math && does not short-circuit on NaN, and the gt 0 evaluator treats NaN as firing. An SLO rule without !is_nan(...) guards flapped overnight on empty windows. Guard every ratio that feeds a condition.

4.7 The histogram whose buckets moved

An OpenTelemetry SDK upgrade switched a service from the SDK’s default histogram boundaries (…5000, 7500, 10000, which were being read as milliseconds) to the semantic-convention boundaries in seconds [17]. The SLI selected le="7500". Before the upgrade that bucket caught 100% of requests, so the SLI had silently measured 5xx only and never latency. After the upgrade the bucket vanished, good went to 0, and the slow burn fired with a real p99 of 0.9s. Diagnostic: count(..._bucket{le="<value>"}) == 0 means the boundary is gone. List live boundaries with sum by (le) (rate(..._bucket[1h])). Prevention: add a sev3 rule that fires when an SLI’s selected bucket has zero series.

4.8 Recording rules that evaluate but never persist

In Grafana 12, a Grafana-managed recording rule without target_datasource_uid, and with no [recording_rules] default_datasource_uid in grafana.ini, evaluates successfully and then fails to write: remote write failed: data source uid not specified and no default set. No sli:* series exist, and every SLO tile shows “No data”. Diagnose with /api/prometheus/grafana/api/v1/rules and look for health=error and lastError. Set the field explicitly [6].

4.9 Instrumented metrics that were never there

Several “obviously present” metrics turned out to be empty or wrong:

• Never instrumented. A nonce-resync counter referenced by a dashboard had 0 series. The same was true of ingress-controller metrics whose metrics.enabled: true block no chart template ever read.
• Collided names. Two OTel counters (external_api_requests and external_api_requests_total) both exported to one Prometheus name. The result was two series families with disjoint label sets. A bare sum double-counted, and status!~"2.." matched the status-less family, so every success counted as an error.
• Divergent label conventions. App OTel metrics used deployment_environment_name, kube-state and the RPC proxy used env, the HTTP checker used deployment_environment, and Loki used environment. A rule with the wrong one matches nothing and, with no_data = OK, reads green forever.
• Label name differs between exporters. Tempo’s span-metrics labelled the service as service, while app OTel metrics used service_name.

Rule: before merging, run the rule’s query through the datasource proxy and paste the result into the PR. Keep an “observability metrics: what exists, what doesn’t” doc. Every negative fact in it (“this metric has 0 series”) saves the next person an hour.

4.10 increase() and the first event

Classic Prometheus increase() needs two samples in the window. For a counter that only appears on a fault, the birth of the series (the first event) yields no result, or vector(0) substitutes 0, and a zero-tolerance rule stays green on exactly the regression it was written for. VictoriaMetrics considers the last sample before the window and doesn’t extrapolate [10], so it doesn’t have this problem. That changes if a rule ever moves to a Prometheus-semantics datasource. Portable form:

sum(increase(m[10m])) or sum(m) or vector(0)

Don’t “fix” it with sum(m) - sum(m offset 10m). Each pod roll drops the old series, so that reads negative for a full window after every deploy and masks anything that happens during it.

4.11 Short-lived pushers and instant queries

A metric pushed by a CronJob is stale between runs. Prometheus-style lookback drops it after about 5 minutes, so a */30 probe is absent around 83% of the time. That looks the same as a probe that never runs. Read such metrics with last_over_time(...[a few cadences]) and size windows off the job’s cadence, not by copying a neighbouring rule’s [15m].

4.12 Annotation templating bugs that mislead the responder

• $values.<ref>.Value pointing at a threshold node. A threshold expression’s value is 0/1, so a summary templated on it always read “1%” instead of the real success ratio. Template the math or query ref.
• Unencoded LogQL pipes (|=, |~) inside view_logs URLs break in Slack, which treats | as a link separator. Encode them as %7C.
• Wrong label in log links. A Loki link used the trace service name (api-ts) instead of the Loki app label (api) and returned zero lines for months.
• Dashboard annotation boolean as a string. useValueForTime: "false" is a truthy string. Markers land in 1970 and never render.

4.13 Bulk NoData storms that aren’t outages

Ten or more rules flipping to DatasourceNoData at once almost always means Grafana can’t reach the metrics store: an ingest gap, a TSDB under memory pressure, or a node pool shrunk too tight. It is rarely the services. One platform had repeated sev1 storms that traced back to 75–105 second ingest gaps after its observability node pool was consolidated onto two nodes. Give the storm its own runbook section (“confirm services are actually down before declaring”), and treat repeated storms as a capacity incident on the telemetry stack.

4.14 Ratios that exceed 1

A ratio cannot exceed 100%. If an SLI’s error rate reads above 100%, the numerator and denominator have stopped corresponding. In one case an expiry counter was labelled with the chain that triggered the sweep rather than the chain of the expired operation, which turned the mainnet/testnet split into nonsense. Write “an error rate above 100% is an instrumentation fault, not an incident” into the runbook. Wrap an optional subtraction term in or vector(0). In PromQL a binary operation against an empty vector yields an empty vector, so without the wrapper the healthy case (zero expiries) records nothing and the 30-day budget under-counts.

4.15 Proxy SLIs blind to the obvious failure

“Created minus expired over created” looks like an intent-completion SLI. It is blind to a broken expirer: stuck items never reach the expired counter, so the SLI reads perfect. Every proxy SLI needs a written statement of what it can’t see, and a companion liveness rule (here, “the expirer deployment has zero Running pods”, with or vector(0) so a deleted deployment still fires).

4.16 Paging fixes that never reached prod

A ticket that says “Done” isn’t evidence. On one platform the “prod approval gate” and a secret-scoping fix were both closed via documentation-only PRs, and the underlying change was never made. For alerting, the equivalent is a rule PR merged but never applied (§2.2), or applied to a separate Grafana org. Verify through the provisioning API, and fire a synthetic test alert after any contact-point change.

---

5. Best-practice checklist

5.1 Rule review checklist (copy into your PR template)

• severity label set; sev1 only if a human must act now
• runbook_url deep-links to an anchor that resolves, and the runbook section changed in this PR
• Query run through the datasource proxy; result pasted into the PR; env-label convention correct for this metric family
• Every value of each aggregated-away label listed; each emits this metric family
• no_data_state chosen deliberately, with the reason in a comment
• Ratios are NaN-guarded and volume-gated
• Freshness uses timestamp(m) - m; push-only metrics use last_over_time sized to cadence
• Loki counts use sum(count_over_time(...)) or vector(0) with a last reduce
• Log-pattern rule cannot match its own query text
• Annotations template the value ref, not the threshold ref; pipes URL-encoded
• If a new metric or scrape job is involved: it is already live (up == 1 per target) in a prior merge
• UID ≤ 40 chars; group name change understood to reset state

5.2 Day 1 (seed / first on-call)

1. Rules, contact points and the notification policy in Terraform, with plan on every PR [5].
2. Three severities; only sev1 pages; unlabelled → Slack.
3. PagerDuty escalation with primary and secondary, and a loop.
4. Two to four SLOs on the money path, with multi-window burn-rate alerts [1]. Fast burn only where volume allows.
5. Wallet balance floor alerts on every signer and fee-payer, per chain, min by (address).
6. Telemetry canaries (scrape and push) with no_data = Alerting.
7. An external heartbeat (Watchdog → Healthchecks.io or a PagerDuty heartbeat) [12][13].
8. A runbook section per alert and the co-change CI check.

5.3 First year (Series A)

1. Anchor-resolution CI check; a script to print one runbook section.
2. Recording rules for SLIs generated from the same map; budget-viz dashboards.
3. Runway (predict_linear) alongside floors; nonce-wedge rules; RPC network-failure ladder.
4. Freshness rules for every backup, restore-check, indexer and batch job.
5. Mainnet/testnet split everywhere a chain label exists; testnets one notch down.
6. An authoring guide, appended after each defect, and an observability-metrics doc of what exists and what doesn’t.
7. A weekly page review: every sev1 is classified actionable, false positive or duplicate, and each non-actionable page gets a ticket [15].
8. PagerDuty change events from CD; Event Orchestration priority stamping.
9. HA for the external-probe collector, with dedup.

5.4 At scale (Series B+)

1. team × severity routing; per-team rotations.
2. Generated runbook metadata (UID, severity, for, threshold) from Terraform, so it cannot drift.
3. Dashboards templated from the SLO map, or budget recorded as a series.
4. HA for the metrics store (or an explicitly accepted risk); evaluator isolation.
5. Quarterly claim audit: re-verify every documented “this metric exists or doesn’t” and every runbook metadata line.
6. Synthetic end-to-end probes on the money path (quote → intent → fill) feeding an SLI.
7. An error-budget policy agreed with product [3].

---

6. Maturity roadmap

Stage	Alerting	SLOs	Paging	Meta-monitoring	Hygiene
0 — Ad hoc	UI-clicked rules, CPU and pod restarts	None	Everything to one Slack channel	None	None
1 — Codified	Terraform rules, severity labels, runbook links	2–4 burn-rate SLOs	sev1 → PagerDuty, primary and secondary	Telemetry canaries	Co-change CI
2 — Trustworthy	Per-dimension aggregation everywhere; freshness rules; on-chain floor, runway and nonce rules	Recording rules; volume gates; NaN guards; mainnet/testnet split	Escalation loop, change events, priority stamping	External heartbeat; HA probe collector	Anchor CI; authoring guide; weekly page review
3 — Self-verifying	Rules linted against the authoring guide in CI; bucket-existence and metric-existence meta-rules	Budget as a recorded series; dashboards generated; error-budget policy	Team routing	HA metrics store; game-day tests of the watchdog	Generated runbook metadata; quarterly claim audit

Both the Well-Architected Operational Excellence pillar [19] and DORA’s monitoring capability [20] frame this as continuous improvement, not a one-off project. Most startups that think they are at stage 2 are at stage 1 with gaps in the right-hand columns. Those gaps are where silent failures live. The highest-leverage single step from stage 1 is usually the external heartbeat plus a per-dimension aggregation audit. Together they close the two classes of failure that produce weeks-long silent incidents.

---

7. A worked week: putting it together

To show how the pieces interact, here is a plausible week on a mid-stage relayer platform running this design.

• Monday 02:10. A testnet faucet wallet drains. Without the testnet split, the completion SLO pages sev1. With it, the testnet slow burn posts to Slack at sev3, and the runway rule posts sev3 about two hours before zero. Nobody wakes up. The morning triage tops up the wallet.
• Tuesday 14:30. A deploy lands. The /quotes p99.9 sev3 starts firing within ten minutes, then the 30m/6h slow burn reaches sev2 an hour later. The PagerDuty change event and the Grafana deploy annotation put the deploy next to the regression, and the team rolls back. The leading indicator gave about 50 minutes of warning before the budget alert.
• Wednesday 09:00. The weekly page review finds that a sev1 for a background worker’s restart storm was non-actionable: it self-healed. They lower it to sev3 and add a zero-Running-pods sev1 in its place.
• Thursday 04:45. The metrics store’s node runs out of memory. Thirty rules flip to DatasourceNoData. The scrape canary pages sev1 at 04:55. The runbook’s NoData-storm section tells the on-call to confirm services through the external probe dashboard first. They are fine, so the incident is scoped to the telemetry stack.
• Saturday 11:20. Grafana’s database fails over and Grafana’s pods crash-loop on reconnect. No in-Grafana rule can fire. The external heartbeat misses its grace window and pages through a PagerDuty integration that doesn’t touch Grafana, 7 minutes after the last ping.

Each of these outcomes depends on one specific design choice above. Take that choice away and the event becomes either a 3 a.m. false page or an undetected outage.

---

References

1. Google SRE Workbook, “Alerting on SLOs” (Ch. 5). https://sre.google/workbook/alerting-on-slos/
2. Google SRE Book, “Monitoring Distributed Systems” (Ch. 6). https://sre.google/sre-book/monitoring-distributed-systems/
3. Google SRE Workbook, “Implementing SLOs” (Ch. 2). https://sre.google/workbook/implementing-slos/
4. Grafana documentation, “No Data and Error states”. https://grafana.com/docs/grafana/latest/alerting/fundamentals/alert-rule-evaluation/nodata-and-error-states/
5. Grafana documentation, “Use Terraform to provision alerting resources”. https://grafana.com/docs/grafana/latest/alerting/set-up/provision-alerting-resources/terraform-provisioning/
6. Grafana Terraform provider, grafana_rule_group resource (fields no_data_state, exec_err_state, keep_firing_for, record.target_datasource_uid, notification_settings). https://github.com/grafana/terraform-provider-grafana/blob/main/docs/resources/rule_group.md
7. Prometheus documentation, “Alerting” best practices. https://prometheus.io/docs/practices/alerting/
8. Prometheus documentation, “Recording rules” (naming and aggregation). https://prometheus.io/docs/practices/rules/
9. AWS Well-Architected Framework, Reliability Pillar, “Monitor workload resources” (REL06). https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/monitor-workload-resources.html
10. VictoriaMetrics documentation, “MetricsQL” (increase() semantics vs Prometheus). https://docs.victoriametrics.com/victoriametrics/metricsql/
11. Prometheus documentation, “Query functions” (timestamp, predict_linear, absent_over_time). https://prometheus.io/docs/prometheus/latest/querying/functions/
12. Prometheus Operator runbooks, “Watchdog”. https://runbooks.prometheus-operator.dev/runbooks/general/watchdog/
13. Healthchecks.io documentation (dead man’s switch, period and grace time). https://healthchecks.io/docs/
14. Google SRE Book, “Being On-Call” (Ch. 11). https://sre.google/sre-book/being-on-call/
15. Google SRE Workbook, “On-Call” (Ch. 8). https://sre.google/workbook/on-call/
16. PagerDuty developer documentation, “Events API v2 overview”. https://docs.pagerduty.com/developer/events-api-v2-overview
17. OpenTelemetry semantic conventions, “HTTP metrics” (http.server.request.duration bucket boundaries). https://opentelemetry.io/docs/specs/semconv/http/http-metrics/
18. Grafana Loki documentation, “Metric queries” (count_over_time). https://grafana.com/docs/loki/latest/query/metric_queries/
19. AWS Well-Architected Framework, Operational Excellence Pillar. https://docs.aws.amazon.com/wellarchitected/latest/operational-excellence-pillar/welcome.html
20. DORA, “Capabilities: Monitoring and observability”. https://dora.dev/capabilities/monitoring-and-observability/