Self-Hosted Observability on EKS: Grafana, Loki, Tempo, VictoriaMetrics and OpenTelemetry

Abstract. This guide is for senior platform, DevOps and SRE engineers at AWS-native startups, crypto and web3 infrastructure companies in particular, who run or are thinking about running their own metrics, logs and traces stack. It covers the economics of self-hosting compared with a SaaS vendor. It sets out an opinionated reference design: Grafana, Loki, Tempo and VictoriaMetrics on a dedicated EKS cluster, fed by a two-tier OpenTelemetry Collector topology, with S3 as the durable tier. It also covers the decisions that design forces on you: dedicated versus in-cluster, single-replica versus HA, agent versus gateway collectors, and DaemonSet log shipping versus OTLP. Most of the guide is about failure modes. A self-hosted observability stack rarely fails loudly. It fails by going quiet, by returning a plausible number that is wrong, or by paging fifteen people for an outage that never happened. The lessons below come from a production platform that moved from a single EC2 docker-compose box to Helm on EKS. They are generalized and anonymized.

---

1. The problem space

1.1 Why this is harder at a startup than it looks

A startup’s observability stack has two jobs that pull against each other. It has to be cheap and low-maintenance, because nobody is staffed to run it. It also has to be the most reliable thing you own, because it is how you find out everything else is broken. The Google SRE book frames monitoring as the thing that tells you what is broken and why, and warns that pages must be actionable and that noisy alerting destroys trust in the system [1]. A self-hosted stack adds a third concern the SaaS vendor normally hides: the monitoring system itself is now a production system with its own failure modes. Those failure modes are correlated with your incidents. Observability gets queried hardest exactly when something is on fire.

Crypto infrastructure raises the stakes in particular ways:

• Silent failure is the expensive kind. A relayer that stops filling, an indexer that stops advancing, or a gas top-up job that fails 100% of the time all look like “nothing happening” on a dashboard. A missing metric and a healthy zero read the same unless you design for the difference.
• Hot-wallet balance and signer health are monitored signals. If your metrics pipeline is blind, so is your low-balance alarm. When it goes blind, the next sign of trouble is a stuck transaction or an angry partner.
• Partners and auditors ask about it. “How do you know when settlement is degraded?” is a standard due-diligence question. “We have Grafana” is not an answer. “Multi-window burn-rate SLO alerts, a telemetry dead-man canary, and an external heartbeat on the alerting engine” is.
• Vendor data residency and data-sharing concerns. Logs often contain wallet addresses, transaction payloads, partner identifiers and occasionally things that should never have been logged. Some teams prefer that data never leave their AWS account.

1.2 The forces

Force	Pulls toward
Tiny team, no dedicated o11y owner	SaaS
Telemetry volume grows faster than revenue (traces, chatty indexers, per-chain metrics)	Self-host, or very disciplined SaaS usage
Cardinality from multi-chain dimensions (chain × token × route × provider)	Self-host (series-based pricing punishes this)
Data stays in-account	Self-host
“It must work when prod is degraded”	Separate failure domain, which either SaaS or a dedicated cluster provides
Alerting must never silently stop	Either works, as long as there is an external watchdog

AWS’s Well-Architected Operational Excellence pillar asks you to implement observability as a designed capability. That means identifying KPIs, instrumenting for them and alerting on them, not just deploying tooling [3]. Neither SaaS nor self-hosting gives you that by default.

---

2. Self-host vs SaaS: the economics

2.1 What self-hosting really costs

A representative small self-hosted footprint for a Series-A-scale platform looks like this. It has roughly 15–25 services, around 500 metric names, about 100k active series, tens of GB of logs a day, and traces from the services that matter.

Item	Rough monthly cost (us-east-1, on-demand)
EKS control plane for a dedicated tools cluster	~$73
2× Graviton general-purpose nodes (4 vCPU / 16 GiB class)	~$200–250
VictoriaMetrics PVC, 600 GiB gp3	~$50
S3 for Loki chunks + Tempo blocks + metric backups (low hundreds of GB, 90-day retention)	~$10–40, plus request costs
NLB for ingest + internal ALB for the UI	~$40–60 plus LCUs
Small RDS Postgres for Grafana state	~$30–60
Cross-AZ / NAT data transfer for telemetry	Highly variable; often the surprise line
Infrastructure total	~$450–700

That number is real, and it is small. It is also not the cost. The cost is engineering attention. The platform this guide draws on spent, over roughly six months, meaningful time on all of the following: one storage meltdown, one cardinality incident, two bulk false-page storms, a trace pipeline that dropped 100% of spans for over a day, a cluster that shipped no telemetry at all for weeks, an auth-proxy forgery finding, and a steady stream of “this query returns the wrong number” corrections. Price two to four engineer-days a month of senior time into the self-host column at steady state, more during migrations. AWS’s EKS cost-optimization guidance on observability makes the same point from the other side. Most observability spend is driven by what you choose to collect and keep, not by the tooling [24].

2.2 What SaaS really costs

SaaS pricing models differ by vendor, but they all charge on some mix of ingested log GB, active metric series (or custom metrics), indexed spans, and hosts or containers. Check current public price pages rather than trusting any number in a blog post. These structural points hold regardless:

• Series-based pricing punishes the dimensions crypto platforms care about. A metric labelled by chain_id × token × route × provider × status can produce tens of thousands of series from a single counter. On a self-hosted TSDB that is a little RAM. On series-priced SaaS it is a line item that someone will eventually ask you to cut, and the dimension you cut is the one that would have shown “chain X is failing 50% while the global rate looks fine.”
• Log volume is spiky and incident-correlated. An error loop can 10× daily volume in an hour, exactly when you can least afford to sample it away.
• The free or startup tier is a trap only if you design for it. Grafana Cloud’s free and low tiers, and vendor startup credits, are excellent for seed stage. Using them is a sound choice. Building a cardinality budget around them, and then discovering the cliff, is the trap.

2.3 A decision rule

• Seed / pre-product-market-fit (≤5 engineers): use SaaS. Grafana Cloud is the natural choice if you want to stay on the OSS query languages (PromQL/LogQL/TraceQL), which keeps a future self-host migration cheap. Instrument with OpenTelemetry SDKs from day one so the backend is swappable [9][10].
• Series A (telemetry bill trending past a few thousand dollars a month, or a hard data-residency requirement): self-hosting starts to pay off, if one engineer is willing to own it and you adopt the reliability guardrails in §5 and §6.
• Scale (dedicated platform team): either path works. Self-hosting at this point usually means moving to the clustered or microservices topologies (VMCluster, Loki simple-scalable or distributed, Tempo distributed), or to a hybrid where long-term storage stays self-hosted and alerting has an external watchdog.

Revisit signals: the observability bill exceeds roughly 10–15% of total AWS spend; you are cutting labels to save money rather than to fix design; or you have had a “monitoring was down during the incident” retrospective.

---

3. Reference design

3.1 Topology

The key properties:

1. Separate failure domain. The stack lives in its own cluster, in its own VPC, so a prod node-pool exhaustion, a bad CNI upgrade or a runaway HPA does not take monitoring down with it.
2. S3 is the durable tier wherever the component supports it. Loki chunks and index (TSDB schema v13), and Tempo blocks (vParquet). Local disks hold only WALs and caches.
3. Two collector tiers. Per-cluster collectors near the apps handle batching, resource attribution and pull-based receivers. A gateway tier in the observability cluster handles sampling decisions and fan-out [9][10].
4. Alerting is evaluated by Grafana, and Grafana is itself watched from outside.

3.2 Deploying it: Helm via Terraform

Manage the releases as helm_release resources in Terraform (or as ArgoCD Applications; either works). Pin both chart and image versions. The VictoriaMetrics release from the reference platform, trimmed:

resource "helm_release" "victoria_metrics" {
  name       = "victoria-metrics"
  repository = "https://victoriametrics.github.io/helm-charts/"
  chart      = "victoria-metrics-single"
  version    = "0.40.1"            # pin; read the chart's values.yaml for this version
  namespace  = "observability"
  atomic     = true

  values = [yamlencode({
    server = {
      retentionPeriod = "90d"
      extraArgs = {
        "search.maxConcurrentRequests" = "4"
        # Keep queued queries under Grafana's 30s eval/dataproxy timeout,
        # or a burst of rule evaluations turns into DatasourceError pages.
        "search.maxQueueDuration" = "20s"
        "memory.allowedPercent"   = "60"
        # Collapse duplicate samples from HA collector pairs (see §4.4).
        "dedup.minScrapeInterval" = "15s"
      }
      persistentVolume = { enabled = true, size = "600Gi", storageClassName = "gp3" }
      resources = {
        requests = { cpu = "500m", memory = "2Gi" }
        limits   = { cpu = "1",    memory = "4Gi" }
      }
      # The PVC is the ONLY live copy of metric history. Back it up.
      extraContainers = [{
        name  = "vmbackup"
        image = "victoriametrics/vmbackup:v1.145.0"   # match server version
        # loop: snapshot via :8428/snapshot/create, vmbackup to s3://…, sleep 86400
      }]
    }
  })]
}

Two details in that block cost the reference team real time:

• volumeClaimTemplate size is immutable on a StatefulSet. Growing the VictoriaMetrics PVC through Helm fails on the live StatefulSet. Plan the migration as: resize or re-create the volume out of band, orphan-delete the StatefulSet, then let the next apply recreate it and adopt the existing pod and PVC.
• VictoriaMetrics single-node has no object-storage tier. Unlike Loki and Tempo, losing that PVC loses history back to your last vmbackup. A daily backup sidecar plus a scheduled restore check is the minimum. A backup that has never been restored is a hope, not a backup.

3.3 Loki: SingleBinary on S3, with cardinality limits set from day one

deploymentMode: SingleBinary
loki:
  auth_enabled: false            # single tenant
  commonConfig: { replication_factor: 1 }
  schemaConfig:
    configs:
      - from: "2024-01-01"
        store: tsdb
        object_store: s3
        schema: v13
        index: { prefix: index_, period: 24h }
  limits_config:
    retention_period: 2160h      # 90d
    max_streams_per_user: 100000 # a ceiling, not a target; healthy is hundreds
    query_timeout: 30s
    split_queries_by_interval: 30m
    max_entries_limit_per_query: 50000
  compactor: { retention_enabled: true, delete_request_store: s3 }
  querier: { max_concurrent: 4, query_ingesters_within: 4h }
  query_range:
    cache_results: true
    results_cache: { cache: { embedded_cache: { enabled: true, max_size_mb: 300 } } }
singleBinary:
  replicas: 1
  persistence: { enabled: true, size: 10Gi, storageClass: gp3 }
backend: { replicas: 0 }
read:    { replicas: 0 }
write:   { replicas: 0 }
minio:   { enabled: false }

Loki’s deployment-modes documentation positions monolithic mode for small volumes (roughly up to tens of GB a day) and simple-scalable or microservices modes beyond that [22]. SingleBinary on S3 is the right starting point for most startups. It is durable because the chunks are in S3, and it is cheap. Its one weakness is that a pod restart is a brief ingest gap, which the log shipper’s retry buffer absorbs.

3.4 Tempo: two replicas means a real ring

If you run more than one Tempo replica from the single-binary chart, you must switch the target to scalable-single-binary and use memberlist for the rings. §5.4 explains why.

target: scalable-single-binary
memberlist:
  join_members: [tempo-headless]      # create this headless Service if the chart doesn't
  abort_if_cluster_join_fails: false
ingester:
  lifecycler:
    ring:
      kvstore: { store: memberlist }
      replication_factor: 2           # every trace on both ingesters until flushed
compactor:
  compaction: { block_retention: 2160h }
  ring: { kvstore: { store: memberlist } }
storage:
  trace:
    backend: s3
    block: { version: vParquet4 }
    wal: { path: /var/tempo/wal }
metrics_generator:
  ring: { kvstore: { store: memberlist } }
  processor:
    service_graphs:
      dimensions: [deployment.environment.name]
    span_metrics:
      dimensions: [deployment.environment.name, chain.id, error.type]
  storage:
    remote_write:
      - url: http://victoria-metrics.observability.svc:8428/api/v1/write
overrides:
  metrics_generator_processors: [span-metrics, service-graphs]

Tempo’s deployment documentation describes the monolithic, scalable-monolithic and microservices modes and their tradeoffs [13]. Put deployment.environment.name on the service-graph dimensions from the first day. Without it, every dev and prod edge lands in the same series, and the derived graph cannot be read as a statement about prod.

3.5 Grafana: stateless pods, state in Postgres, HA alerting

Grafana can run more than one replica only once its database is external (Postgres or MySQL). Per-pod SQLite on a ReadWriteOnce volume forces one replica and the Recreate strategy. With an external database you can run two replicas, a PodDisruptionBudget of minAvailable: 1, soft anti-affinity, and unified alerting in HA mode via memberlist gossip over a headless Service [21][20]:

[unified_alerting]
enabled = true
ha_peers = grafana-headless:9094
ha_listen_address = ${POD_IP}:9094
ha_advertise_address = ${POD_IP}:9094

Use soft (preferred) anti-affinity on a small, fixed node pool with no autoscaler. A hard rule can leave the second replica Pending forever when one node is briefly unschedulable, and nothing will correct that on its own.

Manage dashboards, datasources, alert rules, contact points and notification policies as code (the Grafana Terraform provider), and treat UI edits as drift that the next apply overwrites.

3.6 OpenTelemetry Collector topology

Agent tier (per workload cluster). The agent receives OTLP from SDKs, adds Kubernetes resource attributes, batches, and exports. It also hosts pull-based receivers, such as prometheus scrape jobs (kube-state-metrics, cAdvisor, third-party exporters) and httpcheck synthetic probes. OpenTelemetry documents both the agent pattern and the gateway pattern [9][10].

Gateway tier (observability cluster). The gateway receives from every agent, applies sampling and policy, and writes to the backends.

A minimal agent config with the processors that matter for resilience [19]:

receivers:
  otlp:
    protocols: { grpc: {}, http: {} }
processors:
  memory_limiter: { check_interval: 1s, limit_percentage: 80, spike_limit_percentage: 20 }
  k8sattributes: {}
  resource:
    attributes:
      - { key: deployment.environment.name, value: prod, action: upsert }
  batch: { send_batch_size: 512, send_batch_max_size: 1024, timeout: 5s }
exporters:
  prometheusremotewrite:
    endpoint: https://o11y-ingest.internal.example:8428/api/v1/write
    resource_to_telemetry_conversion: { enabled: true }
    retry_on_failure: { enabled: true, max_elapsed_time: 300s }
    remote_write_queue: { enabled: true, queue_size: 10000 }
  otlphttp/traces:
    endpoint: https://o11y-ingest.internal.example:4318
    sending_queue: { enabled: true, queue_size: 5000 }
service:
  pipelines:
    metrics: { receivers: [otlp], processors: [memory_limiter, k8sattributes, resource, batch], exporters: [prometheusremotewrite] }
    traces:  { receivers: [otlp], processors: [memory_limiter, k8sattributes, resource, batch], exporters: [otlphttp/traces] }

Split pull-based receivers into their own release. A synthetic-probe collector (httpcheck against every public and internal health endpoint, plus TLS-expiry checks) is the single source of truth for every “is service X up” alert on the platform. It deserves its own Deployment with two replicas, anti-affinity and a PDB. Both replicas probe the same targets with identical labels, and VictoriaMetrics’ -dedup.minScrapeInterval (set ≥ the probe interval) collapses them into one series [12]. No query or alert changes, no double counting. §4.4 covers this in more detail.

3.7 Log shipping

The reference platform’s architecture doc said for months that “logs are emitted over OTLP, there is no DaemonSet.” That was false. A Promtail DaemonSet in every cluster was the only log shipper, and the team found out when it was OOMKilled. Whichever path you choose, write it down correctly.

The practical default on EKS today is a node-level DaemonSet tailing container stdout/stderr: Grafana Alloy, or Promtail if you still run it, though note Promtail reached end-of-life on 2 March 2026 and Grafana recommends migrating to Alloy [14][25]. It captures everything, including crashes before an SDK initializes, sidecars and third-party images, without app changes. Keep promoted labels to a minimal, bounded set:

# Alloy / Promtail stage: the ONLY labels that become Loki stream labels
- labels:
    namespace:
    app:
    container:
    level:        # bounded: debug|info|warn|error
# trace_id, span_id, request_id, wallet, tx_hash, error message: stay in the line.
# Query them with `| json | trace_id="..."` or structured metadata, never as labels.

OTLP logs straight from the SDK are attractive because they keep trace context natively. They cannot capture output from anything you didn’t instrument, though, and they put the export path inside your application’s process. See §4.5.

3.8 Grafana authentication

The pattern used on the reference platform is an internal ALB reachable only over VPN, with Cognito (federating Google SSO) as the authenticate action. A sidecar proxy in the Grafana pod translates the ALB’s identity header into Grafana’s auth.proxy headers:

The AWS documentation is explicit about what the target must do. Verify the ES256 signature of x-amzn-oidc-data using the public key fetched from https://public-keys.auth.elb.<region>.amazonaws.com/<kid>, and confirm the signer field in the JWT header equals your ALB’s ARN, before doing any authorization based on the claims. The other two headers (x-amzn-oidc-identity, x-amzn-oidc-accesstoken) are unsigned and must never be trusted [4]. The sidecar’s job, in order:

1. Strip any client-supplied X-WEBAUTH-* headers.
2. Parse the JWT header. Require alg == ES256. Fetch the key by kid and cache it per kid.
3. Verify the signature. Require signer == <expected ALB ARN>. Check exp, which the ALB places in the header, not the payload, with small leeway.
4. Apply the email-domain policy to the verified payload.
5. Set X-WEBAUTH-USER / X-WEBAUTH-EMAIL and proxy to Grafana on localhost.

On the Grafana side, accept proxy headers only from the sidecar, and default new users to Viewer [5]:

[auth.proxy]
enabled = true
header_name = X-WEBAUTH-USER
header_property = username
auto_sign_up = true
whitelist = 127.0.0.1, ::1
[users]
auto_assign_org_role = Viewer

If you need API access for automation, the usual approach is an ALB rule that lets /api/* requests carrying Authorization: Bearer skip the authenticate action, so they authenticate by service-account token only. Treat that as a second auth path, and audit it as one.

The simpler alternative is Grafana’s native OAuth/OIDC ([auth.generic_oauth], or the Google/Okta/Azure-specific providers) pointed at your IdP [6]. It removes the sidecar, the header-trust problem and the JWT-verification code entirely, and adds role mapping from IdP groups. Prefer it unless you need ALB-level authentication in front of Grafana’s own login surface. The decision record in §4.7 covers this.

---

4. Decisions and tradeoffs

4.1 Dedicated observability cluster vs in-cluster

	In the prod cluster	Dedicated cluster (shared “internal tools” cluster)
Cost	Lowest; no extra control plane	+~$73/mo control plane, plus nodes
Failure independence	Poor: node exhaustion, CNI, DNS, or a bad upgrade takes monitoring down with prod	Good, if networking is also independent
Multi-environment	Each env monitors itself; cross-env views are awkward	One pane for dev + prod
Networking	Trivial	Requires cross-VPC ingest (NLB/PrivateLink), SG allow-lists, and correct in-cluster vs external URLs
Blast radius of a mistake in o11y config	Can starve prod workloads	Contained

Recommendation: dedicated from Series A onward. A shared “internal tools” cluster that also hosts ArgoCD, back-office and BI tools is a reasonable compromise, but enforce resource isolation (a dedicated node group or at least strict requests) and NetworkPolicy. §5.12 shows what happens without them.

Revisit when the shared tools cluster’s other tenants begin competing with the stack for headroom, or you need per-tenant isolation.

4.2 Single replica vs HA, per component

There is no single answer. Decide component by component, based on what a pod restart actually costs:

Component	Starting point	What a restart costs	When to go HA
Grafana	2 replicas + external DB	UI blip; with one replica, all alert evaluation stops	Day one, once you have an external DB. It is the alerting engine.
Probe / httpcheck collector	2 replicas + dedup	Every synthetic health alert flips to NoData at once	Day one
Agent collectors	2 replicas	Gap in push metrics/traces; SDKs retry briefly	Early
Tempo	2 (scalable single binary, RF=2)	Unflushed traces lost; queries partial	When traces drive incident response
Loki	1 (SingleBinary, S3)	Short ingest gap; shippers buffer and retry	When daily volume or query load outgrows one pod, move to simple-scalable
VictoriaMetrics	1 (PVC + daily backup)	Every metric alert goes NoData during restart	When the “accepted risk” is no longer accepted: VMCluster, or two independent single nodes behind dedup

The weak point people miss is that making collectors HA does not help if every collector writes to one VictoriaMetrics pod. The reference platform made its probe collector HA after a node failure, then saw both healthy replicas fail at the same moment because the single downstream TSDB was too slow to answer within the remote-write deadline (§5.11). Either record “metrics path = chain of single pods” as an explicit accepted risk, with the dead-man canary in §5.13 as compensation, or make the TSDB redundant.

4.3 Agent vs gateway collectors

• Agent-only. Simplest. Every cluster exports directly to the backends. Sampling and policy are duplicated per cluster.
• Agent + gateway. Central place for tail sampling, PII scrubbing, routing and rate limiting. Costs an extra hop and an extra component to keep alive. Tail sampling requires all spans of a trace to reach the same gateway instance, so a horizontally scaled gateway needs trace-ID-aware load balancing (the loadbalancing exporter) [11][18].
• DaemonSet agents vs Deployment agents. DaemonSets suit node-local scraping (kubelet/cAdvisor) and log tailing. Deployments suit OTLP receive and cluster-scoped scrapes. Many teams run both.

Recommendation: agent and gateway, with the gateway kept dumb until you have a reason. Every processor you add to the gateway is code on the critical path of all telemetry. §5.3 shows what happens when a sampling processor misbehaves.

4.4 HA for pull-based receivers: the dedup trick

Naively raising replicaCount on a collector that scrapes or probes produces duplicate data, because each replica scrapes independently. Three options:

1. Identical labels + TSDB dedup. Both replicas emit byte-identical series, and VictoriaMetrics keeps one sample per dedup.minScrapeInterval window [12]. Zero downstream change. The interval must cover the slowest duplicated job’s interval, so split fast and slow jobs into separate collectors if the intervals differ widely.
2. Target allocator (OpenTelemetry Operator) shards targets across replicas. No duplication, but losing a replica loses its shard until rebalance.
3. Replica label + query-time dedup. Works, but every query and alert must deduplicate by hand.

Option 1 is the right choice for synthetic probes. Option 2 suits large scrape fleets.

4.5 DaemonSet log shipper vs OTLP logs

	DaemonSet (Alloy/Promtail)	OTLP from SDK
Coverage	Everything that writes to stdout, including crashes before init	Only instrumented code paths
Trace correlation	Parse trace_id from JSON lines	Native
Failure isolation	Shipper OOM loses logs on one node	Export backpressure lives in your app process
Resource cost	One pod per node; sized by the noisiest node’s volume	Spread across apps
Scheduling gotchas	Must tolerate every taint (arm64, GPU, dedicated pools)	None

Recommendation: run a DaemonSet as the baseline, with JSON logs that include trace_id/span_id as fields, and use Grafana’s derived fields for log-to-trace links. Revisit when you adopt the OTel Operator fleet-wide and want one pipeline for all three signals.

4.6 Storage: local disk vs S3

For Loki and Tempo the answer is S3, always. Their compactors are designed around object storage, and running them on block storage turns compaction into an IOPS problem (§5.5). For VictoriaMetrics single-node, local gp3 is the only option. Size its IOPS and throughput explicitly, give it its own volume, and back it up to S3.

4.7 Grafana authentication options

Option	Upside	Downside
ALB authenticate + auth-proxy sidecar	Nothing reaches Grafana unauthenticated; one IdP for all internal tools	You own JWT verification code; header-spoofing risk if verification is wrong or skipped; plaintext ALB→pod hop allows replay until exp unless the target group uses HTTPS
Native Grafana OAuth/OIDC	No sidecar; group→role mapping; maintained upstream	Grafana’s login page is exposed (mitigate: internal ALB + VPN)
VPN-only + Grafana local users	Simple	No SSO, offboarding is manual

Recommendation: native OIDC behind an internal load balancer and VPN, unless you have a concrete reason for ALB-level authentication. If you keep the sidecar, verify the JWT exactly as AWS specifies [4], and add NetworkPolicy so only the ALB can reach the pod port. Note that “only from the ALB” often cannot be written as an ipBlock when the ALB and pods share subnets. You need security groups for pods or a subnet split.

---

5. Failure modes and field lessons

Every item here happened on a real production platform. Most of them share a property: nothing errored. The pipeline was configured, the pods were Running, the dashboards rendered, and the data was missing, partial or wrong.

5.1 A whole cluster shipped nothing for weeks, with every exporter “configured”

The tools cluster that hosted the observability stack sent its own apps’ telemetry to the stack’s public NLB hostname, the same endpoint dev and prod use from their VPCs. From inside the same VPC that path hairpinned out through NAT and back to a load balancer whose security group allow-listed the dev and prod NAT IPs, not this VPC’s own NAT. Every export was dropped at the SG. Metrics, traces and logs for that cluster were absent from the day it was built until someone noticed weeks later.

Lessons:

• Workloads in the same cluster as the stack must use in-cluster Service DNS names (loki.observability.svc.cluster.local:3100). External clusters use the external endpoint. Derive both from the same source (for example, the Service names that the NLB’s TargetGroupBindings reference) so they cannot drift.
• Verify each source, not the pipeline. After onboarding any cluster, run a presence query per signal: count by (cluster) (up), sum by (cluster) (count_over_time({cluster="X"}[1h])), a Tempo search scoped to the environment. “The exporter is configured” proves nothing.

5.2 The stats API said zero streams for a stream that was actively shipping

Loki’s index-stats endpoint, over its default window, returned {"streams": 0} for a selector while a metric query over the same selector counted thousands of entries in 90 minutes. Confirm presence with sum by (app) (count_over_time({...}[90m])), not with stats or label-value APIs, before concluding a stream is gone.

5.3 The tail-sampling processor dropped 100% of traces while reporting success

A gateway collector ran tail_sampling with a policy to keep all traces except one very chatty service, of which it would keep errors plus 10%. On the pinned contrib version, the processor’s own metrics showed correct sampled=true decisions, but it never forwarded spans to the next consumer. Every trace from every service was dropped for over a day. The team diagnosed it by splicing a debug exporter into the traces pipeline: removing tail_sampling alone made data flow again.

Lessons:

• A processor’s self-reported metrics are not evidence that data reached the backend. Alert on backend-side ingestion (Tempo’s received-spans rate, or a “distinct services with spans in the last 15m” count), not collector-side counters.
• Keep the gateway minimal. If you need volume reduction, probabilistic_sampler at the agent is stateless and much harder to break. Adopt tail sampling deliberately, with load-balanced routing by trace ID and a canary [18].
• Pin collector versions, and treat collector upgrades as deployments with verification.

5.4 Two Tempo replicas silently returned partial results

The team went from one to two Tempo replicas for HA using the single-binary chart. The config said kvstore: memberlist. The running process said inmemory. Tempo’s own startup code unconditionally forces the ingester and metrics-generator rings back to inmemory (and ingester RF to 1) when the target is the single-binary default. It does this after parsing the config file. Each pod therefore had a private ring. Distributors wrote to whichever pod the Service picked, and the querier on the other pod had no idea that data existed. The result was not an outage. Queries returned some traces.

Lessons:

• Use target: scalable-single-binary, which is Tempo’s documented mode for multiple replicas of the monolith [13]. Create the headless Service if the chart doesn’t, and use RF=2 if you want an ingester crash not to lose unflushed traces.
• Check the effective config, not the ConfigMap. Tempo, Loki and Mimir all expose /status/config (or /config). Diff the runtime config against your intent after every topology change.

5.5 Tempo on local disk ate the IOPS budget, and the cascade paged everyone

On the original single-EC2 docker-compose deployment, Tempo was briefly switched to local EBS storage. Within a week its blocks reached ~280 GB, and continuous compaction exhausted the IOPS budget of the volume it shared with VictoriaMetrics. VictoriaMetrics reads starved, queries timed out, the collector backed up and was OOMKilled, every metric alert went NoData, and PagerDuty fired.

Lessons:

• Tempo and Loki belong on S3. Their local disks should hold only WAL and cache.
• A single box running every component means one resource (IOPS, memory, CPU) becomes a shared failure domain for all signals. This is the strongest argument for moving off the docker-compose box, ahead of scalability.
• If you must share a volume, set provisioned IOPS and throughput explicitly. Don’t rely on gp3 baseline.

5.6 trace_id as a Loki label: 377k streams and 60-second query timeouts

One environment’s log shipper promoted trace_id, span_id and a free-form context field to Loki labels. Every request produced a new, permanently unique stream. The result was 377k+ streams (against ~100 for a healthy environment), 1.7M chunk files and a 235 MB index. Any query with a non-trivial limit exceeded Loki’s 60s timeout. After the labels were removed, retention was temporarily cut to 48 hours to force the compactor to delete the poisoned chunks, and query latency went from timing out to under 200 ms. The cost was five days of log history.

Grafana’s label best-practices guide is blunt: use labels for low-cardinality values that describe the source of the log, and keep high-cardinality values (IDs, user identifiers, IPs) in the line or in structured metadata [7][8].

Guardrails worth adopting before you need them:

# Alert: new-stream creation spike (coarse guardrail, tune to your baseline)
sum(increase(loki_ingester_streams_created_total[30m])) > 2000

• Scrape Loki’s own /metrics (keep only loki_ingester_memory_streams, loki_ingester_memory_chunks and loki_ingester_streams_created_total to stay lean).
• Set max_streams_per_user and max_label_names_per_series so a bad deploy gets rejected rather than absorbed.
• Allow-list shipper labels (as in §3.7) rather than relying on review to catch new ones. For crypto platforms the tempting label is tx_hash or wallet_address. Same answer: keep it in the line.

5.7 A chart silently ignored the resources block

Tempo’s chart at the pinned version reads resources from .Values.tempo.resources. A root-level resources: key, the convention used by every other chart in the same file, was silently ignored. The pod ran for weeks with no requests or limits at all, which is invisible until node pressure picks it for eviction. The same chart hard-coded its Service port and ignored the override.

Lesson: for every chart, helm template with your values and grep the rendered manifest for the fields you set. Then confirm on the live object (kubectl get pod -o yaml). Values keys are not validated.

5.8 One dead node blanked fifteen health alerts

A single EC2 instance failed its system reachability check. The kubelet stopped heartbeating, and Kubernetes’ default 300-second not-ready toleration delayed rescheduling. That node happened to host the single-replica collector whose httpcheck receiver produced the health metric for every service on the platform. About fifteen unrelated alerts flipped to DatasourceNoData simultaneously while almost every service was healthy. The one real outage in the same window, a single-replica ingress controller on that node, was buried in the noise.

Lessons:

• Find the components that are the sole source of a signal family (probe collectors, the kube-state-metrics scraper, the remote-write exporter) and make them HA first. See §4.4.
• Build a “bulk NoData storm” runbook entry. When 10+ rules go NoData together, the first check is “are services actually down?” (curl the health endpoint, list non-Running pods), then look at the pipeline.

5.9 TSDB compaction looks exactly like an outage

VictoriaMetrics’ background merges spike CPU periodically. On an undersized pod, Grafana’s queries time out during the merge and every metric alert flips to NoData together, then resolves within 2–5 minutes. Separately, rule evaluations arriving in bursts exhausted search.maxConcurrentRequests, and a queue duration of 10s turned the queueing into DatasourceError pages.

Mitigations: give the TSDB real CPU headroom. Set search.maxQueueDuration below Grafana’s evaluation timeout but long enough to absorb bursts (20s worked with a 30s Grafana timeout). Stagger rule-group evaluation intervals. Set a for: on NoData handling that outlasts a merge.

5.10 A scheduling fix doubled a DaemonSet’s real footprint

A log-shipper DaemonSet had never tolerated the taint on a new arm64 node group, so it silently ran on only a subset of nodes. When the toleration was added and the fleet moved fully to arm64, the shipper ran on every node for the first time. Its 512 MiB limit, sized for the old mostly-unscheduled footprint, OOMKilled on the busiest node. Per-node working sets ranged from 67 MiB to 395 MiB in one snapshot.

Lessons:

• Size DaemonSets for the noisiest node, not the average.
• After any taint or toleration change, treat “this workload now runs in more places” as a load change that needs its own resource review.
• Alert on kube_daemonset_status_number_misscheduled and on desired vs scheduled pods, so a DaemonSet that only covers part of the fleet doesn’t go unnoticed.

5.11 HA collectors, single TSDB: synchronized ingest gaps

After the fix in §5.8, the two probe-collector replicas, on different nodes and in different AZs of the prod cluster, began logging Exporting failed. Dropping data. Permanent error: context deadline exceeded at the same instant, roughly every three hours, for 60–120 seconds. Each time, a rotating set of health alerts went NoData and self-resolved. VictoriaMetrics logged no errors. It was simply too slow to answer. Earlier that day, the tools cluster had been consolidated to two nodes. At investigation time, memory limits summed to 113–131% of node capacity and CPU limits to 86–206%. There was no metrics-server, so throttling could not be confirmed directly.

Lessons:

• Overcommitted limits on a small fixed pool with no autoscaler is a latent ingest outage.
• Run metrics-server (or scrape cAdvisor CPU throttling counters) on the observability cluster itself.
• Add the exporter’s otelcol_exporter_send_failed_metric_points to your dashboards. It is the earliest signal that the backend is the bottleneck.
• Size the observability node pool so that losing one node still leaves the TSDB and Grafana scheduled with headroom (N+1).

5.12 The Grafana auth proxy trusted an unverified JWT

The sidecar base64-decoded x-amzn-oidc-data and regex-matched the email domain. It did not verify the signature. It did strip client X-WEBAUTH-* headers, and Grafana accepted proxy headers only from localhost. But the cluster enforced no NetworkPolicy (no network-policy agent was enabled). Any pod in the shared tools cluster could therefore send a forged header straight to the sidecar’s pod port and sign in as any existing user, including admins. Grafana holds datasource credentials and an IAM role.

The fix followed the AWS specification [4]. It verifies ES256 against the regional key by kid (cached), requires alg == ES256, requires signer equal to the ALB ARN, and checks the header exp. The image is pinned. A config-checksum annotation forces a restart on ConfigMap change, because the subPath-mounted config had never reloaded in running pods. A live test showed forged, unsigned and missing tokens returning 401 while real logins passed. Two residual risks remain and should be written down: replay of a token sniffed on the plaintext ALB→pod hop until exp, and the fact that the cluster still has zero NetworkPolicies.

Two more lessons from the same area:

• VPN users can often reach the datastores’ HTTP APIs (:8428, :3100) directly, bypassing Grafana auth entirely. Decide whether that is acceptable. VictoriaMetrics’ delete APIs and Loki’s delete requests are reachable on those ports too.
• Grafana itself is the alerting engine. Treat its compromise as a monitoring outage and a credential exposure.

5.13 Nothing watched the watcher

Most alert rules sensibly set no_data_state = OK, because many metrics are legitimately absent when nothing is happening. The consequence is that a total pipeline failure evaluates the entire alert estate to OK. The compensating control is a telemetry dead-man canary: a small number of rules that alert on the absence of data, anchored on series that exist independently of product traffic:

# Scrape path: is anyone writing scraped samples at all?
count(up{job="kube-state-metrics", env="prod"}) or vector(0)        # alert if < 1 for 10m

# Push path: are apps' OTLP metrics arriving? (baseline: 13 services, flat for a week)
count(count by (service_name) (rate(http_server_request_duration_seconds_count{env="prod"}[10m]))) or vector(0)   # alert if < floor

or vector(0) is essential. Without it, “series gone” is NoData, which then depends on no_data_state rather than being an ordinary threshold breach.

The canary runs inside Grafana, so it cannot detect Grafana being down. If Grafana stops evaluating, whether from its database, the cluster, the namespace or a bad config rollout, every page stops silently. Close this with an external heartbeat: a Grafana rule that always fires, routed to a PagerDuty heartbeat integration or a service like Healthchecks.io, which pages when the pings stop. This was an open High finding on the reference platform even after Grafana went HA. Two replicas don’t help if both share a broken database or a broken config.

5.14 An alert rule shipped before its metric existed

Alert rules applied through a fast, separate pipeline (Grafana provider) while the scrape job feeding them applied through a slower infrastructure pipeline. Merged together, the rules were live about nine minutes before the metric existed. With no_data_state = Alerting, that was nine minutes of sev1 pages with an empty label set.

Rule: scrape job first. Verify up{job="<job>"} returns one series per target, all reading 1. Then merge the rules. Prefer no_data_state = OK on up-based rules. up == 0 is a real outage, while absent up is a missing scrape job, and total loss is the canary’s job.

5.15 Metrics that look right and return the wrong number

This class of bug costs the most in aggregate, because the query runs, the panel renders, and the number is plausible. A catalogue, generalized:

Trap	Symptom	Fix
Three env-label conventions. App OTel metrics carry deployment_environment_name; kube-state/cAdvisor and third-party exporters carry env; httpcheck carries deployment_environment (no suffix); Loki uses environment with different values (e.g. prod in one, prod-v1 in the other)	Filters silently match nothing; dashboards hack around it with =~".*$env.*"	Pick one resource attribute (deployment.environment.name per OTel semconv [15]) and normalize at the agent with a resource/transform processor. Normalize Loki labels to the same values
Same attribute, different label name. Tempo span-metrics label the service service, but app-emitted OTel metrics use service_name	Copying a filter between the two families returns empty	Document per-family label names; test alert queries for non-empty results before merge
Collided counter names. Two OTel counters (external_api_requests and external_api_requests_total) both export as external_api_requests_total, one with a success label, one with status	sum(rate(...)) double-counts; status!~"2.." counts the status-less family’s successes as errors	Lint for collisions at the exporter’s _total suffixing step; rename app-side [16]
Error counter inflated by retries/hedges. Errors counted per attempt, requests per logical call	“Error ratio” reaches 3.6; clamp_max(…,1) still trips a 50% threshold	Never build a ratio from counters with different increment semantics; alert on the logical outcome
Success ratio from two counters over churning pods	Read 0.08 “success” with a flat-zero failure counter during a rollout	Alert on failed / total, not successful / total; churn breaks the pairing in both directions
Global aggregate dilution. sum() across chains or tenants	One chain at 50% failure, global at 0.5%, no page	Aggregate by (chain) (or tenant/route); a global aggregate is safe only for > 0 on a count, or when no dimension can break independently
CronJob-pushed metrics go stale between runs. Prometheus-style staleness drops a series ~5 min after its last sample [17]	A */30 job is “absent” ~83% of the time; instant queries see nothing	Use last_over_time(m[<several cadences>]); size alert windows from the job’s cadence
Freshness with time() - m	Includes scrape age: a 0–30s sawtooth unrelated to real lag; aliasing with eval interval satisfies for: on healthy targets	Use timestamp(m) - m
Loki count alert with reduce: sum	Sums the rolling count across every range step, so the rule fires on ≥1 error regardless of threshold	sum(count_over_time(...[5m])) or vector(0) with reduce: last
Grafana recording rules without a target datasource	Rules “evaluate fine” but persist nothing; SLO panels show No data	Set the recording rule’s target datasource (or the global default) and check rule health via the rules API
Annotation templating the threshold node	Summary always says “1%”	Template the ref that holds the real value, not the threshold expression
Inert metrics config. An ingress controller’s values file carries metrics.enabled: true, but no chart template reads it	Zero series; dashboards built against counters that never existed	Before building on a metric, confirm it has series
Service graph treated as a census	“Declared but not observed” flagged the most critical path as dead	The derived service graph is a lower bound. It needs CLIENT spans and context propagation; queue- and WebSocket-carried edges never appear; uninstrumented callers (and services calling your own public hostname) show as client="user"
Per-client attribution by label	by (project_id) lists only tenants that use one feature	Check which metric families carry the label; complete attribution may only exist in traces

The underlying discipline is in the SRE Workbook’s alerting chapter. Alert on symptoms tied to SLOs, with multi-window, multi-burn-rate conditions (the classic 14.4× / 6× pair), volume gates and NaN guards [2]. A burn-rate rule over a diluted global ratio is still wrong, though. The table above is what makes the SLI itself trustworthy.

5.16 Docs drift is an observability failure

Twice on the reference platform, the architecture doc described a stack that didn’t exist: “no log DaemonSet” when one ran everywhere, and retention values that disagreed between the doc and the Terraform. At 3am the doc is what gets read. Generate inventories from config where possible, and edit claims in place rather than appending corrections.

---

6. Migrating from a single EC2 docker-compose box to Helm on EKS

The docker-compose box is the right first self-hosted step. One t3.xlarge or m6g.xlarge, Compose, a big gp3 volume. It hits its limits in predictable ways: IOPS contention (§5.5), no redundancy, manual config edits over SSM, and upgrades that require downtime. The migration that worked:

1. Stand up the new stack in parallel in the observability cluster, with S3 buckets and IRSA roles, receiving no traffic. Gate the cutover behind a Terraform variable.
2. Keep the ingest endpoint stable. Put an NLB (or the existing one) in front, and move its target groups from the EC2 instance to pod IPs using the AWS Load Balancer Controller’s TargetGroupBinding. Clients never change hostnames.
3. Cut over one signal at a time. Traces first (least stateful, easiest to verify), then logs, then metrics. For metrics, either backfill with vmctl or accept a history gap and keep the old box readable for the retention window.
4. Grafana last. Export dashboards and alerts as code before the move. Move Grafana state from SQLite to Postgres as a separate step from going multi-replica.
5. Stateful pod moves need a plan. A single-replica StatefulSet with a zonal ReadWriteOnce EBS volume can only be rescheduled into that volume’s AZ. When moving to a new node group (for example, an arm64 migration), make sure the target group already has a ready node in the PVC’s AZ, or the pod sits Pending. Do these moves one component at a time, and do the TSDB last.
6. Decommission the old box only after a presence check per source and per signal (§5.1), and record the dead instance IDs somewhere searchable so old runbook recipes (“SSM in and docker ps”) hit an explicit dead end.
7. Delete or rewrite every doc, diagram and memory note that describes the old topology in the same change.

---

7. Best-practice checklist

Day 1 (any stage, including SaaS)

• Instrument with OpenTelemetry SDKs; one resource attribute for environment (deployment.environment.name) everywhere [15].
• Mandatory structured log fields: service, env, level, trace_id, span_id, timestamp, as fields, not labels.
• Log-shipper label allow-list; no IDs, hashes, addresses or error strings as labels [7].
• Health-probe collector runs 2 replicas with dedup.
• Telemetry dead-man canary for both scrape and push paths, using or vector(0).
• External heartbeat that pages when the alerting engine stops evaluating.
• Every alert has a severity and a runbook link; only sev1 pages.
• Grafana reachable only over VPN/internal LB; SSO; new users default to Viewer.
• If using ALB auth, verify the x-amzn-oidc-data signature, signer and exp [4]; otherwise use native OIDC [6].

First year (self-hosted)

• Dedicated cluster or isolated node group; N+1 headroom; metrics-server installed.
• Loki and Tempo on S3; VictoriaMetrics PVC backed up daily with a scheduled restore check.
• Grafana on an external database, 2 replicas, PDB, HA alerting.
• Tempo multi-replica only with scalable-single-binary + memberlist; verify /status/config.
• Loki cardinality guardrails: stream-creation alert, max_streams_per_user, label-name limits.
• Collector exporter-failure metrics on a dashboard; memory_limiter and sending queues configured [19].
• Multi-window burn-rate SLO alerts per critical user journey [2].
• Alert-authoring guide listing every “wrong number” trap you have hit (§5.15), reviewed on every rule PR.
• Rule-after-metric merge order enforced.
• NetworkPolicy enforcement enabled on the observability/tools cluster, with ingress restricted to known sources.
• Migrate off Promtail to Alloy [14][25].

At scale

• VictoriaMetrics cluster (or redundant single nodes) to remove the TSDB single point of failure.
• Loki simple-scalable/distributed and Tempo distributed when volume demands it [22][13].
• Trace-ID-aware load balancing before any horizontally scaled tail sampling [11].
• Per-team cardinality budgets and usage reporting.
• Label-convention linting in CI for new metrics [16][23].

---

8. Maturity roadmap

Stage	Metrics	Logs	Traces	Alerting	Reliability of the stack
0: Seed	SaaS or Grafana Cloud free tier; OTel SDKs	SaaS	Sampled SaaS	A handful of symptom alerts	Vendor’s problem; still add an external heartbeat
1: First self-host	VictoriaMetrics on one EC2 box	Loki on the same box	Tempo on the same box, S3 backend	Grafana alerting as code	Single box = single failure domain; back up the volume
2: Series A	VM single on a dedicated EKS cluster, PVC + vmbackup	Loki SingleBinary on S3, DaemonSet shipper	Tempo 2× scalable single binary on S3	Burn-rate SLOs, canary, heartbeat, runbook links	HA Grafana + HA probe collectors; documented accepted risks
3: Growth	VM redundant or VMCluster	Loki simple-scalable	Tempo distributed, tail sampling with LB exporter	Per-service SLOs; alert linting in CI	N+1 everywhere; NetworkPolicy; restore drills
4: Scale	Multi-region or long-term tiered storage	Tenant isolation	Exemplars everywhere	Error budgets drive release policy	Observability has its own SLO and on-call

---

9. Closing

A self-hosted stack on EKS is affordable and, done carefully, more capable than a startup-tier SaaS plan, especially for crypto platforms whose most important dimensions (chain, token, route, provider) are exactly what series-based pricing punishes. The infrastructure is the cheap part. The work is in three habits: prove each signal arrives rather than trusting configuration; treat every query as guilty until it returns a number you can independently confirm; and make sure something outside the stack notices when the stack goes quiet.

---

References

1. Google, Site Reliability Engineering, ch. 6 “Monitoring Distributed Systems.” https://sre.google/sre-book/monitoring-distributed-systems/
2. Google, The Site Reliability Workbook, ch. 5 “Alerting on SLOs.” https://sre.google/workbook/alerting-on-slos/
3. AWS Well-Architected Framework, Operational Excellence Pillar, “Implement observability.” https://docs.aws.amazon.com/wellarchitected/latest/operational-excellence-pillar/implement-observability.html
4. AWS, “Authenticate users using an Application Load Balancer” (user claims encoding and signature verification). https://docs.aws.amazon.com/elasticloadbalancing/latest/application/listener-authenticate-users.html
5. Grafana Labs, “Configure auth proxy authentication.” https://grafana.com/docs/grafana/latest/setup-grafana/configure-security/configure-authentication/auth-proxy/
6. Grafana Labs, “Configure generic OAuth2 authentication.” https://grafana.com/docs/grafana/latest/setup-grafana/configure-security/configure-authentication/generic-oauth/
7. Grafana Labs, Loki, “Label best practices.” https://grafana.com/docs/loki/latest/get-started/labels/bp-labels/
8. Grafana Labs, Loki, “Cardinality.” https://grafana.com/docs/loki/latest/get-started/labels/cardinality/
9. OpenTelemetry, “Agent deployment pattern.” https://opentelemetry.io/docs/collector/deployment/agent/
10. OpenTelemetry, “Gateway deployment pattern.” https://opentelemetry.io/docs/collector/deployment/gateway/
11. OpenTelemetry, “Scaling the Collector.” https://opentelemetry.io/docs/collector/scaling/
12. VictoriaMetrics, “Single-node VictoriaMetrics” (deduplication, -dedup.minScrapeInterval). https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/
13. Grafana Labs, Tempo, “Deploy Tempo” (deployment modes). https://grafana.com/docs/tempo/latest/setup/deployment/
14. Grafana Labs, Loki, “Promtail agent” (end-of-life notice). https://grafana.com/docs/loki/latest/send-data/promtail/
15. OpenTelemetry Semantic Conventions, “Deployment environment” resource attributes. https://opentelemetry.io/docs/specs/semconv/resource/deployment-environment/
16. Prometheus, “Metric and label naming.” https://prometheus.io/docs/practices/naming/
17. Prometheus, “Querying basics” (staleness). https://prometheus.io/docs/prometheus/latest/querying/basics/
18. OpenTelemetry Collector Contrib, “Tail Sampling Processor.” https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor
19. OpenTelemetry, “Collector resiliency” (sending queues, retries, persistent storage). https://opentelemetry.io/docs/collector/resiliency/
20. Kubernetes, “Specifying a Disruption Budget for your Application.” https://kubernetes.io/docs/tasks/run-application/configure-pdb/
21. Grafana Labs, “Configure high availability” for Grafana Alerting. https://grafana.com/docs/grafana/latest/alerting/set-up/configure-high-availability/
22. Grafana Labs, Loki, “Deployment modes.” https://grafana.com/docs/loki/latest/get-started/deployment-modes/
23. Prometheus, “Instrumentation” best practices. https://prometheus.io/docs/practices/instrumentation/
24. AWS, Amazon EKS Best Practices Guide, “Cost optimization — observability.” https://docs.aws.amazon.com/eks/latest/best-practices/cost-opt-observability.html
25. Grafana Labs, Alloy, “Migrate from Promtail to Grafana Alloy.” https://grafana.com/docs/alloy/latest/set-up/migrate/from-promtail/