Full-Stack Monitoring with Prometheus & Grafana

This sets up a Prometheus + Grafana stack that scrapes real targets, alerts on symptoms users actually feel rather than on raw resource metrics like CPU% by default, and provisions the dashboard as code.

Scraping targets with prometheus.yml

Prometheus pulls metrics rather than receiving them, so the core of the config is just: where to look, and how often.

global:
  scrape_interval: 15s
  evaluation_interval: 15s
 
scrape_configs:
  - job_name: "checkout-api"
    metrics_path: /metrics
    kubernetes_sd_configs:
      - role: pod
        namespaces:
          names: ["payments"]
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        regex: checkout-api
        action: keep
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
 
  - job_name: "node-exporter"
    static_configs:
      - targets: ["node-exporter:9100"]
 
rule_files:
  - "alert.rules.yml"
 
alerting:
  alertmanagers:
    - static_configs:
        - targets: ["alertmanager:9093"]

kubernetes_sd_configs does service discovery so you're not hand-maintaining a target list every time a pod reschedules, and relabel_configs filters that discovery down to the one app you actually asked for, skip it and Prometheus scrapes every pod in the namespace. The keep action with a regex match on a pod label is the pattern I reach for first.

Picking what to alert on: RED and USE

The fastest way to alert fatigue is alerting on whatever the exporter happens to expose. Two frameworks fix that:

• RED (for request-driven services): Rate, Errors, Duration. If a service handles requests, these three cover what a user actually experiences.
• USE (for resources): Utilization, Saturation, Errors. Use this for the infrastructure underneath: nodes, disks, queues.

CPU utilization is a USE metric about the node. "Requests are timing out" is a RED metric about the user's experience. Alert on the second, and use the first as debugging context once you're already paged.

A real alerting rule: error rate with for:

Here's the alert I ship for an HTTP service, built around RED's error rate:

groups:
  - name: checkout-api.rules
    rules:
      - alert: CheckoutAPIHighErrorRate
        expr: |
          sum(rate(http_requests_total{job="checkout-api", status=~"5.."}[5m]))
          /
          sum(rate(http_requests_total{job="checkout-api"}[5m]))
          > 0.02
        for: 10m
        labels:
          severity: page
          team: payments
        annotations:
          summary: "Checkout API error rate above 2% for 10 minutes"
          description: "{{ $value | humanizePercentage }} of requests to checkout-api are returning 5xx over the last 5 minutes."
          runbook_url: "https://runbooks.internal/checkout-api-errors"

The for: 10m is the single most under-used field in Prometheus alerting. Without it, the alert fires the instant the expression crosses 2%, even on a single 30-second blip during a deploy, then immediately resolves. Requiring the condition to hold for 10 straight minutes filters that out while still catching anything real and sustained. I tune the duration per alert: 2-3 minutes for something genuinely urgent, 10-15 minutes for anything that self-heals from normal retry/backoff behavior.

Routing and grouping in Alertmanager

Prometheus decides whether to alert; Alertmanager decides who hears about it and how often. Route by the labels your rules already set:

route:
  receiver: "default-slack"
  group_by: ["alertname", "team"]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - match:
        severity: page
        team: payments
      receiver: "payments-oncall-pagerduty"
      continue: false
 
receivers:
  - name: "default-slack"
    slack_configs:
      - channel: "#alerts-general"
  - name: "payments-oncall-pagerduty"
    pagerduty_configs:
      - routing_key: "{{ .PaymentsPagerDutyKey }}"

group_by is what stops a single bad deploy from paging you fifteen times for fifteen pods of the same rollout. group_wait gives related alerts a short window to arrive together before the first notification fires; repeat_interval controls how often an unresolved alert re-notifies, which should be long enough that on-call isn't repaged every few minutes.

Provisioning the Grafana dashboard as code

The dashboard that matters is the one that ships with the service, not the one someone builds by hand in the UI. Provisioning does that:

apiVersion: 1
 
providers:
  - name: "checkout-api"
    orgId: 1
    folder: "Payments"
    type: file
    disableDeletion: false
    updateIntervalSeconds: 30
    options:
      path: /etc/grafana/provisioning/dashboards/checkout-api
      foldersFromFilesStructure: true
 
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true

Pair that with a dashboard JSON file (generated once from the UI, then checked into the repo), and the dashboard, its datasource, and alert rules all deploy together through the same pipeline as the service. Skip that step and the dashboard drifts the moment someone tweaks a panel by hand, quietly, until it's the broken panel nobody trusts.

How the pieces fit together

Closing

Scrape config, one alert rule, a routing tree, one provisioned dashboard. What separates a team that trusts its alerts from one that mutes the channel is whether every alert answers "is a user affected right now," backed by a for: window long enough to ignore the noise.