Prometheus

Prometheus is a systems and service monitoring system. It collects Metrics from configured targets at given intervals, evaluates rule expressions, displays the results, and can trigger alerts if some condition is observed to be true.

It is one of the core tools used for Observability.

When used with Kubernetes it will deploy a node exporter pod to each node in your cluster to collect metrics.

Examples

A typical scrape target might look like this:

scrape_configs:
  - job_name: "app"
    static_configs:
      - targets: ["localhost:8080"]

Common PromQL examples:

rate(http_requests_total[5m])

This shows the per-second request rate averaged over the last 5 minutes.

histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

This estimates the 95th percentile request latency.

sum(rate(http_requests_total{status=~"5.."}[5m]))

This counts server errors over the last 5 minutes.

An example alert rule:

groups:
  - name: app-alerts
    rules:
      - alert: HighErrorRate
        expr: sum(rate(http_requests_total{status=~"5.."}[5m])) > 5
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "High error rate detected"