First scrape target
Add one scrape job for node_exporter on localhost:9100 with a 15s interval, then verify target health in /targets.
DevOps & Tooling / Knowledge library
Prometheus
open-source metrics monitoring and alerting system
Why learn Prometheus?
Prometheus gives operators a reliable way to collect, store, query, and alert on time-series metrics so they can detect incidents fast, track SLOs, and understand system behavior over time.
Where it came from
Started at SoundCloud in 2012 to solve microservice monitoring needs. Open-sourced early and built around a pull model plus dimensional labels. Joined CNCF in 2016. Became a CNCF Graduated project in 2018. PromQL, Alertmanager, and exporters drove wide adoption in cloud-native ops.
- Started at SoundCloud in 2012 to solve microservice monitoring needs.
- Open-sourced early and built around a pull model plus dimensional labels.
- Joined CNCF in 2016.
- Became a CNCF Graduated project in 2018.
- PromQL, Alertmanager, and exporters drove wide adoption in cloud-native ops.
Build your understanding
Work through these exercises in order. Try the task before opening a reference.
- Add one scrape job for node_exporter on localhost:9100 with a 15s interval, then verify target health in /targets.
- Write a PromQL query for per-service 5xx request rate over 5 minutes using a counter metric.
- Create an alert rule that fires when instance CPU usage stays above 80% for 10 minutes, with clear labels and runbook annotation.
- Tune a high-latency alert using rate windows, for clauses, and label grouping to reduce flapping while preserving fast detection.
- Design a small architecture for long-term retention using remote_write and a backend (for example Thanos or Mimir), and explain trade-offs.
More ways to practice
Error-rate query
Write a PromQL query for per-service 5xx request rate over 5 minutes using a counter metric.
Actionable alert
Create an alert rule that fires when instance CPU usage stays above 80% for 10 minutes, with clear labels and runbook annotation.
Reduce noisy alerts
Tune a high-latency alert using rate windows, for clauses, and label grouping to reduce flapping while preserving fast detection.
Scale path design
Design a small architecture for long-term retention using remote_write and a backend (for example Thanos or Mimir), and explain trade-offs.
Documentation and sources
- Prometheus overview (prometheus.io)
- PromQL basics (prometheus.io)
- Configuration (prometheus.io)
- Alerting rules (prometheus.io)
- CNCF project page (www.cncf.io)
Catalog research notes
- 2.x releases improved TSDB efficiency and query performance at scale.
- Agent mode enables lightweight scrape + remote_write for edge and short-lived nodes.
- Native histograms and exemplars advanced for better latency/trace correlation (often feature-flagged).
- Kubernetes service discovery and rule tooling continue to improve operational usability.
Connect the ideas
Explore next
Real World Example
Dynamic Microservice Traffic Observability
A logistics firm managing a fleet of autonomous delivery drones that rely on a distributed microservice architecture to coordinate routing and battery management.
The business problem
The engineering team struggles to identify which specific microservice instance is experiencing latency spikes during peak delivery hours, leading to delayed routing updates and potential drone battery depletion.
How this topic solves it
Prometheus provides a multi-dimensional data model that allows the team to attach labels to metrics, such as instance IDs and service versions, enabling precise identification of problematic nodes via PromQL queries.
- Instrument the drone routing microservices using Prometheus client libraries to expose HTTP endpoints for metrics.
- Configure the Prometheus server with service discovery to automatically detect new drone routing instances as they scale.
- Define scraping intervals in the Prometheus configuration to pull metrics from the discovered service endpoints.
- Develop PromQL queries to aggregate request latency across specific service versions and instance labels.
- Set up Alertmanager to notify the operations team when latency thresholds are breached for specific service clusters.
This diagram shows how Prometheus interacts with microservices to collect and visualize data.
Identifies active drone routing instances
Scrapes and stores time series data
Exposes metrics via HTTP
Queries Prometheus for dashboards
- Service Discovery → updates target list → Prometheus Server
- Prometheus Server → pulls metrics over HTTP → Microservice Instance
- Visualization Tool → queries via PromQL → Prometheus Server
Expected business outcome
The operations team gains granular visibility into service performance, allowing them to isolate and restart specific failing instances without impacting the entire delivery network.
Limitations and checks
Prometheus is designed for reliability rather than absolute data accuracy; it is unsuitable for per-request billing because samples may be missed during scrape intervals. To check for data gaps, monitor the 'scrape_series_added' and 'scrape_samples_scraped' metrics to ensure expected ingestion rates.