Prometheus 101 (thread)
1⃣ Metrics
A metric is a feature (i.e., a characteristic) of a system that is being measured.
Typical examples:
- http_requests_total
- http_request_size_bytes
- system_memory_used_bytes
- node_network_receive_bytes_total
2⃣ Labels
Prometheus scrapes metrics from instances (read, servers) it monitors. The same metric can be collected from multiple instances.
How to differentiate samples produced by different instances? Label them!
*Labels usage is broader than that, but this is a good start.
3⃣ Time series
Prometheus scrapes metrics with some predefined interval between two consequent scrapes (typically, 10-30 seconds).
A sequence of samples that belong to the same metric and share exactly the same key-value set of labels forms a time series.
4⃣ Instant vectors
You can ask Prometheus to show all the time series satisfying some criteria at a given timestamp (i.e, instant).
Example: http_requests_total{}
In the general case, multiple series are returned for a single timestamp. Hence, the result is a _vector_.
5⃣ Range vectors
Instant vectors are _instant_ - they cannot be used to calculate some value over a time window (e.g., 5min avg request rate).
Solution - add _time interval_ notion to the query.
Example: http_requests_total{}[5m]
A range vector is a two-dimensional matrix!
6⃣ Vector operations
Instant Vectors support different types of operations:
- vector <op> scalar
- vector "bool" <op> scalar
- vector <op> vector
*Range Vectors cannot be used as operands. You must aggregate them back to Instant Vectors using a built-in _function_.
7⃣ Vector matching
vector <op> vector is a tricky one!
Since vectors are on both sides, a by-element matching is required to perform such operations.
The actual matching rules depend on the type of the operation - arithmetic, comparison, or set operation.
Cheatsheet 🔽
8⃣ Hands-on learning
The best way to learn Prometheus and PromQL is by trying to feed a Prometheus node with some well-known dataset, issuing queries to it, and analyzing the results.
Here is my set of scripts to create a playground:
github.com/iximiuz/promet…
9⃣ Prometheus Learning Series on iximiuz.com
Check out the Prometheus learning series. It covers the above stuff in more detail, but also it touches upon some tricky topics like:
- Instant query vs. Range query
- What is a lookback delta? 🔽
iximiuz.com/en/series/lear…
Share this Scrolly Tale with your friends.
A Scrolly Tale is a new way to read Twitter threads with a more visually immersive experience.
Discover more beautiful Scrolly Tales like this.