Prometheus 101 (thread)

1⃣ Metrics

A metric is a feature (i.e., a characteristic) of a system that is being measured.

Typical examples:

- http_requests_total
- http_request_size_bytes
- system_memory_used_bytes
- node_network_receive_bytes_total
2⃣ Labels

Prometheus scrapes metrics from instances (read, servers) it monitors. The same metric can be collected from multiple instances.

How to differentiate samples produced by different instances? Label them!

*Labels usage is broader than that, but this is a good start.
3⃣ Time series

Prometheus scrapes metrics with some predefined interval between two consequent scrapes (typically, 10-30 seconds).

A sequence of samples that belong to the same metric and share exactly the same key-value set of labels forms a time series.
4⃣ Instant vectors

You can ask Prometheus to show all the time series satisfying some criteria at a given timestamp (i.e, instant).

Example: http_requests_total{}

In the general case, multiple series are returned for a single timestamp. Hence, the result is a _vector_.
5⃣ Range vectors

Instant vectors are _instant_ - they cannot be used to calculate some value over a time window (e.g., 5min avg request rate).

Solution - add _time interval_ notion to the query.

Example: http_requests_total{}[5m]

A range vector is a two-dimensional matrix!
6⃣ Vector operations

Instant Vectors support different types of operations:

- vector <op> scalar
- vector "bool" <op> scalar
- vector <op> vector

*Range Vectors cannot be used as operands. You must aggregate them back to Instant Vectors using a built-in _function_.
7⃣ Vector matching

vector <op> vector is a tricky one!

Since vectors are on both sides, a by-element matching is required to perform such operations.

The actual matching rules depend on the type of the operation - arithmetic, comparison, or set operation.

Cheatsheet 🔽
8⃣ Hands-on learning

The best way to learn Prometheus and PromQL is by trying to feed a Prometheus node with some well-known dataset, issuing queries to it, and analyzing the results.

Here is my set of scripts to create a playground:…
9⃣ Prometheus Learning Series on

Check out the Prometheus learning series. It covers the above stuff in more detail, but also it touches upon some tricky topics like:

- Instant query vs. Range query
- What is a lookback delta? 🔽…

• • •

Missing some Tweet in this thread? You can try to force a refresh

Keep Current with Ivan Velichko

Ivan Velichko Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!


Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @iximiuz

30 Dec 21
I often search for container images, and here is my pain:

- There is more than one place to go - Docker Hub, GCR, Quay
- Some software has multiple image providers - e.g Envoy
- Image trustworthiness and difference are obscure

My recent cool finding - 🔽
I'm always hesitant to add a new tool to my toolset - in the end, it's yet another dependency and extra time investment.

But Slim .AI has all the chances to pass my high barrier 🙃

Seems like it can really help me start with the right image faster. ImageImage
The difference between some images has been obscure to me for a while.

Of course, I could fetch them and compare locally, but since it's just for some ad-hoc playgrounds, I've never done it.

With Slim, I can know the difference right in my browser: ImageImage
Read 5 tweets
22 Dec 21
Kubernetes UI/GUI/CLI projects

Gathered a list of (mostly open-source) projects offering Kubernetes UX enhancements.

Starting from the well-known solutions and down to projects mounting the Kubernetes API as a FUSE filesystem 🔽
github .com/lensapp/lens
stats: TypeScript / 16.8k stars
descr: Lens - The way the world runs Kubernetes
github .com/derailed/k9s
stats: Go / 14.6k stars
descr: Kubernetes CLI To Manage Your Clusters In Style!
Read 24 tweets
8 Dec 21
How to Run a FaaS Platform On Your Own Terms

FaaS is a higher-level kind of Serverless tech where the smallest deployable unit is a Function.

AWS Lambda, Azure Functions, GCP Cloud Function are all super handy, but what if you can't use them for some reason?

Meet OpenFaaS! 🔽
OpenFaaS is an open-source project that turns a piece of lower-level infra into a high-level FaaS solution.

Sounds too abstract?

Kubernetes cluster + OpenFaaS = FaaS API
Single VM + containerd + OpenFaaS = same FaaS API!

where FaaS API is:
- mgmt. methods
- invoke functions
At a high level, OpenFaaS has a universal architecture that allows it to be installed on almost any kind of infra:

API Gateway (concrete) + FaaS-Provider (abstract)
Read 7 tweets
30 Oct 21
How Kubernetes differs from Docker in the way it deals with containers 🔽

Under the hood, Kubernetes and Docker both rely on the same/similar lower-level components to run containers.

Often, both use containerd and runc. However, Kubernetes makes the container runtime pluggable
Even when Kubernetes uses exactly the same container runtime as Docker, the implementation of Pods differs from the implementation of Containers.

Pods are more than just groups of containers. Containers in a pod share net, ipc, and uts namespaces making pods more similar to VMs.
Not all Containers are Linux containers!

OCI Runtime Spec doesn't limit container implementation to namespaces and cgroups. And VM-based containers is a real thing!

Some of the container runtimes supported by Kubernetes (e.g., Kata), launch a micro-VM per Pod.
Read 6 tweets
20 Oct 21
Computer Networking in Layman's Terms (thread)

L2 - Ethernet
L3 - IP
L4 - TCP

Lots of server-side folks are fluent with L4/L7 concepts. But understanding L2/L3 is often important too!

Especially if you deal with containers, Kubernetes, or DC networking constraints 🔽
Example ✏️

By default, Docker/podman/containerd/etc. use a `bridge` network to interconnect containers on a single host. But what is a Bridge?

In the case of containers, a Bridge is a virtual device. However, it simulates a real-world L2/L3 networking device called a Switch.
A Switch is a multiport device that combines two logical functions: L2 bridging and L3 routing.

Bridging creates an L2 broadcast domain out of connected nodes. A single broadcast domain _usually_ forms an L3 network.

Routing creates an (inter)network out of L3 (sub)networks.
Read 8 tweets
3 Oct 21
Cloud-Native Learn-by-Doing Platforms (part 2) ✏️

I closely reviewed 10+ platforms providing a hands-on learning experience to study:

- Linux
- Containers
- Orchestration
- other Cloud-Native stuff

My findings (thread) 🔽

#kubernetes #docker #containers #linux #sre #devops
1. A Cloud Guru (acloudguru .com)

A really massive platform for individuals and businesses.

- 360+ courses and 1,600+ hands-on labs
- Clouds: AWS, GCP, etc.
- Tech: Kubernetes, Docker, Serverless

- Servers
- Temporary cloud accounts

2. Developer Sandbox for Red Hat OpenShift (developers.redhat .com/developer-sandbox)

A free platform with a private OpenShift playground:
7 GB RAM, 15GB storage 🔥

Almost no educational content, but combines well with kubebyexample .com.
Read 7 tweets

Did Thread Reader help you today?

Support us! We are indie developers!

This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal

Or Donate anonymously using crypto!


0xfe58350B80634f60Fa6Dc149a72b4DFbc17D341E copy


3ATGMxNzCUFzxpMCHL5sWSt4DVtS8UqXpi copy

Thank you for your support!

Follow Us on Twitter!