Ivan Velichko Profile picture
Jan 15, 2022 9 tweets 4 min read Read on X
Prometheus 101 (thread)

1⃣ Metrics

A metric is a feature (i.e., a characteristic) of a system that is being measured.

Typical examples:

- http_requests_total
- http_request_size_bytes
- system_memory_used_bytes
- node_network_receive_bytes_total
2⃣ Labels

Prometheus scrapes metrics from instances (read, servers) it monitors. The same metric can be collected from multiple instances.

How to differentiate samples produced by different instances? Label them!

*Labels usage is broader than that, but this is a good start.
3⃣ Time series

Prometheus scrapes metrics with some predefined interval between two consequent scrapes (typically, 10-30 seconds).

A sequence of samples that belong to the same metric and share exactly the same key-value set of labels forms a time series.
4⃣ Instant vectors

You can ask Prometheus to show all the time series satisfying some criteria at a given timestamp (i.e, instant).

Example: http_requests_total{}

In the general case, multiple series are returned for a single timestamp. Hence, the result is a _vector_.
5⃣ Range vectors

Instant vectors are _instant_ - they cannot be used to calculate some value over a time window (e.g., 5min avg request rate).

Solution - add _time interval_ notion to the query.

Example: http_requests_total{}[5m]

A range vector is a two-dimensional matrix!
6⃣ Vector operations

Instant Vectors support different types of operations:

- vector <op> scalar
- vector "bool" <op> scalar
- vector <op> vector

*Range Vectors cannot be used as operands. You must aggregate them back to Instant Vectors using a built-in _function_.
7⃣ Vector matching

vector <op> vector is a tricky one!

Since vectors are on both sides, a by-element matching is required to perform such operations.

The actual matching rules depend on the type of the operation - arithmetic, comparison, or set operation.

Cheatsheet 🔽
8⃣ Hands-on learning

The best way to learn Prometheus and PromQL is by trying to feed a Prometheus node with some well-known dataset, issuing queries to it, and analyzing the results.

Here is my set of scripts to create a playground:
github.com/iximiuz/promet…
9⃣ Prometheus Learning Series on iximiuz.com

Check out the Prometheus learning series. It covers the above stuff in more detail, but also it touches upon some tricky topics like:

- Instant query vs. Range query
- What is a lookback delta? 🔽

iximiuz.com/en/series/lear…

• • •

Missing some Tweet in this thread? You can try to force a refresh
 

Keep Current with Ivan Velichko

Ivan Velichko Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

PDF

Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @iximiuz

Jan 28
Docker vs. containerd vs. Podman 🧵

Containers are everywhere, and Docker is the most popular (and user-friendly) way of running them. But it's definitely not the only way!

I prepared a series of exercises to help you explore the alternative single-host runtimes 👇 Image
To set up a baseline, I recommend starting with Docker.

Try launching a container and inspecting it:

- What is it exactly that you just launched?
- Is it a single process? A lightweight VM?
- Can you find the IP address of the container?

labs.iximiuz.com/challenges/sta…
Image
Docker relies on containerd, a lower-level container runtime, to run its containers. It is possible to use containerd from the command line directly, but the UX might be quite rough at times.

contaiNERD CTL (nerdctl) to the rescue!

Try it out 👉 labs.iximiuz.com/challenges/sta…
Image
Read 5 tweets
Jan 10
What Actually Happens When You Publish a Container Port? Mini-🧵

docker run -p 8080:80 nginx

Have you ever wondered what `-p 8080:80` in the above command does? Then read on! Image
When you launch Nginx (or any other service), it opens a socket on a certain address - e.g., 172.17.0.3:80.

Clients that can reach this IP address can access the service. Image
However, when a service runs in a container, its socket will likely be on the container's primary IP address.

...which may or may not be reachable from the host!

Hence, port forwarding. Or, as Docker calls it - port publishing. Image
Read 8 tweets
Nov 27, 2023
How Container Networking Works 🧵

1. Network namespaces - a Linux facility to virtualize network stacks.

Every container gets its own isolated network stack with (virtual) network devices, a dedicated routing table, a scratch set of iptables rules, and more. Image
2. Virtual Ethernet Devices (veth) - a means to interconnect network namespaces.

Container's network interfaces are invisible from the host - the latter runs in its own (root) network namespace.

To punch through a network namespace, a special Virtual Ethernet Pair can be used. Image
3. The need for a (virtual) switch device.

When multiple containers run in the same IP network, leaving the host ends of the veth devices dangling in the root namespaces will make the routes clash. So, you won't be able to reach (some of) the containers. Image
Read 6 tweets
Nov 17, 2023
What is Service Discovery - in general, and in Kubernetes 🧵

Services (in Kubernetes or not) tend to run in multiple instances (containers, pods, VMs). But from the client's standpoint, a service is usually just a single address.

How is this single point of entry achieved? Image
1⃣ Server-Side Service Discovery

A single load balancer, a.k.a reverse proxy in front of the service's instances, is a common way to solve the Service Discovery problem.

It can be just one Nginx (or HAProxy) or a group of machines sharing the same address 👇 Image
2⃣ Client-Side Service Discovery

The centralized LB layer is relatively easy to provision, but it can become a bottleneck and a single point of failure.

An alternative solution is to distribute the rosters of service addresses to every client and let them pick an instance. Image
Read 7 tweets
Oct 25, 2023
What Happens When You Publish a Container's Port? 🧵

"Port publishing" might be a term coined by Docker.

But "port forwarding" or "port mapping" - as a form of socket redirection - was a widespread trick well before the invention of containers. How are the two different? Image
Learn about different port forwarding techniques and how container runtimes implement them in this heavily illustrated blog post 👇

- Docker Engine
- Docker Desktop
- containerd
- nerdctl
- Lima

iximiuz.com/en/posts/docke…
Even when regular port publishing stops feeling like magic, there is still a way to extend your skillset!

Did you know that it's possible to access ports of the already running containers?

What about services listening on the container's localhost? 🧙‍♂️

iximiuz.com/en/posts/docke…
Read 5 tweets
Oct 22, 2023
🧵 Efficient Learning Path to Master Containers 📚

1/ When I started with Docker in 2015, I viewed containers as lightweight VMs with fast startups. But this oversimplified view posed risks:

- Misunderstanding capabilities
- Misusing the technology
- Misjudging safety means Image
2/ The "container = VM" simplification was helpful at first but it quickly became inadequate.

To truly understand what I can and cannot do with containers, I had to dive deep into Docker's internals. However, available materials were either too basic or too complex. Image
3/ After a lot of trial and error, I came up with a learning path that finally worked for me:

- Linux Containers: The raw details
- Container Images: Why and How
- Container Managers: Docker's role
- Orchestrators: Kubernetes and co.
- Non-Linux Containers: Broadening horizons
Read 7 tweets

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Don't want to be a Premium member but still want to support us?

Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal

Or Donate anonymously using crypto!

Ethereum

0xfe58350B80634f60Fa6Dc149a72b4DFbc17D341E copy

Bitcoin

3ATGMxNzCUFzxpMCHL5sWSt4DVtS8UqXpi copy

Thank you for your support!

Follow Us!

:(