Thread by @copyconstruct

Now the talk I’m looking forward to the MOST - @colmmacc on PID loops (pronounces P-I-D loops, and not “pid” (like sid) loops as I’ve always been pronouncing it in my head.

This is going to be so good! I’ll try to live tweet.

Haha @colmmacc put my DM to him the slides - as an example of observations and feedback in practice! 😁 #QConNYC

Giving a shoutout to @vllry’s previous QCon talk.

The second book wasn’t written for a distributed systems audience per se, but it has a lot of patterns and lessons in it that applies to our field. @colmmacc at #qcon

When and where control theory is directly applicable but rarely applied.

@colmmacc at #QConNYC

[ed - be still, my beating heart!]

The most classic example of control theory in practice is a ... furnace.

Control theory has a lot to teach us about how to effectively operate a furnace. The biggest thing it has to say is - measure the system in terms of the errors. @colmmacc at #QConNYC

A proportional controller (the P in the pid) deals with responding “proportional” to the error.

An integral component can help offset [ed - I missed what came next]

A thermostat is a “PI” controlled system. But even a “PI” controlled system isn’t entirely stable.

If you can build a mental model analyzing systems through the PID framework, you can very easily determine if a system isn’t stable.

In the case of an auto scaling system (a glorified furnace), we’re measuring for instance the CPU. If it’s way below say 20%, you scale it down.

AWS recently launched machine learning based forecasting.

To @colmmacc this is a fancy integral (the I component of the PId controller) #QConNYC

How can PID loops help with a problem like placement (so users are directed to the closed POP)

Cloudfront runs a control system which constantly measures the utilization of every center, and change the radius or every edge node. It’s a big system interconnected with PID loops.

Common lessons from control theory that are missed in real world systems:

1. Open loops
A system as simplified in the code shown results in an open loop. These kind of systems can’t deal with big shocks in the system.

If an action isn’t taken very often, that’s an open loop.

Chaos engineering and observability help close “open loops”.

Measures that can help “close” open loops. @colmmacc at #QConNYC

H/t @caseyrosenthal @nora_js

A really common open loop is ... certificate management.

[ed - I’ve never thought about this before, but OMG yes, this is such a clear example of open loops in practice]

More ways of dealing with open loops:

- trying to close loops should be a part of system design.
- formal verification.

2) power laws

Imagine a distributed system as one big uniform collection of boxes. Systems can exhibit power laws - where an single error can have a massive blast radius.

Ways of dealing with power laws:

- compartmentalizations. Reduce the blast radius [ed - recent google outages come to mind here]
- rate limiters are like derivators.

There’s a whole science in control theory called “loop tuning” [ed - now I WANT TO LEARN ABOHT THIS ALL]

3. Liveness and lag

In dist sys we use workloads that can take variable amounts of time, and getting metrics back can also become lagging.

The lag starts driving the system in case of sudden spikes.

Doing everything everywhere in constant time is an impractical solution here.

In AWS, health checking has to work at ALL times. They are built as constant time systems, since it’s important for them to work and return information is related at all times.

If queues get too long, it’s better to return errors

LIFo is commonly overlooked - it always prioritizes liveness.

4. Look for false functions.

In a dist sys, there are many dimensions of utilization. Caching, memory utilization etc all go up at different rates and have different functions. We often end up measuring some synthetic metric of all three systems.

Examples of false functions - LOAD AVERAGE — [ed - 🙏 ]

5. Edge triggered systems.

Edge triggering invites modal behavior. If you’re going to build an idempotent system, might as well make it a level triggered systems.

Summary

Fin.

[ed - that was the most inspiring talk I’ve seen. The room was packed, the audience completely arrested, and @colmmacc now gets to put on his resume that he performed at Broadway!

Context - QCon is happening at the Marriott marquis right on broadway, nestled between all the great theatres