Thread by @dvassallo

TIL what EC2's "Up to" means. I used to think it simply indicates best effort bandwidth, but apparently there's a hard baseline bottleneck for most EC2 instance types (those with an "up to"). It's significantly smaller than the rating, and it can be reached in just a few minutes.

For example, a c5.xlarge gets throttled to 1.14 Gbps after just 13 minutes.

A c5.2xlarge gets throttled to 2.28 Gbps after 26 minutes. So the baseline is 2X higher and takes 2X longer to reach. It also costs 2X more.

This stuff is so obscure that I bet 99% of Amazon SDEs that use EC2 daily inside Amazon don't know about these limits. I only noticed this by accident when I was benchmarking S3 a couple of weeks ago:

https://twitter.com/dvassallo/status/1113641894641143809

. Today I wanted to try to figure out what was going on.

The only place I see this documented is by @andreaswittig here: cloudonaut.io/ec2-network-pe….

@awscloud - Why not document EC2's baseline and peak network bandwidth? What's the point of hiding this information? How is this customer obsessed?

Someone on reddit pointed out that stopping & starting the instance resets the throttling: reddit.com/r/aws/comments…. I just tried it. It's true! Left image shows the throttling reset following a stop/start. Right image shows continuous throttling on an uninterrupted instance.

The throttling behavior is mentioned here: docs.aws.amazon.com/AWSEC2/latest/…. h/t to reddit.com/r/aws/comments…. But nothing about baseline rates or burst durations.

If anyone wants to know the network behavior of any instance type, let me know. I can easily test it, and I'll post the data.

I couldn't sleep well not knowing when EC2 would start throttling my network, so I woke up early today and tested all the "up to" instances of the C5/C5N/M5 families with a download loop from S3.

Summary of results in the next tweet 👇. Lmk if you want to know about other types.

Burst Gbps, burst duration, & baseline bandwidth:

C5 & M5
L = 6 for 7 min, then 0.6
XL = 9 for 10 min, then 1
2XL = 9 for 20 min, then 2
4XL = 9 for 40 min, then 4

C5N
L = 6 for 55 min, then 2.5
XL = 13 for 40 min, then 4
2XL = 20 for 45 min, then 9
4XL = 20 for 75 min, then 13

If you want to run these tests on your instance, I've added the throttling test in the s3-benchmark tool: github.com/dvassallo/s3-b…

./s3-benchmark -throttling-mode=true

It runs a download loop from S3 saturating the network. Then monitor NetworkIn from CloudWatch.