Fun #AWS project at work this afternoon: calculating the configuration values for #DynamoDB provisioned concurrency with autoscaling.
Background: we have a DDB Table with a very seasonal load pattern. It's currently configured for on-demand pricing, which is expensive.
Background: we have a DDB Table with a very seasonal load pattern. It's currently configured for on-demand pricing, which is expensive.
Provisioned capacity with auto scaling requires us to define the minimum WCUs, the maximum WCUs, and the target utilization levels. But what values should we use?
The first two numbers are easy - just set them to a baseline and some safe maximum (we chose 3x the current peak).
The first two numbers are easy - just set them to a baseline and some safe maximum (we chose 3x the current peak).
The target utilization is more complex. A post on SO described how to calculate it:
> What is the biggest change in capacity usage you see over a time period of 15 minutes expressed as a percentage? Leave this amount of room in your target utilization.
stackoverflow.com/a/50018158/160…
> What is the biggest change in capacity usage you see over a time period of 15 minutes expressed as a percentage? Leave this amount of room in your target utilization.
stackoverflow.com/a/50018158/160…
This leaves the question how to calculate the biggest change in throughput capacity. Luckily CloudWatch Metrics Math supports the `DIFF(metric)` function, which contains the difference between any data point and its predecessor. The function `(m2-DIFF(m2))/m2` yields this graph.
This is a good start, but it also shows a target utilization above 100%, which doesn't make sense. The configuration limit is 90% anyway.
So let's add a max limit at 90%, which starts to look useful. The lowest value in the blue chart could be the right threshold value.
So let's add a max limit at 90%, which starts to look useful. The lowest value in the blue chart could be the right threshold value.
Last problem though: the threshold is very spiky at night, when load is low.
By taking the minimum WCUs into account, which is set to the horizontal line in this diagram, we can filter out any auto scaling at night.
By taking the minimum WCUs into account, which is set to the horizontal line in this diagram, we can filter out any auto scaling at night.
The lowest dip of the red line is now our accurate target utilization number. If we configure it at 60%, the provisioned capacity would be 16667 WCUs when the consumed capacity is 10000 WCUs.
This leaves exactly the right amount of room to scale with our workloads, leading to a big cost savings and no throttling at any time.
PS. This entire thread is also available on Mastodon, where it's much easier to read because of the 500 character limit there. mastodon.online/@donkersgoed/1…
• • •
Missing some Tweet in this thread? You can try to
force a refresh
