Profile picture
, 15 tweets, 5 min read Read on Twitter
A few weeks back, we've noticed our snapshot bills to grow weirdly high.. after some debugging, I've found the snapshot billing and way snapshots stored as an interesting concept..
As the graph says, the usage vs snapshot billing isn't growing proportionally linear.. (1/n)
I decided to debug how snapshots are billed.. fundamentally, snapshots are block-level backups of EBS volumes.. since AWS can't see data inside your block, they read disk blocks occupied and take a copy of them and store them as objects in their s3.. (2/n)
Every time there's a data change inside your EBS volume, by nature of linux and hard disks, the new blocks is where data is written.. So, AWS knows there's change of data (this could be deletion of data too) and marks those blocks as the difference in data..thus incremental (3/n)
snapshots. So, all data points are simply stored in s3 as objects in incremental format. When you restore a snapshot, you get all the underlying objects automatically and that causes the volume to take some time to warm up to make OS read all the blocks (4/n)
One of the common practices is to delete older snapshots in order to save costs. While AWS misses the terminology appropriately, we internally started to call snapshots as "recovery points" and the snaphots as backups. (5/n)
When you delete a snapshot from console, you're deleting recovery points and not the actual backups. Assume you've spawned up a 100gb disk and been taking snapshots over the last 3 years for every 2 hours, and the nature of your app is to delete data older than 10 days. (6/n)
Now, over the years, your usage of ebs volume remained at 100gb while some of the data difference could've been much larger. Assume we're deleting 10gb everyday, that's 10,950gb deleted over 3 years. (7/n)
While you're continuing to take snapshots, you're paying price for 10,950gb (compressed in s3) while you may need only last 1 week of data for restore. and deleting snapshots won't work as you're deleting recovery points only. (8/n)
To deal with situation, I've turned off the scheduled jobs to take backups every two hours.. deleted all last recovery-points of the disk. As all recovery points are deleted, AWS would get rid off corresponding s3 objects too. I gave it a day before taking next snapshot.(9/n)
After a day of such operation on several disks, I've brought down my snapshot count near to zero (I left root volume snapshots and AMIs as is).. now, I've turned on my scheduler again to make the system think it's the first snapshot ever. (10/n)
time taken by AWS to take a snapshot has always been mysterious and I've expected this one to take long. So it did, about 4 hours for a 100gb volume. And I waited 2 days for cost explorer to reflect the new costs. (11/n)
Waking up on a Monday morning and seeing the console first thing, hurray! it worked. The costs came down.. so, key takeaway now had been a decision of wiping away all the snapshots of a volume every month and starting afresh so that we'll pay only for a month's data drift. (12/n)
So, instead of paying for 10,950gb, we'll pay for 300gb (compressed). Also, there's more to it. If you take a snapshot S1 of V1, create a new volume V2 and make a new snapshot S2.. S2 will use the same underlying data points as S1.. thus you're not paying for it twice. (13/n)
Instead, you're paying for the underlying data blocks (including drift) once and independently for data that differs in V1 and V2. It gets interesting once you delete V1 and you know, S1 is never actually gone until S2 and above are going on.. (14/n)
Here's an AWS support related to the last point.. Thanks for reading it through.. hope it helps you plan your backups in a better way.. Also, AWS has launched Backup service. I find it easy to use and manage. (15/n)
#AWS #backup #devops #ebs #snapshot #tech #cloud @awscloud
Missing some Tweet in this thread?
You can try to force a refresh.

Like this thread? Get email updates or save it to PDF!

Subscribe to Srujan
Profile picture

Get real-time email alerts when new unrolls are available from this author!

This content may be removed anytime!

Twitter may remove this content at anytime, convert it as a PDF, save and print for later use!

Try unrolling a thread yourself!

how to unroll video

1) Follow Thread Reader App on Twitter so you can easily mention us!

2) Go to a Twitter thread (series of Tweets by the same owner) and mention us with a keyword "unroll" @threadreaderapp unroll

You can practice here first or read more on our help page!

Follow Us on Twitter!

Did Thread Reader help you today?

Support us! We are indie developers!

This site is made by just three indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3.00/month or $30.00/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!