My Authors
Read all threads
Kicked off @AWSreInvent 2019 by attending @houlihan_rick’s @DynamoDB modeling session. As expected, it was a 60 minute firehose of #NoSQL knowledge bombs. There was *A LOT* to take away from this, but here are some really interesting lessons that stuck out to me. #reInvent
DynamoDB performance gets BETTER with scale. Yup, you read that correctly, the busier it gets, the faster it gets. This is because, eventually, every event router in the fleet caches your information and doesn’t need to look up where your storage nodes are.
Big documents are a bad idea! It’s better to split data into multiple items that (if possible) are less than 1 WCU. This will be a lot cheaper and cost you less to read and write items. You can join the data with a single query on the partition key.
You don’t want to denormalize highly mutable data. Data that is likely to change often should probably be referenced, which can then be retrieved with a separate query (when necessary). Immutable data (like DOB, name, email, etc.) can be denormalized for optimizing data shapes.
Don’t use auto-generated primary keys as partitionKeys if they are not used by your primary access pattern. If your application doesn’t use those ids directly, then you are creating a "dead index" and you need to create additional GSIs to index meaningful references. Forget 3NF!
Avoid running aggregation queries, because they don’t scale. Use DynamoDB streams to process data and write aggregations back to your DynamoDB table and/or other services that are better at handling those types of access patterns.
Build idempotency into your DynamoDB stream processors. DynamoDB streams + Lambda = decoupled stored procedures. DDB Streams *guarantee* “at-least-once” delivery, which means your Lambda *could* receive the same event twice. Plan for that.
When you need to add a new access pattern, "You don’t necessarily have to remodel everything and throw the baby out with the bathwater.” ~ @houlihan_rick. Run an ETL and decorate some existing items or change some values. Maybe add a new index.
When possible, push certain logic down to the client. Rather than filtering out old versions or calculating availability by diff’ing existing items, let the client do that math instead of your application layer. #RickProTip
Use write sharding to store large indexes with low cardinality. Rick gave the example of languishing support tickets. Write these to multiple partitions on a GSI and then use a once-a-day look up to decorate other items. Parallelization is an added benefit. 👊
Use an attribute versioning pattern to store deltas for large items with minimal changes between mutations. Don’t duplicate a massive item every time some small part changes. Instead, store each attribute as an item with the same partitionKey.
Use NoSQL Workbench for DynamoDB. You can model your table, add indexes, and pivot your data visualizations. Plus, Rick uses it to model his tables and create the diagrams for his slide decks. Do you really need another reason?
Your workload most likely fits into a single DynamoDB table. Rick's latest example shows 23 access patterns using only *THREE* GSIs. Is it easy? Not really. Can it be done? Yes, and I have faith in you. #SingleTableForEveryone
Looking forward to the video of this talk being posted so that I can go through his examples in excruciating detail. And if you’ve not yet jumped into the DynamoDB pool, come on in, the water is fine! 🏊‍♂️ #NoSQL modeling is going to be a hugely valuable skill. Make 2020 your year!
And if you need some extra help working with DynamoDB, check out my open source project at
Missing some Tweet in this thread? You can try to force a refresh.

Enjoying this thread?

Keep Current with Jeremy Daly @ AWS re:Invent

Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

Twitter may remove this content at anytime, convert it as a PDF, save and print for later use!

Try unrolling a thread yourself!

how to unroll video

1) Follow Thread Reader App on Twitter so you can easily mention us!

2) Go to a Twitter thread (series of Tweets by the same owner) and mention us with a keyword "unroll" @threadreaderapp unroll

You can practice here first or read more on our help page!

Follow Us on Twitter!

Did Thread Reader help you today?

Support us! We are indie developers!

This site is made by just three indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3.00/month or $30.00/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!