Thread by @astuyve

@fitzsimons_dev @dynamodb Oh! You're one of today's lucky 10,000!

DynamoDB streams are GREAT! I use them in almost every place I use a DDB table with #Serverless. I suggest doing non-essential work via stream vs in one long function call, so the end user gets the fastest experience possible.

Here's a 🧵

@fitzsimons_dev @dynamodb 1. Sending a confirmation email after a user registers.

This is an important function, but users shouldn't have to wait for my API to call an email provider before getting a response from the backend. Stream it to another function!

@fitzsimons_dev @dynamodb 2. Stateful/synchronizing needs! Lambda functions are a distributed system, no way around that (unless you set function concurrency to 1, but I digress). So you need to build idempotency into your app.

Example 👇

@fitzsimons_dev @dynamodb 2.cont Say you receive a Github PR webhook and want to kick off a job, like a CI task. Use the first invocation to ack the webhook, and load a new task in your DDB table.
Then use the stream-triggered function and a conditional update to guarantee your job runs only one time!

@fitzsimons_dev @dynamodb 3. A user swipes their card for your paid service, but you don't want to update multiple records in the callback from your payment service, because you can't risk dual charging.

Use a DDB stream to update the necessary record(s) once the initial invoice is processed!

@fitzsimons_dev @dynamodb 4. You can subscribe MANY functions to ONE stream. Consider Twitter for Pets. When someone important like @QuinnyPig's dog tweets, you will probably want to promote that in other TLs. Do that in a function triggered by DDB streams, don't hold up Corey's dog!

@fitzsimons_dev @dynamodb @QuinnyPig 5. DDB Streams work on a batch, and you can set a timeout window. Imagining Twitter for Pets again - you can use a stream with a batch of events to produce/collect trending topics. 5,000 dog park tweets in the last 20 minutes? Something is happening!

@fitzsimons_dev @dynamodb @QuinnyPig 6. Flaky APIs are a bummer. You can use of course use SNS and SQS to retry a flaky API with a backoff, but that's just as easy to do with a DDB stream, and DDB streams work off of shards (like kinesis, but DDB records are durable) - this can help prevent a noisy-neighbor issue

@fitzsimons_dev @dynamodb @QuinnyPig 7. Say you've got a busy app and are writing to your table frequently, but want to report aggregated metrics somewhere. You can use a DDB stream to read a batch and emit aggregated metrics, in a way that won't slow down your writing functions/code.

@fitzsimons_dev @dynamodb @QuinnyPig 8. Another concrete fan-out example. Your app denormalizes data to Elasticsearch for app search, but your hotshot new CIO wants it in redshift.

You can add a specific function (with its own error handling, biz logic, etc) for ES as well as Redshift - on the same stream

@fitzsimons_dev @dynamodb @QuinnyPig 9. Say I'm creating a slackbot for my custom deploy script for my custom k8s cluster (because I've entirely lost my mind. If this ever happens - send help).

Store the request & ack the webhook from Slack, then use DDB streams to trigger a function and run the deploy

@fitzsimons_dev @dynamodb @QuinnyPig 9.cont You can keep using conditional updates and treat your DDB record as a state machine - conditionally updating the record to update progress & ensure idempotency.

@fitzsimons_dev @dynamodb @QuinnyPig 10. Let's end with something more concrete - removing related data when a user deletes their account. (Not single table design in this case)

You won't want to open a transaction and remove 100s of records - that could time out or fail partway through.

Use a DDB stream!

@fitzsimons_dev @dynamodb @QuinnyPig 10.cont when the user record is deleted, you can trigger a function to size up a batch delete. It can perform it (and retry on failure), or fan out to other systems that need to clean up data. GDPR and DDB streams are meant to be

@fitzsimons_dev @dynamodb @QuinnyPig To sum it up - end your function as quickly as possible, and do the rest of the work asynchronously with streams!

The more you practice, the more opportunities you'll see to perform background/async work with DDB streams