Thread by @theburningmonk

Great question from my current cohort of students, paraphrased:

"Should you always use Step Functions to chain together a few Lambda functions? Are there patterns to simplify this? How about using SQS between the functions?"

Here are my thoughts 🧵

Firstly, on the broader topic of orchestration vs choreography, I've written my thoughts before. TL;DR is that I prefer orchestration for intra-service workflows, and use events for inter-service communication.

theburningmonk.com/2020/08/choreo…

If you have a dead-simple workflow then Step Functions can be overkill, especially if you're new to it.

Simplest approach:
1. implement workflow inside a single function
2. use Lambda destinations to chain several functions together

Some considerations to think about...

❓ How is the workflow started?

Lambda destinations don't work with synchronous invocations. This is actually one of my #awswishlist items, to be able to use destinations with sync Lambda invocations.

❓Does the caller need the end result of the workflow?

If so, then you're better off implementing the workflow in a function or using Step Functions' synchronous workflows. OR, you'd need something like the decoupled invocation pattern

theburningmonk.medium.com/applying-the-d…

❓ Do you need to be able to restart a failed workflow from where it failed?

Doing this usually requires you to implement the logic to skip/fast forward to the previously failed state yourself. But it's trivial when you chain functions via destinations or EB/SQS.

While Step Functions add complexity (for really simple workflows), it also adds value.

There's the obvious value in having visualization in both the design view, but also the execution view (great for customer support teams).

It also simplifies error handling when you move the error handling and retry logic to the state machine definition, it removes the tension between having a reasonable timeout value (for Lambda) vs allowing extra times for retries and exponential backoffs.

When you implement workflows in a Lambda function, it's tricky to work out what timeout value to use. Each step can fail, and each step should have some retry with exponential backoff, so execution time can have a big range between the happy path and the worst-case scenario.

What's wrong with just using a high timeout value?

Usually nothing.

But, if you make a mistake, or get attacked (e.g. regex DOS), then the high timeout value amplifies the cost of those Lambda invocations. Have seen a few clients get stung by this...

Finally, if you were to chain functions via X, should you use SQS or EventBridge?

It depends. Do you need batching, ordered delivery, archiving or replay of these messages? Those would heavily influence your decision here.

https://twitter.com/radzikowski_m/status/1513941279175942155

Also, there's a difference between events ("something happened") vs tasks ("do this thing").

My rule of thumb is to use SQS for tasks, and EventBridge for events. But for convenience's sake, I sometimes masquerade tasks as events to avoid adding lots of SQS queues...🙈

And that's a wrap!

If you enjoyed this thread:

1. Follow me @theburningmonk for more of these
2. RT the tweet below to share this thread with your audience

https://twitter.com/245821992/status/1537782149176246274