I'm sad to report that the PostNL Event Broker project I'm responsible for has broken its two-year streak of no production incidents.

In fact we failed to deliver about 40k events, and *did not even notice* until a customer told us 3 days later.

So let's learn from it. We also successfully delivered 17 million events in the same time frame, so 0.235% of events were impacted. But the affected customers don't care about this stat. They lost 100% of their events.

And the work of real humans - our delivery drivers - was impacted.

How a single-line bug cost us $2000 in AWS spend...

We recently refactored a Lambda Function. We extensively tested its functionality and released it into production. And everything still worked as expected. But then the billing alarm went off..

(repost with sanitized images) This Lambda Function calls credential-protected downstream APIs. It does so about 4.5 million times a day. The credentials are stored in AWS Secrets Manager, and you can see where this is going...

Fun #AWS project at work this afternoon: calculating the configuration values for #DynamoDB provisioned concurrency with autoscaling.

Background: we have a DDB Table with a very seasonal load pattern. It's currently configured for on-demand pricing, which is expensive. Provisioned capacity with auto scaling requires us to define the minimum WCUs, the maximum WCUs, and the target utilization levels. But what values should we use?

The first two numbers are easy - just set them to a baseline and some safe maximum (we chose 3x the current peak).

55.4% of 523 votes knows how to improve Lambda performance! Good! For the rest of you, here are the three answers I was looking for (thread):

https://twitter.com/donkersgood/status/1554060652876075008

1. At memory configurations under 3008MB, the CPU is throttled. Increase memory to gain more CPU power. From the documentation:

> At 1,769 MB, a function has the equivalent of one vCPU.

At 3,008 MB, a function has the equivalent of two vCPUs

docs.aws.amazon.com/lambda/latest/…

My perspective on the top Serverless features and cost reductions announced at re:Invent, in one giant thread. Enjoy! 1. Lambda now supports event filtering for SQS, DDB & Kinesis Data Streams.

It's always been inefficient to process irrelevant messages in Lambda, only to drop them immediately. Native filtering removes this responsibility from our code.

Announcement: aws.amazon.com/about-aws/what…

I recently found a new enumeration vulnerability in AWS. It allows me to identify valid account IDs and any IAM principal in it. I had a call with AWS security, and they say it’s by design. Well then, let’s take a look! Short summary: we can use API Gateway resource policies to verify if any AWS account ID, IAM user or IAM role exists. In any account, without a trace in the target's CloudTrail or any other (user accessible) log.

I’m building a tool that collects every @awscloud resource through List* and Describe* calls. This could have been easy with consistent APIs. After many hours of suffering, let me tell you a few of the many ways AWS APIs are everything but.

A thread 👇 1/ Let’s start with the basics: you would expect that a List operation returns a list of resources and a Describe operation returns the resource’s details. Right? Wrong. For example, in RDS you list resources with DescribeDBClusters, DescribeDBInstances, DescribeDBSnapshots…

OMG I did it! I got all 12 AWS certs 🤓🚀 💻 🎉 💯

To celebrate, a thread with some thoughts on each exam👇 1/12 Cloud Practitioner. The only cert in the Foundational category, its questions cover topics like 'what is the difference between block and object storage' and 'should you host MySQL on EC2'.