We had downtime today. It was a stressful 3 hours trying to diagnose the issues and repair our systems.

Sharing some learnings and takeaways below 👇 but the TLDR: Re-evaluate trade-offs regularly. 1) Account for the Redis Engine CPU cost of TLS connection negotiation during a storm of new connections due to auto-scaling (over-scaling) API servers to support an unexpectedly large surge of users and activity.

Want to make your real-time app feel even more magical?

Queue up offline socket events for 90 seconds after a user backgrounds the app and resend them upon foreground.

Don’t make your users pull to refresh every time they re-open the app How do we do this?

When we get the client's socket disconnect event (from Redis, API Gateway, or insert your pub/sub of choice), we create a Sorted Set in Redis with a TTL of 90 seconds and a new key that resembles our connection key (i.e dcon:{connectionId} )

Today we cut our Redis Engine CPU Utilization by 80% and survived to live (and grow) another day!

We were CPU constrained & maxed out scaling limits for our ElastiCache cluster. We had 2 hours to reduce utilization before our next traffic surge or we were toast.

What did we do? First, we connected to our cluster via redis-cli and ran MONITOR to identify the key prefixes used at the highest frequency.

Overwhelmingly, one data model stood out as the most frequently accessed