Thread by @cyen

"How We Used Serverless to Speed Up Our Servers" 💥✨🐝

woohoo, here we go #strangeloop!

(bear with me everyone, livetweeting is a skill i am still leveling up)

https://twitter.com/jessitron/status/1444363956466880514

how meta! a @jessitron shout-out to @samstokes's #strangeloop talk from several years ago, on a prior evolution of our datastore

@jessitron @samstokes ✨ visualization of how we pull column segments to service a given query.

(the state of our datastore/query engine as of sam's talk 4 years ago)

via @jessitron #strangeloop

@jessitron @samstokes narrator: time to brace for scale

"now we have more data, and when you have more data, what do you do?"

"put it in S3, of course" 🙃

via @jessitron #strangeloop

@jessitron @samstokes "the limiting factor [for this query process] isn't IO, it isn't memory, it's CPU."

we're constantly aggregating + deciding whether to keep or toss some data -> so we're constantly CPU bound.

the naive state: exceeding "sip of coffee" time range

via @jessitron #strangeloop

@jessitron spoiler: farm that compute out to lambda! to get that sublinear graph

via @jessitron #strangeloop

@jessitron real talk about using lambdas in this way—

3-4x as expensive as EC2 measured by compute-second (feels like a lot, except when you recognize the extreme spikiness)

via @jessitron #strangeloop

@jessitron "lambda scales. very quickly... until it stops"

"we're using it as burst compute; AWS... doesn't sell it that way. 😬"

via @jessitron #strangeloop

@jessitron observability helps here ;)

it's now trivial to say "hey, how many lambda invocation spans are running *right now*"

(this CONCURRENCY operator is now available to everyone in @honeycombio ✨)

@jessitron "these [lambda] sleeps are not restful sleeps—the sleeps are more like carbon freezes"

cleanup is pushed to the next invocation

via @jessitron #strangeloop

@jessitron at 99.9% success rate, and the scale of lambdas that service any given large honeycomb query—"failures are totally normal, retries are totally normal, so tune those parameters"

("the go SDK by default will retry for 5 minutes" 🤯)

via @jessitron #strangeloop

@jessitron lambda workaround: requires JSON as input into the function?

dump the compressed data into S3, in a format you control, and send a link via JSON

via @jessitron #strangeloop

@jessitron "lambda is *so opaque* compared to running on your own servers."

"observability makes a big difference here in figuring out what the heck is going on."

via @jessitron #strangeloop

@jessitron "we have a @honeycombio trigger that tells us if any single customer has cost us more than $300/day in lambda costs alone."

"functions have a mysterious amount of cost, and we have to react to how our customers are using it" 🤷‍♀️

via @jessitron #strangeloop

@jessitron @honeycombio cool screenshot of using @honeycombio to test out the performance of our lambdas on the new graviton2 processor (touted to have better price performance), over the last few days

via @jessitron #strangeloop