Thread by @jeanqasaur

A few months ago, as users started throwing more API traffic at the Akita client, we started seeing multi-GB memory spikes. 🙀

At first, we hoped for an easy solution. There was none.

This is the story of @markgritter's 25-day journey taming memory usage in a #golang client. 1/

@markgritter The first thing @markgritter did, as you would do in most languages, was to profile the system for obvious bottlenecks.

There was one, in the #gopacket reassembly buffer. Memory was persisting even though it wasn't getting used.

Mark fixed this, but the problem persisted. 2/

@markgritter At this point, the team and I weighed possible next steps.

Since #golang is memory-managed (2x memory overhead no matter what 🙀) and does not expose much control over the garbage collector, we were worried that it was not possible to be able to get memory usage low enough. 3/

@markgritter When I asked about other solutions, @markgritter said, "One solution is to rewrite the CLI in #Rust."

As Rust gives control over memory management and our code base was not huge, a rewrite was viable.

We ended up talking through the pros and cons of this option seriously. 4/

The major factor in our rewrite decision was speed to a functional, scaling CLI.

A #Rust rewrite was bounded time, but a lot of it. We didn't know how long it would take to fix our #golang memory issues.

A rewrite meant that we'd be trading off one set of issues for another. 5/

And some people think we're joking here, but "ability to hire good people and build faster" was a very real factor in our decisions.

Recently, we've noticed that not only have Rust rewrites become a Startup Thing, they're actually working in terms of attracting talent! 🙀 6/

We ended up deciding @markgritter would spend a bounded number of weeks fixing the memory issues in #golang.

If memory usage remained too high, we would seriously consider a rewrite.

If Mark could fix the issues, then we could tell everybody how we managed to buck the trend. 7/

After a couple of weeks in the depths of despair, @markgritter finally got our #golang CLI to an acceptable level of memory usage.

While identifying fixes often required understanding the Go GC, many of the fixes themselves had to do with using libraries more efficiently. 8/

It turns out that @markgritter's initial reaction that rewriting in #Rust would have traded one set of problems for another was right.

If we had done a rewrite from #golang to #Rust, we would have had to go through an analogous process of finding inefficient library usage. 9/

Now @markgritter has suffered so you don't have to! He's written this EXCELLENT blog post about his experience taming #golang memory here:
akitasoftware.com/blog-posts/tam…

And I've actually been learning quite a bit about the discussion in the HN comments:
news.ycombinator.com/item?id=286080…

end/