Thread by @QuinnyPig

So it's probably time I pick this back up.

Let me begin by saying I'm *pretty* sure that this isn't sour grapes; I left ten years ago as of six weeks from now. I have no financial stake in Expensify, but I do have stories. I was the "Director of TechOps" for a year.

https://twitter.com/QuinnyPig/status/1450850872271900678

Let me further disclaim that my information is thus a decade old. I hope it's all ancient history, but these stories have a way of helping people out. I hope and trust that as we all have, the culture there has evolved significantly.

First, the good parts. I've never seen a place that was as good as Expensify was about onboarding junior engineers and training them basically from scratch. "Senior engineers are expensive, so we'll hire them before they get there" is a great philosophy.

I've also never seen a startup as willing to build hard things from scratch. This was a double edged sword, but we'll get there downthread.

I'm not a big fan of looking back. Times change, people change, and it's best to move forward.

But for five years after I left Expensify I still got phone calls from their data centers about various issues. This is Bad.

It's hard to let go when they won't leave you alone!

Speaking of leaving things alone, the kerfuffle over their Mechanical Turk scandal and "we fire people" and "why we don't hire .NET developers" posts has mostly disappeared from the public awareness.

But if you want a good example of the culture at the time I was there, this is short and fairly representative.

web.archive.org/web/2015040200…

I met @mike_julian in person when he interviewed there. The company took him out for a boozey lunch, got back to the office, put him in front of a whiteboard in front of the whole company, and hit him with this (as they did EVERY candidate):

https://twitter.com/mike_julian/status/1379997837673463811?s=20

The post-leaving character assassination drove me nuts. This particular gem was "DNS is notoriously unreliable, so use Saltstack to manage /etc/hosts instead." SURPRISE, it broke after I left, and I got posthumously blamed for this.

https://twitter.com/QuinnyPig/status/1450846297603264518?s=20

The people who left inherently "sucked." I'm sure it had nothing to do with "not wanting to work on a PHP expense reporting app for a decade or more without an exit." At the time there was a "you work 50 hours a week" expectation. I still hate the concept.

The day I decided to leave was when the CEO called and fired my direct report out from under me.

It was 9PM and we'd just worked all day in a data center.

That's not how you manage people.

The bedrockdb.com kerfuffle? Yeah, that was fun. "We'll build our own!"

HackerNews may love such things, but not being allowed to drop $5K on Palo Alto firewalls and instead spend months kicking PFsense into shape kinda soured me on the idea of "build it ourselves."

"We have a retirement plan but don't bother contributing to it because your equity in the company is your real retirement plan" may be the worst thing I've ever heard someone tell a new grad.

Rather than going with native apps, they built their own cross-platform library to build mobile interfaces. As per their S-1 they're finally knocking that off and going with React. SO SOON?

For one month out of the year they took the entire company on an international off-site to someplace far away. Amazing and unique experience, but also "people with families need not apply."

There was a cofounder (Witold, I miss you) who's apparently been completely written out of the entire Expensify origin story. Not cool; he set a lot of critical path things in motion during my tenure there.

There was also the standard startup fare of "mobile engineer --> VP of Eng --> CFO --> VP of Different Eng --> gone" in less than three years. That's vaguely terrifying for, y'know. A fintech.

An interesting point was "take yourself out to eat every day at a restaurant, and expense it." It was a great way to get new grads exposure to expense reporting, since they'd never done it.

That was the year I learned "eating out every lunch costs me ~5K a year."

It wasn't all bad! Expensify is where I met @kkuchta, who became a close friend. If you imply that he's not a good engineer because he chose to leave the company, you are a goddamned clown.

All in all, it's what you'd expect from a disciple of Travis Kalanick (which he was! He's name-checked in @MikeIsaac's "Super Pumped" book!) who even now makes a company that's going public all about himself instead of the team.

Now, the BedrockDB database nonsense.

It's been ten years but OH MY GOD. It'd be fast and zippy and then one day start bogging down. You'd have to vacuum the SQL database node by node (too many at once, you lose quorum and probably your job). This took HOURS.

They split the key to start the thing up between multiple people. You submitted your part of the key to the process by, and I swear I'm not making this up, hurling it across the wire via netcat.

It kicked a lot of the multi-primary model up into the application layer, but I remind you: this is an app for expense reports! That's one of the most downtime-tolerant use cases there is! Teach your mobile apps to retry in the background and you can take 12 hours a day down!

The access pattern for receipts is fascinating as well. You upload the receipt, it gets "Smart Scanned" (read as: humans transcribe it), added to your report, and while you need to be able to produce that receipt image upon demand, you will never view it again.

There was also this screamingly funny post about what a rip-off @awscloud is compared to bare metal. It could just as easily have been titled "I have not read the EC2 docs and made a bunch of unfounded assumptions." Which, hey, we all do that. But come on.
blog.expensify.com/2018/01/08/sca…

I *strongly* suspect that the call I had with the CEO to resign was intended as a "you're fired" conversation, I just got to it first.

I was told that day could be my last day, and I'd be paid 2 weeks' pay as severance. Then got an email an hour later "since I resigned there will be no severance."

Classy!

The best part was an email a day or so later asking me to sign a non-disparagement agreement for $0. NOPE.

All in all there were some amazing people there who built amazing things, there was a bunch of nonsense involved, and there were fascinating stories that came out of it.

@mike_julian always teased @kkuchta and I about our "mean time to Expensify" since it invariably came up.

There were cultural problems symbolic of the era, I've never before or since encountered a stronger "reinvent from first principles" mindset, and the app solved a painful expensive problem I was passionate about.