Thread by @carmatrocity

RIP TravisCI. A community-backed startup that commoditized CI/CD in the GitHub era, giving it free to open source, & playing a huge part in shaping the industry we find today.

It was an incredible group of people who built that together. Still proud to be a #TravisAlum

https://twitter.com/carmatrocity/status/1284540376892702721

Putting a top-level yml file in the repo to define your build and pipeline? I'm pretty sure that idea started with @svenfuchs : and the entire cottage industry that tamed that for K8s, for better or worse.

YAML engineers have Travis to thank for first thinking of it.

Understanding the power of GitHub API to do more than source code hosting and sharing? That was Josh and Sven, in 2011. GitHub's popularity, if it didn't partner early on with small companies like Travis, might have suffered the same fate at Source Forge, Google Code, BitBucket

What a small group of thoughtful, committed engineers, customer support, design, accomplished for just under a decade is humbling in the impact it had.

Anyone who used GitHub for personal projects between 2011-2018 probably used Travis.

The infra team I was on was, in hindsight, in over their heads. Looking at error budgets and SLOs/SLAs and the scope for each operations eng / SRE / Sys Eng has at Google, we each were providing availability, reliability, monitoring for build environments in as many as 5 clouds

To say Travis Infra did multi-cloud was an understatement

BlueBox
Packet
AWS
GoogleCloud
MacStadium

(and, as macOS has no native virtualization, ask @emdantrim, @mjmoriarty, @rynchantress, @sarahhodne what adventures that meant to instantiate ~13-15K mac builds per day

On top of this we wielded these builds to Heroku on the web and Platform side. (Heroku dynos would reset/resart every 24 hours IIRC, which is why we needed different clouds for the workers to sit on)

This was all 5-10 years ago, when clouds still were really rough around the edges and TravisCI engs needed to figure out their bespoke APIs and services. @meatballhat @igorwhileflase @bogdanavereha @solarce could tell you some stories...

Then there's the imaging and automation. Instantiating, running, shutting down, & deleting containers, VMs and directly on bare metal for multiple platforms, operating systems, dependencies, In the tens of thousands, every day. @chef + @HashiCorp packer + terraform = besties.

The observability toolchain ended up being my favorite, with much credit to @igorwhilefalse for exploring/experimenting. Travis was one of @honeycombio 's earliest adopters, and that was in 2107-2018. The product was amazing then, I can only imagine what it is now.

Building tracing for the workers on Travis for all the clouds (in Go 💙) on Google's Cloud Trace w/ Open Telemetry was something I was proudest to have implemented. Wrangling the macOS infra with MacStadium (the giant rube goldberg machine that @mjmoriarty tamed) was the hardest

Then there was the adventure of hitting quotas on all these platforms! CI/CD is Not Your Average User, and not getting the right quota from cloud providers or GitHub often meant outages. To be a TL on a team often meant you had your cloud Customer support engineer on speed dial

This might all sound easier in 2021 with more mature K8s solutions and even multi-cloud offerings like Anthos, but it was really emergent and ambiguous and we had to figure out how to change the engines in running cars along the way.

CI/CD is hard work. It's an ever moving target of cloud evolution and abuse (dag you stupid bitcoin miners, ask @kerrizor and @s0ulshake about their abuse detection). The security threat footprint is large, and the threat model (as we found out) is intense...

security researchers in year's past reached out to disclose some really gnarly stuff. The way the entire team would come together to remediate, disclose, communicate to users, and credit was something I was new to, learned a lot from.

I pass this baton on other companies still in the game, and salute you for carrying on the good work.

@ctrlsysrq could tell you stories here!

There's a WHOLE other side of Travis I didn't get go into, Platform eng, Web devs, Enterprise eng. Our small army of thoughtful Customer Engineers: imagine having to debug build failures for something like 17 languages, runtimes, and even more tools, dbs, and other such things)

Even Travis' logo, Travis and Tessa, is a testament to @ctrlaltjustine's thoughtful approach towards inclusivity and representation. It meant the WORLD to me and my family, expressed here.

https://twitter.com/carmatrocity/status/1062850307699499008

DOH :facepalm: I meant @mjmoriarity

There's so many people I didn't mention that just shaped me and grew me in ways technical, social, societal in that group. I think I grew more in my 3+ years at Travis than at any other time in my adult life.

Other people at Travis CI:

@nickstenning astute, thoughtful sizing up of situations
@ana_rosas always finding a blind spot in our perspective
@aakri_tea for always explaining things to me on the platform side
@carlaD for always articulating and voicing the systemic problems

@carlaD again for onboarding + internal documentation for engineers. So few startups invest in that, and AFAIK, Carla played a large part in transferring that institutional knowledge out of the most seasoned engineers' heads and into diagrams and docs. She inspired me to do same

There's some one liners that forever imprinted on me from @gigglegirl4e, who would eventually become our CEO. When you can condense an entire conflict, scenario, situation into one line, that is boss.

The memories are like smoke, and I feel like now that I've started, I can't stop.

I'll say it again. Travis CI was a special place to work because of its people, who were Travis' greatest asset, by far.

@plaindocs is THE technical writer for CI/CD. He got his start at Travis, and has helped other companies wrangle their docs. For that reason he probably knows more about the internal workings of CI/CD than anyone I can think of.

And then there's the Travis Foundation, @travis_fnd staffed by @langziehohr and @alicetragedy. The work these two did with @RailsGirlsSoC is a big reason why tech is more diverse than it was a decade ago.

https://twitter.com/travis_fnd/status/1189122868640178176

to add to these efforts, Many on the Travis team @svenfuchs and Joe Corcoran spent so many Monday evenings mentoring newcomers at @RubyMonstas. The Ruby community, especially in Berlin is vibrant thanks to the time they donated.

Speaking of Ruby, Hiro Asari: github.com/BanzaiMan was probably the engineer with the longest tenure at Travis, among the first hires and the last to leave. I remember meeting someone at a Go conference a few years back who followed all of Hiro's GH commits and said Hiro's...

commit comments, code organization, and general style was emulated by him AND his entire team. Hiro led though doing.

While Travis didn't invent pipelines, we did enable others to wire pipelines through GitHub easily in an era where Jenkins, Team City, etc. felt too hard to parse, and certainly not accessible to small scale OSS projects.

https://mobile.twitter.com/rikki_js/status/1441495159393316865

And trying to wrangle cost models in all of these! IIRC, our cloud costs were on par with salaries. We ran benchmarks with different images. We called it #costops
I remember @QuinnyPig offering his services at Scale16? 17X in LA (2017) to figure it out.

https://mobile.twitter.com/carmatrocity/status/1441459696817225728

Maybe Travis should have hired Corey @QuinnyPig for his services. We wouldn't be the first to rue this mistake.

@QuinnyPig

https://mobile.twitter.com/panda_rocks2020/status/1441495232596439040

Still have conflicting feels. I remember that fateful GitHub Universe when actions were launched, to everyone's surprise at Travis. Wish we could have gotten a heads up that showed transparency + respect.

For the GitHub Actions launch, many of us learned a hard lessons: ~how NOT to communicate to stakeholders.
~The sharp realities of hot ecosystems, ours was a fast-growing interest in, and thus many engineers and companies devoted to, CI/CD.

https://mobile.twitter.com/carmatrocity/status/1441459707844177922

I wanted to correct the handle. It was @mjmoriarity. Within a *few short months*, Matt implemented K8s + GitOps + deployment solution for the Mac Infra. He also refactored the image automation to play better in the Mac space. I learned so much from Matt.

So much is coming back to me. @bogdanavereha is one of those engineers that would chase a performance or scaling problem down a rabbit hole. In this case it was our queueing system with Heroku's managed Rabbit MQ/ AMQP.

Travis had hit so many queuing problems and even outages where we would blindly fiddle with `heroku cli` knobs + kick the servers to get it back online. For years we always were able to calm a retry storm, or scale up platform resources as a bandaid solution

but it was @bogdanavereha that untangled all the hand-off points in our distributed system to actually fix the problem b/t platform services running background processing, RabbitMQ instances on heroku, & pooled hand-offs to the workers in 4 different clouds.

Gosh, and I can't even with what @igorwhilefalse did towards reliability + o11y. They could deep dive into Postgres thread pool documentation, understand redis internals, improve our querying performance, clean up deployments. I am remembering only about 5% of what they did.