Thread by @colmmacc

Tuesday Tweet thread time! Warning, because this one is *shameless* promotion and a very minor, but still crass, personal victory lap 🤣. I'm going to compare the work it took at my former startup jobs that would now take ... oh, seconds, with EC2.

At my last job, I built 6 multi-rack installations of servers. Most were SUN (RIP) boxes in Level3 (RIP) colos. Our biggest was in our own datacenter, which we converted from office space. Job before that I built out two room-sized installations.

Anyway, at my last job (Joost), we were streaming video using p2p (until we abandoned p2p). We needed to build our own "Long Tail Sites" CDN. We couldn't use commercial CDNs, because we used our own protocol. That was very dumb, but there you go.

Before we could build any sites, we needed to get colo space. We didn't have a lot of time for negotiating and getting through vendors, so we focused on vendors that could sign one contract that covered many sites. So folks like Level3 and Equinix.

Still that took months. Meetings between me, our purchasing manager (Menno), our Lawyer (Melinda) ... just to get to good commercials. We ended up going with Level3 because they could *also* sell us bandwidth, which meant one fewer contract to negotiate.

We had to do all of this too for which servers to buy. We ended up going with SUN hardware, though x86 not Sparc, because we wouldn't have people nearby the sites and they had an excellent reputation for reliability. Also Mads, one of our sysadmins, had a thumb on the scale.

Oh also, the SUN boxes had a great ILOM (Integrated Lights Out Management) which meant we could actually boot and reimage the things remotely, which was important. We had a "Golden Image" deployment pattern.

I think it was maybe 3 months between deciding we needed sites and actually having everything signed and ordered, and another 6 weeks before hardware would come and the space actually be usable. This was really really fast and involved me pestering a lot of people.

Oh we also had to choose a network hardware vendor: I remember cold calling Cisco through contacts in Ireland and asking who could be the best solutions provider that support RMAs globally. The answer was "no-one".

Ended up with a European support provider (BT Infonet I think) and support in the US via Cisco directly. Oh well; all this just so that we could have Cisco 3650's in a bunch of places.

The boxes would all ship to the colos, and I'd fly to each one for a few days to set them up. Except for London, I drove to that one.

Let me tell you, when you show up in 2008 at Harwich Ferry Terminal, with a working class Dublin accent, and a heavy van full of equipment, and tell them you're going directly to Lodnon's financial district ... they take you aside.

I was coming from our HQ in Holland, and en.wikipedia.org/wiki/1988_IRA_… is a nice round up of some relevant and awful mutual Irish, Dutch, British history.

All setup, the London site looked like this. Count the servers, all this work for just 11 boxes and two routers! flickr.com/photos/colmmac…

I set up similar sites in New York, Los Angeles, and other places I've forgotten. Driving a van in Manhattan was fun! Not. Oh and Level3 left the combination lock codes inside the cages! I had to jimmy my camera in under the door one time, take a photo of the codes, then unlock.

We could drop-ship supplies to the nearest CDW, things like screws and cables. Everyone knows US and EU have different power standards, but did you know that they have completely different cage-nut types? Things you learn!

Our site in Luxembourg was one of our biggest, for advantageous tax reasons. I was even asked to trombone our Dutch office internet via the Luxembourg site so we could geo-ip as there. Oh and it was in an old unused telco switch room below a Pizzeria.

flickr.com/photos/colmmac…

Building our own transcoding facility was even more epic. Trucks and trucks of gear. Took about a year to build out ... and we were a 2 year old startup!

flickr.com/photos/colmmac…

It was around this time that EC2 first launched, and literally the day it went public, I asked our CTO if we could just use it instead. I worked out the numbers, and even at EC2's day one prices (which would now seem exorbitant) ... the time savings seemed worth it.

The transcoding stuff I thought EC2 was perfect for, but I was told we couldn't use EC2 for some hand-waving reasons to do with MPAA security requirements. Sadness.

The summer after we built all this, I interviewed at AWS. I interviewed at Amazon in Dublin, and I told every interviewer I only wanted to work on AWS. Turns out AWS was building a CDN, but couldn't tell me, and it'd be a good fit.

I remember telling one of my interviewers (Doug) that the kind of stuff I was doing now was dumb-ass and that AWS would make it history, and that I wanted to be on the right side of that. He was more modest ... that AWS/Cloud could carve out a niche.

There's a long way to go still, plenty of workloads yet to move. This thread isn't a victory lap for Cloud/EC2. But good god is it hard to imagine a startup in 2019 repeating what we did in 2007/2008. It would be *insane*.

The entire capacity we built out, which was certainly 10s of people years of effort; lawyers, admins, sysadmins, developers, flights, drives, etc ... I can spin that up in seconds using EC2. Seconds. In more locations! Cheaper!

That truck worth of servers that took months of negotiation and logistics? Trucks like that pull up to AWS every day! You can even order a SnowMobile truck of ours to pull up to your own site and then come to us. It's mind-blowing.

One of my favorite parts of my job is that I go to SFBA every month or two and talk with several startup customers. It's so fun to meet people who don't know any other way; who never had to do it the burdensome way. That is very very satisfying! /End-of-thread or AMA.