It's ten years since we launched Amazon CloudFront! Building it was super fun, my first day at Amazon was spent getting stuck into the CloudFront routing software and making it actually work. A few weeks later, but before we launched, we hit my favorite bug ever ...
of course it involves YAML, and may be the root of my hatred for that format! So CloudFront is a DNS based CDN, the "magic" happens at the DNS layer, where our DNS servers figure out what the best location to serve your traffic is ...
we built this gigantic live map of the internet ... for every network out there, we figure out what the correct ordering of CloudFront locations should be. E.g. if I'm Seattle, then it's something like SEA, PDX, SFO ... you get the idea ...
The map is based on latency measurements and geo-data. For the geo-data, we had import a huge geodb (we have a few of these) and import them into the CloudFront routing later. That import happened in YAML format.
Anyway, the bug was that the YAML parser ignored the negative signs ... so negative GPS coordinates became positive ones. We caught it before launch, obviously, but it's still my favorite bug because it was so subtle.
The bug affects 75% of the Earth's surface, but by geographic coincidence or Greenwish winning the meridian wars, it just happened that our entire dev team was unaffected. Both Seattle and Dublin are in the "positive, positive" quadrant of the world!
So we never experienced it ourselves, or our test cases, and we didn't notice it until we really started looking at other locations. I always think of that when I remember that even "dumb" and "obvious" bugs can be subtle and invisible because of your perspective.
*Greenwich in my previous tweet. Anyway, fun bug! CloudFront had an incredible launch team, including 8 then-or-now Principal Engineers, which is staggering!
