1/ One of the most stressful days of my career was when I found out that we lost ownership of our domain at @xdotai
2/ Having a domain issue is worse for us since x.ai was both our marketing website and our product. Customers schedule meetings using our personal assistant by emailing amy@x.ai or andrew@x.ai
3/ I was at some NYC entrepreneur lunch thing when my Slack started blowing up. Our DevOps team noticed that our app's error rate was shooting up. They have been debugging for 1/2 hr without a sense of the root cause.
4/ I excused myself and went straight back to the office. It was all hands on deck.
5/ Everything in our system seemed fine except that communications across the platform were failing randomly. Finally, one of our engineers looked up our domain and found out that we no longer owned it
6/ All of a sudden, our domain was owned by some dude in Nebraska. Our first reaction was that we got hacked. The tech team went into a frenzy going through security logs.
7/ In parallel, I triple-checked our domain registrar to ensure that we were paying our bills. We were and we were on an auto-renew plan
8/ I also looked up the new owner on Facebook, LinkedIn, and Twitter. The guy seemed to be a regular joe working in corporate America.
9/ I reached out to the customer service + security team at the domain registrar. They claimed that everything seemed to be in order. They can't explain why the domain was reassigned.
10/ Next step of debugging required the registrar to contact the government office that managed the top-level .ai domain. .ai is administered by the government of Anguilla, a beautiful Caribbean country that is not necessarily known for its fast-paced business processes.
11/ By this point, it was Friday afternoon. I was told that we'll have to wait until Monday. On top of that, Anguilla only updated its domain list once a week. Our company was set to be down for many days.
12/ Instead of giving up, we took our Ninja googling skills to work and found the contact info of the government official in charge of domains in Anguilla.
13/ After taking a deep breath, I called the administrator's home. His wife picked up the phone and said that her husband was on her way home. I apologized and asked for a callback.
14/ 20 minutes later, I got the return call and told our sob story. The admin sympathized with us and connected us with the IT person who owned the actual domain list.
15/ Turned out that ALL of the .ai domains were managed with an Excel sheet on his laptop. During the last update, there was a copy/paste error and our domain was assigned to someone else.
16/ Given the error, he made the exception and corrected the list immediately. After a few hours of DNS propagation, the domain was returned to us. By the weekend, we were fully back to normal
17/ Don't give up. #neversurrender. Don't take no for an answer.
@threadreaderapp unroll
• • •
Missing some Tweet in this thread? You can try to
force a refresh
