Thread by @gravislizard

@hypercalcium ok so 2012, i'm tier 1 support at a voip provider, we have to deal with peoples ISP problems a lot. i'm talking to this customer who's having issues with their internet service dropping out.

@hypercalcium this shouldn't be our problem except that we provide the customers primary router, because at the time no other router was trustworthy for VoIP. so I have to figure out if it's our devices fault or their ISP.

@hypercalcium so i ask, who's your ISP. it's some name i've never heard of. i ask, is it fiber? DSL? "no, it's microwave."

in 2012 you couldn't get a decent pipe in san fran. i assume that's changed some, but god, internet in 2012 was *rough* and has come a HUGE distance.

@hypercalcium since burying new copper or fiber in SF is a laughable proposition for a startup and the LECs move at their own pace, microwave was a neat "hack" - you put dishes up on buildings that have fiber coming into them and PtP to your subscribers. actually a very clever solution

@hypercalcium now, my instinct was "hey, you have wireless internet, that's your problem" because we knew that wireless PtP between buildings was often spotty, but the ISP assured us the link was tested and clean

@hypercalcium i can't recall the exact diagnostic process but I believe what I did was run a continuous ping to their WAN IP and observed that when their service cut out, I stopped hitting my router out there, but pings continued working at a different latency.

@hypercalcium if you speak layer 2/3 pretty well this one's obvious - ARP contention. but on a WAN IP? how? i verified they had nothing else going into the media converter, since that is indeed all their "modem" was - a layer 1 bridge between ethernet and RF

@hypercalcium the ISP was clearly a startup because the guy I was talking to was obviously one of their three total techs and knew way more than tier 1 ought to, so I engaged him on discussing the headend layout. now, at this point i knew how T1/DSL and cable were delivered

@hypercalcium your internet connection goes into the phone line or cable and at the far end it converts back to ATM or ethernet and has to enter the ISPs network. you don't use "a router" for this, you use a BRAS, a thing with separate logical interfaces for every subscribe

@hypercalcium i asked the guy how their network architecture at the headend worked. "are we going into a BRAS or just a router?"

"it's a switch. all the RF converters terminate into a switch, and then..."

@hypercalcium mother FUCKERS had NO IDEA how to run an ISP. all their subscribers were ON THE SAME LAN SEGMENT. THE SAME BROADCAST DOMAIN. you could just ARP to take another subscriber's IP. ZERO security. i was horrified

@hypercalcium what they had going was not topologically different than just having all their customers pringle canned to a wifi AP plugged into a fiber modem. it was really tragic. i told the ISP to figure out which customer was stealing this one's IP and closed the ticket