Lots of things can ruin the average person's Christmas holiday. In 2019, one network engineering team ruined their Christmas by combining IP SLA operations, track objects, and static routes.

Let's find out how! 🧵

Prefer a blog post format? Click here: chrisjhart.com/TAC-Tales-How-… It was Christmas Day of 2019, and I was working the holiday shift in Cisco TAC. Working Christmas is enjoyable - it tends to be quiet, and in the rare case you need to assist with an issue or outage, customers are nice and in good spirits.

A #network administrator's worst nightmare can be intermittent network congestion - it's impossible to predict, short-lived, and has major impact. Can #Python help us find and fix it?

Let's find out! 🧵

Prefer a blog post format? Click here: chrisjhart.com/Practical-Pyth… A case I've seen in TAC is where customers observe intermittently incrementing input discard counters on interfaces of a Nexus 5500 switch. This is usually followed by reports of connectivity issues, packet loss, or application latency for traffic flows traversing the switch.

Excellent thread from Nick on this topic! A big point I'm a fan of:

"...most juniors don't have an immediately accessible lab on their laptops or cloud environment, because they don't spend much time labbing. Most mid-levels can spin up a topology on demand."

https://twitter.com/nickrusso42518/status/1485250684492656643

Labbing something does not have to be a arduous, time-intensive process. Being familiar with lab resources available to you and knowing how to efficiently use them is paramount to getting definitive answers to questions quickly.

A common misunderstanding engineers have about Equal-Cost Multi-Pathing (ECMP) and port-channels is that they increase the bandwidth that can be used between two network devices. This *can* be true, but isn't *always* true.

Curious why? 🧵 First, let's review our topology. Three Cisco Nexus switches are connected in series. Traffic generators are connected to Switch-1 and Switch-2 through physical interface Ethernet1/36. Switch-1 and Switch-2 connect to Router through Layer 2 port-channels.

"I see a lot of packet loss when I ping my switch" 🚩🚩🚩🚩🚩🚩🚩🚩🚩🚩🚩🚩

Wait, why is this a red flag? Let's dig into this behavior in a bit more detail... 🧵 First, let's take a look at our topology. We have two hosts in different subnets that connect to a Cisco Nexus 9000. One host connects via Ethernet1/1, and the other connects via Ethernet1/2. Ethernet1/1 has an IP of 192.168.10.1, while Ethernet1/2 owns 192.168.20.1.

On Cisco Nexus switches in production environments, avoid working within a configuration context on the CLI unless you're actively configuring the switch. Otherwise, you might accidentally cause an outage by trying to run a show command.

Curious how that's possible? 🧵 Cisco IOS and IOS-XE require you to prepend show commands with the "do" keyword to execute them within a configuration context.

Discovered an interesting issue at home today - when I ping a Nexus 9000v running in CML from an Ubuntu host, I see duplicate replies. At first glance, you might think the Nexus is duplicating replies. Meaning, a single ICMP Echo Request packet enters the switch, and the Nexus sends two ICMP Echo Reply packets.