Thread by @_ChrisJHart

"I see a lot of packet loss when I ping my switch" 🚩🚩🚩🚩🚩🚩🚩🚩🚩🚩🚩🚩

Wait, why is this a red flag? Let's dig into this behavior in a bit more detail... 🧵

First, let's take a look at our topology. We have two hosts in different subnets that connect to a Cisco Nexus 9000. One host connects via Ethernet1/1, and the other connects via Ethernet1/2. Ethernet1/1 has an IP of 192.168.10.1, while Ethernet1/2 owns 192.168.20.1.

The architecture of most network devices has three "planes" - a data plane, a control plane, and a management plane. We'll focus on the first two. The data plane handles traffic going *through* the device, while the control plane handles traffic going *to* the device.

If we visualize the data plane and control plane of our switch within our topology, it would look like this. Notice how the control plane connects to the data plane through an inband interface. Also notice how the control plane hosts various software processes, such as ICMP.

ICMP traffic between the two hosts flows through the data plane of the switch. This makes sense, because traffic between the two hosts will go *through* the switch - it is not destined *to* the switch.

However, what if the switch gets an ICMP Echo Request packet destined to itself (e.g. 192.168.10.1)? The data plane will recognize that the switch itself owns IP 192.168.10.1 and forward the packet to the control plane's inband interface. This action is called a "punt".

When the control plane receives this packet through the inband interface, it will inspect it and "route" it to the ICMP software process so that the ICMP process can handle it accordingly.

The ICMP software process will most likely generate an ICMP Echo Reply packet, which will be sent to the control plane's inband interface, which is dequeued by the data plane and forwarded back out of Ethernet1/1 towards the host.

However, what if the switch was receiving a *lot* of ICMP traffic all at once? For example, a malicious actor may be sending the switch more ICMP traffic than it can handle, or maybe a network monitoring system (NMS) is aggressively monitoring switch reachability through ICMP.

This could clog the inband interface and control plane with unnecessary traffic or cause high CPU utilization, which would inadvertently affect other control plane protocols (such as BGP, OSPF, Spanning Tree Protocol, etc.) and cause instability in the network.

This is where the concept of "Control Plane Protection" comes in. We need a mechanism to rate limit the amount of control plane traffic sent to a network device so that the control plane of the network device does not get overwhelmed with traffic.

On Cisco Nexus switches, Control Plane Protection is primarily implemented through "Control Plane Policing" - a feature better known by its acronym, CoPP. This is enabled by default, but you can confirm it's configured with the "show copp status" command.

CoPP is implemented within the data plane of the switch and enables the data plane to drop a specific class of traffic if the rate of traffic for that class exceeds a threshold. It's essentially a QoS (Quality of Service) policer for the control plane.

The output of the "show policy-map interface control-plane" command can show us how classes in the CoPP policy are organized, what type of traffic corresponds with each class, and each class's policer rate.

In our scenario, the "copp-system-p-class-monitoring" CoPP class handles ICMP traffic. We can see that by default, a 360 kbps CIR (Committed Information Rate) of ICMP traffic is allowed with a 128 kilobyte Bc (committed burst).

We can also see that 32 megabytes of ICMP traffic has been allowed by this CoPP class, while about 279 megabytes of ICMP traffic has been dropped by this CoPP class since the last time the counters were cleared. Clearly, this switch is being *blasted* with ICMP traffic!

As it turns out, I had another SSH session open to my host that was mindlessly slamming the Nexus with ICMP traffic. Oops, silly me! Thankfully, this isn't production! 😅

Now that I've stopped the other ping command, we can see that my original ping works as expected with no packet loss.

What's the moral of this story? There is a fundamental difference between data plane traffic and control plane traffic. Intermittent packet loss observed when pinging a network device is not necessarily symptomatic of packet loss observed when pinging hosts on the network.