Thread by @KWF

I have two objectives tonight:

1) Sit on the floor and enjoy this pineapple cider.

2) come up with the dumbest way possible to load test an authoritative DNS server.

Some of you might ask "but why not load test your DNS server like any... normal(?) person?"

And to that I say, "my bottle opener is a wrestler and can kick your ass"

We have these tools at our disposal:
* A potato laptop I pulled out of the E-waste.
* Some Ethernet cables, which I pulled out of the e-waste
* An Arista 7050T-64, which I pulled out of the e-waste
* My DNS server DUT, guess where I got it
* My wits

To get us started, three cables into the switch. DNS server, laptop, uplink to the rest of my network.

Create a new vlan 100 on the switch, set them all up as access ports for vlan 100. This is all easy peasy

Quick check in the switches' mac address-table, and we can see the DNS server on eth45 and the laptop on et47.

Good deal. We'll need that later.

Log into DNS server, write stupid dummy zone to ask about.

Log BACK into DNS server, and re-write the zone so it's something that BIND is actually willing to load...

But I mean, come on, isn't that satisfying? We can get answers to questions you didn't even know we were asking.

So at this point, we have a functional DNS server with our example zone and 13k other random zone files I grabbed from... the e-waste?

So how do we break it?

This is where things get dumb.

We plug an Ethernet cable into Et1 and Et3.

On a dumb switch, this would be bad, and cause a broadcast storm.

On a smart switch... spanning tree is involved, so it's still bad.

Checking spanning tree, we can see that, yes, just as you would hope, spanning tree has placed Et3 in discarding and nothing bad is going to happen.

We can fix that.

So now we can see that both Et1 and Et3 are up, and they're on the same VLAN, so they're primed and ready to go off like a Juniper in production the moment they see any broadcast traffic...

But we don't want broadcast traffic. We want lots of DNS traffic.

So what looks like a broadcast storm, but isn't a broadcast storm?

Well... Each vlan has its own MAC address table, and we know that any DNS traffic will be headed to the DNS server's MAC address... so what if we point a static entry on vl200 out Et1?

So, recap, normally, the switch has a mac address table for vlan 100 and the DNS server's mac is dynamically learned by the fact that the switch sees traffic sourced from it.

We have now set up a separate vlan as a whirlpool of L2 death, and pointed a static entry for the DNS server out the first port...

Another sanity check before we break things.

The whirlpool is still quiet, because there's nothing else on vlan200 and the only traffic there is LLDP, which gets trapped at the other end and punted to CPU so EOS can learn about itself

Now we need to create an on-ramp to the whirlpool, so we add Et5 to the vlan, BUT, we apply an ACL to it to only allow traffic destined to the DNS server to come in...

Now WHERE do we have some good sources of traffic destined for the DNS server... 🤔

So there's the awesome thing about managed Ethernet switches; they have this feature called port mirrors or monitor sessions, where you can take the traffic passing through one port, and mirror it on another port to pipe it all to the CS^H^H your network monitoring tools

So we have mirrored traffic coming in from our laptop onto port 7

Then used a cable to pipe that back into 5, used the MAC ACL on 5 to filter for only the traffic we want

So now we can probably toss something into this whirlpool we made!

Generating a little bit of DNS traffic from our laptop isn't too hard...

Alright, so we turned on the monitor session going out Et7, fired three DNS A queries, then shut off the monitor session and shut Et5,7 real fast.

And the result?

a stable 1.04Gbps/1.5Mpps whirlpool of DNS queries.

So now that we have this maelstrom of packets, we want it back on vlan 100!

No sweat. We just set up another two ports to sample the tsunami and pipe it back into vlan100

We set et13 to "no switchport" since we don't need it to do anything other than be the destination for the monitor session, add Et15 to vlan100, and fire it up.

Survey says...

Well, we definitely can't ssh into this box right now, so cell phone picture of dstat on a monitor.

Well... huh...

7600 pps is a little less than I expected, but not by much, since we're playing with an Atom here...

But to be fair, we're kind of pummeling it with an entire 1Gbps of line rate DNS queries, so maybe it's not an entirely reasonable test.

And up until now we've only been dicking around.

To be science, we need to take TWO measurements.

This is a lot more what I expected. So slightly less than what it was able to do full throttle, and we have a decently well loaded DNS server.

8200pps out to the DNS server, 6600pps back in is kind of a lot of packet loss...

But then again, 8200qps is kind of a lot of DNS traffic for a little box that's going to be authoritative for a few zone? I think?

Finally, to be absolutely sure we're only pulling DNS traffic here, we can apply an IP ACL on the output of the whirlpool tap to only allow UDP DNS traffic.

We could have also applied this inside the "Ethernet Frame Particle Accelerator [TM]" I guess if we were really worried.

A HA!

As I expected, the full line rate was working against us when we were getting 7600pps as a maximum.

If we set the whirlpool tap to something slightly higher than that, we can get 8300 answers per second at ONLY 20% packet loss.

Eeep. Ok, so that's really about it.

So 8500qps out of an authoritative DNS server running on a Dell FX160 Atom box I pulled out of the e-waste. 🤷‍♂️

But the possibilities here with this technique are pretty endless as long as a lot of repetitive traffic is useful for your tests.

My whirlpool is a 10GbaseT loop, so the reason it was only running at 1.04Gbps was because I only had 3 packets in flight and... the speed of light

Well, speed of light and the switching latency between Et3 and Et1 inside the whirlpool...

But I've seen these test loops on 400G DACs, and it's just a matter of injecting enough packets into the loop to fill it to a level which you need for your test.

And on that note, it's almost midnight, and it's a work night, so I hope you were entertained, tip your waiters, wear a mask, and have some fun in these interesting quarantimes.