Thread by @freebsdfrau

When you wield God-like abilities, people come to you with humble requests like this and within seconds you just blow their mind -- alternate introduction: "what can you do with bpftrace on Linux?" Let's go …

We're sunsetting an NFS server, replacing it with Ceph. Problem: There are hundreds of systems using the NFS server. Here is the request that has come-down from on-high:

Using a bit of kit called bpftrace on Linux (or dtrace on BSD), we can run this tool on the NFS server to tap into the kernel and provide statistics on NFS activity github.com/FrauBSD/nfsdtop

You shove this into a cron-job (adjusting the "-iSEC" and/or first "*" to something like "*/5" to determine how much sampling you want to do ... "*" and "-i60" is 100% sampling, "*/5" with "-i30" would be 10% sampling):
* * * * * root nfsdtop -JN1 -i60 >> /var/log/nfsd_stats.log

You then create a database, run telegraf configured to shove the information into said database, and now you've got time-series data representing all NFS activity centrally derived from the NFS server itself in real-time

You now have a very slick database with very slick contents from which you can create slick dashboards

So once you've got that sweet sweet data flowing, now we can answer the original ask by time-series data. As we try to sunset this server, we want to see the data come to a halt. Clearly some people need to be told to stop ... they did this morning, evidenced by the graphs

That's it. That's all there is to it. Now, I wouldn't necessarily call it over-engineering. You saw what the manager asked. It just happens that most people when faced with such an ask would probably berate the manager with "you don't computer, do you?"

Because let's be honest. "No" was a perfectly acceptable answer here. I just happened to have a wartime arsenal of tricks up my sleeves, so "Yes" was a possibility as well as getting it done in less than 1H. But guess what? I gave the tools away for free …

So while I have not YET released the dashboards (wait until this sunsetting operation is complete), you've got access to the tool that generates the data from the kernel plumbing. So have it. Become God-like to your peers and indispensable to managers …

Managers that, 12 years after you have left are still telling your replacements "well (insert name) could do it"

So what are you waiting for? If you run an NFS server, you should be using dtrace/bpftrace to plumb some JSON stats for database injection so you can look like a hero github.com/FrauBSD/nfsdtop