Thread by @vsoch

Y'all we need to talk about developer productivity in HPC. It's terrible. Yes, it's great for simulations & all that but what about me? My ~/.profile is laughing at me as I struggle to create some semblance of a reproducible, comfortable working setup. "It will never work again!"

First, about editors. "Oh, we have Jupyter notebooks!" Y'all a notebook is not a developer environment. I do not work in cells. Remote displays are slow. I figured out a way to work on files locally and ssh back to save. I was blissful! One less barrier to productivity!🎉

Now, about environments. Oh, let's set up something simple with conda/pip, and maybe spack? For an early experiment with some simple data-science oriented Python I was up and running! This is developer bliss - we want to go off and have sword fights while everything *just works.*

Now comes complexity. I went to use software that I've run in containers and locally, and I saw about 10 new snowflake errors. "Oh, just change these niche details in these configs you've never had to use before" After 2 hours, it still didn't work. This is our reality😭

Here's the thing. The time that we aren't spending getting the same thing working again, that's when we are happy, creative, & thinking up interesting ideas. These things need to just work. Progress is what suffers when you don't think about / optimize for developer environments.

Why do we flock to GitHub or similar? We are empowered to develop there. They allow us not just to develop comfortably, but to work together. The builds reproduce. Why can't we feel this way about resources provided by our institution? Because they haven't changed in decades.⏰

I started working with containers in 2015 & they gave me some of this power, at least to build and install, and then run on HPC and avoid the struggle. But *not* to develop. Because it was so hard, I've made entire life decisions to just avoid needing HPC. Let that sink in.

I had so much hope that we would see change. But coming back years later, the problems are still here! On one side of the pit we have people flocking to repeat the same things. On the other side we have innovators trying to make companies and sell me something. Uh, neither?

And then you have the conflict of the users. There are actually three distinct cultures. The admins want stability and "get off my lawn!" with that new shiny thing. The users just want to do science. The developers are trying to balance between those two and make everyone happy.

The reality of being an #rseng that uses HPC is that you try to avoid it. You do a lot of developing locally, and then debugging snowflake environments. Software is tested on GitHub to be deployed in completely different environments! How does that make sense? Why so many places?

We create these problems for ourselves. We *choose* a shared setup where we can support all-the-things all-the-time over a more controlled one that provides a reliably working set of software. No, I don't need to load Python 2.7, and why is it even there, let along the default?😕

I talked to a software engineer in industry, and he was laughing. He doesn't have these problems, because there are entire teams ensuring the successful build and availability of environments. There is a single software stack and you get heads up about changes to test your code.

These problems are embedded in our culture. We place emphasis on GPUs/flops/power/top500 over simple things like developer environments. Can we pause for a second, and consider not building an entirely new system every 4 years to rethink how to make what we already have better?

I can't help but notice how much easier it is to do just about everything off of HPC, whether it's in some online service, a cloud provider, or locally. Entire teams supporting developer tools is a thing! I don't care if HPC has been like this forever, it needs to be better.

Yes, its selfish for me to give attention to developer productivity because I am a developer. It's fair to say "Get off my lawn" and point at the amazing discoveries afforded by our systems. I think they are great too! But I am a developer, and I know what I do matters too.

Have you noticed that so much of what we do is trying to get around the problems of HPC instead of trying to change the underlying cause? As a developer, I am not empowered to think of new ideas or designs. I have the same limited permissions as a cluster user, understandably.

We do the same routines every year with talks & conferences. We aim for technical impressiveness. We like to hear ourselves talk. There is a lot of talk, but often not a lot of doing. I used to ignore this & just pursue what I loved, but the older I get, the more wrong it feels.

And thus we lag behind everyone. So let's not just talk, but make a plan to test new ideas for developer environments because this isn't good enough. Institutions that care about their outcomes should care about their software and consequently need to care more about developers.