Thread by @pappapez

Thread. A long thread. I have had two weeks of my worst and best open source development time ever, where the bad parts have contributed to the good parts. #OpenSource

It has been a mix full of learnings, many disappointments, moments of joy, overconfidence, humbling events, utter fear of failure, great hope of failure avoidance, great hope of success, wonderful help from colleagues and the community, >

and, hopefully, it will be a great success event in the history of Calva.

Let’s see if I can relay some of the feeling. But first some background:

I work as a part time open source developer. My employer, @agical, pays me to spend one day a week on Calva. I have GitHub sponsors supporting me as well. github.com/sponsors/PEZ

Most of my free time is spent on this. An estimate is that I average at least 40 hours a week on Calva. The bulk of that is spent on support.

Calva is an extension for VS Code, providing an IDE for Clojure and ClojureScript developers. calva.io

Clojure is a LISP and thus very dynamic. The mode or programming is very different from what most people are used to. We call it Interactive Programming, some call it REPL Driven Development. #Clojure

Another part of Clojure being a LISP is that it is a very structured language. Clojure is defined in terms of its own data structures. If you have never encountered it, you can compare it with XML.

This opens up for what we call structural editing. It is not line based, as with many languages. You edit by modifying the structure. I mention this structural editing because it is what last week's major challenge for me has been about.

As LISPers many of us rely on a system called Paredit for this. It goes way back to early ZMacs implementations. calva.io/paredit/

There have also been many attempts at other editing systems. A very recent and eye-opening one is Vlojure a graphical Clojure editor.

In terms of LISP history there is another recent take on structural editing, although it is maybe ten years old or so: Parinfer, invented by Shaun Lebron. Here you edit the structure by using indents. shaunlebron.github.io/parinfer/

Yes, like with Python, Haml, Yaml, and some other indentation based languages. Clojure is much more structural than Python, though. So maybe compare it with Haml first, (which is indentation based HTML).

Clojure itself could not care less about the indentation. It cares, deeply, about brackets (of some different kinds, commonly referred to as parens from now on).

In this spirit, Parinfer does not hide the parens, instead it takes care of placing them for you based on indentation. Many people find it easy to see the structure from indentation. A major function of Parinfer is therefore to lower the barriers of entry into Clojure land.

If the parens overwhelm you in the beginning, you can almost ignore them. Parinfer has got your back!

(Parinfer has a very solid mathematical foundation. You really should read about it just to learn stuff. It’s technology taken to the level of art.)

Lowering the barriers to Clojure happens to be a very important part of the Calva project. See Getting Started with Clojure as an example of this. For this goal, the way Calva’s auto-formatter has interfered with the VS Code Parinfer extension has been a big problem.

It pains me to admit it, but I have had to focus away from this problem. When developing an IDE as full featured as Calva, a lack of great problems is never a thing.

It is some comfort that Calva has a stellar Paredit implementation (I dare to say with a straight face). Structural editing is covered.

However, for many beginners it is a bit like the parens that they are overwhelmed with get even more prominent. Paredit does not cover all the editing bases that Calva wants to cover in order to be a great choice for all Clojure beginners.

It is not just the lack of good Parinfer support that is a bother. The bigger problem is that VS Code makes it very hard to implement ”real” Parinfer reliably.

Fact is that even Shaun Lebron himself, as well as the current maintainer of Parinfer, @oakmac1, has found the challenge too big of a time sink.

Oh, wow, that was some background. Let’s transition into the challenge…

And now I can't add more tweets to this thread! I'll need to continue it some other way...

Wait. I could add another tweet! Maybe this can go on. If I have any readers left after that intro...

A couple of years ago I spent more than a month’s worth of Calva development time on trying to implement Parinfer SmartMode, a mode where you can either modify the indentation or the parens and both will work totally intuitively. (The current Parinfer extension uses two modes.)

That ended in a total failure. VS Code just didn’t provide the API I needed to fulfil the need of SmartMode to deliver its promise.

Shaun Lebron is right in noting: ”Editors simply are not yet designed to allow an ideal version of Parinfer to exist—probably because nothing like Parinfer has demanded them before.”

After that I decided that I simply couldn’t let other parts of Calva suffer any more from lack of focus, and put away my ambitions to ever make Parinfer a Calva thing.

Until some half a year ago, when I started to think differently about the problem. What if I could combine the strong Calva auto-formatter and Parinfer’s paren inference in a mode-less way?

After all, the indent inference of Parinfer is really just a relaxed formatter. Some time after I started to think like this I spoke to a user who mentioned using Parinfer this way together with something called ”aggressive indentation” (I think that is an Emacs mode)

I didn’t act on it then, because of the non-lack of big problems I mentioned earlier.

Then, two weeks ago, a pull request arrived on a piece of Calva that deals with paren inference (this actually is not completely lacking from Calva, maybe I forgot to mention). It came at a moment where I had cut out a big chunk of some other Calva work to focus on.

I decided to put that other work aside for a moment, maybe I could afford to do some Parinfer experiments?

Aaaaand. Down in the rabbit hole I went. Again. I must be crazy.

https://twitter.com/pappapez/status/1457646079961440259

In theory my idea for implementation should be quick to test. And it was. Shaun and Chris had made the hard work of actually inferring parens from indentation. And I had already made a lot of the integration work.

(Or so I thought...)

I started with making some needed refactorings, to expose some functions I wanted to reuse. The idea itself quickly got validated. I could create a mode-less Parinfer experience this way! At least nothing so far was telling something different.

I decided to go for releasing the new cool feature as a disabled setting and spent some day’s worth of Calva time on creating guards and make it not interfere too much with users that did not enable Calva Parinfer.

I tested it and I created builds that other Calva users tested as well. It all seemed good. The implementation was a bit rough, but as a proof of concept it would be quite awesome.

The change was still so big that I needed to be available to deal with any problems the release may cause. So I released it on a Calva day (my Wednesdays are my Calva days).

I then started to prepare for making a Youtube video demonstrating the feature and try encourage people to test it so that I could get information on how to proceed from here. I was in celebration mode!

About 10 minutes after the new Calva version was available on the VS Code Marketplace a colleague at @Pitch (where I spend my workdays that are not Calva days) asked in our VS Code channel if something was up with Calva.

All of a sudden he couldn’t enter parens into the files. Parens are mighty important when editing Clojure code, as we have already noted in this thread.

Together we quickly isolated it to be about editing files with CRLF line endings, I couldn’t reproduce this, but everything indicated it. My colleague could not reproduce it on the same files when they had only LF line endings.

Not knowing the scope of the problem, but judging from the short amount of time between release and the first problem, it was obvious I needed to roll back. The problem with that is that our CI pipeline is only prepared for successes. Rollbacks, huh?

I managed to do it swiftly anyway, but in the stress (the stress level when something like this happens is hard to describe) I created a mess in the git repository.

So before I could attend to try figure on the actual bug I had to clean up and it took me a while to get things back into a way where it is prepared for releasing new versions of Calva and isolate the Parinfer changes neatly so that I could continue to work on them.

My Calva day was up before I could continue any Parinfer work.

The marketplace Calva was only broken for an hour, maybe less. I don’t think that too many people got affected. I allowed myself to be happy about my decision to release this on a Calva day.

At the same time I started to get user feedback on the actual feature. People were thrilled and the TODO-list grew with things that needed to be addressed, apart from the horrible bug.

I couldn’t reproduce, but since my colleague could, and since he is an awesome person he shared some of his time with me and we debugged Calva on his machine. We didn’t quite find the bug, but collected a lot of clues. It had to do with the auto-formatter.

When not having access to my colleague’s time I couldn’t really work on the bug. So I worked on the other items on my growing todo-list. github.com/BetterThanTomo…

I had fun, although I was deeply worried about what on earth could be causing the bug. Maybe something about the Calva infrastructure was broken in ways that would not allow Parinfer after all?

These are worries that a struggling tool smith does not cope with very well...

Since LF files are the most common, I could work without being affected by it and my kind testers could verify that my work was going in the right direction. I was soon at a point where I was almost feature complete for the experiment to be supported well.

Just a few more tweaks… And then. Wut?

Why does the code break when I undo this change? It looks very similar to that CRLF bug… Maybe they have a common root?

I could reproduce this one, but I couldn’t for the life of me even start to form a theory about why it happened. It was late. I updated the pull request about that I might have run into a stumbling block that would stop the whole thing. Off to bed.

During the night I remembered something. Someone at the @Pitch slack had mentioned git bisect the other day and what a wonderful tool that was.

After some sleep, with some fresh ideas, today being a Calva day, and determined to put git bisect to use, things looked brighter.

I mean, the reality was the same, but I wasn’t in doom ‘n gloom mode about it now. If this was the same root cause as the CRLF issue I was in a much better position now, right?

I love git bisect! ♥️

It quickly found me the commit where I had the issue and I could put the file I suspected the most in a three way split in VS Code. The current file, in full width, at the top, and I placed the diff that git bisect had found below, in vertical split.

My eyes wandered around in a triangle for a while. Ha, there I see the bug! What kind of crazy person had coded that? Someone that can’t really cope with the mutation that Calva is built on…

Anyway, easily fixed! Maybe I have fixed the CRLF bug too? That would be a bit strange because why would this particular bug cause those problems?

My colleague wasn’t available to test it for a while so I chose to ignore that clue and pretended everything was good now. Some final touches on the experiment and this should be good to go any day now.

Of course the CRLF bug wasn’t fixed.

The new build behaved exactly like before on my colleague’s machine. We decided to run git bisect on his machine next week when he would have some jak shaving time allotted at work. That would probably mean next Wednesday.

What to do today then? I decided to look over the documentation and plan the Youtube video I never had recorded.

While testing and figuring out how to best record it I happened to open one of the CRLF test files I had created and the bug hit! Exactly the same bug. In an LF version of the file: no bug. In this one: bug.

I was so happy I could have started to cry!

git bisect time again!

And here we are. The actual bug perfectly matched the clues we had collected that first screen sharing session, my colleague and I. (It was in that first refactoring I mentioned way up in the thread.) The latest build works! The bug is gone also for my colleague.

This was some hours ago. I should have spent my time preparing a release. But instead I got this urge to write it all down. I have no blog, so a Twitter thread will have to do.

I take away so many things from this. I’m actually still in the middle of it so can’t really summarize. But I can tell you that hunting around in the innards of the Calva editing infrastructure has been extremely valuable to me.

Yes, I realize there is a lot of technical debt there, but I also can see how trusty the code base is. It delivers even when I have new crazy things for it to do.

I can also say, with a straight face, that even if this would have ended badly, in another failure to bring Parinfer to Calva users, I would have celebrated.

It is fun to work on a feature that has so devoted fans and that demands the best developer that I can be. It is not fun to cause breakage for my users, and not fun all the time when hunting ghost bugs. But when they materialize and I have git bisect in my corner. Ha! Awesome.

I now realize that I might have made it sound like Parinfer is mostly a beginners tool. That's not at all how it is. It's an intuitive and powerful way to perform structural edits without explicit commands. I will run with it enabled, for sure.

Also. It is an absolute requirement on Calva's implementation of Parinfer that it should work together with Paredit. There is no either/or there. These systems complete each other in the most beautiful way.

You can follow the development of Calva Parinfer in this thread btw. I started it once the first PR was a draft.

https://twitter.com/pappapez/status/1457646079961440259

The thread is now unrolled here threadreaderapp.com/thread/1461018… Thanks, @count_lovelace !

This didn't really work, as readers of the thread know.

https://twitter.com/pappapez/status/1461019672925642759

I think my mistake was to put the feature guards in as an afterthought. It wasn't really possible to fully isolate the change to only those who chose to enable the experiment. >

Are there any general advice out there about how to plan and act from start when releasing an opt-in experiment with minimal risk for the users who chose not to participate? Some massive temporary code duplication, maybe? If you know where I can learn about it, please share! 🙏♥️

Please cross your fingers for me as I try once again to launch this experiment! calva.io/parinfer/ #parinfer #Clojure

As an added bonus we actually launch two experiments today. 😀 Somewhat related. Anyway, if you are not into experimenting with Parinfer you might want to try out the new ”aggressive” auto-formatter mode.
github.com/BetterThanTomo…