i'm finally ready to share my personal story about this whole #golang dependency management process.
Consider it a counterpoint to the story Russ shared last month: . (Please note that he also publicizes private communications in that thread.)
Two weeks ago, i published a screencast that distills the single most essential technical issue (information loss) with the module system that shipped with Go 1.11.
Reasonable fixes should still be possible.
Reasonable fixes should still be possible.
But the screencast covers the technical bits. Though it will intertwine with some technical bits, here, i'm going to talk about the process.
Process is incredibly important. Collaboration is one of the most foundational values for any open source community.
Process is incredibly important. Collaboration is one of the most foundational values for any open source community.
i know this has been a difficult process for folks to watch. There are so many things i wish i'd done differently.
Most of all, i'm sorry that i've not provided a clear, comprehensible explanation of my concerns. i hope that this story, plus the screencast, fix that.
Most of all, i'm sorry that i've not provided a clear, comprehensible explanation of my concerns. i hope that this story, plus the screencast, fix that.
Healthy collaboration makes for healthy communities. And healthy communities are a necessary condition for the long-term health of the software in the ecosystem.
Unhealthy collaboration hurts us in both the short and long term.
Unhealthy collaboration hurts us in both the short and long term.
Now, a number of community members have asked me to keep this story to myself.
i'm speaking up about it anyway, as i believe that an honest accounting of damage done is prerequisite to moving forward, and defining our values as a community.
i'm speaking up about it anyway, as i believe that an honest accounting of damage done is prerequisite to moving forward, and defining our values as a community.
Important preface: i continue to believe what i wrote in the post i made when Russ first announced vgo: it would be hypocritical of me not to get on board with a truly better technical solution. sdboyer.io/blog/vgo-and-d…
Also, please understand that i am not arguing (nor have i ever), ""dep was fine, use it as-is."" i get to have new ideas, too.
i understand why it's tempting to make dep-vs-modules comparisons, but it's usually not relevant, if not misleading and harmful.
i understand why it's tempting to make dep-vs-modules comparisons, but it's usually not relevant, if not misleading and harmful.
It might help to understand that dep has three basic parts:
1. the visible bits: CLI commands, Gopkg.* files, etc.
2. the version selection engine
3. the VCS source management system
i always expected 1 to change. Really, to disappear. 2&3, i'd hoped to keep and evolve.
1. the visible bits: CLI commands, Gopkg.* files, etc.
2. the version selection engine
3. the VCS source management system
i always expected 1 to change. Really, to disappear. 2&3, i'd hoped to keep and evolve.
However, dep is a fitting starting point for this story - in particular, the idea that somewhere along the line, dep became unacceptable to the Go team.
That point, IMO, was before dep even had a name. And it's because we were using a SAT solver as the core algorithm for version selection.
The very first email Russ ever sent me inquired about this.
The very first email Russ ever sent me inquired about this.
And, a couple weeks later, the first words he ever wrote on the topic were:
“Dependency hell is NP-complete. But maybe we can climb out.” research.swtch.com/version-sat
“Dependency hell is NP-complete. But maybe we can climb out.” research.swtch.com/version-sat
Now, certainly, avoiding NP-complete problems is preferable, when possible. They come up with surprising frequency in compiler development, and dodging around them is just “what you do.”
But i was skeptical, from the very beginning of the interaction, that there existed a solution that would avoid NP-completeness, without just redistributing the complexity elsewhere.
i emailed him specifically to that effect, inquiring (among other things) as to how his hypothesis - that the *goal* was avoiding the NP-hard search - might be falsified.
No response.
No response.
(If talking about algorithms seems overly academic, don't worry. The key takeaway here is just that this has been an argument about the core version selection algorithm from the very start.)
We released dep in Jan 2017, warts and all. The package management committee met with Russ once more in February, i believe, around which time i shared this gist with him: gist.github.com/sdboyer/def9b1…
That gist is significant. It was intended as a first step towards a language proposal. i wrote it up this way because we knew there was no way we could write a proposal for something of such massive scale without guidance.
So, to the point that ""there was never a dep proposal""?
True. But not for lack of trying on our part - our efforts to collaborate on such a mammoth undertaking were ignored.
True. But not for lack of trying on our part - our efforts to collaborate on such a mammoth undertaking were ignored.
In retrospect, it might seem obvious that we should've formally written one, but try to think about the state of things 18 months ago.
We had no idea what was on or off the table. Or what negative community consequences (wasted work/goodwill) a misfire proposal might have.
We had no idea what was on or off the table. Or what negative community consequences (wasted work/goodwill) a misfire proposal might have.
Shortly after the gist, there was this golang-nuts thread that folks have often pointed to, sometimes as evidence that we were overstepping our bounds, other times as a reason we should’ve expected this outcome all along.
Honestly, the best i can say there is that i was just sloppy with words. We hadn't found "official experiment" yet; it was late, and i was stupidly thinking of "merge" in a more general sense, not realizing how people would interpret it.
i'm sorry.
i'm sorry.
i was certainly hoping it would end up being a fairly direct path, but i was not, even then, expecting dep to just "become" the solution in the toolchain.
The committee heard from Russ again around the beginning of June, with his first writeup that he’s now made public.
It took me a few weeks to assemble a response. docs.google.com/document/d/1NA…
That took up all my dep contribution time during those weeks.
It took me a few weeks to assemble a response. docs.google.com/document/d/1NA…
That took up all my dep contribution time during those weeks.
As part of this discussion, i asked, again, about grounds for falsfiying Russ’ premise that the primary goal was avoiding the NP-hard search. Again, no response.
In fact, i had no idea until half a year later whether Russ had even read the response i prepared, as he never commented or emailed. Discussion just stopped.
"This was disheartening, but the idea that *all* of our work would just be discarded still seemed beyond the pale.
Even when a Go team member told me in early July that word was Russ was planning to just ~""rip out all that crap"" (rough quote), it still seemed absurd."
Even when a Go team member told me in early July that word was Russ was planning to just ~""rip out all that crap"" (rough quote), it still seemed absurd."
So, i gave my keynote at Gophercon 2017, and we encouraged people to migrate to dep as the "official experiment," and a stepping stone into the eventual toolchain integration.
But it was radio silence from the Go team again after Gophercon. So, in quiet limbo, we plugged away at our self-defined roadmap, waiting for guidance on next steps.
It's worth noting that, by the late fall, i was really hoping to be able to hand off this work to the Go team. i wanted to share, and ultimately relinquish, control, as this snip of an email to Sameer indicates.
Near the end of November, Russ emailed the committee with the draft of his essay on Semantic Import Versioning. We set a meeting to discuss it.
During that meeting, some of us were quite resistant to the idea. At minimum, four-on-one wasn't a good dynamic.
During that meeting, some of us were quite resistant to the idea. At minimum, four-on-one wasn't a good dynamic.
First, it is entirely false that Peter and i argued dep should just become `go dep`.
It's also false that we refused to integrate SIV into dep.
It's also false that we refused to integrate SIV into dep.
My take was that he was miffed that we didn’t immediately appreciate his idea.
To the extent i was defensive, it was primarily from having to field yet another greenfield idea, after months of silence.
To the extent i was defensive, it was primarily from having to field yet another greenfield idea, after months of silence.
He also suggested i was a “disciple of Yehuda Katz” - i.e., that i'd no capacity for original thinking.
(Yehuda is the original author of Bundler (Ruby) and Cargo (Rust), and a friend)
He apologized, though i now see this for the red flag that it was.
(Yehuda is the original author of Bundler (Ruby) and Cargo (Rust), and a friend)
He apologized, though i now see this for the red flag that it was.
So, i firmly reject the implication Russ made in his tweets that somehow, our reaction is what set him on a divergent path. AFAICT, he'd been on that path for a long time already.
Besides, he's BDFL. Why would our rejection even matter?
Besides, he's BDFL. Why would our rejection even matter?
In any case, i suggested that Russ and i start meeting one-on-one to discuss things. He agreed. We began our twice-weekly scheduled meetings, which ran from mid-December through early July.
At first, the meetings were exhilarating. Suddenly, possibilities were at hand - e.g., changes to `go get` - that i had been dancing around for *years*, as i i'd tried to read the tea leaves about the Go team's intent.
Plus, a ton of face time with Russ Cox. Amazing!
But it quickly became clear that we weren’t discussing incorporating dep, in any form. Whatever my aims might have been, my fears were confirmed - he'd already set a different course.
i pushed back. After two particularly fractious meetings, i inquired yet again about how his premise that ""avoiding SAT is the goal"" might be falsified.
Finally, i got an answer.
Finally, i got an answer.
Note that highlighted bit: my relentless problem-finding i've been doing, without defining alternatives? That's me trying to follow the rules and "find the line" - the precondition Russ defined for not throwing away our work.
It's also important because it's easy to get the impression from, say, these tweets that Russ gave specific, actionable reasons for sidestepping dep right up front.
He did not. Those issues (which all have answers) were pulled together for his Singapore keynote. Some he never even mentioned to me directly; identifying them was not about fixing them.
So, no: our work was disqualified before discussion even began.
So, no: our work was disqualified before discussion even began.
But these were interesting questions! And, i was already expecting - even hoping - for a rigorous teardown of ideas.
So, i trusted him, and trusted the process - however academic it may have been.
So, i trusted him, and trusted the process - however academic it may have been.
February was a big month. We found out my wife was pregnant (!!🎉!🎉!!), ushering in fifteen weeks of morning (reality: all day, every day) sickness hell.
The new baby is why i'm not at Gophercon - too close to due date!
The new baby is why i'm not at Gophercon - too close to due date!
Russ also finished the remainder of the articles that became the vgo blog posts (research.swtch.com/vgo). i provided ongoing feedback.
In reading these early drafts, i started seeing through the polite veneer.
Some of that was disdainful comments, like the closing line of research.swtch.com/vgo-mvs
Some of that was disdainful comments, like the closing line of research.swtch.com/vgo-mvs
It's bizarrely hostile to announce a novel system, and in the same breath suggest others' approaches are *intentionally* wasteful by comparison.
More than one baffled package management friend from other languages asked me, ""Why is he even doing this?"
More than one baffled package management friend from other languages asked me, ""Why is he even doing this?"
Other parts were just odd or misleading.
For example, the Kubernetes/YAML issue that featured so prominently was a problem with glide, not dep. (It's easiest to link to in the proposal: github.com/golang/proposa…)
For example, the Kubernetes/YAML issue that featured so prominently was a problem with glide, not dep. (It's easiest to link to in the proposal: github.com/golang/proposa…)
Or, flattening the vendor tree. This was a major design goal for dep, and we definitively solved it. Not novel.
Do such details matter? Only if you're trying to have a respectful engineering discussion about tradeoffs.
Do such details matter? Only if you're trying to have a respectful engineering discussion about tradeoffs.
But true misinformation came in the form of an omission, when discussing dep's "latest transitives" behavior.
This would also later be the second pillar of his Singapore talk:
This would also later be the second pillar of his Singapore talk:
Back in January, i had pushed to include "preferred versions" in our discussion scope. PVs are a longstanding-but-unfinished system in dep that gives it the same "current version" behavior vgo. (Better, actually, because of phantom rules: sdboyer.io/vgo/failure-mo…)
At the very least, finalizing PV seemed like a prudent fallback, in the event of unanticipated problems with vgo.
Russ was flatly uninterested. IIRC, he shut down that entire conversation with "this (vgo) is the path Google is interested in pursuing."
Russ was flatly uninterested. IIRC, he shut down that entire conversation with "this (vgo) is the path Google is interested in pursuing."
i realize now that changing that behavior in dep would've significantly weakened these arguments, none of which even acknowledge that PV could work.
(Yes, i probably should've just finished PV. i still may.)
(Yes, i probably should've just finished PV. i still may.)
After the vgo blog posts went public in late February, i retreated from the world.
This was partly an emotional defense mechanism. When the rug is pulled from under me, i tend to hide from public view until i've found somewhere stable and safe.
This was partly an emotional defense mechanism. When the rug is pulled from under me, i tend to hide from public view until i've found somewhere stable and safe.
Once i'd hidden away, i still tried to follow the rules, and find "the line" - the clear issue that justifies use of a more complex algorithm.
And i did. i refer to the single most critical problem with the design as "information loss."
And i did. i refer to the single most critical problem with the design as "information loss."
The screencast covers it, but really, it's just how conflating "min version" and "current version" into the single "require" statement creates perverse behavioral incentives, and a new flavor of dependency hell.
It can be mitigated if "everyone is careful," but that's never been, and likely never will be, how people interact with their dependencies.
It's something you'll have to think about all the time, as we were discussing over here:
It's something you'll have to think about all the time, as we were discussing over here:
But Russ effectively dismissed it as a non-problem, and continues to, in this sole time he's publicly mentioned it:
(i've explained labor extraction many times - and again, in the screencast.)
(i've explained labor extraction many times - and again, in the screencast.)
Now, one could argue it's an acceptable tradeoff for a simpler algorithm.
But a non-problem? No. 3 days after modules were merged, someone hit it and emailed the list: bit.ly/info-loss-stri…
But a non-problem? No. 3 days after modules were merged, someone hit it and emailed the list: bit.ly/info-loss-stri…
i would've found it far easier to think of it as just a tradeoff, if not for the way Russ' private dismissal mirrored the public arguments being made about vgo.
See, at a basic level, the implication behind the early arguments for vgo were that they "fixed dependency hell."
Which is just silly - subtle behavioral incompatibilities will always exist. It's Hyrum's law, which everyone involved agrees on. hyrumslaw.com
Which is just silly - subtle behavioral incompatibilities will always exist. It's Hyrum's law, which everyone involved agrees on. hyrumslaw.com
Nevertheless, his arguments implied that the Go ecosystem would somehow converge towards such an outcome, *because* of these changes in vgo. Pillar 1 of the Singapore talk , or the initial vgo blog post are illustrative research.swtch.com/vgo-intro
i could just see these new koans - e.g., "software moves forward" and "an import name has one meaning," - becoming Go Proverbs: received wisdom, intoned to justify and erase the user pain arising from information loss.
In this way, the dismissals i experienced in private seemed like a precursor to dismissing those same concerns when they arose in the community.
(This behind-the-scenes view has also led me to wonder what user pain other Go proverbs might be concealing.)
(This behind-the-scenes view has also led me to wonder what user pain other Go proverbs might be concealing.)
There is good reason for such concern. The Go team has a history of deafness to community pain - after all, it took years for them to even really admit there was a problem with `go get`.
So i continued to advocate privately.
By early May, said advocacy had led to three meetings devolving into the same argument.
After the third, i tried to extend an olive branch. (Remember, this is before the proposal was formally accepted)
By early May, said advocacy had led to three meetings devolving into the same argument.
After the third, i tried to extend an olive branch. (Remember, this is before the proposal was formally accepted)
Googlers' chat histories are disabled, and this is from after his messages disappeared.
But yes, my two messages are quite disjoint. That's because his response to my olive branch was, roughly, "MVS and SIV are happening, and you need to accept it."
But yes, my two messages are quite disjoint. That's because his response to my olive branch was, roughly, "MVS and SIV are happening, and you need to accept it."
This has been my experience of Russ: reliably polite, but forcefully in control. And not just of the code, but of the narrative surrounding it, too.
Perhaps that narrative control is why my concerns were mischaracterized to the committee as "not handling all situations." github.com/golang/go/issu…
Or why fixing shortcomings (like incompat declarations) were only mentioned after the proposal was accepted. github.com/golang/go/issu…
Or why fixing shortcomings (like incompat declarations) were only mentioned after the proposal was accepted. github.com/golang/go/issu…
Or why he's strategically avoided mention of information loss or preferred versions.
Or why his story, and apology, came long after the committee's decision to accept his proposal.
Or why his story, and apology, came long after the committee's decision to accept his proposal.
Or why he continues to promote the misleading notion that it had to be some all-or-nothing decision with incorporating dep.
(i struggle to accept the sincerity of that apology, by the way, as the package management committee got a similar one back in February, before things got really bad.
Apologies are only meaningful when they come with changes in behavior.)
Apologies are only meaningful when they come with changes in behavior.)
It's difficult for me to see this process as anything but a sham. i've felt railroaded since December. That same experience of railroading made a lot of people just check out quickly. So, "broadly"? The sampling is flawed.
Now, one *could* make an "ends justify the means" argument - that whatever pain the process might cause, an objectively better result is worth it.
And, to be clear, i think modules are mostly better than dep *as it stands today*. (But, as i said early on, that is not, and never has been, the right comparison)
But that brings us around to this tweet.
But that brings us around to this tweet.
If incompatibility is inevitably such a minefield, then it means the arguments implying vgo would *cause* a convergence towards a compatible ecosystem aren't true, and never were.
These were the arguments i found so problematic. And the "convergence" idea was a main pillar taken up by others reddit.com/r/golang/comme…
If these arguments weren't true all along? That's engaging in bad faith.
If these arguments weren't true all along? That's engaging in bad faith.
So all it leaves is this - Russ doesn't trust SAT enough to use it for automation.
Which i get. But now, we're not talking about high-minded design, or hard choices in pursuit of the best outcome, or the most scalable solution for an ecosystem.
That's just workflow preferences.
Which i get. But now, we're not talking about high-minded design, or hard choices in pursuit of the best outcome, or the most scalable solution for an ecosystem.
That's just workflow preferences.
And workflow preferences generally don't have "right" answers. Picking one over another isn't a good basis for capsizing a community project.
i've learned a lot from this process. Scars are good teachers. (Though i wouldn't pretend for a moment that i've had the worst 2018 - give @bketelsen a hug if you see him this week!)
i love communities, and helping them thrive. Those are my values. i've tried hard to keep my trusting, communitarian attitude, in the face of all this.
Most important, though, is what all this says about the future of Go.
No matter your take on the outcome, this process was extraordinarily toxic. We can't chew up and spit out community members on every major change.
No matter your take on the outcome, this process was extraordinarily toxic. We can't chew up and spit out community members on every major change.
Now, It's important that we embrace creative destruction when necessary. Keeping code around because we want to spare the original authors' feels is a surefire path to collapsing under our own weight.
But we also can't be cavalier about cutting. Open source contributions are made in good faith, in a spirit of collaboration and openness. Failing to honor that spirit is the best, fastest path to a dried up community.
i'm glad that Russ seems to be looking to the Rust community for inspiration here. i think they've made a lot of good decisions.
However, it doesn't seem like the community should look to Google/the Go team for leadership on good collaboration.
However, it doesn't seem like the community should look to Google/the Go team for leadership on good collaboration.
Go has a real problem with governance. Nothing the Go team "gives" to the community can fundamentally change that.
If we, as a community, want to see better process and outcomes, the better governance model will need to come from us.
(fin)
If we, as a community, want to see better process and outcomes, the better governance model will need to come from us.
(fin)
