Thread by @_rsc

@mattfarina @Crell @sdboyer This is an absolutely fair criticism - we have not handled the community process around dependency management well. The core Go team was not involved early or often enough for that process to lead to a smooth landing. As tech lead for Go, that was my fault, and I apologize.

@mattfarina @Crell @sdboyer Ever since I wrote “go get,” we on the core Go team hoped the community would take care of dependency management. Keith Rarick's goven and then godep, and Matt Butcher's glide, and Dave Cheney's gb all seemed like promising steps toward a solution.

@mattfarina @Crell @sdboyer In spring 2015, Jason Buberel spent a while surveying these efforts and trying to understand how to unify them. The evidence he gathered clearly showed that a major impediment for all these tools was the lack of support in the go command for what we now call vendor directories.

@mattfarina @Crell @sdboyer The evidence Jason gathered convinced me and Rob Pike that the go command needed to add vendor directory support. We did just that, following a proposal originally by Keith Rarick. Go 1.5 added experimental support and Go 1.6 enabled it by default.
golang.org/s/go15vendor

@mattfarina @Crell @sdboyer Daniel Theophanes led writing a spec for a standard vendor.json config, so dependency tools could interoperate. I used vendoring as a community success story in blog.golang.org/open-source. But we were missing a shared language to express dependency requirements.

@mattfarina @Crell @sdboyer In summer 2016, there was a community package management discussion at Gophercon that led to a new package management working group and advisory committee. I was on paternity leave that summer, and Andrew Gerrand represented the Go team.
groups.google.com/forum/#!msg/go…

@mattfarina @Crell @sdboyer In late November, as Andrew prepared to move on to other things, I started to get involved. The group was (rightly) focused on building a standalone tool usable immediately. I wanted to make sure the overall design would work for the eventual go command integration too.

@mattfarina @Crell @sdboyer In December 2016, I began to mail Sam to discuss the design. I wanted version resolution to become part of “go build,” to avoid a second step in everyone's workflow, and I asked whether that would be too expensive. That led to my Version SAT post: research.swtch.com/version-sat

@mattfarina @Crell @sdboyer We also discussed allowing multiple versions of a package in one build, to handle programs with conflicting dependencies, and Sam spoke positively about Cargo's experience with that, although he was still in favor of using SAT.

@mattfarina @Crell @sdboyer We also discussed elimination that go command integration would require additional compiler and build support, including possibly changes to the semantics of vendor and GOPATH.

@mattfarina @Crell @sdboyer I said that although the final go command integration would necessarily be different, it made sense to keep working on dep to make something available now and also to learn from it. Sam seemed to agree.

@mattfarina @Crell @sdboyer In January 2017, in advance of the announcement of dep, its name was briefly changed to hoard. In that discussion I mentioned again that dep would not simply become “go dep” (or “go hoard”).
github.com/golang/dep/iss…

@mattfarina @Crell @sdboyer In March 2017, Sam mailed golang-nuts with a link to a “roadmap for merging into toolchain.” That roadmap seemed to say that dep would just move directly into the main distribution.
gist.github.com/rsc/c4472e4e29…

@mattfarina @Crell @sdboyer I replied to that thread, reiterating my concerns about what needed adaptation to merge into the go command as well as concerns that Sam was holding out dep as more of an official solution than it was.
groups.google.com/d/msg/golang-n…

@mattfarina @Crell @sdboyer Sam's reply later in the thread introduced the term “official experiment,” which I still think is a good description. An experiment is an action taken to test a hypothesis, advance your knowledge about something.
groups.google.com/d/msg/golang-n…

@mattfarina @Crell @sdboyer In June 2017 I shared a doc with the package management group sketching thoughts about go command integration. There are some major flaws that Sam and others helped me see. I've now published it here: research.swtch.com/go-pkg-june-20…

@mattfarina @Crell @sdboyer I met with the package management group a few days later to discuss the design. I took notes and wrote them up for Andrew, who missed the meeting. I've now published them here: research.swtch.com/go-pkg-june-20…

@mattfarina @Crell @sdboyer One topic at the meeting was the status and future of dep. Again I emphasized that the whole point of this process was to understand the problem space better and to work toward smooth integration in the go command.

@mattfarina @Crell @sdboyer We discussed dep at the GopherCon contributor summit. Matt Farina says that I said I could do better if I went off on my own and built something. I'm sure I didn't; likely I said that dep's design needed to change to be integrated into the go command.
codeengineered.com/blog/2018/gola…

@mattfarina @Crell @sdboyer Matt's writeup also makes clear that at least some in the community did not have the same expectations about what would happen with this “official experiment.” I didn't understand that. I'll return to that in a few minutes.

@mattfarina @Crell @sdboyer I realized that go command integration would require significant reworking of the go command internals, so I spent the middle of 2017 working on what became Go 1.10 build and test caching. The build cache obsoletes GOPATH/pkg, which makes no sense in a versioned world.

@mattfarina @Crell @sdboyer After GopherCon I took a vacation, then had to help another project, and finally returned to completing build caching for Go 1.9. Once build caching worked, I was able to demonstrate building a program that had multiple versions of the same package. Now it was November 2017.

@mattfarina @Crell @sdboyer In late November I sent the package management group a draft of what became semantic import versioning and I met with them to get their thoughts. I had not written any code - I was still trying to work out what go command integration meant.
research.swtch.com/impver-draft-2…

@mattfarina @Crell @sdboyer My draft ended by suggesting that dep add support for semantic import versioning, and I met with the pkg mgmt group to discuss that. Sam and Peter argued that it was too late to change anything in dep and that it should just become “go dep,” a fait accompli. No dep changes.

@mattfarina @Crell @sdboyer I wanted to post the import versioning draft and have a public discussion, but Sam asked me to hold off, to avoid spoiling his holidays. I agreed. Starting in December, Sam and I met once or twice a week to talk through the design of go command integration.

@mattfarina @Crell @sdboyer Since Dep was not going to add support for import versioning, I hacked up an implementation in the go command to test the ideas. I also found the nice fit between import versioning and minimal version selection and implemented that too.

@mattfarina @Crell @sdboyer At the start of January 2018, I had a design I wanted to talk about publicly and no implementation to test whether it was any good. But Sam was talking about Dep at FOSDEM on Feb 3, which I didn't want to complicate. So I kept talking to Sam but didn't publish anything.

@mattfarina @Crell @sdboyer By mid-February I had a working demo of all the pieces and was confident there weren't major problems with the design. Sam's FOSDEM talk was over, so I finally published the blog posts and the prototype implementation. Boom.

@mattfarina @Crell @sdboyer Everyone knows most of the rest: the design was revised in public for a few months, became an official Go proposal, was accepted,implemented, merged into the main tree, and is slated forrelease as an opt-in preview in Go 1.11.

@mattfarina @Crell @sdboyer The original “go get” will be about ten years old when we replace it. The goal is to build something that's going to serve Go users well for the next ten years, especially as code bases and team sizes continue to grow and grow.

@mattfarina @Crell @sdboyer Dep was a successful experiment. It tested the hypothesis that Go should adopt the “tag versions, record Boolean constraints, solve with SAT” approach used in Bundler and Cargo. I certainly learned a lot from Dep:

@mattfarina @Crell @sdboyer - I learned that establishing community behaviors around tagging versions is critical.
- I learned that people are comfortable with and even expect semantic versioning.

@mattfarina @Crell @sdboyer - I learned (confirmed by gathering stats from Cargo) that the dominant Boolean constraint is “version X or later.”
- I learned that using SAT to find constraints can take a very long time and still does not imply a single best answer.

@mattfarina @Crell @sdboyer - I learned that constraints recorded in the tagged repository are necessarily incomplete, because you can't write down an incompatibility with a new version of a dependency that hasn't been released yet.

@mattfarina @Crell @sdboyer - I learned that users have a hard time understanding the behavior of a dependency manager that has too many features and too many special cases.

@mattfarina @Crell @sdboyer Although it was a successful experiment, Dep is not the right approach for the next decade of Go development. It has many very serious problems. A few:

@mattfarina @Crell @sdboyer - Dep does not support using multiple major versions of a program in a single build. This alone is a complete showstopper. Go is meant for large-scale work, and in a large-scale program different parts will inevitably need different versions of some isolated dependency.

@mattfarina @Crell @sdboyer - Dep's very flexible config conflicts with the convention-based “go build” approach. Another complete showstopper. Even Sam admits Dep is too flexible. Rich configs are good for experimentation but bad for day-to-day use. No one understands how to use Dep well.

@mattfarina @Crell @sdboyer - Dep's use of a SAT solver adds more problems. My tweetstorm yesterday hit the highlights there.

https://twitter.com/_rsc/status/1022164929137336321

@mattfarina @Crell @sdboyer Why does the Go team have a say in how the go command supports versions? For one thing, we created Go, and I designed the go command. One of the most important things we can offer is consistency of vision, to help keep Go recognizably Go.

@mattfarina @Crell @sdboyer Part of the vision for Go is to make large-scale software development better, including internet-scale open-source work. It's early still - there's no playbook for what approach works best here - but we can identify ways other systems don't scale and try to avoid them.

@mattfarina @Crell @sdboyer We have a defined process for changes to go, large and small. Basically, write a proposal. The core Go team helps point out the important concerns to address to stay true to Go's design and vision, and we guide a community conversation toward consensus.
golang.org/s/proposal

@mattfarina @Crell @sdboyer Community consensus is not always possible. If we don't get there, then the core Go team decides. Technically I am the final decider but what actually happens is that a bunch of long-time Go team members talk through the decision to get to a consensus among ourselves.

@mattfarina @Crell @sdboyer There was never a Dep proposal. For the Go modules proposal, the community was broadly in favor, though certainly not Sam and a few others. I left the decision to the other usual proposal reviewers, and they considered the issues and the objections and formally accepted it.

@mattfarina @Crell @sdboyer The Go module design is not perfect. But it addresses the problems listed above, and others, and looks to be a good foundation for the next decade. We will gladly work with anyone who wants to help us improve it.

@mattfarina @Crell @sdboyer Looking back, as I said at the start, we the core Go team mishandled the community process, and I apologize. We should have had more consistent involvement, to help make small course corrections to bring the effort to something that would fit into the rest of the go toolchain.

@mattfarina @Crell @sdboyer FWIW, I really admire the Rust team's community engagement. To be successful, the people who understand the core concerns and own the relevant pieces have to be fully engaged with the community contributors. For too long in this process I wasn't.

@mattfarina @Crell @sdboyer We let Dep go its own way and end up somewhere unacceptable, making Go modules seem a very large course correction. Worse, the course correction surprised a lot of people, because we'd only shared concerns with the package management group.

@mattfarina @Crell @sdboyer I thought I could focus on the technical details and let the pkg mgmt group run community interactions. Somehow that led to the entire community believing that Dep was the official endgame, even though my discussions with the pkg mgmt group were clear it was not on that track.

@mattfarina @Crell @sdboyer In retrospect I made lots of mistakes, but the biggest one was not communicating concerns with Dep and plans for go command integration more widely and publicly. I wanted to let the package management group speak with one voice.

@mattfarina @Crell @sdboyer It seemed to me most productive to talk directly to Sam and the others about what was needed to bring package management into the go command proper. But those concerns were basically ignored, with the result that Dep is and remains unfit as a design for go command integration.

@mattfarina @Crell @sdboyer And one way or another, Dep was presented and became known as the final answer for Go package management, even though we'd been clear with Sam as early as December 2016 that it was only a step along the way and should be expected to be replaced.

@mattfarina @Crell @sdboyer Making the discussions public instead probably would have been better for adjusting everyone's expectations, and maybe it would have resulted in Dep's design focusing on the showstoppers for go command integration and being easier to incorporate.

@mattfarina @Crell @sdboyer Again, I apologize for all the mistakes I made during this process. I've learned a lot about what a successful community collaboration does and does not look like, and I hope it helps the next one run more smoothly.