Thread by @GeePawHill

Upstream Uptime #1: Grasping the Problem. Increasingly, we build apps by composition with other apps. Granting that this approach is viable, still it comes with a cluster of related problems, and to really win with this style, we'll have to address them.

An "upstream" is an independent program that my application's correct behavior depends on at runtime. It is typically 1) across a transport mechanism, 2) in a separate process and/or machine.

We've of course had upstreams in our lives forever: virtually all apps that use a database system deal with that system as an upstream. It's an external program we speak to across a transport layer.

Looking at the our experience with database systems, we notice several things about them that are different from our modern upstream situation, and in that, there are some clues.

First and foremost: the database is remarkably *stable*. Notice I don't mean by that that it's necessarily bug-free. Rather, it's that it doesn't change its behavior over time in unknowable ways. (This can vary from vendor to vendor, but it's still a good generalization.)

And of course, that leads us to the second difference: we didn't write our database, someone else did, and that someone else doesn't work for or with us. The development of the database system is as entirely external as the development of the language compiler.

And we hit a third difference and a fourth right there: 1) we usually both know and choose the *version* of a database. 2) the database is typically "road-tested" by tens or hundreds of thousands of people every day.

And a fifth, and final for now, difference: the database is not under anything even remotely like the "rate of change" of our application. In the majority of application development, we are adding function every day or two. COTS database systems do this once every year or two.

To recap, then, most modern upstream-based development uses upstreams that are 1) unstable, 2) owned by us or people we know, 3) weakly versioned, 4) road-tested by very few, 5) rapidly changing in parallel to our application.

Now I want to make an observation: most of the orgs I've seen relying on upstream-centric architectures waste *staggering* amounts of time, energy, mind, and money because they develop "as if" the streams they make & use are substantially the same as database systems.

Be careful, here, and don't go telling everyone I hate upstream-centric architectures. I certainly don't. What I am saying is that our current *practice* with these modern upstreams is based on a metaphorical mapping -- to the proven database approach -- that isn't really there.

If we're going to win using upstream-centric approaches, we need to account for those differences, and either eliminate or find clever ways to work with them. If we don't, well, look for the airline magazines your CTO reads to begin decrying upstreams right damned quick.

I've just got two more points to make for the moment, and we'll push out some proposals in a later thread. For now, though I want to get you to reach to the meta- a little. This pattern isn't unique to this tech, but is common in our thin culture.

The first meta-point: We can see this entire problem as, once again, an imbalance in the triple focus that is our central concern: the Made, the Making, and the Makers. In particular, our failing style fails because it does not account adequately for the making.

(This is why I am refusing to shut up about this M+M+M thing, because I see not just imbalance everywhere I look, but because the majority of the imbalances I see are about not emphasizing that software is in a permanent state of making.)

The second meta-point: the difference between trad development and my agility is in its stance towards change, and change is exactly the problem here. The old school will say "control & restrict change". The new school, the absolute heart of my agility will say "embrace change".

So. What then, is the "embrace change" approach going to tell us?

I have some proposals. I will be making them soon in a thread near you.

Meanwhile, it's a surprisingly cool Wednesday afternoon here. I hope you have a surprisingly cool day, too. :)