Thread by @deontologistics

Honestly, I wish people would just realise that algorithmic bias and bureaucratic stupidity are *exactly the same thing* so we could start unpicking the rationalisations implicit in both, as they're synergetic: you have to get them both to tackle either successfully.

Putting aside whether this is even a good use of the term 'algorithm', which you can usually substitute for 'wizard' without any loss of meaning, the issue is that we keep pretending we can *trivially* solve certain sorts of problems with certain sorts of tools, when we can't.

It doesn't matter whether the implicitly specified knowledge representation generated through training is encoded in some distributed set of educated human neurons or some artificial kludge of ML systems, it's implicitness is a logical feature of the problem it is targeting.

There are exceptions to this. If you want to calculate the prime factors of a number, or run a cryptographic algorithm based on it or similar mathematical problems, then there can simply be *best* ways to do it, and they can be automated. If we ran RSA on HR we'd be in trouble.

At the very least, we'd need some impressive error checking procedures. It's not *running* such programs that is computationally hard, it's *discovering* them. We can say some very precise things about what proof search is and is not. But simulation search? That's trickier.

The only tacit models of searching for mathematical rules that let us adequately simulate some system in our environment in ways that might allow us to interact with it usefully, within tolerances determined by that usefulness, is mathematised empirical science itself.

But there's an all too tempting impulse to take a shortcut and assume that, not only can empirical discovery be fudged till it looks like mathematical discovery, but this can be fudged till it looks like mathematical deduction. Throw enough probabilities in there and it'll work.

This is a deep and abiding form of logical false-consciousness I find everywhere, and it's generally a refined form of the engineer's intrinsic optimism: "give me a couple days and some materials and I'll hack something together that'll do the job well enough!"

The problem being that this kind of optimistic satisficing all too easily gets hyperbolically projected into the assumption that there are satisfying optima for every such problem. To quote Voltaire: 'the perfect is the enemy of the good.' Perfectionist fallacies get everywhere.

In the process, we forget that the problems in relation to which these hallucinatory optima make any sense are something *captured* by mathematics, rather than *defined* by it. The assumption that such apparent optimal solutions are stable under perturbation is *optimism*.

I intend every last connotation loaded into that word by Candide and fling it in the face of the information technicians of contemporary society, be they designing human behaviour, computational infrastructure, or, increasingly, both at the same time.

"Just one more form" they say, balancing a stack on the back of their charges, blissfully unaware of the strain or its potentially catastrophic consequences.

"Just one more variable, one more tranch of data" they insist, sure that this time, the bugs will magically disappear.

The dynamics of information imply a politics. The first rule of this politics is that nothing is free. One cannot simply acquire more data, let alone process it, in ways that guarantee efficiency will increase linearly. There are pervasive and pernicious non-linearities here.

I beg you. Look at the examinations system. Just look at it, and ask yourself these questions: "What is it supposed to do? What is supposed to measure, and how is this supposed to modulate the process it is measuring in a way that improves this thing?" Education is fucked.

This is the closest I come to getting on the Adorno-Heidegger train regarding the evils of instrumental reason. The slightest nudge and the metrics we've balanced on the things we care about slide off, onto nearby goals it's easier to optimise for. Mandatory stupidity awaits.

The only thing that keeps cognitive processes (individual/collective, natural/artificial, multi-modal hybrid) locked on to the objects they intend are independent mutually correcting/reinforcing representations: i.e., more opportunities for error and its correction.

This is a sort of mutual responsibility: keeping each other in check. This 'personal' model is how we should think about the integration of administrative and computational infrastructure. More often we get an 'impersonal' one in which mutual blame diffuses such responsibility.

I give you the Trinity of automatic incompetence:

The Unholy Spirit: "Not my problem mate, I just collect the data."

The Idiot Son: "Not my problem mate, I just follow the rules."

The Absent Father: "Not my problem mate, I just use the data they give me to make the rules."

Perverse incentives will creep in wherever the checks aren't balanced. The more delicate the balance you've created, the easier it will slip when the problem is perturbed by any exceptional circumstance: the easier these exceptions become the rules, rather than changing them.

Optimism encourages us to ratchet this balance as far as it will go, stretching metrics to *do* more than they were intended to, to *include* more than they were designed for, to permit levels of *precision* and *comparability* that they cannot sustain while stably referring.

I've talked about technical debt a little recently. This is what we might call 'technical leveraging'. The irony is that those who do it think they're creating solutions to ranges of small problems rather than creating newer bigger ones. Worse, their efforts are synergetic.

There's an analogy with anomalies in scientific research programs to be had here. The little changes and extensions are not unlike ad hoc explanations of exceptions to a model's predictions. The accumulated weight of minor explanatory sins that drags a paradigm down.

The difference is that in organisations with practical rather than strictly theoretical goals, these are often unforced errors: accumulated good intentions that pave its road to hell. The ad hoc fixes are how its members work around these decisions to pursue the original goal.

This is the epistemic resistance to rationalisation that James C. Scott is so concerned to advertise and analyse. People keep on doing their jobs even as their organisation's implicit representations of what these jobs are drift further away from its stated intentions.

Technical leverage and representational drift. These are our eternal enemies. They will not go away, no matter how much toxic optimism we flood our organisations with, be it managerial or computational, which is to say, no matter how cybernetic. True cybernetics is fallibilist.

Which is not to say 'conservative'. There's a wisdom to conservatism that radicals ignore at their peril, but it's a wisdom that can and should be turned against the object it fears: the mistakes that accompany every (necessary) change. The liberal mean is blissful ignorance.

The liberal mean is cruel optimism. The blank, indifferent face of the bureaucrat. The sleak, inflexible interface of the kiosk. The rank, degenerate incompetence of the managerial class refusing to hold each other responsible for any fucking thing, no matter how important.

The conservative bemoans (inevitable) change, or worse, capitalises on it for fun and profit. The radical confronts its inevitability, and the inevitability of the errors that come with it. They are motivated by one simple thing: that whatever they know, *this* doesn't work.

I like to think that I'm a progressive, but here's the tawdry truth of progress shorn of radicalism: every claim that things can be improved without anyone shouldering the risk of improving them is an *evil* fucking lie; a pleasant hope that starves children and breaks the frail.

Commitment requires constraint. Enthusiasm tempered by a friendly word from a cheerful cynic. The more the merrier. Not all risks are great. There are undoubtedly many low risk improvements to the present order. It's finding and convincing people to try them that's hard.

There are too many fucking forms to fill in, for one thing.

You've got to laugh, lest you'll cry. There's strange joy to be found in absurdity and I only hope I for the wisdom to keep finding it there, amongst other exceptional things. Solidarity in failure, you hopeless radicals.🖖

CODA: As ever, there's a few other bits of writing directly relevant to this.

1. 'Immanentizing the Eschaton' - on theological assumptions about perfectibility common to various strands of thought: deontologistics.co/2019/10/22/tfe…

2. 'Incompetence, Malice, and Evil' - a more considered discussion of the relationship between incompetence and evil and its contemporary manifestations: deontologistics.co/2019/10/22/tfe…

3. 'Universal Leverage' - a thread on the decline of universities that discusses leveraging and the plague of metrics:

https://twitter.com/deontologistics/status/1350739061452001280?s=20