Thread by @alexandrosM

What if we had a do-over on this whole "medical science" thing? 🧵

I'm increasingly worried that our approach to science when it comes to complex systems generally, and medicine specifically, is deranged.

A rant, followed by a concrete suggestion for a different way forward:

We try to isolate a slice of a system, nail down as many variables as we can, and produce generalizable conclusions. Here's the thing though:

Every variable we isolate our experiment from, exponentially compounds experiment costs. So the experiments that can isolate the greatest number of variables are only run by the largest companies for the most profitable things. Not a good way to allocate resources. It gets worse:

Besides the ever-increasing pressure to make the expensive experiments succeed (promoting either extreme conservatism or corruption) we also have the problem of never being able to isolate enough variables. So we focus on the commonly accepted "important" ones. As for the rest:

We pretend not to notice and whack everyone making noises with the hammer that has "Trust The Science" written on it. And if it turns out we missed something important, it goes under the rug. Maybe we add another variable to the "important" ones every decade, changing nothing.

But the problem isn't the number of variables we care about. The problem is that we ever believed we could slice a tiny part of a complex system from the rest, run a very expensive experiment on it, and draw conclusions that are generalizable in any sense. Reality intrudes.

It intrudes from every conceivable nook and cranny. Not only have we reduced the number of actors who can contribute to extending knowledge in important fields to a small handful with deep conflicts, we've also rendered their contributions incredibly shaky. Can you say deadlock?

I don't see how the current approach can be rescued. There doesn't even seem to be any movement towards making the acquisition of knowledge faster in any way. If anything we're piling on more ineffective regulations that only encourage corruption. We may just need a do-over:

In less regulated fields like startups, we've built tools that are far, far more nimble and continuous. You may have heard of A/B testing, which is of course highly flawed, but can still work on occasion. en.m.wikipedia.org/wiki/A/B_testi…

Understanding the potential flaws, especially the wasted revenue and relative slowness, people have moved on to ideas like multi-armed bandits. Critically, multi-armed bandits can reduce wasted revenue (which is why they're called "earn as you learn") en.m.wikipedia.org/wiki/Multi-arm…

The idea of applying bandits to clinical trials isn't new. But it's also not widely applied to my knowledge. We still make gut calls when to halt trials, which are far less efficient and endlessly disputed. Maybe in a decade or three. pubmed.ncbi.nlm.nih.gov/20018711/

And multi-armed bandits aren't even close to the cutting edge. Recognizing that one-size-fits-none, the software industry has created contextual bandits, among other interesting variants. Those can reach different conclusions for different contexts. link.medium.com/3BeIvOxKnjb

And as the link above makes clear, "you can think about reinforcement learning as an extension of contextual bandits." We're now touching the boundaries of machine learning. This 2019 article from Google's Cloud Bandits Solutions team is a good primer. cloud.google.com/blog/products/…

And make no mistake, this is the beginning, nowhere near the end. Modern AI/ML can get far more competent than this. But the distinctive characteristic is that learning is continuous and integrated with real use. Just like your brain does. Just like any other biological system.

Nowhere in nature do we see an equivalent to the billion-dollar RCT. Facebook engineer Jeff Hammerbacher is famously quoted for having said: "The best minds of my generation are thinking about how to make people click ads". Maybe they weren't totally wasting their time after all?

The ultimate endpoint for something like this would be that every data point, no matter how small, would continuously feed into a massive bayesian model that would be ever-adapting, ever-shifting to the winds of evidence. If we varied every medical treatment by a minute amount,

...stsrting from its current best guess given the context, and adding just a bit of randomness, such a system, I suspect, could discover a wealth of solutions, customized to each, yet far more imaginative than we can construct from within the straitjacket we've put on ourselves.

By making every treatment 99% best practice and 1% experimental, we can also end the endless ethical dilemmas around who participates in trials, etc. Everyone who partakes of medicine 2.0 also contributes in the same motion. Another counterintuitive outcome is that we'd want to..

...include as many people as possible in this system. The more people it helps, the faster it learns, the better it gets for all of us. It's a complete no-brainer. And even possible privacy issues have well-known if somewhat underdeveloped solutions.

The orders of magnitude better at saving and improving human lives something like this would be is very hard to overstate. To help visualize, just imagine if Google had to iterate its recommendations via RCT. What it does today would simply be impossible.

Yet we keep our best technology to ad clicks, financial markets, gaming, and we're decades away from applying it where it can do help the most. When it comes to learning from streaming, large-scale, medical data, what we get is... VAERS. A system the 80s would repudiate in shame.

To get a sense of what's possible at the forefront of our capabilities today, I recommend tesla's AI Day as well as its previous Autonomy Day.

If you're interested in an even deeper dive into what's possible with large-scale learning, you can do much worse than @karpathy's talk on Software 2.0

And if you want to get an intuitive feel for what it means to use every single shred of information one has at their disposal, this short story by @ESYudkowsky has been incredibly formative for me. lesswrong.com/posts/5wMcKNAw…

If you sense I'm frustrated by what passes for quality from today's medical establishment, now you know why. This isn't science fiction. Except for the short story at the end. The rest is here today, and for reasons that the future will judge us severely for, essentially ignored.

I guess I'll go sit in the corner and read something light while I wait for the trolls who haven't bothered to look me up in Google Scholar to descend. What should I read though? Ah, this will do: journals.plos.org/plosmedicine/a…