Thread by @eigenrobot

in monetary economics, greshams law states that good money drives out bad

ie: if there exist different currencies with the same _legally-mandated_ value but different market values--say, quarters made of silver vs zinc--people will pay using the shitty ones and hoard the others

this is almost trivially true, but an easy check of the law can be conducted by looking at the number of quarters in circulation from the years after 1965 and comparing them to the number of quarters from 1964 and earlier, when quarters still contained silver

oh yeah sorry baby is running amok

greshams law is "bad money drives out good"

anyway i have a related hypothesis about science that i dont quite trust but perhaps seems under-considered

https://twitter.com/acidshill/status/1491245513806872579?t=2bg7-NqY6dWxPiqIzaWq-g&s=19

in science somehow defined success is determined by publications and i think theres most likely an element of bad research driving out good. heres its not a matter of hoarding good papers but rather their not being produced at all

the specific mechanism of the debasing im proposing is methodology

the development of new tools is the most important means of driving progress in science but unfortunately i suspect it may also be a common cause of the destruction of a field

when new methods and tools have worked well in the past they have opened entirely new vistas. consider the effect of van Leeuwenhoek's early microscope on biology or Galileo's telescope on astronomy. more recently methods for extracting ancient DNA are revolutionizing prehistory

new tools and methods are being introduced all the time, and unfortunately not all of them turn out to be as useful as the aforementioned. some of them in fact turn out to be complete trash. (more frequently they're just mediocre or badly used but this is sufficiently Bad, here)

the main feature of a shitty method is that it can be used to apparently reject a hypothesis that it does not actually reject

suppose that a shitty method is invented and used to publish a paper. a few things will obtain

first, the paper has a decent chance of being wrong in rejecting a hypothesis

second, other researchers will start to talk about the method and consider how they can use it themselves

if you get very lucky, someone will think to use the method on an old and unimpeachable result and find it doesn't work, and quickly home in on the problem and nip it in the bud

this is less likely than youd think because lol who replicates anything you cant publish that shit

more likely, it will take years before someone comes around and identifies the real issues with the method. in the mean time lots of papers will be published with it, and careers made and tenures granted, and you have an entire generation of ppl who will defend it to their graves

in a really unfortunate case, the adoptees of the shitty method will use it to "overturn" good results, to establish entire subfields, and also to weed-like choke out competing fields of resources and practitioners. you might never be rid of this garbage

speculating, what might be some especially dire features of a shitty method?

1. generalizable. it can be used in a large array of research cases in a field

2. easy to "use" and cheap. if it requires a ton of money or talent its use may be constrained

3. difficult to invalidate

in the worst case scenario, you might encounter some extremely broadly generalizable and incredibly cheap method for rejecting hypotheses that works beautifully sometimes--just enough so that ppl come to trust it--but usually fails silently in practice

before you were able to figure out what had happened, you might see subfields overrun by people using this method and failing silently, although its hard to imagine that an entire discipline on the scale of say nutrition or psychology would be be vulnerable

what would this look like? papers would be written, students taught, results recorded and iterated on. all manner of theories would be established and over time become conventional wisdom. but none of them would be useful in building or creating anything except for more papers

given the insularity of the disciplines and subdisciplines, the academic prerogatives of researchers to police their own borders and speak with authority, and multiple generations of students bought into the system, it might become irredeemable. youd have to start anew to fix it

we should all be grateful that no such methodological paradigm has arisen to stunt research across many disciplines. science might be set back for generations as stupid ideas become gospel because of shitty research methods and this is a thing too terrible to contemplate

fin