Economists are known for typically having a Whig view of their science, as @PeterBoettke nicely summarizes in this piece: econlib.org/library/Featur… . Other social sciences often follow the same reasoning, with the implicit view in e.g. management being that new contributions must
"build on" (not question or criticize) what has already been published. The rationale economists use for subscribing to the Whig view of theory is fundamentally mistaken, however. It is based on applying a view of markets as selection mechanism onto the development of theoretical
knowledge. The problem is not that the market principle may not always be applicable on the theory development process, but that what is applied is a hagiographic misconception of the market process. The market process does not select the "best" ideas; it selects the most
effective given specific expectations of market actors. When the market process "selected" VHS over Betacord, it was not because the VHS was "better" in its own right (which many argue it was clearly not), but that VHS was better fit to satisfy consumers' wants at that point in
time. As wants change over time, so does also what is selected by the market process. But it is an inefficient process that is based on the ability of entrepreneurs to anticipate what consumers will want. While errors are quickly weeded out, these are "errors" only relative other
attempted solutions. What remains is not the efficient (or optimal) solution, but the one that most fits consumers' real wants. In times where entrepreneurs collectively make errors, as is the case in an unsustainable boom, or where there is immense uncertainty, such as during
the "regime uncertainty" of the 1930s, entrepreneurship fails at finding the better solutions. And entrepreneurs profit from their undertakings to the relative extent they satisfy specific consumer wants, and how urgent those wants are. Similarly for theory development: insights
are selected for specific purposes and not for their "objective" quality (long-term contribution). In academic publishing, this means new ideas and insights get published and are cited because they help other researchers, subject to the "publish or perish" requirement, produce
more (publishable) research. In other words, important insights that undermine previous research and/or do not facilitate more publishing are selected against. Hence the aforementioned "build on" principle, which in practice virtually rejects both radical new ideas and ideas that
do not facilitate publishing. The error is in assuming the existence or possibility of an efficient market for ideas using a flawed model of the market process. The point is not that scholarship is inefficient compared to markets, but that markets are inefficient (yet, as I show
in my book The Seen, the Unseen, and the Unrealized, unbeatable: rowman.com/ISBN/978073919… ). So also with scholarship and theory development, where - from the point of view of actual knowledge generation - improper incentives direct the selection mechanism toward inefficient and
limitedly valuable (for society) ends. In other words, scholarship qua scholarship requires serious study of the history of thought, where one can often find more interesting, more clearly developed, and better scrutinized ideas than in contemporary journals. The Whig theory of
science (or of anything) only applies to the degree selection is made on proper grounds, which is why manipulations and distortions of the open-entry, free-enterprise principle is so destructive and harmful.
