Thread by @mlipsitch

This tweet got me thinking again about a topic that's been on my mind for the last several weeks and throughout the pandemic. In principle I fully agree with @flodebarre that people should evaluate arguments for logical soundness and consistency with facts, not who makes them.

https://twitter.com/flodebarre/status/1392937748886134787

But many people have asked me (most recently @AmyDMarcus) how thoughtful people should know whom to trust in getting information (science) and advice (for personal actions) and opinions (about policy) on a topic like COVID

Consistent with @flodebarre's tweet, my first response was you shouldn't trust anyone intrinsically, but should trust good arguments. As a scientist, that is how we are (or should be, there is still too much hero worship in our field) trained.

But for any particular topic, it is out of the expertise for 99.99...% of the world to evaluate the arguments. Ask me about COVID epidemiology, and I'll evaluate the arguments. Ask me about dengue epidemiology, and I'll try but not fully trust myself.

Ask me about diabetes epidemiology, same. Ask me about cardiac electrophysiology, probably can't. And those are all in my broad field of health science. On what our foreign policy should be in (pick a country) I'm at the mercy of the experts: whom to trust?

If I apply my nostrum of "evaluate the arguments not the speakers" to whether the Afghan withdrawal was a good idea, for example, I have a strong sense but also realize that I am too ignorant about the history, the current politics, & other considerations to really trust my views

So fair enough that a person (most of the world) who spends their time doing something other than studying COVID is looking for heuristics for whom to trust.

I think there is no perfect classifier of "reliable expert" vs "other." There are factors that probably make the prediction better than random. Here are the ones I use, as best I can unpack my mainly subconscious process.

Matters if it's someone I know or someone new to me. Lots more options available for someone I know. For someone new: 1. Expertise in one or more relevant fields. A degree is correlated with expertise, but is neither necessary nor sufficient.

The most insightful conversation I have had this week about COVID was with someone whose current highest degree is a master's degree. And a degree can signify broad expertise or very narrow knowledge.

But still, all else equal, a degree in a relevant field is reassuring. And that in itself requires judgment. An MD (alone) does not make one an expert on ventilation and virus transmission, any more than I'd ask @j_g_allen for how to treat lower back pain.

Nutritional epidemiology is different from infectious disease epidemiology. So it depends on what is being asked about.

2. Where someone works is again probably correlated with their likelihood of being knowledgeable when they speak out on a topic, but we all know exceptions in COVID of people from prestigious institutions saying nonsense or worse. Imperfect but not meaningless.

3. Does the statement show an awareness of considerations outside someone's narrow field, if it should? Most decisions about the pandemic involve epidemiology, other parts of public health, and also economic, social, and psychological considerations.

People who either a) limit their statements to their own field (an epidemiologist talking about how much distancing reduces transmission) or admit the relevance of considerations outside their field to a decision are more trustworthy than people who ...

start with "distancing reduces transmission x amount" and conclude "we must implement distancing" without at least admitting the implicit assumption that the costs thereof are worth it, for a certain time frame etc.

4. Does the person show in their statement that they have considered contrary evidence? And refute it in a way that is not ad-hominem and otherwise conforms to fair argumentation? Sometimes hard to tell, as the whole thought process may not be described, but sometimes it is.

5. Is it a person or an institution speaking? As far as one can understand, are there bureaucratic (cover-yourself, risk-averse) or self-interested motives that could explain the position they take? Again, imperfect, as a self-interested statement can also be true. But relevant.

6. (now getting into territory where "knowing" someone (personally, from long twitter following, or whatever in between) can help. Asking myself: can I guess what this person's position will be on a topic before I read it? If so,uninformative: they could be right and predictable.

but if not, their view is of greater interest and to me more credible.

7. (ht @HelenBranswell) do they ever say "I don't know" or "that's beyond my expertise" or do they have a view on everything? I trust the former more.

8. Do they distinguish between fact, educated speculation, and opinion? blogs.scientificamerican.com/observations/h… Here @BillHanage and I wrote about this distinction in the context of the current pandemic

Probably will think of more. Bottom line, figuring out whom to trust is hard. The most trustworthy people are sometimes wrong. (ah that reminds me, 9. do they correct errors publicly when they make them? Good sign).

No heuristic or combination of them is perfect, and ultimately the more one can evaluate arguments and not speakers, the better. And an informed opinion is not one downloaded from the MOST TRUSTED SOURCE but one informed by broad reading, discussion, etc.

A corrollary is that for most of us who work for a living and/or have other demands on our time, having a truly informed opinion on most matters is beyond our time constraints.

Thanks to someone who brought up peer review. This is whole other ball of wax but yes peer review increases confidence, but is not infallible. In short my view is that peer review is the penultimate safety net / slice of swiss cheese in science.

Science works because we have multiple filters: scientists being trained well and working carefully; scientists wanting to be the ones to see the flaws in own work before someone else (reviewer) does; peer review; replication

when all that's in place, science does well; when one or more layers of net are frayed, caveat emptor