Thread by @dontwantadothis

🧵 Just because something is published doesn't mean it's good science. Below, I'll talk more about how you can evaluate research articles you may come across on social media or that might be cited in the press.

1/20

https://twitter.com/dontwantadothis/status/1818390979721703447

1st, when reading a media article that cites/links to a research pub., look at what the research piece itself says. Journalists may get things wrong or present a biased/incomplete picture. They may imply there's a causal relationship b/t variables when that's not the case.

2/

Second, evaluate the content of the journal article. Is a hypothesis clearly stated? Do the study authors test it in a way that makes sense? What are their conclusions? (You will find those in the paper's "discussion" and "conclusions" sections.)

3/

Do the study conclusions address the hypothesis and match the data? At the end of the piece, do the authors note any limitations to their findings? Do they suggest future research that will advance understanding of the issue? Ideally, answers to those questions will be yes.

4/

Evaluating a study's methodology can be challenging-to-impossible for readers who aren't experts in that field. But note the time period during which data were collected. Has anything happened since then that might make findings stale or obsolete?

5/

Note how many participants were inc. (+ is better, as long as participant inclusion criteria are good) & who they were. Does it seem reasonable to generalize findings to the whole population? Sometimes, we inappropriately draw conclusions about one group of people based on a

6/

study of very diff. folks.

Note what's said about "effect sizes" - do authors suggest these were large? Were results statistically significant? Generally, anything less than p = 0.05 is a sig. result (meaning the result is not overly likely to have been due to chance), but

7/

smaller p values like p < 0.01 are better.

And check to see if the study's conclusions are similar to those of other work. If they contradict other studies, do the authors address why, & does their explanation seem reasonable?

8/

Next, you'll evaluate the source itself. Where did authors do the work - a research uni., or somewhere else? Who are the first few authors? Do they list conflicts of interest at the end of the article? Is their other work controversial? (You may need to do some sleuthing.)

9/

In the example in the quoted tweet at the top of this thread, the first two authors appeared to be undergraduate students, not grad. students or credentialed research scientists (and they did not test any hypotheses). That's a problem.

10/

What about the journal itself? Are submissions "peer reviewed"? Experts in the field should be evaluating a study's methodology & soundness of conclusions before the work is published. (The journal noted in the original post has a sketchy history w/ peer review.)

11/

Does the work the journal publishes correspond with the discipline that's named as its topic? Are the journal editor & its board experts in the field? Does the name of the journal include identifiers (like a nationality or subject) that match where it's published & by whom?

12/

If the answers to those questions aren't yes, or if you see random advertisements on the journal's web page, or a lot of typos/errors in the published work, those are bad signs.

13/

You also want the journal to be "indexed" by a variety of organizations; this will give you a sense of whether the journal's ethical & review policies and editorial board transparency are good.

14/

For example, the journal in the OP above was founded by someone the BMJ (a reputable journal) once referred to as a "snake oil salesman," & was involved in HIV/AIDS denialism. It is published by Elsevier, but has been de-listed by Pubmed & Medline. Bad signs.

15/

In fact, according to Wikipedia, when Elsevier demanded adoption of a peer review process for the journal, "to avoid publication of 'baseless, speculative, non-testable and potentially harmful ideas,'" ed. board members responded that "not having peer review 'is an integral

16/

part of our identity, indeed our very raison d'être,'" & said they'd resign if peer review were required.

That's a *really* bad sign.

17/

You can also google a journal's "impact factor" to eval. it (though it won't mean every study they publish is necessarily poor, fair, or good). The IF is a ratio of the # of times a journal's articles were cited elsewhere to the # of articles it published over a 2 year pd.

18/

There are other variables to consider, too, but in gen., you want the IF of a journal you're reading to be high, b/c it's a proxy for quality of the work they publish. The IF of The Lancet (a v. reputable & well-known journal) is 98.4; the IF of the journal in the OP is 4.4.

19/

Last, note how you heard about the research. Was it from a trusted source who's an expert in the field, or from a random person on soc. media? Even experts occasionally share research that misleads or misinforms, but they're less likely to do so than an anonymous guy on X.

20/