People seem interested in how a small group of researchers with an agenda can 'rig' a "systematic review" in medicine to make it say whatever they want, albeit dressed up in objective-sounding rhetoric. Here is a follow-up for those who want to see the details of how it is done. 
The case study from my previous thread was this highly-cited review by Morris & Krieger on the potential (adverse) sexual effects of male circumcision, published in the leading journal in this area, @jsexmed (bit.ly/2EWwBUy). M&K reported "no conflict of interest," but
as I wrote, M is co-founder & chief spokesperson of a pro-circumcision lobbying organization petitioning 4 government insurance to cover medically unnecessary circumcisions, which would directly financially benefit several board members (see bit.ly/2QYZmWY), and K
had submitted a patent application months before the review came out for a new circumcision clamp (bit.ly/2An1uO1). If removing sensitive tissue from the penis (the foreskin) turned out to have negative sexual implications, this could be bad for business. Conveniently,
then, M&K discovered - after rating the available studies using SIGN criteria (bit.ly/2rZcBrC) - that the high-quality studies "uniformly" showed no negative effects of circumcision, whereas all studies that DID report negative effects were of low quality. But as
@JennBossio pointed out in a little-noticed letter to the editor (bit.ly/2QXhZub) SIGN guidelines require that quality ratings be done by a multidisciplinary panel carefully assembled to minimize any possible bias among raters, whereas M&K did all their *own* ratings.
And as Boyle noted in an even more obscure rebuttal (bit.ly/2AkuNAy), several of the ratings M&K assigned for quality seemed to track the results of the study (that is, whether it supported their conclusion or not), rather than the actual quality. So how was this done?
In the rest of this thread, I'll walk through a couple examples, so you can see the sleight-of-hand authors may sometimes use to assign ratings to studies that support their desired conclusion. Take a study by Masters & Johnson, which M&K rated 2++ for quality, which refers to
"High-quality systematic reviews of cohort or case and control studies; cohort or case and control studies with very low risk of bias and high probability of establishing a causal relationship." Masters & Johnson reported "No clinically significant difference between the
circumcised and uncircumcised glans" in exteroceptive and light tactile discrimination (bit.ly/2LGPif8). Okay, so why should we think this study deserves the highest possible quality rating for a case-control study (2++)? The answer is: we shouldn't. The "study" consists
of a single paragraph (reproduced below) in the 1966 popular book by Masters and Johnson, "Human Sexual Response," in which they refer to some "routine neurological" tests they performed on a small sample of men. No description of the tools or methods used, no
description of statistical analyses performed, no description of participant sample characteristics, no peer review, no nothing. It is literally impossible to evaluate the quality of the study by Masters & Johnson. Yet based on a single paragraph mentioning a "brief clinical
experiment" in passing in an out-of-print 1966 book, M&K rated this evidence as being of the *highest possible quality* (2++) for a case control study! At the same time, they rated a similar experiment by Sorrells et al. published in 2007 in the widely respected, peer-reviewed
British Journal of Urology as the *lowest possible quality* for a study of this kind (bit.ly/2JIgpbv), despite more than twice the sample size, a priori power analysis, and exhaustive description of methods and tools used. How could this be so?
One possibility is that they didn't like the main findings reported by Sorrells et al. (2007), namely, that "the glans of the circumcised penis is less sensitive to fine touch than the glans of the uncircumcised penis" and that the foreskin - which is removed by circumcision -
contains the "most sensitive parts" of the penis to light touch - a finding later confirmed by Bossio et al. (bit.ly/2LDQGzi) (yet bizarrely reported in the media such as @voxdotcom as supporting the exact opposite conclusion, as I discuss here: bit.ly/2LEn44H).
When explaining their lowered rating, M&K cite a letter to the editor, referring to "those who criticized" the Sorrells et al. study [53] <-- if you look at the reference, this is a letter by Morris himself, along with a co-author. Other researchers have noticed that this is
consistent with a more general strategy employed by Morris (see screenshot below, from bit.ly/2QddGpw), in which he writes a letter criticizing *any* study that appears to show a negative effect of circumcision, on whatever grounds he can think of, then later cites this
in the third-person as *definitive* reason to reject the study & its conclusions, while failing to cite (much less engage) with the author responses 2 the original critique. Van Howe has recently documented this pattern in detail (see screenshot, from bit.ly/2LEZDsb).
Nevertheless, if M&K wish to downgrade a study's quality based on a letter-to-the-editor critique, they should do so consistently. For example, consider the Randomized Control Trials they rated as quality 1++ (the highest possible level of quality for any study design). The
first thing to notice is that one of these trials was conducted *by Krieger* et al., so he is rating *his own study* as of highest possible quality. The potential for bias here is obviously substantial. Now, you might think that because it is an RCT
it should automatically be rated as high-quality: RCTs are generally considered to be the 'gold standard' for establishing causation ... but *only* if they are well-designed and conducted. 'RCT' refers to a type of study design; it doesn't tell you anything about the quality of
materials actually USED in the RCT, including whether they were fit for purpose, capable of detecting a difference should it exist, etc. This screenshot shows *just* the limitations mentioned in the study by Krieger et al., which M&K apparently judged not to matter for quality.
But critiques by others had been raised before the M&K review, which they certainly were aware of because they came in an author reply letter responding to one of Morris's 'rebuttals' (see screenshot below, from bit.ly/2QddGpw). In short: the questionnaires used in
RCTs were badly designed & likely couldn't have shown a difference in sexual outcomes anyway (so absence of evidence is not evidence of absence). Other studies have shown *major* differences (see below, from bit.ly/2LJPyKE). But these were rated as "low quality" by M&K.
This is just a couple of the studies they rated, and you can already see how there is enormous room for subjective judgments about whether a study of a given design should be 'upgraded' or 'downgraded' in terms of quality, based on whether one favors the study results or not. So
the more general lesson is: when you see a "systematic review" (or a meta-analysis, or an RCT, or any other scientific artifact claimed to represent the highest quality of evidence), you should not take this at face-value. Meta-analyses are often GIGO ("garbage in, garbage out"),
RTCs are often poorly designed, carried out, and/or analyzed, and then "prettied up" with an attempt by the authors to "spin" their findings in the most dramatic and conclusive direction they can manage; and systematic reviews are often partisan opinion in disguise. Yet
doctors, policymakers, and ordinary citizens, rely on such studies, analyses, and reviews to decide what to believe about any given topic of importance to health and society. Even respected, professional organizations like the @AmerAcadPeds can reach unsupported conclusions based
on biased reports and biased interpretations of those reports by whoever happens to be selected to serve on a task force or committee (as I detail at length here bit.ly/2AgagNF). In fact, like M&K, the AAP 2012 circumcision task force members failed to declare
their financial and other conflicts of interest in their 2012 policy statement, only later adding a mention in a reply to a critique by international experts (see screenshot below). AAP task force member Dr. Andrew Freedman also forgot to declare his personal conflict of interest
stating in a separate interview that he circumcised his son for "religious, not medical reasons" (bit.ly/2VcxTzz), and acknowledging only later that the 2012 AAP committee had explicitly non-scientific and even political considerations in mind as they reviewed the
the empirical evidence on circumcision (see screenshots below, from bit.ly/2rYp2Eh). And even the anonymous working group at the @CDCgov, in echoing the stance of the AAP, has relied on none other than Brian Morris in support of its CENTRAL contention that
the benefits of circumcision outweigh the risks - despite the fact that no other researcher has ever been able to reproduce the figures quoted by Morris (using the same tactics outlined above): see screenshots below, from bit.ly/2QYZmWY. Science and medicine are
complicated endeavors. There is often quite a big back-story to whatever ends up getting published, cited, and incorporated into policy. This has been just one paper in one corner of one area of the literature I happen to know well. I have no doubt others have similar stories.
@threadreaderapp unroll please
