Thread by @jsm2334

@K_Sheldrick BTW, I checked your p-values for Fisher's exact tests and they appear correct.

Indeed, it appears unusual for there not to be more smaller p-values given the expected uniform distribution, assuming they reported all factors and not just a subset that appeared similar ...

@K_Sheldrick Now, it is possible that they compared the 2 groups for more than these 22 factors, and only presented these 22 (perhaps because they appeared to demonstrate balance in randomization) and omitted the others.

@K_Sheldrick If we assume that did not occur and these are the only 22 they looked at, then we can use permutation tests to assess the rarity of observing p-value distributions like this under the null hypothesis of no association between randomization and any of these factors.

@K_Sheldrick We can do this by conditioning on the marginals for each of the 22 2x2 contingency tables, but randomly reallocating the counts within the contingency table, and then computing the Fisher's exact test for each.

@K_Sheldrick If we repeat this process for many (e.g. 10,000) random permutations, we get a set of 10,000 Fisher's exact test statistics (and p-values) for each of the 22 factors.

@K_Sheldrick To see if the distribution across the 22 factors is unusual, we can look at the distribution of Fisher p-values across the 22 factors across the 10,000 permutations.

For example, below is the permutation distribution of minimum p-values from among the 22 factors.

@K_Sheldrick This shows the expected distribution of minimum Fisher p-values across the 22 factors, conditioning on their marginals, assuming no association, and that these 22 factors were not selected from a larger subset to be more similar across randomization groups.

@K_Sheldrick The actual minimum Fisher p-value across these 22 factors is 0.409, which is unusually large -- only 3/10,000=0.0003 of the permutation distributions had a minimum p-value that large.

So, this is definitely evidence enough to reject the specified null hypothesis.

@K_Sheldrick If we look at the 2nd smallest p-value, 0.486, we can compare with the corresponding permutation distribution, and find only 32/10,000=0.0032 are at least this large, again providing strong evidence that the null hypothesis is not true:

@K_Sheldrick This is also true if we look at many of the other ranks.
Of course, the small sample sizes in some of the cells lead to an artificially high number of p-values of exactly 1.0, but this permutation distribution takes these artifacts into account.

@K_Sheldrick As Sheldrick suggests, this does suggest something unusual here, since we would expect randomization to not produce groups this well matched by random chance alone.

@K_Sheldrick However, I would not be confident making claims of fraud on the basis of these results -- there are other possible explanations:

@K_Sheldrick 1. Stratified randomization: if randomization were stratified by some factors, even factors not in this table, the association of those factors with the ones in the table could produce groups more alike than expected under unconstrained randomization.

@K_Sheldrick 2. Table does not present all factors considered: If the 22 factors represent not all factors for which they tested for comparability, but rather a select subset, this changes the context & could lead to a table like this in which the groups are more alike than expected by chance

@K_Sheldrick For example, suppose they looked at 40 factors, only presented these 22, with the other 18 not presented being less balanced. This type of selection would change the expected distribution of Fisher p-values across factors, as well, making them overrepresented for large p-values.

@K_Sheldrick This would not be good practice either. I am in no way accusing the authors of this, but hypothetically this would be another explanation for these unusual results.

@K_Sheldrick So, in conclusion, I agree with @K_Sheldrick that this table suggests something unusual - that the groups are unusually well-balanced across these 22 factors for unconstrained randomization - but I am not willing to infer the study is fraudulent on the basis of this result alone

@K_Sheldrick BTW, to be more precise, I did these calculations via Monte Carlo simulation conditioning on the estimated marginals as the truth when simulating the contingency tables under the specified null distribution.

@K_Sheldrick Here is the R script I used -- pretty simple:
twitdoc.com/BL34

@K_Sheldrick These results are even more unusual when we consider that since these were observational cohorts one would expect the groups to be even more different than under randomization, again unless they did matching or presented only subset of factors matching closely