NIST finally released (5 years later!) its notorious Mix13 study, available here: fsigenetics.com/article/S1872-… Good coverage here: forensicmag.com/news/2018/08/n…. Notorious because of "case 5," in which 70 of 108 crime labs falsely implicate an innocent suspect. /1
MIX13 is a follow-on to MIX05. Both were efforts to do in forensic science what is routinely done in clinical/medical context: interlaboratory comparisons. E.g., if you get a blood test at lab A, B, and C, will all three give you the same results? Important, obviously. /2
So in MIX13, NIST sent five "cases" to 108 participating labs, who each had between 8-25 analysts who reported "results." NOTE: they only sent electronic test results files -- they did not send biological samples. /3
That's significant because they even eliminated one of the critical steps: how does your lab actually handle the testing process itself. Do your instruments work, are your analysts doing quant correctly, etc. But here, they got results &t the only task was interpretation. /4
As study says, work has "demonstrated: (1) that labs have instruments with different sensitivities, (2) different levels of experience and training play a part in effective mixture interpretation & (3) amount of input DNA affects the ability to detect the minor component" /4
Also - forensic work is challenging in part bc of real world pressures. Have to meet performance numbers or get in trouble. Prosecutor pressuring you for certain answers. Extraneous info biasing analysis, etc. /5
So even with none of that on the table, how did analysts do? First, defense attorneys take heart: even though labs specifically asked to include protocols in results, only "a portion" did. Curious as to why - overlooked, or protocols a mess? /6
Second, intrigued that most labs using FBI population stats database. 12 use state or local pop database. 10 use combo. Majority also used PopStats software bundled with CODIS. /7
Also wide range of stats produced. Most common was the Combined Probability of Inclusion (CPI), which as we know can implicate innocent people. Also Likelihood Ratios (LR) and random match probabilities (RMP) used in some cases. /8
Thus, even in the simplest case (1) -- a 2 person mixture with a 1:1 ratio of contributors, all labs correctly included the "known suspect" but "a wide range of variation between methods was observed in the statistical values reported." 
The use of CPI also led to false inclusions in other cases. But the bombshell, as it were, is Case 5. Case 5 simulates a robbery involving 3 suspects. 2 suspects are named by an informant, and their DNA is provided. A third "associate"'s DNA is also provided. /10
The "crime scene sample" is from a ski mask found at the scene. It actually has four DNA contributors in equal amounts (1:1:1:1), but it looks like a 2 person mixture bc no locus has more than 4 alleles. In other words, a lot of allele sharing & drop out. /11
"The purpose of MIX13 Case 5 was to explore whether labs would consider this mixture too complex to interpret, and whether they would include the non-contributing reference profile (5C) and provide a matching statistic." I.e., do the labs false implicate the innocent suspect? /12
Result? "Of 108 laboratories contributing to the MIX13 study, a total of 74 (69%) included “suspect” 5C (along with suspects 5A and 5B) and provided CPI statistics in a Caucasian population ranging from 1 in 9 to 1 in 344,000."
So, yes. /13
So, yes. /13
The paper tries to minimize this as a learning exercise etc. But this kind of scenario -- a mixture of multiple people -- is actually quite common. Plus the most difficult part -- doing the actual testing -- was not in the test. /14
So here's what I learned: labs are falsely implicating people as contributors to DNA samples. Probably pretty regularly. /15
I should add that only 6% of labs (7) correctly excluded the innocent person, but only 2 did so for the "right" reasons. /16
There were also labs using probabilistic genotyping (TrueAllele, Lab Retriever, STRMix, DNAView) systems not counted in the study, and ALL of them properly excluded the innocent sample. Which affirms my sense that PGS systems- WITH PROPER OVERSIGHT-should be the standard. /17
Notably, for instance, those systems were correct when they assumed a 3 or 4 person mixture. Some do that on their own, but others provide for manual input/override, and in this case it might have looked like a 2 person mixture. /18
A final important finding of the study is the high degree of INTRA-lab variation. I.e., analysts within the same lab -- ostensibly using the same protocols, having received the same training, etc. -- reported starkly different results. /19
Within the same lab, for instance, some analysts would compute the CPI while others did an RMP and others a LR. That's nutty. That's like not only sending out your blood for testing & having different labs say A! O+! AB! but also getting divergent results from the same lab! /20
"When labs & analysts were presented simple, straightforward two-person mixtures such as MIX05 Case 2 or MIX13 Case 1 and Case 4, participants drew correct conclusions & reference samples were correctly included or excluded. More complex samples & scenarios, or situations..." /21
..where a great degree of allele overlap existed in the mixtures, produced more variation in responses." [Given the same case in the same lab,].. /22
under nominally the same interpretational protocol, 50% of the analysts effectively said, “I don’t know”; 30% of the analysts said, “He’s not there”; and 20% of the analysts said, “He’s not only in the mixture, but I can exclude greater than 99.9% of the population.”" /23
So, you know, there is work to be done. DNA is not a cure-all. It can be overextended. The more complicated the sample (which isn't always facially obvious) the more critical training and restraint may be. And lawyers should not just plead to avoid the fight. /fin
