In 2 weeks, @PublicHealthUMN will remove its number from ETS, going #GRExit for all programs. Our decision was largely based on the results of a RANDOMIZED assessment of how GRE scores influence admissions decisions.
What we did and what we found: a thread. 1/n
What we did and what we found: a thread. 1/n
Quick #GRExit background: there is published literature looking at whether GRE scores predict success in grad school. Most show it doesn't (much), but it's tough to define/measure "success", and selection bias clouds interpretation of study results. 2/n
We decided to ask a simpler question: does seeing the GRE score actually affect how admissions committee members score an application?
Now *that's* a question we can design a randomized study to answer! 3/n
Now *that's* a question we can design a randomized study to answer! 3/n
First, we figured out how to redact GRE scores from application PDFs. This was actually one of the most challenging parts of the process. We settled on a tool called AutoRedact by @evermap which allowed us to automate the process. 4/n
Next, we used a randomized incomplete block design (function 'ibd' from R package of same name) to assign redacted or unredacted applications to admissions committee members. Several admissions committees across @PublicHealthUMN participated. 5/n
Committee members scored applications per their usual system. At @umnbiostat, we use a 1-5 scale; other admissions committees use a 1-10 scale. 6/n
After compiling the scores, it was analysis time! 7/n
(Aside: Final admissions decisions were made after a committee discussion where we did not attempt to maintain blinding to an applicant's GRE score. Our analysis focuses on the preliminary score only.) 8/n
Methods tweet: Outcome was the preliminary score (1 to 5) and predictor of interest was whether the committee member saw a version of the application with the GRE included or not. We used a mixed model with random committee member and applicant effects. 9/n
Result #1: On average, redacted Biostat PhD applications were scored 0.03 points higher than unredacted applications (p = 0.61) 10/n
Result #2: On average, redacted Biostat MS/MPH applications were scored 0.01 points lower than unredacted applications (p = 0.84) 11/n
Result #3: There was no difference in the (essentially null) effect of redaction based on whether or not students were US citizens or permanent residents. 12/n
Result #4: Underrepresented (minority/first gen) students were scored 0.41 points higher by reviewers who did not see the GRE. Numbers were small, so not significant, but notable. 13/n
Results were similar for other committees which implemented our randomized protocol. 14/n
Overall conclusion: In the presence of other application components, the GRE score does not appear to substantially influence the preliminary score given by admissions committee members. 15/n
Implication: Eliminating the GRE requirement is unlikely to substantially change how applications are evaluated at @PublicHealthUMN. 16/n
Discussion: All sorts of caveats apply. Results may depend on admissions committee processes, admit rates, applicant mix, etc. Also, blinding was likely imperfect b/c we couldn't redact scores from, e.g., resumes and CVs that applicants included. 17/n
Many thanks to everyone who supported and participated in this process. A special shout out to @rtdemmer and @emdunsworth. 18/n
As we embark on a GRE-free future, this evergreen tweet seems like a good place to end. 19/n
