Thread by @jbcarmody

In response to this thread about using #USMLE Step 1 scores for residency selection, many people have asked some variation of this question:

“But without Step 1 scores, how will we possibly select residency applicants?”

I have a few things to say about that. (1/x)

https://twitter.com/jbcarmody/status/1079183928911638529

First, a reminder. The USMLE is a licensing exam. It is intended to discriminate between test-takers with sufficient content mastery to be licensed as a physician, and those who do not have such mastery.

It was not intended to discriminate among higher scores. (2/x)

So first of alll, it’s not entirely clear what you are measuring when you have an applicant with a 225 and a 245 and a 265.

(But probably, you are measuring test preparation, test-taking strategy, and memorization of increasingly arcane minutae.). (3/x)

This is frankly and honestly acknowledged by the NBME CEO in the same article I highlighted yesterday.

“Does Step 1 performance predict residency success? To our knowledge, no study has been done to answer this question.”

journals.lww.com/academicmedici…

(4/x)

What is true, however, is that USMLE scores correlate with residency in-service examination scores and specialty board exams. And selecting residents who will pass the boards is a reasonable goal. But as a program, is that really all you want to do?

(5/x)

So if you ask, “How will I pick residents if Step 1 is reported as pass/fail?”, I ask: What kind of residents do you want to have in your program?

What is your program’s mission?
How do you define success in your program?
What do successful residents/graduates look like?

(6/x)

Do you want residents who pursue academic careers? Practice in under-served areas? Demonstrate excellent procedural skills? Provide excellent patient care?

Define what it is you’re trying to find. Then look at your own data and figure out what predicts that outcome.

(7/x)

If your program’s goal is simply to graduate residents who score well on their in-service and board exams - then by all means, keep using Step 1 scores.

But my guess is, USMLE scores are not going to be the best predictor of other things.

(8/x)

Using Step 1 scores to screen applications is lazy (at best). At worst, it systematically disadvantages under-represented minorities, inefficiently identifies talent, and perpetuates an ‘arms race’ that will ultimately hurt patient care (if it hasn’t already).

(9/x)

Here again are data from the NRMP program director’s survey.

If you a PD who said that USMLE Step 1 scores were important but denied that the others were, you should be ashamed.

You cannot tell me that you can’t pick a good class with just the factors below line 1.

(10/x)

If you are a program director, YOU are in the best position to know what leads to success in your program or your specialty.

Stop outsourcing that responsibility to the NBME.

(11/x)

I am also hearing the argument, “But the students who excel on Step 1 are the best students anyhow. If you give them a different ‘hurdle’ to jump over, they will, and you’ll end up picking them anyway.”

Two things on that.

(12/x)

First, I doubt that the premise of this argument is true.

Success on Step 1 is undoubtedly collinear with some traits (perseverance, work ethic, etc.) that are beneficial in other areas. But there are lots of talented students who underperform on the test.

(13/x)

But even if you accept that argument as being true:

Is this how you want our brightest students spending their time? If it’s true that the “best” students score the best on Step 1, then let’s stop memorizing esoterica and put their talent to work on things that matter.

(14/x)

Acting as if our only choice is between using Step 1 data or having no data to select residents is a false dichotomy. ERAS applications are data-rich - but most of that data is never seriously analyzed.

(15/x)

“But my highly selective program receives XXXX applications for X slots. Without scores, I could never triage them all.”

Then get someone to help you. You owe it to your program to make the best selection decisions you can, not just the most time-efficient.

(16/x)

To me, this argument is like a physician saying, “I’ve got so many patients to see today that I don’t have time to take a history. I’m just going to take a quick look at the patient’s WBC and go with it.”

(17/x)

I recognize that PDs are not full time admissions officers. But I also think that it’s an important job, and one that needs to be done right. Tell your chair that you need time or personnel to help with applications. Use those resources to evaluate your own data.

(18/x)

Let’s also bear in mind that in the long term, we are not limited to considering only the data contained on the current ERAS application. I’m sure ERAS is happy to collect whatever PDs say is important to them. But we have to stop being lazy and define what that is.

(19/x)

Last, you don’t have to agree to an alternative to USMLE Step 1 scores in order to accept the argument that scores are being inappropriately used in residency selection, and that it is hurting both UME and GME. The first step to a solution is admitting we have a problem.

(20/x)