Thread by @MatthewAKraft

[THREAD] Today @RANDCorporation released a major study of the @gatesfoundation's efforts to improve teacher effectiveness through evaluation & human capital reforms. I've waded through the 500+ page report so you don't have to. Here is what I took away.

rand.org/pubs/research_…

If you are looking for an overview, this won't be it. Even the executive summary was 25 pages! Check out these articles published today by @matt_barnum in @Chalkbeat and @KevinMahnken in @The74:

chalkbeat.org/posts/us/2018/…

the74million.org/study-multi-ye…

First - the headlines will all focus on the impact estimates, but IMO the real contribution of the study is the rich qualitative and survey data the researchers collected over multiple years. The story here is really about implementation - what actually happened on the ground.

In many ways this study is an analysis of a proof of concept. The 3 districts & 4 CMOs were specifically selected because of their strong commitment to the reforms, and they had unprecedented financial support - $575 million in total which amounted to $800 - $3,500 per pupil.

Despite the buy-in & funding, capacity constraints & the tension between using evaluation for both formative & summative purposes proved to be major implementation challenges. Evals were huge burden on principals & the new system resulted in few teachers rated below Proficient

Ratings consistently shifted upward over time. I originally thought the opposite would happen when the first new eval ratings came out a few years ago. I assumed there would be an implementation learning curve, but I underappreciated the interpersonal elements of the eval process

Principals want to recognize and reward effort/growth. Ratings get high quickly when almost all teachers start at a 2 or 3 and there are only 4 or 5 rating categories.

The use of rigorous classroom observation rubrics that provide a common language for talking about high-quality instruction was widely viewed as a positive outcome. 20 years from now I'd bet this is one of the most enduring legacies of teacher eval reforms.

The districts/CMOs struggled to use evaluation as an engine for professional development. The observation and feedback process was limited due to capacity and funding constraints (yes, they had $$$, but feedback is time intensive and people's time is a major driver of costs).

IMO, it was a real lost opportunity that the districts/CMOs did not attempt to redesign teacher compensation with much higher top-end base salaries. They stuck to modest supplemental bonuses. Teachers repeatedly said a few $1,000 did not motivate them to change their practice.

Career ladder opportunities were "add-ons" rather than distinguished job promotions. I can't imagine modest supplemental bonuses and "add-on" roles are a very visible or effective way to recruit top talent.

Which leads me to my favorite quote of the report . . .

"Our teaching staff will only be as good as what we are allowed to select from. We have to strengthen our pool of eligible candidates . . . . "

- School Leader p.113

The study found the eval reforms did not attract more effective novice teachers into the districts/CMOs

The study also highlighted a rarely discussed headwind in the eval reform era - the challenge posed by the substantial turnover in district/CMO leadership. When reformers hold up D.C. as a model, let's not forget how much they benefited from consistent leadership and transitions.

My quick read of the impact results are that they likely rule out the moderate to large positive effects reformers envisioned. It is hard to say much more given the limited treated sample (n=7), short pre-period, & nature of the comparison group that also implemented eval reforms

If you are holding out for any type of good news, all 3 districts succeeded in meaningfully reducing forced teacher placements and seniority transfers where less-experienced teachers are "bumped" from their positions.

This is an incredibly rich study that I hope is not pigeonholed into being just the "nail in the coffin" of teacher eval, but rather informs how districts invest their resources & build systems to attract, develop, & retain effective teachers in the new ESSA era.

[END THREAD]

+1/ Here is another nice summary of the study in @edweek by @madeline_will with insightful comments from @rickhess99 @CEDR_US & more.

edweek.org/ew/articles/20…