The first ManyClasses study is released! Final registered report now in press at AMPPS and preprint posted at
What's it all about? Hey, we're glad you asked… 1/19
People have believed (for at least a hundred years) that psychology should be able to improve education; psychologists often recommend what teachers and students should do to improve learning outcomes. 2/19
But when these recommendations are tested in the classroom, it's usually just with one class (often a psychology class), and it's unclear whether the theories or research findings generalize. That's where ManyClasses comes in. 3/19
ManyClasses is a model for embedding an experiment across lots of classrooms, where materials and measured outcomes are authentic to each individual class. In this first study (ManyClasses1), we had 38 college classes across 15 campuses participate. 4/19
We care a lot about ethics. Students consented to share their data and participate in the study. About 80% of students opted-in, a final sample of 2,081. Everyone got all conditions (within-subject design), equating treatment for all students. 5/19
ManyClasses1 examined how the timing of feedback on online homework quizzes affects student performance. Sometimes students saw their grades and feedback immediately after submitting a quiz in @CanvasLMS, and other times they had their feedback delayed by a few days. 6/19
Teachers had flexibility to select and report outcome scores that were authentic and relevant to the manipulated homework quizzes. These were usually scores on exam items that assessed material on the quizzes. 7/19
So what did we find? Our analysis found no generalizable effect of feedback timing across these 38 classes. It wasn't that there was uncertainty in our global estimate -- we found a global effect of feedback timing that was tight around zero. 8/19
We also ran analyses of 40 different candidate moderators, and found no strong evidence of differences in the effects of feedback timing between students or class designs -- but with 38 classes, class-level moderator analyses are kinda underpowered. 9/19
Looking deeper, we found hints of possible benefits for delayed feedback in classes with open-ended test questions and/or with lots of quizzes, but these classes were rare, and these patterns did not meet our threshold for drawing clear inferences. 10/19
So what're the takeaways? We’re glad you asked! We'd like to highlight three of them. It's the magic number after all. 11/19
First, there's no general benefit to receiving feedback immediately after a learning activity. This might offer relief to teachers stressed with grading: there's no evidence that delaying feedback by a few days will harm student learning. It might even help. 12/19
Second, pinning down the kinds of classes where an instructional practice works is really hard. Classes are highly variable and getting even coverage across the design space is a challenge. Future ManyClasses efforts will require grander scales than the current study. 13/19
Finally, and perhaps most importantly: We pulled it off! Despite all the challenges, ManyClasses1 stands as proof positive that large-scale, collaborative, distributed experimental research is feasible in education. 14/19
But let's be clear: ManyClasses is *only* feasible through the involvement of dozens of collaborators and stakeholders. In particular, many thanks to @UnizinEd for making this possible! (ftr: we tried to include non-Unizin schools but barriers prevented their involvement) 15/19
Special thanks also to our coauthor, Janelle Sherman, who was on the front line during implementation and data collection. It wasn't easy. The project benefited tremendously from Janelle's insight, organization, outstanding communication, and attention to detail. 16/19
As @boreel3 affirms, "This isn't the end, or even the beginning of the end, but this is the end of the beginning." @BenMotz is now working on a research platform that can streamline future ManyClasses studies, called Terracotta 17/19
There's plenty more to be said about ManyClasses - preregistration, statistics, experiment design, open science, data sharing, and transparency. But for now, we're just gonna throw our mortar board in the air and celebrate the culmination of 4 years (!) of hard work. 18/19
Press release:…. The ManyClasses lead team is @FyfeEmily @joshdeleeuw @paulo_fcarvalho @Boreel3 and @BenMotz . 19/19

• • •

Missing some Tweet in this thread? You can try to force a refresh

Keep Current with ManyClasses

ManyClasses Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!


Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

Did Thread Reader help you today?

Support us! We are indie developers!

This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!

Follow Us on Twitter!