Thread by @manyclasses

The first ManyClasses study is released! Final registered report now in press at AMPPS and preprint posted at psyarxiv.com/4mvyh/
What's it all about? Hey, we're glad you asked… 1/19

People have believed (for at least a hundred years) that psychology should be able to improve education; psychologists often recommend what teachers and students should do to improve learning outcomes. 2/19

But when these recommendations are tested in the classroom, it's usually just with one class (often a psychology class), and it's unclear whether the theories or research findings generalize. That's where ManyClasses comes in. 3/19

ManyClasses is a model for embedding an experiment across lots of classrooms, where materials and measured outcomes are authentic to each individual class. In this first study (ManyClasses1), we had 38 college classes across 15 campuses participate. 4/19
osf.io/tjkyr/

We care a lot about ethics. Students consented to share their data and participate in the study. About 80% of students opted-in, a final sample of 2,081. Everyone got all conditions (within-subject design), equating treatment for all students. 5/19

ManyClasses1 examined how the timing of feedback on online homework quizzes affects student performance. Sometimes students saw their grades and feedback immediately after submitting a quiz in @CanvasLMS, and other times they had their feedback delayed by a few days. 6/19

Teachers had flexibility to select and report outcome scores that were authentic and relevant to the manipulated homework quizzes. These were usually scores on exam items that assessed material on the quizzes. 7/19

So what did we find? Our analysis found no generalizable effect of feedback timing across these 38 classes. It wasn't that there was uncertainty in our global estimate -- we found a global effect of feedback timing that was tight around zero. 8/19

We also ran analyses of 40 different candidate moderators, and found no strong evidence of differences in the effects of feedback timing between students or class designs -- but with 38 classes, class-level moderator analyses are kinda underpowered. 9/19

Looking deeper, we found hints of possible benefits for delayed feedback in classes with open-ended test questions and/or with lots of quizzes, but these classes were rare, and these patterns did not meet our threshold for drawing clear inferences. 10/19

So what're the takeaways? We’re glad you asked! We'd like to highlight three of them. It's the magic number after all. 11/19

First, there's no general benefit to receiving feedback immediately after a learning activity. This might offer relief to teachers stressed with grading: there's no evidence that delaying feedback by a few days will harm student learning. It might even help. 12/19

Second, pinning down the kinds of classes where an instructional practice works is really hard. Classes are highly variable and getting even coverage across the design space is a challenge. Future ManyClasses efforts will require grander scales than the current study. 13/19

Finally, and perhaps most importantly: We pulled it off! Despite all the challenges, ManyClasses1 stands as proof positive that large-scale, collaborative, distributed experimental research is feasible in education. 14/19

But let's be clear: ManyClasses is *only* feasible through the involvement of dozens of collaborators and stakeholders. In particular, many thanks to @UnizinEd for making this possible! (ftr: we tried to include non-Unizin schools but barriers prevented their involvement) 15/19

Special thanks also to our coauthor, Janelle Sherman, who was on the front line during implementation and data collection. It wasn't easy. The project benefited tremendously from Janelle's insight, organization, outstanding communication, and attention to detail. 16/19

As @boreel3 affirms, "This isn't the end, or even the beginning of the end, but this is the end of the beginning." @BenMotz is now working on a research platform that can streamline future ManyClasses studies, called Terracotta terracotta.education 17/19

There's plenty more to be said about ManyClasses - preregistration, statistics, experiment design, open science, data sharing, and transparency. But for now, we're just gonna throw our mortar board in the air and celebrate the culmination of 4 years (!) of hard work. 18/19

Press release: news.iu.edu/stories/2021/0…. The ManyClasses lead team is @FyfeEmily @joshdeleeuw @paulo_fcarvalho @Boreel3 and @BenMotz . 19/19