Thread by @SteveCicala

🚨 New* Research Alert: 🚨
My older work on peer effects with a larger message on the limits of randomized control trials when your treatment isn't what you think it is. In the August JEEA (@EEANews)

Thread:

*`new' in econ publication, or geological time.

Suppose you had a theory that there are peer effects in occupational choice--that it's `contagious'--how else to explain why there are so many agricultural workers in Napa Valley, but computer scientists a few short miles away in Silicon Valley?

Suppose a handful of impeccable RCTs are conducted, randomly sending workers to areas with different agricultural intensities--but they always come up with something different.

One experiment sends subjects to the Williamette valley (agricultural), another to the plains of Nebraska (agricultural), another to Brooklyn (urban), and another to Miami (urban). It's a well-replicated experiment, but they all come up with different answers:

In the Williamette Valley the subjects join everyone else and make wine. Weird then they do construction in the Nebraska plains, which is also agricultural. They work in real estate in Miami, but tend rooftop bee colonies in Brooklyn. Are there peer effects or not?

A labor economist would probably find this exercise a bit baffling--it's completely missing a model of the labor market!

Even though workers are randomly assigned to areas of varying agricultural intensities, it ignores that workers choose occupations based on wage offers, and that agricultural intensity does not pin down the wage.

This paper makes the point that this is effectively the same exercise that has dominated the empirical literature on peer effects.

Having high-achieving peers is not sufficient to pin down the reward to studious activities, and without information on those rewards, estimates of peer effects are going to be all over the place.

Empirically, the paper joins recent work on the impact that class rank has on student outcomes (more on that later), but to me the main contribution is a conceptual one.

We use some old tools to take a different look at an old problem: how does the behavior of a student's peers affect that student's behavior?

If there's someone disruptive in class, and all class time is spent trying to maintain order, there isn't a lot of work getting done. In this sense social behavior is something experienced in common by everyone in the class. It's like a kind of pollution.

This has been the dominant approach to peer effects: looking at how students' outcomes change when they are assigned to a classroom with higher-achieving peers. There's something in the air raising (or lowering) everyone's test scores.

The key distinction we draw in this paper is that the people you spend time with privately have a tremendous impact on your outcomes. But in order to spend time with someone it's not enough to be in the same room: they have to choose you to hang out with, and you likewise.

As, I think many econtwitterers can draw upon their youth and agree: Having some cool kids in your class does *not* mean you get to hang out with the cool kids, and definitely does not make you a cool kid.

There is, in short, a market for peers not unlike a labor market where workers (students) join firms (social groups) based on the supply and demand for skills. (The Market for Peers was the paper's first title...in 2008)

The good news is that this framework can rationalize the otherwise puzzling results of really well-done experiments that found wildly varying peer effects. Sometimes positive, sometimes negative, often zero. Peer effects are somehow everywhere, yet no where to be measured.

The bad news is that it's really tough to predict how a student is going to be affected by a new classroom environment. Maybe they fall in with the right crowd, or maybe they can't keep up and find themselves getting in to trouble.

*This is true even if students are randomly assigned to classrooms.* Ultimately, students are going to spend time with the 2 or 3 kids that shake out in the market for peers that happens after random assignment.

A great demonstration of this can be found in an experiment run by Carrell, Sacerdote, and West (2013) on Air Force Squadrons, previously known as "Beware of Economists Bearing Reduced Forms"

Cadets were randomly assigned to squadrons, so if you want to know if better peers on average raises student scores, all you have to do is regress test scores on the average incoming quality of cadets, right?

Well, that regression indicated a positive effect so if you wanted to raise the scores of your worst-performing cadets, all you had to do was put them in classes with your top students. So that's what they did in an experiment.

Outcome? Spectacular fail. The very top and bottom students were like oil and water.

Instead of spending time with a peer group drawn from a bit higher in the distribution, the students at the bottom were isolated with the worst performing students as their *effective* peers, even though the best students were also in the room.

On the empirical side, all is not lost. One of the predictions of the model is that people select in to groups/activities based on their comparative advantage, which is monotonic in class rank (with some assumptions).

So we look at how class rank itself affects students' outcomes in two RCTs and one observational setting.

The headline result from NYC schools is that if, in the transition from elementary school to middle school (which tends to pool a bunch of elementary schools together), you experience a drop in rank on your incoming test scores,

there is an increase in the probability that you get in to trouble in 8th grade.
(i.e. I was the best in my class, but it was lousy compared to the other classes I'm mixed with in middle school, and now I'm not such a hot-shot student)

So we see a substitution between school work and getting in trouble being induced by changes in class ranking.

Re-analyzing the RCT of Duflo, Dupas, and Kremer (thanks for the data!): take two students of the same ability at the beginning of the year and randomly assign them to classrooms where they differ in rank among their peers.

The better ranked student is going to have higher score *levels* at the end of the year.

Finally, my coauthors (Fryer and Spenkuch) put together an RCT that I think is pretty cool. In the first stage, they had classrooms of students practice solving mazes on computers, knowing that in the second stage they would be paid for each one solved (some of them the same).

They then displayed the scores publicly, so everyone knew their rank, and willingness to pay to practice further was elicited. When there is nothing else to do, those ranked at the bottom were willing to pay more to practice.

In a second group, however, everything was the same except that students were given the option of a new activity: they could spend money to mess with their classmates' puzzles with no benefit to themselves.

The presence of this alternative activity eliminates the rank-willingness to pay for practice gradient. Students at the bottom of the distribution stop practicing, and start causing trouble.

Takeaways: Peer effects depend on who you spend time with, and the labor market-like mechanism of determining who spends time with whom occurs *after* assignment to classrooms, roommates, etc.

This makes it tough to predict the sign or magnitude of peer effects, and an RCT isn't going solve the econometric issues. One consistent finding, even in the presence of this complication, is that rank matters.

Here's a link to the full paper: home.uchicago.edu/~scicala/paper…