Thread by @mattcmccaffrey

My critical review of the literature on #lootboxes, problem gambling, and problem gaming is now available. Short version: the vast majority of this “research” is borderline junk science. papers.ssrn.com/sol3/papers.cf… Here's longer version in 2 threads:

In 2017, the Battlefront II controversy brought widespread attention to loot boxes—randomized digital rewards sometimes paid for using real money. Ever since, academics have been jumping on the bandwagon to show a relationship between loot boxes and problem behavior.

This emerging literature has been cited widely in the media and used by policymakers to propose a range of regulations on the use and marketing of loot box mechanics.

The big takeaway from these studies is a strong correlation between loot box engagement and symptoms of problem gambling and/or gaming. The danger is that loot boxes are a gateway to gambling, especially among children. Conclusion: this is a crisis, and we need to regulate!

The problem? The existing research is almost completely useless. Allow me to explain. papers.ssrn.com/sol3/papers.cf…

The overt objective of the literature is to justify regulations and bans on the use of loot boxes on the grounds that they cause harms similar to gambling. This politicized approach is already bad enough, but the underlying research also has many problems.

To begin, in the dozens of current studies there is essentially no actual observation of player behavior. There are some experimental studies, but these don’t involve real money or players making actual purchases, and contribute virtually nothing to our understanding.

The bulk of publications are empirical, and attempt to study the relationship between loot box engagement and various problem behaviors. These too tend to be of very low quality though.

The studies use self-reported surveys usually gathered via purposive and non-representative sampling. In some of them, researchers literally just posted open survey links to Reddit. (Remember, the most downvoted comment in Reddit history is a pro-loot box statement from EA.)

In other cases, sites like Mechanical Turk were used for data gathering, even though that service has been repeatedly shown to over-sample groups like problem gamblers.

Research is also cross-sectional, despite the fact that observing behavior over time is vital for showing the alleged harms of loot boxes. But longitudinal research is hard, I guess, whereas cross-sectional work can increase your academic impact right now.

Incidentally, children, who are supposed to be at greatest risk from loot boxes, are not studied at all. Only a few studies even include adolescents.

Women are also typically underrepresented. Funnily, the authors of one paper say that a sample with 9% women is fine, because that’s an accurate reflection of their prevalence in gaming. But in a later paper the same authors get a 31% response, and say that’s also fine.

Next, when asking about respondents’ behavior, studies define loot boxes vaguely or inconsistently, and are often unclear about whether they are accessed for free or for money. When speaking of paid loot boxes, the bulk of studies do not even ask how much money was spent.

Further, there is also almost no distinction between loot boxes in mobile games and in AAA titles, or in particular genres.

Next, the tools used to screen for problem gaming/gambling are often unsuited to the study of loot boxes. Instead, traditional gambling screening tools are just adopted without question.

But loot boxes are a new and in some ways unique product: we can’t just assume that tools developed to study traditional gambling will work for the study of loot boxes.

In fact, quite the opposite: tools like the Problem Gambling Severity Index ask questions that in the context of loot boxes are vague and question-begging.

In short, the design of these studies tends to be extremely poor, and all the errors seem to bias the results against loot boxes (what a coincidence).

Setting aside the correlation issue, there is also the crucial question of the practical harms of loot boxes. Here again, published work is no help at all.

When you add up all of the data, methodological, and interpretation issues, there’s not much left of these results. But the problems don’t stop there.

Despite raising the hue and cry about how dangerous loot boxes are, most studies do not provide any kind of measurement of or benchmark for harm, or even define what harm is.

But let’s use a common intuition: maybe loot boxes can lead to excessive spending that hurts the spender. Such harm is not defined in the literature, but let’s think generally about how much people spend.

Most studies do not ask respondents about their exact spending, but when they do, the results are hardly alarming.

Reported average monthly spending on loot boxes varies from about $3-$17. Why is this a problem? A Netflix subscription costs more, and spending like this is certainly cheaper than, say, a night at the bar.