Thread by @taylor_owen

We have just launched our report about the media ecosystem and mis/disinformation in the 2019 Canadian Federal election. What follows is a mega thread detailing our methods, findings and some broader reflections. @mcgillu @munkschool @MaxBellSchool @ppforumca
1/n

There is a big worry about the influence of mis/disinformation and other forms of online media manipulation on democratic institutions, elections and political life. We set out to build and produce a robust and reproducible Canadian study. And we are heartened by what we found.

We assembled a fantastic team of political scientists, computer scientists, communications and public policy scholars to do what we believe to be the most comprehensive case study of an information ecosystem done during a Canadian election to-date.

We analyzed tens of millions of tweets, hundreds of thousands of public Facebook posts, all of Canada-focused reddit, and over four hundred thousand articles posted on Canadian online news media. At the same time, we conducted a weekly national survey of thousands of Canadians.

Our goal was to capture the Canadian political discourse and to look for the behavioural impact of mis/disinformation as well as trends of coordinated disinformation campaigns organized by state or other actors, bot activity, malicious and deceptive advertising, and polarization

*The Takeaway* Canada appears to have a more resilient information ecosystem than other countries, particularly the US. This, along with election integrity policy measures, likely inoculated us from some of the worst forms of election manipulation seen in other democracies.

This resilience is a function of: 1) high trust in media 2) relatively homogenous media preferences and marginal role for hyper-partisan news 3) high levels of knowledge and interest in politics 4) low levels of ideological polarization.

Below I will lay out our findings on media consumption, media trust/literacy, political knowledge, fact checking, polarization, echo chambers, toxic speech, inauthentic activity, political scandals & disinfo. + speak to some wider reflections on method the media, & public policy

1. Unlike many other countries (viz US) partisan news sources play a very small role in the Canadian media environment. Only 10% of Canadian regularly consume this type of news. Social media sharing and consumption is a bit more polarized. Also television continues to rule.

2. Canadians simply trust the media system and have high media literacy. They are skeptical of stories they see on social media.

3. Canadians are interested and reasonably knowledgeable about politics. Those who participate online tend to be even more interested, knowledgeable and believe they can be part of political change.

4. Canadian’s like fact-checking and it seems to sometimes work. BUT fact-checking is no panacea and false beliefs can become entrenched and highly resistant to fact-checking efforts even by trusted sources.

5. Canadians tend to be politically moderate. BUT they do tend to dislike members of ideologically-opposed parties. The election itself didn’t seem to widen the gap, even among the most avid social media users.

6. An echo chamber effect in the full Canadian population is limited BUT Canadians who use social media do tend to receive information from only their preferred party.

7. We didn’t find a lot of toxic speech on Twitter but it did exist on negative hashtags. Generally the language used indicates your political ideology, with the left more likely to use classist terms and the right more likely to use gendered and sexualized slurs.

8. Again just on Twitter but we found little evidence of systematic inauthentic activity. We investigated claims made during the election and found insufficient evidence for election manipulation or interference.

9. We analyzed scandals during the election (ie #blackface trending for days), and saw interesting online dynamics about what keeps a story going on social media. Bottom line: if you have something to say do it quickly or attention will have already moved on.

10. Finally we looked closely at disinformation. We found some disinformation but also note that it was not widespread or high-impact or Facebook (public posts) or Twitter. It existed primarily in partisan echo-chambers and likely changed few minds.

Ok, a TON more findings in the report, avail at link below. But let me add a few broader takeaways about election and mis/disinfo monitoring. mediaecosystemobservatory.com/ddp

First, on data. There remain big blind spots in the online ecosystem caused by limited data access to researchers. We could only see a small fraction of Facebook, and none of WhatsApp or WeChat. This is a huge problem for the policy & research community and needs to be addressed

All of the findings in this report must therefore be understood in the context of the limitations of these kinds of large-scale media ecosystem studies in general, and the challenges of empirically studying disinformation and media manipulation more specifically.

For our study, we relied on three primary data collection streams: a weekly national survey, a metered sample of internet users who provided their browsing data, and large-scale online data collection.

We faced our largest set of challenges in obtaining consistent access to the APIs needed for large-scale data collection from social networks such as Facebook, Instagram, Twitter and YouTube.

These companies have in the past few yrs severely restricted the access that researchers have to reliable streams of data, leading to what some scholars have called a ‘post-API’ age that has made it increasingly untenable for computational social science research w/ platform data

We sought to contextualize our online data collection with survey data, which provided numerous helpful insights. However, it is limited by two factors.

First, the set of Canadians will to be members of survey panels where respondents are paid or otherwise compensated for their participation is probably shrinking. This increases the risk of unrepresentative inferences from survey data.

Second, surveys rely on self-reporting, which is not ideal for getting the most accurate picture of respondent behaviour during an election. Individuals cannot always remember which media they have consumed, and sometimes have reasons of social desirability for misreporting.

For this reason, so-called “metered” samples of internet user browsing histories (where survey respondents install an app or browser extension that tracks, with consent, all of their activity) are a particularly promising tool for studying online behaviour. But it is v expensive.

We strongly believe that further development and testing of this model is needed. But without these data, our ability to make inferences about the impact of media exposure on voting preferences and behaviour (already a difficult and contentious exercise) was seriously curtailed.

Despite all of these issues, we still see value in merging survey researchers w/ data scientists to study mis/ disinformation. We are further developing this method in the context of covid at the @MediaEcosystem. More on that soon.

Second, on the Public Policy of election integrity.

Coming after a number intl elections where disinfo was a serious problem, Canada were able to learn from the tactics of both those seeking to undermine election integrity and those trying to protect it. This allowed the government to mobilize.

While it is impossible to directly measure the impact of particular public policies in this space, Canada had the benefit of learning from the vulnerabilities in the digital ecosystem seen in preceding elections in other jurisdictions.

The gov't limited spending leading up to the election; banned advocacy groups from using money from foreign entities to conduct partisan campaigns; mandated an online ad registry for all internet platforms selling political and issue ads;

made it Illegal to impersonate or share false information about a candidate; and created a five-person panel of senior bureaucrats tasked with sounding the alarm if a serious attempt to meddle in the election by a foreign actor was detected.

It is very possible that these policy initiatives helped guard against the “low hanging fruit” of election interference.

Third, on the Media.

It was a core premise of our research that journalists bring a critical tool set for understanding disinformation, and that better collaboration between journalists doing that work and researchers adding additional analytic focus would be valuable.

We hosted a two-day workshop for journalists where we brought in global leaders in the study and reporting of disinformation during elections. And a lot of journalism orgs stood up truly impressive disinfo reporting capacity. This is a great thing.

While it was very important that journalistic capacity for reporting on disinfo was scaled up prior to the election, this could have created an incentive to find examples of it. Journalism requires news, and the lack of a problem is a more difficult story to tell.

Ultimately, disinformation reporting in an election is a public good, and the presence of journalists in this space plays an important disincentivizing and accountability role. In this case, the absence of news is good news for our democracy.

On our end, we observed some real tensions between the twin mandates of our project: conducting academically rigorous analysis, and communicating regular findings to journalists and the public.

Our research design was stretched by providing weekly briefings to journalists, and this type of real-time monitoring demands a far larger data analysis team than that of scholars taking a longer-term and more scientifically oriented research approach.

Fourth, on the ecosystem based methodology. We believe there is clear value in this methodology, but it needs iteration and large-scale trials, both in Canada and internationally as well as during elections and outside of them.

However, due to the data-gathering and methodological limitations described above, we are still only scratching the surface on how to understand the role digital information plays in an election.

The absence of robust open-source frameworks and tools for this work has become real problem in the interlinked research, journalistic and policy communities.

In effect, every project or investigation building a monitoring infrastructure must do so from the ground up, applying different methods and piecemeal datasets, leading to limited comparability and reproducibility across studies and contexts.

Developing an open-suite of tools, along with better APIs for research should be a priority, and policy solutions that seek to incentive mechanisms for ethical, reproducible and privacy-preserving research may well be necessary.

Going forward we will continue to ask: How can we ensure greater ethical data access? How can we scale these efforts to function outside of elections? How can we deal with private online spaces? And how can the methodology evolve to better capture behavioural impact?

Read the full report for detail on these findings, reflections on methods, and broader reflections on mis/disinfo monitoring. Above all, thank you to the great team: @AengusBridgman @EricMerkley @PeejLoewen @derekruths @ozhilin1 @rgorwa @smaclellan & to our parter the @PPFForum

Follow our ongoing work at the @mediaecosystem
and the full report, LESSONS IN RESILIENCE: Canada's Digital Media Ecosystem and the 2019 is avail here:
mediaecosystemobservatory.com/ddp

This project was led by @PeejLoewen @derekruths & myself. Aside from being great to work with, bringing together political science, comp science, media studies and public policy made for a fascinating collaboration. We are continuing to work together through the @MediaEcosystem.