Thread by @JJSchroden

HOT OFF THE PRESSES! My team @CNA_org just finished a pair of reports looking at the #psychology of #disinformation. We sought to answer two main Qs: 1) How does disinfo work on your brain? 2) What, if anything, can #DOD do about it? Summary findings in this THREAD. 1/n

@CNA_org The first report answers the first question: How does #disinformation work on your brain?

You can find the full version of the report here: cna.org/CNA_files/PDF/… 2/n

@CNA_org We identified four key psychological principles related to the absorption & spread of #disinformation: initial information processing; cognitive dissonance; the influence of groups, beliefs, & novelty; and the role of emotions & arousal. How do these work? 3/n

@CNA_org Initial info processing: Our mental “processing capacity” is limited, so our brains take shortcuts to incorporate new info & those can open us up to mistakes. To the extent that we don’t process info as deeply as we should, #disinformation can be construed as true info. 4/n

@CNA_org Cognitive dissonance: People feel discomfort when confronted w/competing ideas. This motivates us to discount the importance of (or ignore) contradictory info or⬆️ the importance of compatible info. If #disinformation supports our beliefs, we’re more likely to believe it. 5/n

@CNA_org Groups, beliefs & novelty: We’re more likely to share info w/people we consider members of our group, when we think it's true, & when it’s novel or urgent. If #disinformation comes from "our group", is consistent w/our beliefs, or is new to us, we’re more likely to share it. 6/n

@CNA_org Emotion & arousal: We’re more likely to share info if we feel awe, amusement, or anxiety than if we feel sadness or contentment. Thus, #disinformation is more likely to be absorbed & shared if it is constructed to be emotional & arousing. 7/n

@CNA_org A key takeaway is that these principles aren't unique to #disinformation—they’re the same tools your brain uses to process all info. Thus, disinfo is absorbed & spread through *normal, routine & adaptive* mechanisms, which malign actors exploit & manipulate for their goals. 8/n

@CNA_org We ID'd 2 techniques in the literature to counter #disinformation associated w/more than one psych principle: 1) preventive inoculation (warning people about the effects of disinfo & how to spot it) & 2) encouraging deeper, analytic thinking about the info people encounter. 9/n

@CNA_org We also ID'd tricks to counter individual principles. For ex, to counter the effects associated w/groups, beliefs & novelty, researchers recommend creating unique content that encourages individuals to identify w/a broader “group” & increases their access to opposing info. 10/n

@CNA_org Researchers also recommend that #disinformation containment policies emphasize behavioral interventions aimed at countering the psych principles activated by disinfo, rather than solely focusing on the use of bots, algorithms & tech.

This is both a tech & a people problem. 11/n

@CNA_org So, what, if anything, can #DOD do about this?

We address that question in the second report, which you can find here: cna.org/CNA_files/PDF/… 12/n

@CNA_org We analyzed 5 cases of #disinformation in which absorption & spread benefited from the psych mechanisms described in the first report. These real-world examples clearly illustrate the human vulnerability to what we might call psychological hacking. 13/n

@CNA_org We examined: Op Secondary Infektion (#Russia campaign to foment division in Western countries); Endless Mayfly (#Iran effort to promote fake content); 2016 POTUS election; Hong Kong protests; & #CoronaJihad (attempt to blame #COVID outbreaks in #India on Muslims). 14/n

@CNA_org To illustrate, some details on Endless Mayfly. Named by researchers at Toronto U’s Citizen Lab, which studied these ops from Apr 2016 thru Nov 2018, it unfolded in 5 overlapping phases, each of which involved the creation, dissemination & amplification of false content. 15/n

@CNA_org The first step was the creation of fake personas, including students, activists & reporters. These were initially basic accounts w/little history, but over time they grew more robust as the “users” acquired bylines on third-party websites such as @BuzzFeed. 16/n

@CNA_org @BuzzFeed In steps 2-3, Endless Mayfly created 72 “lookalike” domains that mimicked well-known, highly credible news outlets, including @guardian, @TheAtlantic, @Independent & @haaretzcom, & populated them with malicious, scurrilous & false “copycat” content. 17/n

@CNA_org @BuzzFeed @guardian @TheAtlantic @Independent @haaretzcom To do this, Endless Mayfly operators used “typosquatting”—exploiting typographical errors in URLs to steer users to fake sites that look real. 18/n

@CNA_org @BuzzFeed @guardian @TheAtlantic @Independent @haaretzcom In step 4, the #disinformation campaign worked to amplify the content its operators had created. Endless Mayfly personas, posing as journalists, students, & activists, actively promoted the operation’s fake content on @Twitter IOT get others to do so as well. 19/n

@CNA_org @BuzzFeed @guardian @TheAtlantic @Independent @haaretzcom @Twitter In Endless Mayfly’s final stage, operators removed the phony articles & redirected their links to the impersonated media sites it had created (eg visitors to the misspelled theguardıan.com would be directed to the real website theguardian.com). 20/n

@CNA_org @BuzzFeed @guardian @TheAtlantic @Independent @haaretzcom @Twitter In terms of psych principles, Endless Mayfly played heavily on initial info processing—it hacked the authenticity of legitimate news sources by making lookalike content & effectively triggered our tendency to engage more superficially w/info that appears familiar. 21/n

@CNA_org @BuzzFeed @guardian @TheAtlantic @Independent @haaretzcom @Twitter So what can #DOD do about this? There are 3 primary vectors for countering #disinformation: 1) At the source, by ID'ing & stopping producers; 2) In the environment, through regulatory intervention; 3) With the audience, by reducing individual vulnerability to disinfo. 22/n

@CNA_org @BuzzFeed @guardian @TheAtlantic @Independent @haaretzcom @Twitter Though the efforts of organizations like @USSOCOM's Joint MISO WebOps Center, #DOD is already actively working to disrupt #disinformation at the source. The reality, though, is that doing this is effectively a game of whack-a-mole that greatly favors the mole. 23/n

@CNA_org @BuzzFeed @guardian @TheAtlantic @Independent @haaretzcom @Twitter @USSOCOM Addressing #disinformation in the environment (eg social media) is outside the control of #DOD. To date, tech has advanced faster than regulations & will likely continue to do so—leaving social media companies to (poorly) regulate themselves. 24/n

@CNA_org @BuzzFeed @guardian @TheAtlantic @Independent @haaretzcom @Twitter @USSOCOM IRT audiences for #disinformation, the USG can do little directly to protect all its citizens. Within the US’ freedom-loving culture, Americans are not going to accept the govt telling them what info they can & cannot—or even should or should not—access via media platforms. 25/n

@CNA_org @BuzzFeed @guardian @TheAtlantic @Independent @haaretzcom @Twitter @USSOCOM That said, there is a sizeable population over which #DOD has a line of direct control: its employees (service members & civilians). There are things it could do to protect those employees, from the direct effects of #disinformation & from serving as unwitting vectors of it. 26/n

@CNA_org @BuzzFeed @guardian @TheAtlantic @Independent @haaretzcom @Twitter @USSOCOM In this light, we offer a number of evidence-based interventions that #DOD should consider, in two categories. The first is “preventative inoculation,” which could include game-based inoculation, awareness campaigns, &/or video & banner reminders on DOD platforms. 27/n

@CNA_org @BuzzFeed @guardian @TheAtlantic @Independent @haaretzcom @Twitter @USSOCOM The second is “cultivation of deeper, analytic thinking,” which could include media literacy training, confirmation before posting requirements, pop-up windows alerting individuals to unregulated sites, pop-up windows sharing accurate info, & fact checking & verification. 28/n

@CNA_org @BuzzFeed @guardian @TheAtlantic @Independent @haaretzcom @Twitter @USSOCOM Per #DOD, #disinformation “poses one of today’s greatest challenges to the US.” As our reports highlight, any consumer of online media is likely to encounter disinfo & our normal psychological mechanisms enable it to act on our brains in beneficial or malicious ways. 29/n

@CNA_org @BuzzFeed @guardian @TheAtlantic @Independent @haaretzcom @Twitter @USSOCOM It’s beyond #DOD's role to directly protect all Americans from #disinformation, but it can do more to protect its employees. We conclude that add'l actions are critical, given DOD’s own assessment of the threat posed by disinfo & the inadequacy of existing means to stop it. 30/30

cc @peterwsinger @firstdraftnews @DFRLab @citizenlab @USASOCNews @WarInstitute @DavidMaxwell161 @NatSecMulligan @MARSOCofficial @USSOCOM