|| New publication
The Misinformation Susceptibility Test (MIST)
#AI-based (#GPT) fake news detection test
Paper (Behavior Research Methods)
Press Release (@Cambridge_Uni)
https://t.co/A69WeRTV8x
Online Quiz (#YourMIST)
https://t.co/P89rQdbhAA
(1/15) https://t.co/2Ci9cfVLuB https://t.co/Sb3sap3yHndoi.org/10.3758/s13428…
cam.ac.uk/stories/mist
yourmist.streamlit.app
twitter.com/i/web/status/1…
The Misinformation Susceptibility Test (MIST)
#AI-based (#GPT) fake news detection test
Paper (Behavior Research Methods)
Press Release (@Cambridge_Uni)
https://t.co/A69WeRTV8x
Online Quiz (#YourMIST)
https://t.co/P89rQdbhAA
(1/15) https://t.co/2Ci9cfVLuB https://t.co/Sb3sap3yHndoi.org/10.3758/s13428…
cam.ac.uk/stories/mist
yourmist.streamlit.app
twitter.com/i/web/status/1…
In 2019, @FriedrichMGoetz and I set out to develop the first psychometrically validated, AI-based, news veracity discernment test.
After 4 years of work, we are happy to announce the #MIST.
Read below more about:
(1) how we developed the MIST
(2) its first applications
(2/15)
After 4 years of work, we are happy to announce the #MIST.
Read below more about:
(1) how we developed the MIST
(2) its first applications
(2/15)
The development of the MIST was done over 4 phases.
* Phase 1: Item Generation
* Phase 2: Item Reduction
* Phase 3: Quality Control
* Phase 4: Psychometrics & Validation
This was then followed by a series of applied studies (more on this below).
(3/15)
* Phase 1: Item Generation
* Phase 2: Item Reduction
* Phase 3: Quality Control
* Phase 4: Psychometrics & Validation
This was then followed by a series of applied studies (more on this below).
(3/15)
Phase 1: Item Generation
400+ news headlines
Fake headlines
We used GPT-2 to generate thousands of headlines based on existing misinformation inventories
Real headlines
We selected headlines from sources rated as least biased and highly factual (using @MBFC_News)
(4/15)
400+ news headlines
Fake headlines
We used GPT-2 to generate thousands of headlines based on existing misinformation inventories
Real headlines
We selected headlines from sources rated as least biased and highly factual (using @MBFC_News)
(4/15)
Phases 2–3: Item Reduction & Quality Control
A multinational expert committee rated the headline pool. We kept items with a 3+/4 consensus on whether it was real/fake news. This was followed by a final quality check.
This led to a set of 100 remaining headlines.
(5/15)
A multinational expert committee rated the headline pool. We kept items with a 3+/4 consensus on whether it was real/fake news. This was followed by a final quality check.
This led to a set of 100 remaining headlines.
(5/15)
Phase 4: Psychometrics & Validation
We used the following methods to select the most predictive and reliable headlines.
* EFA & CFA
* Item-Response Theory (IRT)
* Exploratory Graph Analysis (EGA)
Validation Samples
Four US & UK quota samples (N > 4,000).
(6/15)
We used the following methods to select the most predictive and reliable headlines.
* EFA & CFA
* Item-Response Theory (IRT)
* Exploratory Graph Analysis (EGA)
Validation Samples
Four US & UK quota samples (N > 4,000).
(6/15)
Analysis Framework
We developed a framework ("Verification done") to interpret the results of the MIST that can also used for other tools. We recommend to report each of the following:
* Veracity discernment
* real/fake news detection
* distrust–naïvité (response bias)
(7/15)
We developed a framework ("Verification done") to interpret the results of the MIST that can also used for other tools. We recommend to report each of the following:
* Veracity discernment
* real/fake news detection
* distrust–naïvité (response bias)
(7/15)
Applied: Bad News Game (BN)
We used the MIST-8 to reanalyse the effectiveness of the BN intervention ().
People become better at detecting fake news, but more sceptical of real news. We are now using this data to further improve the intervention.
(8/15) https://t.co/JSE66WIZFi
We used the MIST-8 to reanalyse the effectiveness of the BN intervention ().
People become better at detecting fake news, but more sceptical of real news. We are now using this data to further improve the intervention.
(8/15) https://t.co/JSE66WIZFi
https://twitter.com/RakoenMaertens/status/1314174694308839424
Applied: YouGov Survey
YouGov used the MIST-20 to figure out which Americans are most susceptible to misinformation.
Article (@YouGov @YouGovAmerica)
Most susceptible
* are young
* heavy social media users
* get their news from social media
(9/15) https://t.co/gG2GM3RJzstoday.yougov.com/topics/politic…
YouGov used the MIST-20 to figure out which Americans are most susceptible to misinformation.
Article (@YouGov @YouGovAmerica)
Most susceptible
* are young
* heavy social media users
* get their news from social media
(9/15) https://t.co/gG2GM3RJzstoday.yougov.com/topics/politic…
Applied: Online Quiz
@YaraKyrychenko developed a quiz app that provides you with a detailed misinformation susceptibility score, including feedback.
What is #YourMIST score?
+ her blog explaining how to develop such app
https://t.co/T7GJaXD247
(10/15)yourmist.streamlit.app
medium.com/behavioral-dat…
@YaraKyrychenko developed a quiz app that provides you with a detailed misinformation susceptibility score, including feedback.
What is #YourMIST score?
+ her blog explaining how to develop such app
https://t.co/T7GJaXD247
(10/15)yourmist.streamlit.app
medium.com/behavioral-dat…
Materials & Implementation Guide
All study and development materials, including a guide on how to implement the MIST and Verification done in any project, can be found on the OSF repository at
(11/15)doi.org/10.17605/OSF.I…
All study and development materials, including a guide on how to implement the MIST and Verification done in any project, can be found on the OSF repository at
(11/15)doi.org/10.17605/OSF.I…
Special thanks to the Team
This project would not have been possible without a fantastic team of collaborators
@FriedrichMGoetz (co-lead)
@GolinoHudson
@roozenbot
@CRSchneider3
@YaraKyrychenko
@SciComGuy
Stefan Stieger
@_WPM3_
Karly Drabot
@james_k_he
@Sander_vdLinden
(12/15)
This project would not have been possible without a fantastic team of collaborators
@FriedrichMGoetz (co-lead)
@GolinoHudson
@roozenbot
@CRSchneider3
@YaraKyrychenko
@SciComGuy
Stefan Stieger
@_WPM3_
Karly Drabot
@james_k_he
@Sander_vdLinden
(12/15)
Methodological Innovation: EGA
Interested in the exploratory graph analysis (#EGA) technique that we used to develop the MIST-16?
See this excellent thread by @GolinoHudson, author of the #EGAnet package, explaining the process in detail.
(13/15)
Interested in the exploratory graph analysis (#EGA) technique that we used to develop the MIST-16?
See this excellent thread by @GolinoHudson, author of the #EGAnet package, explaining the process in detail.
(13/15)
https://twitter.com/GolinoHudson/status/1611382966143569920
Hands-On: Using #GPT for Psychology
#AI for test development? The MIST paper inspired us to work on a hands-on GPT item generation paper:
Let the algorithm speak (with @FriedrichMGoetz @loombasa @Sander_vdLinden)
Psychological Methods (@APA)
(14/15)doi.org/10.1037/met000…
#AI for test development? The MIST paper inspired us to work on a hands-on GPT item generation paper:
Let the algorithm speak (with @FriedrichMGoetz @loombasa @Sander_vdLinden)
Psychological Methods (@APA)
(14/15)doi.org/10.1037/met000…
This work has also been featured in the following article by Forbes
Gen Z And Millennials More Likely To Fall For Fake News Than Older People, Test Finds
@Forbes
(15/15)forbes.com/sites/conormur…
Gen Z And Millennials More Likely To Fall For Fake News Than Older People, Test Finds
@Forbes
(15/15)forbes.com/sites/conormur…
• • •
Missing some Tweet in this thread? You can try to
force a refresh
