, 68 tweets, 21 min read
My Authors
Read all threads
Hey everyone! One of my professors in my PhD at @hcdeUW is somehow letting me turn in a tweet thread for my final paper...

So here's me reviewing 10 CHI (@sig_chi) papers about Twitter, on Twitter, to understand
1) why CHI people study Twitter, and
2) how they do it.
Before I get started, I want to highlight a Twitter feature I wasn't using before reading @brookeauxier et al "#HandsOffMyADA" paper. Alt text!

You can add alt text to Twitter images in the "Accessibility" section under your "Settings and Privacy" tab. Instructions below: The Twitter toolbar, from “Home” to “More.” More is labeled.Sub-menu, with “Setting and Privacy” labeled.Further sub-menu, with “Accessibility” labeled.Further sub-menu, with “Compose image descriptions” labeled.
I had always known about alt text, but I didn't think about it when it came to Twitter. As @colegleason et. al put it in the title of their paper on Twitter’s alt text:

“It’s almost like they’re trying to hide it"

cs.cmu.edu/~jbigham/pubs/… Screenshot of paper heading that reads: “It’s almost like they’re trying to hide it
If you think it's just me who's late to the game on alt text, just check the #chi2020 hashtag right now.

As of typing, I counted one image out of 30+ posted since decisions came back that had alt text or image descriptions. Most of us could add more!
As a final note on this topic, I wanted some tips on how to write better alt text for images, so I turned to a few blog posts (and ANOTHER tweet thread). Check them out here!

lifeofablindgirl.com/2018/10/31/6-w…
amberley.blog/2019-05-21-acc…
Anyway, back to CHI Twitter papers. I chose 10 papers from CHI 2016-2019, with no repeating first & last authors.

I'm interested in what CHI reviewers and researchers find interesting about large Twitter datasets, so I searched for papers that analyzed at least 10,000 tweets.
My goal selecting these papers was not to understand how researchers CAN work with Twitter data. Other conferences, such as @icwsm, might have been better in that case.

Rather, as a newbie to CHI, I want to know why CHI people write about Twitter, and how they present results.
Due to the 10-paper restriction, I only chose quant & mixed-methods paper for easy comparison. But there are many more qualitative CHI Twitter studies!

To name one, @shsannon et al's paper on
communicating invisible chronic illnesses
on social media.

shrutisannon.com/wp-content/upl… Screenshot of paper heading that reads:
To kick things off, here are the 10 papers I looked at, which a tweet-length summary of their topic:

(sorry if I get things wrong!)
(also sorry for @-ing all of you)
1. @brookeauxier et al.'s 2019 "#HandsOffMyADA" paper, investigating the effectiveness of sharing/retweeting messages about topics affecting the rights of people with disabilities.

researchgate.net/publication/33… Screenshot of paper heading that reads:
2. @melissajbica et al.'s 2019 "Communicating Hurricane Risk" paper, investigating Twitter reactions to different forms of tweeted hurricane risk imagery, and the online diffusion of those risk tweets.

melissabica.com/assets/files/B… Screenshot of paper heading that reads:
3. @smarmyliberal et al.'s 2017 "Algorithms ruin everything" paper, on resistance to algorithmic change during the #RIPTwitter timeline update, and folk theories within that resistance.

researchgate.net/publication/31… Screenshot of paper heading that reads:
4. Christian Drescher et al.'s 2018 "What Moves Players?" paper, on pairing Twitter and gameplay data from @DestinyTheGame to examine which topics matter to players, and identify influential players online, via their own visualization tool!

dl.acm.org/citation.cfm?i… Screenshot of paper heading that reads:
5. @abdoelali et al.'s 2018 "Measuring, Understanding, and Classifying New Media Sympathy on Twitter after Crisis Events." Specifically looks at differences in Twitter coverage from Western and Arab media sources.

arxiv.org/pdf/1801.05802… Screenshot of paper heading that reads:
6. Ankit Kariryaa et al.’s 2018 “Defining and Predicting the Localness of Volunteered Geographic Information using Ground Truth Data.” Uses paired survey data to assess Twitter user location prediction methods.

researchgate.net/profile/Johann… Screenshot of paper heading that reads:
7. Huyen T. T. Le et al.'s 2017 "Revisiting the American Voter on Twitter." Assesses Twitter opinions of candidates in the 2016 US Presidential election via a framework of “party, personality, and policy,” and correlates that to election results.

dl.acm.org/citation.cfm?i… Screenshot of paper heading that reads:
8. @QVeraLiao et al.'s 2016 "#Snowden: Understanding Biases Introduced by Behavioral Differences of Opinion Groups on Social Media." Looks at long term differences in behaviors of pro- and anti-Snowden Twitter communities.

markusstrohmaier.info/documents/2016… Screenshot of paper heading that reads:
9. @merrierm et al.'s 2016 "'With most of it being pictures now, I rarely use it': Understanding Twitter’s Evolving Accessibility to Blind Users" Uses surveys of blind and sighted users and the Twitter firehose to analyze Twitter after it added images.

cs.stanford.edu/~merrie/papers… Screenshot of paper heading that reads:
10. @katestarbird et al.'s 2018 "Engage Early, Correct More: How Journalists Participate in False Rumors Online During Crisis Events." Analyzes journalist behavior compared against regular and power user behavior on Twitter in wake of different crises.

cs.stanford.edu/~merrie/papers… Screenshot of paper heading that reads:
What maybe most united these 10 papers is that they all, at one point, used the Twitter API to scrape tweet data. The data scraped ranged from ~32,000 tweets, in the case of "Engage Early," to 150,000,000+ tweets in the case of "I rarely use it..." A bar chart titled “Number of Tweets Scraped, 10 CHI Twitter Papers (Log Scale)”. The x-axis is paper title, and the y-axis in the number of tweets scraped . “Engage Early” is the shortest bar at around thirty five thousand tweets. Bar heights mostly linearly increase from there up to one hundred million tweets in “I rarely use it.”
6/10 papers used the Twitter Streaming API, one paper used the Twitter Firehose, one paper used the Twitter Search API, one used the “Twitter REST API”, one used Twitter’s user timeline API, one used Twitter’s rehydration API, and one used TweetScraper.

github.com/jonbakerfish/T…
The number of users assessed ranged from only 132, in the case of "Localness" to as many as 11,000,000+ in "American Voter." The total user are almost certainly higher in "I rarely use it," but this number was not stated in the text. A bar chart titled “Number of Users Scraped, 10 CHI Twitter Papers (Log Scale)”. The x-axis is paper title, and the y-axis in the number of users scraped . After short bars for “Predicting Localness” and “I rarely use it,” the bars jump up to ten thousand and rise to ten million.
(Note that some of these user totals are from after authors filtered irrelevant tweets, so the true total of their users' scraped may be higher)
Across these 10 papers, just about every feature of someone's Twitter profile was fair game. They looked at users' tweet text, tweet time, tweet location tags, profile locations tags, hashtag usage, followers/following/friend numbers, follower gain over time, tweeting rate, (1/2)
reply/retweet/quote tweet usage, how often they were retweeted and replied to, profile images, profile descriptions, header images, account ages, and whether their tweets had images/videos/gifs attached (2/2).
Apparently you can only post 25 tweets at a time, who knew 🤔. More coming in a moment!
Seven papers included some sort of qualitative coding or analysis, which was always done on a reduced portion of the scraped' tweets. This ranged from 50 tweets for "#HandsOffMyADA" to 85,000+ for "Communicating Hurricane Risks". Hats off to the coders on that one… A bar chart titled “Number of Tweets Coded, 10 CHI Twitter Papers (Log Scale)”. Only 7 bars are shown, because 3 papers had no codes. The x-axis is paper title, and the y-axis in the number of users scraped . “#HandsOffMyADA” is the shortest bar at around fifty tweets. Bar heights mostly linearly increase from there up to eighty five thousand tweets in “I rarely use it.”
Qualitative coders coded for type of user (e.g. politician or journalist), type of media attached (e.g. screenshot or photo), sentiment underlying tweet text, perspective underlying tweet text, whether tweet text confirmed or denied a fact, and more.
The rest of these tweets will cover the methods, paper organization, contributions, and citation practices these authors used.

But before all that, I'll start with my first question: Why Twitter?
To get at this question, I read these 10 papers for statements about the value of studying Twitter. Some were very explicit about Twitter's research merit, whereas others let the reader assume. I categorized 4 broad, often overlapping reasons to study Twitter. A table titled “Why is Twitter important? 10 CHI Papers.”. The first column contains paper names, and the first row contains reasons for importance. Namely: “Popular and Important,” “Easy to Study”, “Interesting Concept,” and “Hasn’t Been Done.” If a paper has the reason stated given in a column header, their intersecting point is colored in red. “Popular and Important” is the most highlighted at 7, all the others ha
The first is that, simply, a lot of people use Twitter! And maybe more importantly, Twitter is valuable to them. 7/10 papers made statements alluding Twitter's "popularity", and their study subjects "increasingly using Twitter," as motivation for their study.
A second reason is that, compared to other social media platforms, Twitter is easier to study. Its accessible APIs and public-by-default nature presents fewer barriers to researchers than, say, Facebook. 3/10 papers explicitly referenced this fact.
The third is that the Twitter platform itself is an interesting concept! For example, that it’s an algorithm in #RIPTwitter, that it’s risk communication media in Communicating Hurricane Risks, and that it’s potentially biasing technology in #Snowden. (3/10 papers)
And last was that the paper’s chosen methods hadn't yet been applied to Twitter (3/10 papers). The American Voter was motivated by a previous, non-Twitter study, and the potential to reproduce its results. Localness noted that studies of localness were rare on Twitter.
So, Twitter: it’s popular, it’s easy, it’s interesting, and it hasn’t been done.
The next thing I looked for in these papers was how they structured their analyses. How did they move from massive datasets of scraped Twitter data to their final conclusions?
Inspired by this diagram in “News Media Sympathy,” I drew similar, albeit simpler, diagrams for each of the other 9 papers in this list, and categorized them from there. A screenshot of box flow chart from a paper. Boxes flow from “Crawling and Preprocessing” to “Extracting News Media Tweets” to “Processing” to “Processed Datasets” to “Crowdsourcing” to “Analysis and Prediction.” In each of these sections, there are several sub-boxes with more detail. A caption reads “Figure 1: Overview of methodological pipeline.”
First four diagrams… Flow diagram for #RIPTwitter. Start from tweet data, is qualitatively coded, then coded by machine learning, and then analyzed.Flow diagram for Engage Early. Starts with tweet data, leads to qualitative coding, which then leads to analysis.Flow diagram for Communicating Hurricane Risk. Starts from tweet data, has two sessions of qualitative coding, with both ending in analysis.Flow diagram for #Snowden. Originates in Tweet Data, and splits off into two different qualitative coding sessions. Analysis continues on both branches.
Next four diagrams... Flow diagram #HandsOffMyADA. Start with Tweet data, which is analyzed, and then also qualitatively coded and analyzed. It is then combined with older tweet data and analyzed further. Historical analysis is conducted outside of the main flow.Flow diagram for What Moves Players. Starts from Tweet and Destiny game data, and moves to data analysis.Flow diagram for Localness. Starts from survey data, moves to tweet data, and ends in analysis.Flow diagram for American Voter. Starts with tweet data, leads to analysis.
Last one! Flow diagram for I rarely use it. Start with survey data, which leads to analysis and tweet data. The tweet data leads to qualitative coding and more analysis. Analysis from the survey and tweet data combine into an ML model. A separate analysis via a Twitter firehose dataset is conducted.
(This is how they looked originally, by the way) A pile of papers on top of a desk. They have flow diagrams scrawled in bad handwriting all over them. For some reason, I was taking notes on bridge scorecards, which is noticeable from the picture.
Six of these papers (News Media Sympathy, Localness, What Moves Players, #RIPTwitter, Engage Early, American Voter) followed what I'm calling a linear analysis structure. They start with a dataset, and move progressively from that dataset to derivative analyses.
Four papers (#Snowden, #HandsOffMyADA, Communicating Hurricane Risk, I Rarely Use It) had a different structure. Multiple datasets, multiple rounds of qualitative coding -- somehow they break off from the straight line.
“I rarely use it” is a notable departure, analyzing in parallel survey data, a user-driven scraped set of user data, their qualitative codes on that data, and then a separate massive dataset from the Twitter firehose to draw conclusions about Twitter’s accessibility.
#Snowden” and “Communicating Hurricane Risk” both perform two parallel qualitative coding experiments on the original dataset, breaking the analysis into two pieces.
#HandsOffMyADA” does the traditional linear analysis on a scraped Twitter dataset, but then compares it with an older, different dataset for its “Comparative Analysis” section. It also has a “Historical Analysis” section that uses no Twitter data.
I wanted to specifically note the “Historical Analysis” section in “#HandsOffMyADA”, which recounts a history of disability rights activism. This section is unique among the 10 papers, and I found it a great supplement to Quantitative/Qualitative analysis. A screenshot of #HandsOffMyADA. At the top of is “Table 1: Disability Rights Timeline,” which starts in 1964 with the Civil Rights Act, proceeds eventually to the Americans with Disabilities Act (ADA) in 1990, and then HR620 - ADA Education and Reform Act (Passed The House) in 2018. Below it, a section titled “Historical Analysis” starts and is cut off. It begins, “In the US, civil rights for people with disabili...
So that was how papers got to their results. My next question was: who were these results written for? And what sort of contribution do its authors want to produce?
To get at these questions, I particularly reread sections in the Introduction, Background, Discussion, and Conclusion. I looked for if a paper audience was explicitly mentioned, who that audience was, and how the authors' titled their own contributions.
Some papers were very explicit about their audience, like "game community managers" or my favorite, "[those who work at] Twitter." Others targeted broad communities like the “HCI community” or “researchers and practitioners.”
Papers that did not explicitly call out an audience still implied one via their framing questions. These questions were asked by different groups, like journalists, social media developers, or political scientists.
“Communicating Hurricane Risks” was unique, though, in that it was the only paper that explicitly targets policy-makers in its contribution. Along with “What Moves Players,” it also had one of the most explicit and detailed implications sections among the 10 papers.
I then categorized how authors described their contributions. All authors to some extent were contributing Knowledge to a field, but some papers also explicitly contributed three other categories: Recommendations, Best Practices, and Issues. A table titled “Contribution Types. 10 CHI Twitter Papers.”. The first column contains paper names, and the first row contains reasons for importance. Namely: “Knowledge,” “Recommendations”, “Best Practices”, and “Issues.” If a paper has the reason stated given in a column header, their intersecting point is colored in red. The Knowledge column is entirely red, so all 10 papers contribute knowledge.
Recommendations tended to be aimed towards practitioners. They translate the knowledge produced into steps or guidelines people can take, like adding alt text to Twitter or promoting streamers in Destiny.
Best practices were aimed at researchers and practitioners alike, and described an explicit methodology to accomplish a task. "Localness" gives best practices on determining user location, while "#Snowden" gives best practices for estimating amplification on Twitter.
Finally, "issue" contributions highlight unresolved problems exposed by the paper. "#HandsOffMyADA" highlights difficulties disability activists have in promoting movements on Twitter. "Classifying News Media" shows that crises in the Beirut receive less coverage than Paris.
My reading found that 10/10 papers contributed knowledge (duh), 4/10 gave specific recommendations, 2/10 provided best practices, and 4/10 contributed issues.
For my final analysis, I wanted to look at the citation practices of these papers. What scholarly communities were they coming from?
To do this, I copied each of their lists of references available on ACM's website, and did some "open coding" (don't kill me) to sort them into broad categories. A chart titled “Distribution of Citations by Type, 10 Twitter CHI Papers.” It’s a stacked bar chart, where bars of equal height, each corresponding to one paper, are divided into colors corresponding to citation types. Citation types include: “Non-Academic,” “Other Academic,” “ICWSM”, “CSCW”, and “CHI.” “I rarely use it” has the greatest number of non-academic sources, followed by “#HandsOffMyADA.”
Three conferences kept on coming up: CHI, CSCW, ICWSM, so I counted those. I also kept track of non-academic citations to news and websites, total academic citations outside of those three conferences, and the general fields papers were citing out to.
Most papers had 50-65 citations, with two in the 30s. They cited out to other fields, including disability studies, news studies, communication, game studies, risk communication journals, machine learning studies, algorithm studies, visualization journals, and political science.
It was hard to pull out trends in the citation types, but here's trying! Those papers with more non-academic citations tended to either cite websites associated with tech platforms (Twitter, Destiny), or news associated with their subject (Disability Rights, US Politics).
Those who cited CSCW more tended to have a prior history at CSCW, and these citations contain their authors' previous work. Those who cited ICWSM more tended to cite similar subjects (particularly evaluating people's political preferences from social media data).
I wasn't able to discern much from consistently high CHI citations ¯\_(ツ)_/¯ (emoji caption: shrug!). I do find interesting the level of variety in each of these citation categories from paper to paper, though.
And that’s it! There’s plenty of subjects I didn’t hit on in this thread, which members of our class at @hcdeUW did: ethics/justice of papers, trustworthiness of papers, theoretical and epistemological grounding of papers.
For justice and trustworthiness, I think I’ll hold off before doing a public tweet thread on those 🙃. For theory and epistemology, well, I take Theory next quarter (something something the Streaming API is a boundary object).
Also, sorry to the authors if I got anything wrong! Or if I made a mistake with the charts. I loved every paper. Just having a little fun with at the end of the quarter, and happy to update this thread if I misread your work.
Missing some Tweet in this thread? You can try to force a refresh.

Enjoying this thread?

Keep Current with Andrew Beers

Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

Twitter may remove this content at anytime, convert it as a PDF, save and print for later use!

Try unrolling a thread yourself!

how to unroll video

1) Follow Thread Reader App on Twitter so you can easily mention us!

2) Go to a Twitter thread (series of Tweets by the same owner) and mention us with a keyword "unroll" @threadreaderapp unroll

You can practice here first or read more on our help page!

Follow Us on Twitter!

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just three indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3.00/month or $30.00/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!