Thread by @rapplerdotcom

[WATCH LIVE] #FactsFirstPH briefing: ‘The Emergence of Political Polling Through Kalye Surveys’ #PHVote #WeDecide

On today's #FactsFirstPH research briefing, Ma. Rosel San Pascual, Anjenica Durana, and Myrnelle Cinco outline their findings on the conduct of kalye surveys. | via @reetuquero

#PHVote #WeDecide

@reetuquero The researchers have recorded at least 350 kalye survey videos that have already been uploaded to YouTube. | via @reetuquero

#PHVote #WeDecide #FactsFirstPH

@reetuquero They reviewed a total of 43 kalye survey videos for this study. | via @reetuquero

#PHVote #WeDecide #FactsFirstPH

@reetuquero Lead author San Pascual says PMM has scanned over 350 'kalye survey' videos on YouTube since May 2021. The study used YouTube's API to go through the most-watched video in the platform. | via @aloisinika

#PHVote #WeDecide #FactsFirstPH

@reetuquero @aloisinika San Pascual: Some 'kalye surveys' do not hide their hyperpartisanship.

Since 'kalye surveys' are packaged differently vs formal poll research, should they be expected to follow the 'gold standard' of survey research? The team says yes. | via @aloisinika

@reetuquero @aloisinika San Pascual: Formal surveys use a standardized questionnaire, which contains thoroughly-designed questions. 'Kalye surveys,' on the other hand, use unstandardized questions, such as inconsistent phrasing of questions. | via @aloisinika

#FactsFirstPH

@reetuquero @aloisinika San Pascual: The gold standard in surveys adheres to the principles of scientific and ethical practice across the entire phase of research... When a survey fails to follow the gold standard of practice, survey results could lead to misinformation. | via @reetuquero #FactsFirstPH

@reetuquero @aloisinika San Pascual: Some 'kalye surveys' do not hide their hyperpartisanship.

Since 'kalye surveys' are packaged differently vs formal poll research, should they be expected to follow the 'gold standard' of survey research? The team says yes. | via @aloisinika #FactsFirstPH

@reetuquero @aloisinika San Pascual says kalye surveys chronicle the survey enumerator's administration, including their interactions and conversations with the respondents, unlike mainstream political polling where data collection is neither documented on video nor publicly disseminated. #FactsFirstPH

@reetuquero @aloisinika San Pascual: Formal surveys use a standardized questionnaire, which contains thoroughly-designed questions. 'Kalye surveys,' on the other hand, use unstandardized questions, such as inconsistent phrasing of questions. | via @aloisinika #FactsFirstPH

@reetuquero @aloisinika San Pascual: 'Kalye survey' enumerators also use off-script language when asking questions about election preference. They sometimes feed the answer itself.

These introduce bias, which compromises the onjectivity of the 'survey,' says San Pascual. #FactsFirstPH

@reetuquero @aloisinika San Pascual asks, since kalye surveys are packaged as "katuwaan lang," should they still be expected to follow the same gold standard? "Our team believes so, especially if the intended outputs of the survey is to serve the purpose of public information and vote choice guidance."

@reetuquero @aloisinika San Pascual: But our team also optimistically believes that the gold standard can be learned and incorporated in any survey practice to validly and reliably contribute to sense-making. | via @reetuquero #FactsFirstPH

@reetuquero @aloisinika San Pascual also points out instances where 'kalye survey' enumerators argue vs the person being asked. | via @aloisinika #FactsFirstPH

@reetuquero @aloisinika San Pascual: When a survey fails to follow the 'gold standards' of practice, survey results could lead to misinformation. | via @aloisinika #FactsFirstPH

@reetuquero @aloisinika San Pascual shows examples of unstandardized questions used by enumerators, or varying ways of asking respondents on their presidential vote choice | via @reetuquero #FactsFirstPH

@reetuquero @aloisinika Some also make additional comments based on respondents' answers | via @reetuquero #FactsFirstPH

@reetuquero @aloisinika The researchers also noted instances of hyperpartisan comments from enumerators. | via @reetuquero #FactsFirstPH

@reetuquero @aloisinika San Pascual: There's a formula used for computing the sufficient number of respondents for the study to be able to make inferences. The manner of selection among the target selection is also important. | via @aloisinika #FactsFirstPH

@reetuquero @aloisinika San Pascual: All these are dangerous because we want our surveys to capture our data objectively. And survey literacy training can actually help explain the importance of objectivity. | via @reetuquero #FactsFirstPH

@reetuquero @aloisinika San Pascual: Since content creators function as enumerators, literacy training can help them realize how they can affect respondents' answers, which consequently introduces bias. | via @reetuquero #FactsFirstPH

@reetuquero @aloisinika San Pascual: In 'kalye survey,' respondents are chosen through non-random sampling. The 'host' of the 'survey' simply intercepts people on the street. | via @aloisinika #FactsFirstPH

@reetuquero @aloisinika San Pascual gives an example where a respondent said, 'Hindi po ako taga-rito,' but the 'kalye survey' video framed the content as the pulse of a particular locality. | via @aloisinika #FactsFirstPH

@reetuquero @aloisinika Can surveys be designed 'just for fun'?

San Pascual says there are pop surveys existing, notes the public interest component of surveys especially those that pertain to elections. | via @aloisinika #FactsFirstPH

@reetuquero @aloisinika On sampling, San Pascual says: Size is not enough... You should also be asking the manner of collecting respondents, or selecting respondents, or drawing respondents from your population pool. #FactsFirstPH

@reetuquero @aloisinika San Pascual says it is evident from the kalye survey videos that the respondents were selected through convenience sampling technique, a non-random manner that has repercussions on the results' potential for generalization. #FactsFirstPH #PHVote

@reetuquero @aloisinika San Pascual points out the clickbait and sensationalized titles of videos, when the content itself does not support the premise presented in the title. #FactsFirstPH #PHVote

@reetuquero @aloisinika San Pascual: Kalye surveys should be careful when framing the results of their surveys so as not to misinform the viewers that their results speak for a group beyond the sample of respondents. #FactsFirstPH #PHVote

@reetuquero @aloisinika Can surveys with respondents drawn from non-random sampling techniques be considered scientific? San Pascual says social science research allows for it, but data from such samples must only be used to describe the pool of respondents and not the larger population. #FactsFirstPH

@reetuquero @aloisinika They noted an instance where one respondent even took over the survey enumeration. She led the enumerator to an area where she believed more people would prefer a certain candidate. San Pascual said this introduced sampling bias. #FactsFirstPH #PHVote

@reetuquero @aloisinika San Pascual raises ethical concerns on video documentation done in 'kalye surveys,' where the host would not ask for the respondent's permission to be filmed and posted on social media.

This runs counter to the principle of informed consent in formal surveys. #FactsFirstPH

@reetuquero @aloisinika San Pascual: If the survey can only manage gathering data through non-random sampling, then the framing of the video title and the presentation of results should not mislead the viewers that the results represent the larger population #FactsFirstPH #PHVote

@reetuquero @aloisinika When reading survey results, one should look at the sampling design. San Pascual notes the importance of looking at the methods, even the phrasing of the questions in the questionnaire. #FactsFirstPH #PHVote

@reetuquero @aloisinika The researchers said most kalye survey enumerators have a board as a "show card" where they record their tally of responses, which succeeding respondents can see. #FactsFirstPH #PHVote

@reetuquero @aloisinika San Pascual says this practice not only runs contrary to the data privacy thrust, but also potentially creates a bandwagon effect or underdog effect. #FactsFirstPH #PHVote

@reetuquero @aloisinika The researchers also noted instances when the manner of survey administration can create a groupthink effect. #FactsFirstPH #PHVote

@reetuquero @aloisinika San Pascual also shows "concerning" examples of how some kalye surveys appear to assume that respondents consent to be recorded on video and have it uploaded to YouTube. This runs against the principles of informed consent, confidentiality, and data privacy.

@reetuquero @aloisinika "Survey literacy should empower our agency as members of the public so that we can demand accountability among survey groups that deliberately misuse surveys for misinformation," San Pascual says as she ends her discussion.