1/
Okay @IranahPris and #naccb2022, here’s a not so short 🧵
But let’s try not to open Pandora’s Box
***Note: @ConSocSci 🤖 retweets #HumanDimensions #ConSocSci #MarSocSci #EnvSocSci #EnvHum #EnvHis #EnvPsych #ConPsych #SocEcoSys. Original content and replies will be limited.
Okay @IranahPris and #naccb2022, here’s a not so short 🧵
But let’s try not to open Pandora’s Box
***Note: @ConSocSci 🤖 retweets #HumanDimensions #ConSocSci #MarSocSci #EnvSocSci #EnvHum #EnvHis #EnvPsych #ConPsych #SocEcoSys. Original content and replies will be limited.
@IranahPris 2/
If you must do a survey, if it’s the appropriate method, a good starting point is to remember people are just people, not your guaranteed respondents; treat them like people and design your survey accordingly
If you must do a survey, if it’s the appropriate method, a good starting point is to remember people are just people, not your guaranteed respondents; treat them like people and design your survey accordingly
@IranahPris 3/
And the “survey” includes all component of research design, sampling, etc., not just your instrument/questionnaire. DO NOT start with the qnr, start with your purpose, aims, questions, goals
And the “survey” includes all component of research design, sampling, etc., not just your instrument/questionnaire. DO NOT start with the qnr, start with your purpose, aims, questions, goals
@IranahPris 4/
But remember to ask, “Do I really need a survey?”
A survey is the default, go-to method but is more limited and complicated than expected, and we don’t talk about this enough
Will a survey get you what you need or will it just get you some data for data sake?
But remember to ask, “Do I really need a survey?”
A survey is the default, go-to method but is more limited and complicated than expected, and we don’t talk about this enough
Will a survey get you what you need or will it just get you some data for data sake?
@IranahPris 5/
Survey deskilling is also a serious issue with so many DIY platforms Google Forms, Qualtrics, QuestionPro, SurveyMonkey, Alchemer, KoBo
These focus you on implementation & data collection rather than good research design, sampling, & qnr design
Survey deskilling is also a serious issue with so many DIY platforms Google Forms, Qualtrics, QuestionPro, SurveyMonkey, Alchemer, KoBo
These focus you on implementation & data collection rather than good research design, sampling, & qnr design
@IranahPris 6/
If you go no further on this thread, read and follow this:
aapor.org/Standards-Ethi…
And remember: "The quality of a survey is best judged not by its size, scope, or prominence, but by how much attention is given to dealing with the many important problems that can arise."
If you go no further on this thread, read and follow this:
aapor.org/Standards-Ethi…
And remember: "The quality of a survey is best judged not by its size, scope, or prominence, but by how much attention is given to dealing with the many important problems that can arise."
@IranahPris 7/
Important to remember that we must evolve & improve practices with the survey climate and how it changes, which has changed significantly over just the past 10yrs
What worked in 1971 will not work now. What works in US may not work elsewhere
*aapor.org/Education-Reso…
Important to remember that we must evolve & improve practices with the survey climate and how it changes, which has changed significantly over just the past 10yrs
What worked in 1971 will not work now. What works in US may not work elsewhere
*aapor.org/Education-Reso…
@IranahPris 8/
In the US, good survey case studies are the 1936 Literary Digest prez poll, the 1948 Gallup prez poll, and 2016 prez polls.
*researchgate.net/project/Sociol…
*pewresearch.org/fact-tank/2016…
In the US, good survey case studies are the 1936 Literary Digest prez poll, the 1948 Gallup prez poll, and 2016 prez polls.
*researchgate.net/project/Sociol…
*pewresearch.org/fact-tank/2016…
@IranahPris 9/
Quick aside…
“Survey Climate” is a term defined by Loosveldt & Joye 2016, building on the work of Lyberg & Lyberg 1991; Groves & Couper 1998
researchgate.net/publication/30…
Quick aside…
“Survey Climate” is a term defined by Loosveldt & Joye 2016, building on the work of Lyberg & Lyberg 1991; Groves & Couper 1998
researchgate.net/publication/30…
@IranahPris 10/
It is also important to recognize that people are more distracted than ever and receive almost constant survey requests for their “feedback”, "input", or "experience"
*doi.org/10.31235/osf.i…
It is also important to recognize that people are more distracted than ever and receive almost constant survey requests for their “feedback”, "input", or "experience"
*doi.org/10.31235/osf.i…
@IranahPris 11/
Any product we purchase linked to email or phone# will request feedback
In app ratings, receipts, grocery checkouts, pollsters, kiosks in the public bathroom, and even meeting schedulers are feedback requests
surveystance.com/airport-feedba…
Any product we purchase linked to email or phone# will request feedback
In app ratings, receipts, grocery checkouts, pollsters, kiosks in the public bathroom, and even meeting schedulers are feedback requests
surveystance.com/airport-feedba…
@IranahPris 12/
Then throw in research-focused surveys from universities and organizations and the survey commons gets crowded
researchgate.net/publication/26…
Then throw in research-focused surveys from universities and organizations and the survey commons gets crowded
researchgate.net/publication/26…
@IranahPris 13/
Assume at your own risk. Just don’t assume:
1. Your audience is as interested in a topic as you
2. Has anything to gain by taking your survey
3. Can distinguish your survey from marketing
…It is your job to make all this clear
Assume at your own risk. Just don’t assume:
1. Your audience is as interested in a topic as you
2. Has anything to gain by taking your survey
3. Can distinguish your survey from marketing
…It is your job to make all this clear
@IranahPris 14/
Don’t borrow too much language or copy/paste from other surveys. Customize to your audience, context, & purpose
You must develop communicative language that will gain their attention, spark their interest, & legitimize your work and its benefits to them
Don’t borrow too much language or copy/paste from other surveys. Customize to your audience, context, & purpose
You must develop communicative language that will gain their attention, spark their interest, & legitimize your work and its benefits to them
@IranahPris 15/
Avoid pitfalls & make your survey design and communications engaging & salient to your audience.
Humanize it.
Use introspection, self-reflection, & internal assessment of your own survey research methodology and practices
This is essential.
Avoid pitfalls & make your survey design and communications engaging & salient to your audience.
Humanize it.
Use introspection, self-reflection, & internal assessment of your own survey research methodology and practices
This is essential.
@IranahPris 16/
Recognize you most likely aren’t a survey expert; most in conservation, natural resources, sustainability don’t have adequate training
Recognize & admit your limitations, ignorance, or arrogance. Ask for help, ask for a consult, use your network, build a team.
Recognize you most likely aren’t a survey expert; most in conservation, natural resources, sustainability don’t have adequate training
Recognize & admit your limitations, ignorance, or arrogance. Ask for help, ask for a consult, use your network, build a team.
@IranahPris 17/
Another quick aside…
This goes for all aspects of social research design, theory, methodology, methods, analysis, and reporting, not just a survey
If you aren’t an expert, have limited experience, or don’t know, don’t be over-confident.
*researchgate.net/publication/33…
Another quick aside…
This goes for all aspects of social research design, theory, methodology, methods, analysis, and reporting, not just a survey
If you aren’t an expert, have limited experience, or don’t know, don’t be over-confident.
*researchgate.net/publication/33…
@IranahPris 18/
People often worry about response rate (RR), which is a major issue, but only one of many. Response is 1 of 4 cornerstones, the other 3 are sampling, coverage, measurement
*joophox.net/papers/SurveyH…
People often worry about response rate (RR), which is a major issue, but only one of many. Response is 1 of 4 cornerstones, the other 3 are sampling, coverage, measurement
*joophox.net/papers/SurveyH…
@IranahPris 19/
The survey invite, follow-up requests, how you design your survey flow and structure, among many things will effect RR
*Weisberg, H. F. (2009). The Total Survey Error Approach: A Guide to the New Science of Survey Research. University of Chicago Press.
The survey invite, follow-up requests, how you design your survey flow and structure, among many things will effect RR
*Weisberg, H. F. (2009). The Total Survey Error Approach: A Guide to the New Science of Survey Research. University of Chicago Press.
@IranahPris 20/
7 influences on RR: (1) mode, (2) sponsor, (3) response task, (4) incentives, (5) response request structure, (6) communication content, and (7) attributes potential respondents
7 influences on RR: (1) mode, (2) sponsor, (3) response task, (4) incentives, (5) response request structure, (6) communication content, and (7) attributes potential respondents
@IranahPris 21/
…8 is number of “survey” requests people get but hard to know
…9 is some people won’t respond regardless; see de Leeuw et al Survey Attitude Scale
*rdcu.be/cR0Ol
…8 is number of “survey” requests people get but hard to know
…9 is some people won’t respond regardless; see de Leeuw et al Survey Attitude Scale
*rdcu.be/cR0Ol
@IranahPris 22/
Reflect on your survey research practices…
Are you current with tactics that works for audience?
Does your qnr design that make sense for audience?
Are contact and response modes appropriate for audience and context?
Does mixed-mode, web-push, email augment make sense?
Reflect on your survey research practices…
Are you current with tactics that works for audience?
Does your qnr design that make sense for audience?
Are contact and response modes appropriate for audience and context?
Does mixed-mode, web-push, email augment make sense?
@IranahPris 23/
Also ask if your audience is over-surveyed? Is your survey design likely to cause burden and fatigue because of length and complexity. Could your survey be perceived as invasive, lengthy, or useless?
*Pro-tip: it’s probably too long
Also ask if your audience is over-surveyed? Is your survey design likely to cause burden and fatigue because of length and complexity. Could your survey be perceived as invasive, lengthy, or useless?
*Pro-tip: it’s probably too long
@IranahPris 24/
A pre-notification letter or formal survey invite is your first point of contact with your audience (sometimes it is us if we do onsite or drop-off/pick-up)
Take it seriously.
A pre-notification letter or formal survey invite is your first point of contact with your audience (sometimes it is us if we do onsite or drop-off/pick-up)
Take it seriously.
@IranahPris 25/
Know why YOU are doing the survey. Really. It sounds simple but is often overlooked, assumed, or forgotten as you distract yourself with sampling and question development
Know why YOU are doing the survey. Really. It sounds simple but is often overlooked, assumed, or forgotten as you distract yourself with sampling and question development
@IranahPris 26/
Know your audience. Who are they to you and your topic/issue? Why are you asking them and not someone else?
Know your audience. Who are they to you and your topic/issue? Why are you asking them and not someone else?
@IranahPris 27/
Pilot and pretest. Always.
Bonus: cognitive interviews and similar methods are helpful for both the questionnaire and invitation to assure your intended meaning is what a potential respondent interprets
Pilot and pretest. Always.
Bonus: cognitive interviews and similar methods are helpful for both the questionnaire and invitation to assure your intended meaning is what a potential respondent interprets
@IranahPris 28/
Social exchange theory (SET) is a good starting point for invitation letters. Or use another communication theory and principles that will drive and customize how you present and frame the survey purpose and they language you use in your invitation.
Social exchange theory (SET) is a good starting point for invitation letters. Or use another communication theory and principles that will drive and customize how you present and frame the survey purpose and they language you use in your invitation.
@IranahPris 29/
SET is recommended by Dillman, Smyth, & Christian in “Internet, Phone, Mail, and Mixed-Mode Surveys: The Tailored Design Method” (2014)
SET is recommended by Dillman, Smyth, & Christian in “Internet, Phone, Mail, and Mixed-Mode Surveys: The Tailored Design Method” (2014)
@IranahPris 30/
If his book is #tldr, Don is a fan of “How to Conduct Your Own Survey” and the “Savvy Survey Series” from Glenn Israel @UF_IFAS
edis.ifas.ufl.edu/entity/topic/s…
wiley.com/en-us/How+to+C…
If his book is #tldr, Don is a fan of “How to Conduct Your Own Survey” and the “Savvy Survey Series” from Glenn Israel @UF_IFAS
edis.ifas.ufl.edu/entity/topic/s…
wiley.com/en-us/How+to+C…
@IranahPris @UF_IFAS 31/
There are tons of established professional organizations and groups that can be resources
Groups like WERA-1010 have been discussing and working on these issues for over 25 years
nimss.org/projects/view/…
There are tons of established professional organizations and groups that can be resources
Groups like WERA-1010 have been discussing and working on these issues for over 25 years
nimss.org/projects/view/…
@IranahPris @UF_IFAS 32/
If this thread has miscommunicated anything, it's probably b/c it was written quickly
Some highly recommended reads and follow-ups are:
If this thread has miscommunicated anything, it's probably b/c it was written quickly
Some highly recommended reads and follow-ups are:
@IranahPris @UF_IFAS 33/
Guides from @AAPOR (or @ESRAsurvey). Seriously.
aapor.org/Education-Reso…
aapor.org/Standards-Ethi…
https://t.co/x3QLtIDVGO
Guides from @AAPOR (or @ESRAsurvey). Seriously.
aapor.org/Education-Reso…
aapor.org/Standards-Ethi…
https://t.co/x3QLtIDVGO
@IranahPris @UF_IFAS @AAPOR @ESRAsurvey 34/
Pew Research Center @pewresearch has a good YouTube playlist and tons of other resources and examples:
youtube.com/playlist?list=…
Pew Research Center @pewresearch has a good YouTube playlist and tons of other resources and examples:
youtube.com/playlist?list=…
@IranahPris @UF_IFAS @AAPOR @ESRAsurvey @pewresearch 35/
Research and books by Robert Grove, Don Dillman, Jon Krosnick, Jolene Smyth, Sharon Lohr, Mick Couper, Judith de Leeuw are good places to start
Research and books by Robert Grove, Don Dillman, Jon Krosnick, Jolene Smyth, Sharon Lohr, Mick Couper, Judith de Leeuw are good places to start
@IranahPris @UF_IFAS @AAPOR @ESRAsurvey @pewresearch 36/
And check @AASROorg if a nearby university has a survey research center or survey consulting
aasro.org
And check @AASROorg if a nearby university has a survey research center or survey consulting
aasro.org
• • •
Missing some Tweet in this thread? You can try to
force a refresh
