New publication with Annika Lindholm & Lukas Lauener. @lukaslauener.bsky.social
We have just published our research note on survey modes in the Swiss Political Science Review (SPSR): onlinelibrary.wiley.com/doi/10.1111/...
#SurveyMethods #PoliticalScience #Selects #MixedMode #SPSR
@jobvanexel.bsky.social @qmethodology #qmethods #surveymethods #Qmethodology #subjectivity
www.gcu.ac.uk/research/res...
- Four attention check questions: The classic "please select 'strongly agree' to show you're reading this" type questions. - Two consistency checks: Paired questions where humans should answer in predictable patterns. For example, if the answer to one question was true, the other had to be false (and vice versa) because it indicates consistency. - Two reverse shibboleth items: Questions that your intended sample (i.e., humans) should not know the answer to, but an AI agent likely would. - Three cognitive traps: These are visual illusions designed to exploit fundamental limitations in vision-enabled AI systems. We used a modified Müller-Lyer illusion, a modified cafe wall illusion, and a ‘moving robot task’, all of which have been shown to be highly discriminative between AI agents and human participants. - Comprehensive mouse tracking: Recording every cursor movement and click pattern throughout the survey. - Qualtrics' reCAPTCHA score: The platform's built-in risk scoring system.
We tested five different AI agents, each completing the survey 25 times to match our human sample size: - GPT Agent: OpenAI's GPT configured to complete surveys - Claude: Anthropic's Claude accessed via Cursor - Perplexity: Perplexity AI's search-enhanced agent - Gemini: Google's Gemini via Project Mariner - Custom Agent: Our in-house ‘white hat’ agent, designed by Prolific’s AI research engineers to take surveys and avoid detection The custom agent was crucial to include because it’s specifically designed to take surveys undetected. It’s closer to Westwood’s bot (2025) as an adversarial, customisable agent than the more readily detectable commercial AI agents. To take part in the survey, we provided all agents with an identical prompt that asked them to complete the survey as a human would and exhibit human-like behaviour.
Prolific’s bot authenticity check (100% accurate) Mouse tracking (95.0% accurate) Qualtrics’ reCAPTCHA score (94.2% accurate) Cognitive traps (85.2% accurate) Consistency checks (62.7% accurate) Classic attention checks (59.7% accurate) Reverse shibboleth questions (58.0% accurate)
Can researchers detect #AI bots taking paid surveys?
#Prolific tested humans and #LLM agents with various #dataQuality checks.
- The company says they caught 100% of the non-humans.
- My take-away: #reCAPTCHA and #mouseTracking caught 95%
www.prolific.com/res...
#surveyMethods
An eye‑opening survey: 74% of developers said they’d ship a feature that restricts rights if pressured. Let’s talk ethics, incentives, and how methods shape results. #AISlopEconomy #DeveloperEthics #HumanRights #SiliconValley #SurveyMethods
Questionnaire Design and Survey Research This workshop will provide some basic principles in questionnaire/survey design and give workshop participants an opportunity for hands-on experience designing a questionnaire. Following participating in this workshop, learners will be able to: - Design a blueprint for a survey/questionnaire appropriate to their own application; - Construct and edit questions to avoid common problems in wording and framing; - Select an appropriate response format from a menu of alternatives; - Design the overall format of the survey/questionnaire to facilitate data management and analysis.
Hey #SurveyMethods and #MedEd folks:
In a workshop for #MedSchool faculty about questionnaire design and survey research methods, what
- objectives should be prioritized?
- materials are out there?
- activities are worth it?
- assessments work?
Example: www.aamc.org/what-we...
In a sample of over 5000 English speakers recruited from six continents via Google ads, attention scores were 2.6 out of 3 and ReCAPTCHA (v3) scores 0.94 out of 1.0 (on average).
We've found good results on overt and covert #dataQuality measures by recruiting people via #onlineAdvertising (perhaps because participation incentives aren't financial):
Attention ≅ 2.6 out of 3
ReCAPTCHA (v3) ≅ 0.94 out of 1.0
doi.org/10.1017/S003...
#surveyMethods #cogSci #psychology #xPhi
The #AI survey "respondent’s performance rendered [attention quality] checks [ACQs] effectively obsolete. Across 6,000 total trials..., [it] committed only 10 errors, achieving an overall pass rate of 99.8% and scoring perfectly on 18 of the 20 ACQ types."
#surveyMethods #psychometrics #psychology
We welcome prospective researchers with a passion for population change, social statistics, and advancing methods and data for impactful research.
#PhDOpportunity #ESRC #SocialStatistics #Demography #PopulationHealth #PopulationStudies #SurveyMethods #ResearchFunding
Imagine you want understand how people respond to Al advice • Participants guess the age of a person in a photo. • They see advice from Al with a number and explanation. • They get to revise their answer. • The photo comes from a pool of 200 photos. • They repeat it six times with six different photos. Questions: In Qualtrics, • How long would it take to set up? • What could go wrong?
One-slide Summary • Researchers increasingly need to present stimuli that are highly customized. • Instead of manually adding stimuli into Qualtrics one-by-one, you can upload them, using a "Stimulus Table" to a Qualtrics List. • The Stimulus Table contains details on all combinations of stimuli you want to use. • We detail a six-step process. • We provide examples and a tutorial.
Can you test a ton of stimuli without #Python, #Otree, etc.?
Angshuman Pal, Christina Rader, and Asa B. Palley can!
All you need is:
- a spreadsheet (.csv)
- Qualtrics' Directories/Contact list function
Try it yourself with their materials of two examples at osf.io/vy27c/overvi...
#surveyMethods
📘 𝗡𝗲𝘄 𝗦𝗽𝗲𝗰𝗶𝗮𝗹 𝗜𝘀𝘀𝘂𝗲 𝗼𝗳 𝙎𝙪𝙧𝙫𝙚𝙮 𝙍𝙚𝙨𝙚𝙖𝙧𝙘𝙝 𝙈𝙚𝙩𝙝𝙤𝙙𝙨:
“Survey Climate and Trust in Scientific Surveys.”
Explore how the relationship between surveys and society has evolved.
🔗 Read the 𝗳𝘂𝗹𝗹 𝗶𝘀𝘀𝘂𝗲 here:
ojs.ub.uni-konstanz.de/srm/issue/vi...
#SurveyResearch #SurveyMethods #SurveyClimate #ESRA
Nisbett, R. E., & Wilson, T. D. (1977). Telling more than we can know: Verbal reports on mental processes. Psychological Review, 84(3), 231–259. https://doi.org/10.1037/0033-295X.84.3.231
Xie, H., Xiong, H.-D., & Wilson, R. (2024, August 26). From Strategic Narratives to Code-Like Cognitive Models: An LLM-Based Approach in A Sorting Task. First Conference on Language Modeling. https://openreview.net/forum?id=1Tny4KgGO2
Zhang, Z., Xie, H., Baker, T. E., Peters, M., & Wilson, R. (2025). Linking Strategies to Think Aloud in A Stochastic Learning Task. Proceedings of the Annual Meeting of the Cognitive Science Society, 47(0). https://escholarship.org/uc/item/4sb936m3
Can thinking aloud accurately capture how we decide?
Nisbet & Wilson's 1977 paper famously suggested it can't.
But #NLP and #AI methods may indicate that it can:
- escholarship.org/uc/...
- openreview.net/forum...
#cogSci #psychology #philosophy #surveyMethods #thinkAloud #LLM
Can anyone pls point me towards some resources for establishing survey sample sizes? I'm doing a discrete choice experiment and want to run a pilot for validity, then establish N based on responses. Also some subgroups I'm keen to examine. Ideally #Bayesian!
#surveymethods #stats #statsky #econsky
📢 𝗡𝗲𝘄 𝗶𝘀𝘀𝘂𝗲 𝗼𝗳 𝙎𝙪𝙧𝙫𝙚𝙮 𝙍𝙚𝙨𝙚𝙖𝙧𝙘𝙝 𝙈𝙚𝙩𝙝𝙤𝙙𝙨 𝗼𝘂𝘁 𝗻𝗼𝘄!
The latest issue (Vol. 19, No. 2) of 𝘚𝘶𝘳𝘷𝘦𝘺 𝘙𝘦𝘴𝘦𝘢𝘳𝘤𝘩 𝘔𝘦𝘵𝘩𝘰𝘥𝘴 is here – featuring new research on survey design, data quality, sampling, nonresponse, & more.
See the issue:
🔗 ojs.ub.uni-konstanz.de/srm/issue/vi...
#SurveyResearch #SurveyMethods #ESRA
🎓 Not just what students expect—but how sure they are.
Dr. Stephen Morgan’s #Socius study shows that tapping uncertainty improves predictions of #college access. Insightful for #SurveyMethods and #SociologyOfEducation.
Read: doi.org/10.1177/2378...
#ComparativeResearch #EducationMeasurement #SurveyMethods
Measuring education across countries is complex—but crucial for valid, comparable survey data. This study tests 16 coding strategies using ESS data and finds a strong contender for a new international standard.
📝Reimagine surveys with AI!
Join Wiebke Weber's course at #RECSM Summer School. This course blends classic practices on survey design with cutting-edge AI tools.
📅Jun 30–Jul 1
📍Online & On-site in Barcelona
Early bird discounts until May 25th!
#SurveyMethods #AI
🔗 www.upf.edu/web/survey/2...
Follow @awolaver.bsky.social and Johny Doces. They've more interesting #surveyMethods #research from the continent of #Africa in the pipeline!
🎓 Dr. Wolaver scholar.google.com/citations?us...
🎓 Dr. Doces scholar.google.com/citations?us...
#health #econ #poliSci #culture #psychology #cogSci #xPhi
Does concerned mean they think he will do it or does it mean the act of jailing citizens worries them?
#SurveyMethods
Differences in framing effects and cognitive reflection test performance in Study 1 and in cost/participant in Study 2.
Differences in conjunction fallacy, reflection test performance, incentives, attention, and pre-experiment activities for Study 2.
Differences in exclusion rates (between lenient and strict exclusion policies) by platform and differences in psychological scales (like self-esteem, need for cognition, and self-control) for study 1.
Differences in psychological scales, reflection test performance, conjunction fallacy, attention, etc. for Study 2.
New #surveyMethods paper replicates and extends differences in #dataQuality, attention, naivety, decision style, etc. by
- online #research recruitment platform (#mTurk, #Prolific, #Qualtrics, #Pollfish)
- device (#mobile v. #desktop)
- person's incentive
doi.org/10.3758/s134...
Blog Post 🚀
Learn how to explore longitudinal data in R using tables & summaries. 📊
Step-by-step guide to navigating and understanding your data with ease.
@usociety.bsky.social @ncrm.ac.uk
#LongitudinalData #RStats #DataScience #SurveyMethods #DataExploration
Blog Post! 🚀
Master longitudinal data visualization in R with this complete guide. 📊
Learn how to create insightful plots for impactful analysis.
@usociety.bsky.social @ncrm.ac.uk
#LongitudinalData #RStats #DataScience #SurveyMethods #DataVisualization
Blog Post! 🚀
Understand the longitudinal data workflow—from import to analysis. 📊
Step-by-step guide to mastering data preparation and research.
@usociety.bsky.social @ncrm.ac.uk
#LongitudinalData #RStats #DataScience #SurveyMethods #DataWorkflow
Join Longitudinal Data Analysis – live & online with @essexsummerschool.bsky.social! 📊
📅 21.07 – 1.08.2025 | 💻 Online
Learn to analyze change over time using advanced statistical models.
#LongitudinalData #RStats #DataScience #SurveyMethods #PanelData #SocialResearch
Join Longitudinal Data Analysis – live & online with @essexsummerschool.bsky.social! 📊
📅 21.07 – 1.08.2025 | 💻 Online
Learn to analyze change over time using advanced statistical models.
#LongitudinalData #RStats #DataScience #SurveyMethods #PanelData #SocialResearch
📢 New Blog Post! 🚀
Learn how to include time-varying predictors in multilevel models for longitudinal data. 📊
@usociety.bsky.social @ncrm.ac.uk
#LongitudinalData #RStats #DataScience #SurveyMethods #MultilevelModels
Blog Post! 🚀
Learn how to include time-constant predictors in Latent Growth Models (LGM) using R. 📊
@usociety.bsky.social @ncrm.ac.uk
#LongitudinalData #RStats #DataScience #SurveyMethods #LatentGrowthModels
Join Longitudinal Data Analysis – live & online with @essexsummerschool.bsky.social! 📊
📅 21.07 – 1.08.2025 | 💻 Online
Learn to analyze change over time using advanced statistical models.
#LongitudinalData #RStats #DataScience #SurveyMethods #PanelData #SocialResearch
Blog Post! 🚀
Learn how to include time-varying predictors in Latent Growth Models (LGM) in R. 📊
@usociety.bsky.social @ncrm.ac.uk
#LongitudinalData #RStats #DataScience #SurveyMethods #LatentGrowthModels
