Science communication has never been more important.
In this animation, @yamilrvelez.bsky.social, Donald Green and I break down our research exploring whether AI chatbots can increase political engagement among young, politically unaligned voters.
Link to animation: www.youtube.com/watch?v=iCuX...
Appreciate the shoutout! I ran an AI image discernment experiment a couple of years ago, but haven’t used images as treatments yet. Would love to see what others have done!
Thanks, Dan!
People feeling hopeless about AI/LLMs spoiling all online polling and survey research should take a look at what @yamilrvelez.bsky.social is doing with “Pulse” — a tool to detect proof of life that’s compatible with Quatrics
An AI Voter bot improves knowledge about politics
But, the AI bot has weak effects on downstream outcomes like vote preferences and party evaluations among respondents whose primary issue position aligns closely with one of the parties.
Partisan action is hard to change.
www.pnas.org/doi/10.1073/...
Though LLMs as persuasion tools are (rightly) getting attention, their most valuable civic use may be retrieval: grounded answers with links to the source material using language that is accessible to users. Positive use cases aren’t dominating the discourse, but that’s where I think the upside is.
In ongoing work w/ Alec Ewig, we explore how RAG can enhance measures of representation by retrieving relevant legislation from a 15,000+ bill database mapped to voter preferences. RAG + agentic workflows can uncover info buried in dense policy docs and answer complex queries with citations.
We used retrieval-augmented generation (RAG), a method that pulls relevant text from curated sources directly into prompts. A vector database, multiple API calls, and a lot of trial and error later, it substantially reduced errors and preserved issue-specific language from party platforms.
When we started this project, standard LLMs weren't up to the task. They produced plausible-sounding answers but regularly hallucinated policy stances, especially on niche or novel issues. For a voting advice application, accuracy was critical, so we had to build a more involved approach.
This paper was a blast to work on. The challenge: present party positions across many issues, in real time, using language voters actually use. 🧵 on why we went with a more involved retrieval-based approach and where I think these tools are headed.
Congrats, Damien!
🚨Excited to share our new paper published in PNAS (joint with @yamilrvelez.bsky.social and Don Green)! AI can enhance political knowledge and provide balanced information about politics with proper guardrails and vetted sources (e.g., party platforms).
www.pnas.org/doi/full/10....
I have a 2024 pre-election Verasight survey I carried out with Alec Ewig that used an adaptive survey method to create a kind of dynamic CES. Happy to chat offline if you think it would be useful!
Looking for a tl;dr for these two excellent papers? I've got you covered: www.science.org/doi/10.1126/...
🚨 New in Nature+Science!🚨
AI chatbots can shift voter attitudes on candidates & policies, often by 10+pp
🔹Exps in US Canada Poland & UK
🔹More “facts”→more persuasion (not psych tricks)
🔹Increasing persuasiveness reduces "fact" accuracy
🔹Right-leaning bots=more inaccurate
All of this is to say that I hope I’m also invited to the party, not only because I care about identifying causal effects, but because I also care about measuring theoretical constructs with a level of precision that quasi-experimental and field experimental designs simply can’t deliver.
As we increasingly interact with GUIs via LLMs, targeted ads, and social media, survey experiments are *even more* fit for the task of understanding human behavior, and I hope we continue relying on them to sort through thorny causal questions.
That’s a useful distinction. Another way to put it is whether a treatment *intervenes* on an outcome. On its face, your standard persuasion experiment looks a lot like “do you prefer [Joe/Jose] for this job?,” but the latter design isn’t trying to shift Y. It’s a measurement exercise.
It's likely not a problem *yet*
Thanks for flagging! I know Brave has several privacy layers. Will look into it.
Thanks! Have to channel my existential panic into something!
Surveys are going to need new ways to establish human presence as AI gets faster and harder to detect. Pulse is one step in that direction.
Special thanks to @seanwestwood1.bsky.social who flagged several spoofing paths. This is exactly the kind of scrutiny we’ll need as we adapt to this new era.
This doesn’t solve the “verify-then-handoff” problem. But it does filter out the most aggressive headless-browser attacks and gives us a first layer of defense in an era where attention checks and gotcha questions fail.
Demo and QSF download here: columbiauniversity.qualtrics.com/jfe/form/SV_...
I'm currently working on Pulse, a “proof-of-life” approach compatible with Qualtrics that uses a simple finger-on-lens verification task to confirm human presence. This approach avoids privacy concerns tied to capturing faces or living environments, and it works on laptops and smartphones.
As @seanjwestwood.bsky.social's terrifying new PNAS article demonstrates, LLMs can now pass almost every attention check, mirror personas, stay consistent across pages, and systematically bias responses in the aggregate.
So here’s a different angle: verify physical presence, not text.
An abstract from Political Analysis discussing a Crowd-Sourced Adaptive Survey System (CSAS) for enhancing public opinion surveys with natural language processing and adaptive algorithms.
#OpenAccess from @polanalysis.bsky.social -
Crowdsourced Adaptive Surveys - https://cup.org/4pNclb0
- @yamilrvelez.bsky.social
I used to believe that survey aesthetics have minimal effects on completion rates and attentiveness… until I saw that slime-green color scheme
The latest issue of PA is out now. We have a great collection of papers by @yamilrvelez.bsky.social, @sysilviakim.bsky.social, @mattblackwell.bsky.social, @sophieehill.bsky.social, @dwlee.bsky.social, @melissazrogers.bsky.social, @kaipingchen.bsky.social, @samuelbaltz.bsky.social (1/2)
Models as Prediction Machines: How to Convert Confusing Coefficients into Clear Quantities Abstract Psychological researchers usually make sense of regression models by interpreting coefficient estimates directly. This works well enough for simple linear models, but is more challenging for more complex models with, for example, categorical variables, interactions, non-linearities, and hierarchical structures. Here, we introduce an alternative approach to making sense of statistical models. The central idea is to abstract away from the mechanics of estimation, and to treat models as “counterfactual prediction machines,” which are subsequently queried to estimate quantities and conduct tests that matter substantively. This workflow is model-agnostic; it can be applied in a consistent fashion to draw causal or descriptive inference from a wide range of models. We illustrate how to implement this workflow with the marginaleffects package, which supports over 100 different classes of models in R and Python, and present two worked examples. These examples show how the workflow can be applied across designs (e.g., observational study, randomized experiment) to answer different research questions (e.g., associations, causal effects, effect heterogeneity) while facing various challenges (e.g., controlling for confounders in a flexible manner, modelling ordinal outcomes, and interpreting non-linear models).
Figure illustrating model predictions. On the X-axis the predictor, annual gross income in Euro. On the Y-axis the outcome, predicted life satisfaction. A solid line marks the curve of predictions on which individual data points are marked as model-implied outcomes at incomes of interest. Comparing two such predictions gives us a comparison. We can also fit a tangent to the line of predictions, which illustrates the slope at any given point of the curve.
A figure illustrating various ways to include age as a predictor in a model. On the x-axis age (predictor), on the y-axis the outcome (model-implied importance of friends, including confidence intervals). Illustrated are 1. age as a categorical predictor, resultings in the predictions bouncing around a lot with wide confidence intervals 2. age as a linear predictor, which forces a straight line through the data points that has a very tight confidence band and 3. age splines, which lies somewhere in between as it smoothly follows the data but has more uncertainty than the straight line.
Ever stared at a table of regression coefficients & wondered what you're doing with your life?
Very excited to share this gentle introduction to another way of making sense of statistical models (w @vincentab.bsky.social)
Preprint: doi.org/10.31234/osf...
Website: j-rohrer.github.io/marginal-psy...
