Is there more background information available for the third dataset, especially regarding the specific sources/harmonization you used to create the dataset? Very cool initiative
🚨 #Bots and #LLMs threaten the integrity of online surveys and public opinion research, but we can identify LLM-generated text in open narrative responses by fine-tuning #BERT.
New #OpenAccess article with @jkhoehne.bsky.social @rubac.bsky.social @carohaensch.bsky.social.
👉 doi.org/10.1177/0894...
📄 arxiv.org/abs/2412.13169
🗓️ Monday, July 28, 14:00–15:30
📍Room 1.62, Session 3: IP-Orals
Looking forward to seeing you in Vienna #ACL2025! 👋
#LLMs #PublicOpinion #SurveyResearch
w/ Berk @carohaensch.bsky.social @xinpeng.bsky.social Markus @barbaraplank.bsky.social Frauke @assenmacher.bsky.social
doi.org/10.48550/arX... preprint also out now on arxiv since srm has a bit of a backlog
We are waiting to get the proofs but I will send you the manuscript via Email in the meantimd
Leah von der Heyde Bernd Weiss and myself have an accepted manuscript at SRM doing this comparison for several LLMs plus a finetuned model
Johanna Hölzl from Uni Mannheim is working a lot on the reliability of Google Trends; she would for sure answer questions via Email
Similar to other preprints released in the last few months, we want to highlight the challenges of using LLMs for synthetic data creation or even public opinion estomation. While LLMs like GPT-3 hold promise for data synthesis, researchers must currently be very cautious of biases in predictions!
GPT-3's predictions often didn't match actual survey data from the GLES, with a notable variance in accuracy across different political parties. It performed better in predicting voters of center- and left-leaning parties than right-leaning ones. #syntheticdata
Our approach: We created detailed personas reflecting the German voting population, including demographic and political-ideological characteristics. These personas were used to prompt GPT-3 for voting behavior predictions. This approach is similar to a paper by Argyle et al. with ANES data. #LLM
Two researchers discussing "We can now use LLMs to impute oder synthesize values from surveys and other data collection types!" "This is a bad idea, LLMs were never built for such tasks" The author of the tweet says "Let's find out!"
First new research thread on Bluesky!
Leah von der Heyde's, Alexander Wenz's and my newest preprint examines the use of Large Language Models like GPT-3 in synthesizing public opinion data, focusing on German federal elections.
osf.io/preprints/so...
Challenges
More specific points from the feminist perspective
Important questions to ask
But as we say in germany, "da ist noch Luft nach oben" - we can do better
Some framework proposals already exist!
How can bias be mitigated when building systems?
Robots behaviour can serve as a validation for biased human behaviour!
Study asking: What is the average skin color for different professions in generative AI?
Will it all get worse with Generative AI? Probably yes
She first presents some examples from web search engines and other "old" systems
Hanan Salam at the University of Augsburg (HCAI workshop) on biases in AI Systems
An Alps landscape with lake water in the lower half of the picture, afternoon sun shining from behind the mountains, blue fall sky, dreamy atmosphere and a small boat before the shore
Conference posts coming soon but for now just another blue sky over the Königssee in Bavaria from this Sunday
The gothic town hall of Munich with a very blue and cloudless evening sky
Blue sky in Munich :)
Not surprised that academics chose to embrace the social media closest to a paywalled journal.
Due to the high interest in our GOR poster, we put the poster up on OSF :) von der Heyde, L., Wenz, A., & Haensch, A.-C. (2023). Artificial Intelligence, Unbiased Opinions? Assessing GPT’s suitability for estimating public opinion in multi-party systems doi.org/10.17605/OSF...
If you have invite codes for this place to spare, please think about donating them via the following service: docs.google.com/forms/d/e/1F...
Extremely useful:
A poster at a conference with the headline "GPT can't predict how Germans vote"
Sending my first skeet from #dgof_gor #gor23 :) Presented work with Leah von der Heyde and Alex Wenz on data collection through GPT-3! Feel free to take the poster home here: drive.google.com/file/d/1nrS_...
