#ProcessTracing: "in-depth theory-based case study method that helps evaluators empirically trace the process through which an intervention produces a contribution. It looks not just at results, but at..mechanisms, interactions, & contextual factors that bring about those results" #Evaluation #MandE
Qualtrics Mouse Tracing (QMT) may be the only scalable, code-free way to track mouse activity during experiments, according to Steph Smith (U Chicago).
"If you want complete and total flexibility for your study design, we have a step-by-step version with extensive documentation!" – Steph Smith (U Chicago)
Does the search pattern predict choice? (Yes) • Outcome: Proportion of choices for higher expected-value option • Predictor: Payne Index {option-wise transitions - attribute-wise transitions}
Steph Smith's collaborator is David Dolifka. Find their code, Qualtrics files, etc. on GitHub with the QR code or the URL in the post.
Many cognitive scientists change decisions; few look in the black box to see HOW.
Steph Smith demoed such #processTracing
...in #Qualtrics!
…without any coding!
Find everything you need at github.com/QMT-code/QMT
Follow her work at www.researchgate.net/profile/Step...
#eyeTracking #mouseTracking
Participants and data
Procedure
Results (1 of 2)
Results (2 of 2)
Do reflection test solutions actually involve reflection?
Our think-aloud studies found they usually do (doi.org/10.14264/0f1...), but Ryan Jesson found that solution-prompting insight is often unconscious or spontaneous.
doi.org/10.14264/0f1...
#ProcessTracing #psychometrics
Process Tracing in action! 🤩
Big thanks to Hilde van Meegdenburg and all participants for a vibrant MethodsNET course at CEU @weareceu.bsky.social last week.
Lots of learning, exchange and a creative “PT” hand sign in the front row! 💜
#methods #research #summerschool #ceu #methodsnet #processtracing
📊🔍 Ever wondered how to trace the real impact of complex interventions like policy advice or knowledge work? Process tracing offers a powerful, theory-based way to uncover causal links—especially when outcomes are hard to measure. Learn more: buff.ly/oy6s0ux
#Evaluation #Impact #ProcessTracing
Some examples of the "aha" moments in allegedly "self-reflection" output from AI reasoning models. "An additional important question is whether self-reflection behaviors are associated with improved model performance after RL training. To investigate this, we host DeepSeek-R1-Zero and analyze its responses of the same questions from MATH dataset. While self-reflection behaviors occur more frequently in R1-Zero, we observe these behaviors are not necessarily imply higher accuracy. Detailed analysis can be found in App. D."
Self-reflection does not necessarily imply higher accuracy. To investigate whether self-reflection behaviors are associated with model performance during the inference (acknowledging that self-reflection may improve exploration during training—a potential positive effect outside this section’s scope), we analyze questions that elicit at least one response with self-reflection from DeepSeek-R1-Zero across eight trials. For each question, we sample 100 responses and divide them into two groups: those with self-reflection and those without. We then compute the accuracy difference between these two groups for each question. As shown in Fig. 13, the results indicate that nearly half responses with self-reflection do not achieve higher accuracy than those without self-reflection, suggesting that self-reflection does not necessarily imply higher inference-stage accuracy for DeepSeek-R1-Zero.
What keywords or phrases count as "self-reflection"? Here's what the paper reports: "terms like “wait” and “try again” frequently result in false positive detections. To reduce false positives, we maintain a small, highly selective keyword pool consisting of terms that are strongly indicative of self-reflection. In our experiment, the keyword pool is limited to: recheck, rethink, reassess, reevaluate, re-evaluate, reevaluation, re-examine, reexamine, reconsider, reanalyze, double-check, check again, think again, verify again, and go over the steps."
#AI reasoning models may seem to reason reflectively when they say things like, "Let me rethink that".
But do these "reflective" phrases predict better reasoning performance?
Not in #Deepseek R1 Zero: doi.org/10.48550/arX...
#cogSci #decisionScience #processTracing #psychology
Observational protest data
Experimental results
Mouse-tracking results
Eye-tracking results and a visualization tool that "attenuated the gap between people’s minimum wage prescriptions" (N ≅ 2000).
Why can economic #inequality depress the #minimumWage?
An is-ought #fallacy?
From over 135,000 people in #protests, experiments, and #processTracing studies, scientists found that people seemed to infer what people OUGHT to earn from what they DO earn.
doi.org/10.1037/xge0...
Today is your last chance to register for the @methodsnet.bsky.social summer school and my #ProcessTracing course.
Join me in Vienna and learn all the ins and outs of this method, mechanisms, and within-case analysis and inferences!
methodsnet.org/course/c01-0...
with @weareceu.bsky.social
What better plans for the summer then joining
@methodsnet.bsky.social in Vienna? From 07.07 until 11.07, I will be offering a one week course on #ProcessTracing methods. 1/
Now out in an issue in @journalpa.bsky.social with Aaron Deslatte and the newly minted Dr. Adam Wiechman:
“Embracing the ambiguity: Tracing climate response diversity in urban water management”
#urbanwater #ratemaking #institutionalgrammar #processtracing
onlinelibrary.wiley.com/doi/10.1111/...
Figure 6. Density plots of response times of the first, median, and last switches for each participant in 3 experiments. X-axis is time in seconds. The deliberation phase starts with the presentation of the problem in text format to participants at X = 0. The dashed vertical lines are at X = 60 seconds after which participants could report their final decision.
Do people vacillate more before or after they can decide?
In this paper, most vacillations occurred before people could decide about logic or moral problems, except maybe for moral dilemmas.
Check out the #processTracing method: doi.org/10.1017/jdm....
#xPhi #ethics #cogSci
'Process Tracing Methods' by Derek Beach & @hildevanmeegdenburg.com Webinar is on TOMORROW!
Sign up for free and make sure you don't miss out on this exciting event!
#methods #methodstraining #webinar #methodnets #processtracing #qualitative
Next week Thursday, Derek Beach and I will be hosting a #Free online webinar on #ProcessTracing Methods. We'll talk about what PT is and what it is that we are tracing. We'll also make sure there is time for questions. Join us!
📅: Thursday Feb 27
⏰: 15:00 CET
🌐: docs.google.com/forms/d/e/1F...
🕵️ Need a hands-on introduction to #ProcessTracing methods for your research? ✅ Sign up for our online course this March: https://buff.ly/3Q5pfAo
You'll cover:
❇️ Causal mechanisms
❇️ Evidence collection & analysis
❇️ Case selection
❇️ Practical applications
👀 Sneak peek below
🕵️ Need a hands-on introduction to #ProcessTracing methods for your research? ✅ Sign up for our online course this March: https://buff.ly/3Q5pfAo
You'll cover:
❇️ Core principles
❇️ Causal mechanisms
❇️ Evidence collection & analysis
❇️ Case selection
❇️ Practical applications
👀 Sneak peek below 👀
Byrd, N., & Conway, P. (2019). Not all who ponder count costs: Arithmetic reflection predicts utilitarian tendencies, but logical reflection predicts both deontological and utilitarian tendencies. Cognition, 192. https://doi.org/10.1016/j.cognition.2019.06.007
Byrd, N. (2023). Great Minds do not Think Alike: Philosophers’ Views Predicted by Reflection, Education, Personality, and Other Demographic Differences. Review of Philosophy and Psychology, 14(2), 647–684. https://doi.org/10.1007/s13164-022-00628-y
Byrd, N., Joseph, B., Gongora, G., & Sirota, M. (2023). Tell Us What You Really Think: A Think Aloud Protocol Analysis of the Verbal Cognitive Reflection Test. Journal of Intelligence, 11(4). https://doi.org/10.3390/jintelligence11040076
Byrd, N., Chapkovski, P., & Michalska, K. J. (2024, November). Experiments In Reflective Equilibrium Using The Socrates Platform. Society for Judgment and Decision Making, New York City. https://bsky.app/profile/byrdnick.com/post/3lbry6q5xas2x
Next I presented papers about how better reflection tests and manipulations as well as #processTracing #surveyMethods challenge/refine #DualProcessTheory.
Most of the papers are online. To find them and get alerts when others are online: scholar.google.com/citations?us...
More in today's poster.
Workshop outline (with names and affiliations in case that helps you find and cite the experts).
Ian Krajbich and visual examples of online eye-tracking, mouse-tracking, and text one may want to analyze (e.g., online reviews)
Nitisha Desai showing an example of mouse tracking patterns can reveal about decisions and judgments.
Ada Aka explaining how think-aloud protocols and other forms of text process tracing matter: they get us inside the opaque box of people's reasoning! Did you know think-aloud protocols can also be done online? See Byrd et al 2023 below: Byrd, N., Joseph, B., Gongora, G., & Sirota, M. (2023). Tell Us What You Really Think: A Think Aloud Protocol Analysis of the Verbal Cognitive Reflection Test. Journal of Intelligence, 11(4). DOI: 10.3390/jintelligence11040076
My final session at #SJDM 2024: Online #ProcessTracing!
Ian Krajbich gave the rationale.
Xiaozhi Yang demoed webcam #eyeTracking.
Nitisha Desai showed #mouseTracking code.
@adaaka.bsky.social advised on #coding scalable #textAnalysis with #LLMs.
Files on github.com/krajbichlab/...
#python #AI
#KOMEX2025 just opened for registrations!
If you are looking to brush up your #ProcessTracing skills, come follow my 1wk online course.
Dates: 17-21 Feb 2025
Early bird registration: EUR 440.
All info: afww.uni-konstanz.de/en/microcred...
@komex.bsky.social
🤓 Stimulating DLPFC often impacted performance on NUMERIC cognitive reflection tests (including a base rate neglect task), but not performance on VERBAL cognitive reflection tests (N = 48):
doi.org/10.1016/j.he...
#neuroscience #processTracing #decisionScience #psychology #measurement #assessment
Can decision transcripts be fruitfully analyzed using #machineLearning?
Eight machine learning algorithms seemed to predict most of the variance in med. students' correct and incorrect diagnoses from linguistic features of the transcripts!
doi.org/10.1007/s125...
#dataAnalysis #AI #processTracing
Just published:
Why Incorporate the ECHR? The Domestic Incentives of Human Rights Commitment
International Studies Quarterly
https://doi.org/10.1093/isq/sqae039
#humanrights #echr #ecthr #sweden #denmark #processtracing #IHRL
June 17-21, I offer a #ProcessTracing course for the MethodsNET summer school.
Part of the Summer School in Social Research Methods (#3SRM), it is for those interested in qualitative case studies and studying "causal mechanisms".
short🧵.
ru.nl/en/education...
With the end of the year approaching the count stands 3.466 downloads (or, at least, times opened). Who helps me reach 3.500 by the end of the year?
Process Tracing: An Analyticist Approach: hildevanmeegdenburg.files.wordpress.com/2023/01/vanm...
#Interpretivism #ProcessTracing #Narratives
Gentle reminder that you can still sign up for my upcoming (online) #ProcessTracing course at #Komex2024 @komex.bsky.social. Details ⬇️
afww.uni-konstanz.de/en/microcred...
EGPROC2023 (http://egproc.org is on ...
Abstract Submission Deadline: May 20th, 2023
Registration Deadline: July 1st, 2023
Please submit your abstract through this link: https://forms.gle/DQMJUV7jn3qpWaVS6
#processtracing @susafiedler @rimamrahal
pattern (previews from the #OpenAccess paper in pictures): https://sociologicalscience.com/articles-v10-4-118/
#DualProcessTheory #ProcessTracing #Research #Methods #Sociology (2/2)
Right before #SPUDM29 in Vienna we will have the yearly #EGPROC meeting between August 18th and 19th, 2023 organised by @susafiedler at the #Vienna University of Economics and Business.
Looking forward seeing you and your #processtracing projects there!
