Hello all! I’m recruiting a postdoc to work with my lab and I on methods for analyzing intensive longitudinal timeseries of psychological phenomena, with particular focus on measurement and optimization of interventions! Start would be August 2026
Yes! I tell my class that regression coefficients are the "unique" relationships of a variable. If variables are not "unique", there is less information available to make the SEs "smaller." It's always either multicollinearity or normality assumptions that you've got to disabuse students of!
Parameters aren't real.
Wait, you're telling me that a study in 20 children with ADHD showing an increased correlation between 2 arbitrarily defined brain regions doesn't represent a paradigm shift in how we could treat ADHD with respect to possible targeted pharmaceuticals?
Gif of police officer, dramatic high contrast lighting, looking confused and asking, "Are we the baddies?"
The one simple trick to avoid > 75% of bad uses of AI that AI ethicists don't want you to know!
Read a science fiction book and if your goal aligns with the bad guys in that book don't do it.
-- Credit @stellaathena.bsky.social on the Bad Site.
I mean, why do journals give due dates on RRs if not to encourage us to submit at the very last second?
Me, moments before I unknowingly hit reply all on the faculty listserv: "We are currently clean on OPSEC"
Homebrew warlock patron "The P-factor" that lets you summon and bind fiends by drawing path diagrams for multi-level SEMs.
At level 17 you gain the ability to measure constructs ... without error!
In 2018 I gave a presentation on research data management to a group of graduate students. Not surprisingly, most of the problems and solutions I discussed 7 years ago, are still the same problems and solutions I see today. #databs
🧠 Applying to Psychology PhD programs? Join @UVA's virtual panel April 14 at 12:30PM! Current faculty & PhD students will share tips on choosing programs & creating competitive applications. All welcome, especially those from underrepresented groups. Register via QR code! #PsychPhD #GradSchool
This year I have the honour of serving the Psychometric Society as its President, and we have been working hard on the program for the International Meeting of the Psychometric Society (IMPS) psychometricsociety.org/imps-2025 You can submit an abstract here: imps2025.exordo.com
"You want to stop, but the amount of time and energy you’ve put into this makes it hard to quit."
1/3
Tutorial on exploring ecological momentary assessment data is online at AMPPS, with:
- Accessible ways to visualize data for better understanding
- Models to get some first insights
- Further reading boxes for more advanced topics
- Reproducible pipeline you can run over your own data
New preprint by Anja Ernst and me on Modeling Qualitative Between-Person Heterogeneity in Time-Series using Latent Class Vector Autoregressive Models (osf.io/preprints/ps...)
... to use human raters to reconcile any ambiguities. I think this is fascinating work, and a great example of how to use LLMs as a true tool for analyzing data (rather than just asking an LLM to analyze data). Joy is currently working on an extension to discover new topics, which is very exciting!
We found that ensembles of small LLMs tended to have reasonable (better than chance) performance at identifying topics. It wasn't perfect, but it is good enough to be a reasonable first pass through a large dataset. The idea is to first identify the cases that are easy to classify, then...
We compared the LLM topics to human raters topics. Now, importantly, Joy used an ensemble of LLMs (equivalent to having several human raters). Why? Well, we wanted for this method to be able to run locally, as there are a number of privacy issues with using consumer LLMs to analyze health data.
She applied this method to both a massive dataset of Reddit posts from eating disorder related forums, and a smaller dataset of free-text responses from patients with eating disorders (courtesy of @cherilev.bsky.social), and compared how well these LLMs could identify a prespecified set of topics.
So, Joy decided to use LLMs to query free-text data to extract "topics." For example, if a study collected responses regarding stressful events, then a topic might be a description like "work stressor" or "interpersonal stressor."
Enter LLMs! While I have a number of thoughts on LLMs that I won't get into here, for this project we conceptualized them as "tools that can comprehend free-text," with comprehension being akin to human reading comprehension (do LLMs comprehend? No, but they sure do seem like they do!)
Traditionally, this is done via teams of human raters. Researchers will put together groups of raters (usually grad students, possibly RAs) that will go through the free text responses and quantify them in some way. This takes an enormous amount of time and resources.
Free-text responses are a wonderful way of collecting nuanced information about any number of psychological phenomena, but the issue is, of course, that they are free-text responses. To perform quantitative analysis on them, you need to convert them into some set of numbers.
Can I interest y'all in a preprint in these trying times? My student Joy Qiu recently completed work on using ensembles of LLMs to perform coding of free text response data. A 🧵
We originally thought about this project from an EMA perspective, trying to optimize for participant burden. So that’s where we are coming from!
To your second, I think this matters the most when only one/a couple questions per construct are asked. For a multi item scale, while I still want to see measurement errors of individual items, I don’t think the extra error induced by VAS would degrade wholistic scale performance.
Fortunately, it is an empirical question, and I would be happy with seeing that no, VAS doesn’t meaningfully degrade measurement! I just want to see the studies.
To your first question: yes, but different things matter more or less for measurement quality. For example, I wouldn’t think that a slight change in anchor language needs a revalidation (it could though!)
Found my new standard review comment
