Our lab has the capacity to test ~500 uni students each semester
If you’re a researcher in cognitive psychology or metascience and need data collection support, we’d love to collaborate. We can help collect high-quality data from a large student sample.
Get in touch to discuss potential projects!
Much thanks to the amazing team: @martonaronvarga.bsky.social , @donvanraven.bsky.social , @kekecszoltan.bsky.social, @jimgrange.bsky.social, @balazsaczel.bsky.social & Máté Gyurkovics
We think that research fields where notable "ground truth" effects are investigated (such as the CSE), a similar systematic exploration of the analytical space is necessary to inform the field's community about common arbitrary decision combinations that can lead to higher false findings.
Based on these results we think that the risks of multiple testing (even with common corrections) are higher than expected, thus sticking to a preregistered analytical protocol is immensely recommended.
in repeated-measures ANOVAs, FPRs were not affected by outlier filtering methods; thus, when severe outlier filtering is justified, repeated-measures ANOVA is a recommended choice for hypothesis testing.
In linear models, type I error rates also increase proportionally to the severity of outlier filters. This inflation of FPR poses a significant risk of false findings; therefore, we do not recommend to use linear mixed models along with severe outlier exclusion techniques, especially on skewed data.
Model TPRs on large effect size datasets on different participant numbers, with the 3SD outlier filtering method. True positive rate is indicated on the y-axis, while false positive rate is indicated on the x-axis. Hypothesis testing models are shown with different colors, and numbers on the plot indicate different sample sizes. An assumed maximum FPR of.025 is indicated with a dashed vertical line
Results showed that certain analytical choice combinations (outlier filtering; data transformation; hypothesis testing method) led to highly inflated false positive rates (type I error rates). Decision pathways where linear mixed-effect models were used were especially impacted.
I am happy to announce the publication of our new work on the impact of arbitrary analytical choices on type I and type II error rates. We simulated reaction time data in a conflict task and analyzed the notable CSE effect in a multiverse manner. Worrying results:
link.springer.com/article/10.3...
