Miha Gazvoda

If your frequentist analysis gives the same result as a Bayesian analysis, that means nothing for either paradigm. No frequentist is bothered that Bayes "works" - they use Bayesian calculations all the time. No Bayesian is bothered that a freq estimator agrees - it's interpretation that is doubted.

While these insights likely won’t differ drastically across businesses, performing such analysis with your own experiments can help clarify the (implicit) trade-offs you're making.

mihagazvoda.com/files/nhst-r...

6/6

To address these, a useful alternative I see is applying a shrinkage estimator based on past experiments (and experiment-specific details), combined with explicit cost-benefit analysis. As @statmodeling.bsky.social puts it, let’s make decisions in real-world units: dollars, customers.

5/6

* Sacrifices business impact by excessively focusing on minimizing losses at the expense of gains.
* Misleads learnings due to exaggerated significant effect estimates, poor coverage, and low replication rates—though the estimated direction typically remains correct.

4/6

Adopting this approach reveals that, under realistic assumptions, NHST with conventional error rates often:

3/6

I highlight how NHST’s effectiveness is often explained using an unrealistic scenario—pretending effects are strictly either “null” or “true.” Instead, I suggest performing a meta-analysis of past experiments to estimate the distribution of unobserved true effects.

2/6

I’ve written a personal-view article, “A Reality Check on Null Hypothesis Significance Testing in A/B Testing", soon appearing in Eppo’s Outperform magazine (print).

1/6

Reminder that the blog is here too!

@statmodeling.bsky.social

Please repost 🙏

I like this paper about CI interpretation: pure.rug.nl/ws/portalfil...

Why we (usually) don't have to worry about multiple comparisons Applied researchers often find themselves making statistical inferences in settings that would seem to require multiple comparisons adjustments. We challenge the Type I error paradigm that underlies t...

Since it is inefficient to attempt to educate every reviewer individually, I am yeeting into your feed this clear paper from Gelman, Hill, and Yajima on how Bayesians can do even better than correcting for multiple comparisons. arxiv.org/abs/0907.2478

Gelman, Hill, and Vehtari in Regression and Other Stories suggest fitting a regression with the same variables as were used for matching/IPTW.