Decent summary of the state of the art x.com/peterwildefo...

Many think LLM-simulated participants can transform behavioral science. But there's been a lack of accessible discussion of what it means to validate LLMs for behavioral scientists. Under what conditions can we trust LLMs to learn about human parameters? Our paper maps the validation landscape.
1/

Forecasting Social Science: Evidence from 100 Projects Founded in 1920, the NBER is a private, non-profit, non-partisan organization dedicated to conducting economic research and to disseminating research findings among academics, public policy makers, an...

"Confidence in the accuracy of one's forecasts is perversely associated with lower accuracy."

www.nber.org/papers/w34493

A blog post giving a more thorough take on survey experiments and the credibility revolution: cyrussamii.com?p=4168

1. Randomly draw cities (with replacement).
2. Keep all time series data for each drawn city.
3. Compute your estimate.
4. Repeat many times to get a valid confidence interval.

Use the Block Bootstrap:

Instead of resampling individual observations, resample entire cities to preserve time dependence.

1. Seattle appears multiple times in the data.
2. These data points aren’t independent. For example, if a GenAI-driven sales boom starts in Seattle, its impact persists over time, making observations correlated.

So how do we get valid confidence intervals while respecting these dependencies?

Let’s take a simple example:

We want to measure the impact of increasing ad spend in Seattle, but not in Portland.
We observe both cities before and after the marketing change.

But here’s the problem:

Panel data is everywhere in data science—but many models assume independence when observations are actually correlated over time. Ignore this in simulations, and your confidence intervals will be wrong.

The 4 biggest challenges I’ve faced in data science - and how I’ve approached them:

1. Causality - experiment if you can
2. Messy data - prioritize stable infra
3. Small data - know the limits
4. Culture - show the value, get leadership to care

What did I miss? Wrong ranking?

"Bar is raised because gravity is lower" was a fun sentence to write

Our political economy of field experiments study has just been released by the @nberpubs.bsky.social!

Written with @gubri.bsky.social, who is brilliant and on the market this year!

Here are the first five sets of slides:

01 Introduction: psantanna.com/DiD/01_Intro...

02 Classical 2x2 setup: psantanna.com/DiD/02_two_b...

03 Clustering issues: psantanna.com/DiD/03_Clust...

04 Functional form: psantanna.com/DiD/04_Funct...

05 Covariates: psantanna.com/DiD/05_Covar...

New working paper out today with @epiellie.bsky.social called "Do LLMs Act as Repositories of Causal Knowledge?"

Can LLMs (ie ChatGPT) build for us the causal models we need to identify an effect? There are reasons to expect they could. But can they? Well, not really, no.

arxiv.org/html/2412.10...

How policymakers and the US population update their beliefs on the use of science and the trust they have in government following a field experiment that demonstrated the ineffectiveness of a policy intervention, from Guglielmo Briscese and John A. List https://www.nber.org/papers/w33239

Economics hasn’t fully internalized how difficult climate tipping points are to both predict and reverse. My JMP 🚨 estimates the costs of this unpredictability and irreversibility.

A thread! 🧵 (1/14)
#econjobmarket #econjmp #econtwitter #climate

This seems like a lot until you think about the likely return on this investment.

The Separation and Reunification of Germany: Rethinking a Natural Experiment Interpretation of the Enduring Effects of Communism (Spring 2020) - German separation in 1949 into a communist East and a capitalist West and their reunification in 1990 are commonly described as a natural experiment to study the enduring effects of co...

I was going to suggest East-West Germany comparison then remembered this great JEP piece on why caution is warranted www.aeaweb.org/articles?id=... @essobecker.bsky.social @lukasmergele.bsky.social

Soviets? Old Soviet Joke: We Pretend to Work,They Pretend to Pay Us.

I summarized this thread in a fuller, written form on my blog:

paulgp.github.io/2024/11/06/c...

(I also, as a sidenote, revamped my blog CSS to be able to do sidenotes and figures, which I'm pleased with)

Behavioral economics: people cannot add or subtract

Macroeconomics: *maybe* people cannot deal with the fact that the entire cross sectional distribution is a state variable

Historically, many people have believed that sources of power, like the vote, access to education, and the printing press, should be restricted to only the rich and powerful.

Today, some want you to believe that about data access.

Opening data access is less of a societal threat than closing it.

This is amazing! Would Claude outperform ChatGPT on this task, or is there another reason you chose it?

Ok, this needs a lot of tweaking and more work, but here's a minimal viable product for

An econometrics paper (Deep Neural Networks for Estimation and Inference)
1. paulgp.github.io/2024/11/06/r...

An economic theory paper (Targeting Interventions in Networks)
2. paulgp.github.io/2024/11/06/r...