David Manheim

I hate being late for meetings, but "the air raid siren went off so I had to go down to the bomb shelter" is probably the best reason I'll ever have for doing so.

Any time you see someone argue that their opponents are wrong, instead of arguing that they are right, the conclusion should be a negative update about *both sides*.

(And before you try to interpret this as a subtweet, it is about at least three different things I've been annoyed about recently.)

I hereby posit Manheim's Law of Positive-Sum Badness:

In polarized disputes, evidence that one side is stupid, malicious, or evil increases the probability that the opposing side is too.

"What can Goodhart's Law teach us about AI alignment?"

Good question! Our 2018 paper was explicitly about the problem: "in Artificial Intelligence alignment... the increased optimization power offered by artificial intelligence makes it especially critical for that field."
arxiv.org/abs/1803.04585

That's just Goodhart's law - because most metrics start out as shit, they are just whatever is easiest to measure that seems related.
I even have a paper about this!
www.cell.com/patterns/ful...

"As Iran’s supply of long range weapons dwindles"

I've heard from several sources that's a long way off, and they have lots of stockpiles, albeit fewer an fewer launchers - but given my location I certainly hope you're right.

Sam Altman faced painful internal backlash from the new Pentagon deal, which is why he told staff that he's... also planning new deals with other NATO countries to use in classified settings.
www.wsj.com/tech/ai/open...

OpenAI's plans after signing the contract which they say won't allow domestic surveillance... are definitely gonna eventually involve domestic surveillance.

I think it's strategic brilliance from the perspective of the current leader; he doesn't care about the country, he wants to be able to stay in power, and this will help him.
(I agree that it's horrific in the long term for the country, but we all know voters are myopic.)

It's like the Fifa Peace Prize doesn't mean anything at all.

Anyways, me and my family are basically safe, just running up and down the stairs to the bomb shelter a couple times a day.

And at least I know that the US is using Claude to plan their strategy for another 30 days or something, which is nice and dystopian.

This would be the worst business move, in isolation.

Federal contracts are net time sinks for tech companies, this contract won't pay enough to be worth the pain. The only way they could benefit is if acceding to pressure were somehow used to punish competitors, which... right, that's the point.

Lastly, there's an argument that biotech in general can be dangerous because of pathogen applications, and advances in AIxBio can be worrying because of this.

True - but not doing safe things because very different ideas in another area of bio are scary is a bad objection!
(fin)

(continued) to think a given new tech is on-net dangerous. Self-replication or an exponential trajectory could qualify, but this isn't that.

It seems far more likely that this enables cool and largely safe nanotech applications.

And third, general advances in biodesign are terrifying!

And yes, all new technology can be misused, and that's certainly true for bio. But out (strong) prior should be that very little of the usage of new technology is for dangerous misuse which means that we need some pretty strong reason...

Second, does this allow creating more robust biological systems? (viruses, bacteria, etc.)

Nope. We're years away from having enough understanding of pathogens with AI to build novel things that work very differently. This other thread explains more:
bsky.app/profile/davi...

First, prions.

Prions are terrifying, but this work doesn't relate to them. It's easy to hear "hard to sterilize" and think "Prion" - but optimizing for elongated β strands, or doing any other structural modification using similar methods, doesn't make for misfolding.

Computational design of superstable proteins through maximized hydrogen bonding - Nature Chemistry Nature contains a variety of mechanostable proteins, which all bear extensive hydrogen-bond networks within their β-sheet architectures to sustain high stability under stress. Now through integrating ...

Cool new paper out in Nature on AI-designed proteins that survive 150 °C and nanonewton forces! www.nature.com/articles/s41...

It's very interesting work - but I see people saying it is also directly worrying. I disagree - to explain, here's a short 🧵

I'm not addicted to using language models, I can write and do work without them, I still remember how to search by myself, I keep repeating... as I repeatedly click refresh on Claude and ChatGPT.

Thankfully, Gemini (partially) saves the day.

It's nice to see that #AAAI2026 is avoiding any religious discrimination; it's on Friday, Saturday, and Sunday, so it's equally hostile to Muslims, Jews, and Christians.

Definitely worth a reminder, even though I should certainly know to do so!

If you're among the 2000+ authors citing research evaluated by Unjournal.org, go to unjournal.pubpub.org to learn how commissioned experts rated & assessed that research.

Top 'citers of unjournal-evaluated research': https://bit.ly/3WRHc8W include Esther Duflo, Julian Jamison, & Berk Özler

And I find myself strongly agreeing with almost everything Emma Ruttkamp-Bloem is saying in her #AIES2025 keynote about the future of AI ethics - despite continuing to worry that the vision for how to make AI systems more ethical does not sufficiently address future risks.

Tweet from @davidmanheim Nov 26, 2023; "I hate that #AIEthics is so closely tied to AGI skepticism. Yes, obviously AGI doesn't exist now, and ethics is really important to current ML. But if (when) we get radically transformative AI, they largely discredited themselves. And we don't stop needing AI ethicists then!"

I'm here at #AIES2025, and still worry quite a lot about this.

The deep skepticism about AI systems ever being generally capable, or even human-level in specific domains, doesn't seem to have changed over the past few years.

Low-resolution photo of David Manheim in front of AIES2025 banner

Excited to be here today at #AAAI #AIES2025. Looking forward to meeting more people and discussing governance and societal impacts of AI.

Many things are explainable, but not understood by us. You were asserting that LLMs are understood, not that they are theoretically understandable.

Good to see you at least read the abstract.

Now try the paper, especially about how behaviors of attention heads in a model evolve with respect to the training data distribution, and then tell me that language models are directly coded again.

...what makes someone qualify as a Zionist, in your view?

Because if you mean supporting genocide and/or ethnic cleansing in Gaza, sure, that's obviously horrific. But if you mean wanting a 2-state solution instead of wanting all of Israel wiped off the map, I'm much more concerned.

Structural Inference: Interpreting Small Language Models with Susceptibilities We develop a linear response framework for interpretability that treats a neural network as a Bayesian statistical mechanical system. A small perturbation of the data distribution, for example shiftin...

I'm very unsure if you're shockingly ignorant for someone so confident, or shockingly confident for someone so ignorant, but I'm not sure it matters.

Anyways, here's an expert obviously disagreeing with you for you to ignore: arxiv.org/abs/2504.18274

When Should We Worry About AI Being Used to Design a Pathogen? Concerns that artificial intelligence (AI) might enable pathogen design are increasing, but risks and timelines remain unclear. This brief describes a Delphi study of biology and AI experts who debate...

Bottom line: Treat AI-enabled bio risk as rising but still governable, and aim for clearer threat models, empirical monitoring, and adaptive policies.
(12/12)

And if you don't want to read the 100+ page report, read the 6-page @rand.org brief for details:
www.rand.org/pubs/researc...