๐ข Life update ๐ข
After a wonderful time at @ai2.bsky.social, I've joined @cislmu.bsky.social at @lmu.de as a tenure-track assistant professor in NLP. Thrilled to be back in Europe and to start a lab in Munich's flourishing AI ecosystem! ๐
Demographic cues (eg, names, dialect) are widely used to study how LLM behavior may change depending on user demographics. Such cues are often assumed interchangeable.
๐จ We show they are not: different cues yield different model behavior for the same group and different conclusions on LLM bias. ๐งต๐
Introducing Bolmo, a new family of byte-level language models built by "byteifying" our open Olmo 3โand to our knowledge, the first fully open byte-level LM to match or surpass SOTA subword models across a wide range of tasks. ๐งต
Excited to see our #COLM2025 paper on fluid benchmarking highlighted by @eval-eval.bsky.social! They are worth a follow if you are into LLM eval research. ๐ฌ
Thereโs plenty of evidence for political bias in LLMs, but very few evals reflect realistic LLM use cases โ which is where bias actually matters.
IssueBench, our attempt to fix this, is accepted at TACL, and I will be at #EMNLP2025 next week to talk about it!
New results ๐งต
Check out this #EMNLP2025 paper led by @minhducbui.bsky.social and @carolin-holtermann.bsky.social showing dialect prejudice remains a major issue in current LLMs.
Example: GPT-5 associates German dialect speakers with being uneducated and steers them toward stereotyped jobs (e.g., farmworkers).
๐
Thanks, Jordan! Your ACL 2021 paper was a huge source of inspiration for us!
We did not specifically analyze novel models as your paper did. While I am optimistic that Fluid Benchmarking improves over static IRT-based methods in this regime as well, there are definitely limitations, which we discuss in the paragraph below.
Would be exciting to run more experiments on this!
In our experiments, we find that this dynamic approach consistently outperforms static IRT-based methods. The improvements are especially pronounced in terms of variance, which poses a major challenge for static IRT-based methods. We discuss this in more detail in the paragraph below.
Great question! The key difference is that we use IRT to dynamically adapt the subset of items to a model's capability, rather than to determine a static, "globally optimal" subset of items as in prior work. With Fluid Benchmarking, each model is evaluated on a different subset of items.
LM benchmark design requires 3 decisions, how to:
๐ select test cases
๐ score LM on each test
๐ฆ aggregate scores to estimate perf
fluid benchmarking is simple:
๐ฃ find max informative test cases
๐ฅ estimate 'ability', not simple avg perf
why care? turn ur grey noisy benchmarks to red ones!
Last but not least, a huge shoutout to my incredible coauthors @davidheineman.com, @ianmagnusson.bsky.social, @kylelo.bsky.social, @jessedodge.bsky.social, @maartensap.bsky.social, Pang Wei Koh, Chun Wang, @hanna-nlp.bsky.social, and @nlpnoah.bsky.social! ๐ค
For details, check out our paper, blog, code, and data:
๐ arxiv.org/abs/2509.11106
โ๏ธ allenai.org/blog/fluid-b...
๐ป github.com/allenai/flui...
๐ huggingface.co/datasets/all...
Looking forward to chatting more at #COLM2025! ๐
Overall, our work shows that LLM evaluations can be substantially improved by moving beyond the until-now universal practice of static benchmarking, which assumes a globally optimal set of evaluation questions for all models.
These (and more) advantages are achieved while at the same time reducing evaluation cost.
Example: on MMLU, Fluid Benchmarking results in lower step-to-step variance and higher validity than standard methods while using 50 times fewer questions. โก
Fluid Benchmarking substantially reduces step-to-step variance during pretraining.
It also increases validity: results generalize better to other benchmarks targeting the same capability. One reason: it automatically avoids mislabeled questions, cutting label errors by 99%! ๐คฏ
In our experiments, we apply Fluid Benchmarking to evaluation during pretraining, a setting where capabilities evolve rapidly.
We find that Fluid Benchmarking dynamically adapts to these changes, administering easier questions early in training and more difficult ones later.
Fluid Benchmarking repeats this loop until the number of administered questions reaches the allotted budget.
Adaptive question selection means that LLMs face different sets of questions, but ability estimation aligns results in a common space.
In Fluid Benchmarking, we start with an initial ability estimate from one question.
To select the next question, we use Fisher information. Essentially: a question close in difficulty (b) to the ability estimate (ฮธ) and with high discrimination (a).
Then we update the estimate.
In addition, IRT models each LLM's ability, which can be estimated from its responses to questions with known difficulty and discrimination.
The IRT ability estimate can be used to summarize performance like accuracy, and it accounts for question characteristics.
To get a question's difficulty, we use item response theory (IRT): we analyze responses of hundreds of LLMs to see how often a question is answered correctly.
IRT also measures the discrimination of a question, meaning how reliably it separates stronger from weaker LLMs.
Test theory says: questions are most informative when matched to a test taker's ability.
For LLMs, that means evaluating weaker models on easier questions and stronger models on harder ones.
But how do we know a question's difficulty, or an LLM's ability, before evaluation? ๐ค
๐ข New #COLM2025 paper ๐ข
Standard benchmarks give every LLM the same questions. This is like testing 5th graders and college seniors with *one* exam! ๐ฅด
Meet Fluid Benchmarking, a capability-adaptive eval method delivering lower variance, higher validity, and reduced cost.
๐งต
I am delighted to share our new #PNAS paper, with @grvkamath.bsky.social @msonderegger.bsky.social and @sivareddyg.bsky.social, on whether age matters for the adoption of new meanings. That is, as words change meaning, does the rate of adoption vary across generations? www.pnas.org/doi/epdf/10....
Attending #ICML2025? Don't miss this TokShop panel, which will explore:
๐ฎ The Future of Tokenization ๐ฎ
Featuring a stellar lineup of panelists - mark your calendar! โจ
LLMs can appear unbiased on the surface but still perpetuate racist views in subtle ways.
What causes this discrepancy? ๐
In our upcoming #ACL2025 paper, we find a pattern akin to racial colorblindness: LLMs suppress race in ambiguous contexts, leading to biased outcomes.
๐ฃ We extend the submission deadline by 24 hours to avoid conflict with ACL camera-ready deadline.
๐
New Submission Deadline: May 31, 2025 (23:59 AoE)
๐ฉ OpenReview: openreview.net/group?id=ICM...
Huge congrats, Adam!!! ๐
Got a good tokenization paper under review at COLM, but the scores were a letdown? ๐ฌ
Why bother with rebuttal when the perfect venue is right around the corner!
Submit your paper to the #ICML2025 Tokenization Workshop (TokShop) by May 30! ๐
Beyond text: Modern AI tokenizes images too! Vision models split photos into patches, treating each 16x16 pixel square as a "token." ๐ผ๏ธโก๏ธ๐ค #VisualTokenization
Interested in tokenization? Join our workshop tokenization-workshop.github.io
The submission deadline is already May 30!
