Now accepted to EMNLP Main Conference!
20.08.2025 19:38
π 4
π 0
π¬ 0
π 0
Now accepted to EMNLP Main Conference!
Submit your work to #BlackboxNLP 2025!
Excited to spend the rest of the summer visiting @davidbau.bsky.social's lab at Northeastern! If youβre in the area and want to chat about interpretability, let me know βοΈ
In Vienna for #ACL2025, and already had my first (vegan) Austrian sausage!
Now hungry for discussing:
β LLMs behavior
β Interpretability
β Biases & Hallucinations
β Why eval is so hard (but so fun)
Come say hi if thatβs your vibe too!
10 days to go! Still time to run your method and submit!
Three weeks is plenty of time to submit your method!
What are you working on for the MIB shared task?
Check out the full task description here: blackboxnlp.github.io/2025/task/
New to mechanistic interpretability?
The MIB shared task is a great opportunity to experiment:
β
Clean setup
β
Open baseline code
β
Standard evaluation
Join the discord server for ideas and discussions: discord.gg/n5uwjQcxPR
In this work we take a step towards understanding and mitigating the vision-language performance gap, but there's still more to explore!
This was an awesome collaboration w\ Yossi Gandelsman, @boknilev.bsky.social, led by Yaniv Nikankin π€©
Paper and code: technion-cs-nlp.github.io/vlm-circuits...
By simply patching visual data tokens from later layers back into earlier ones, we improve of 4.6% on average - closing a third of the gap!
4. Zooming on data positions, we show that visual representations gradually align with their textual analogs across model layers (also shown by
@zhaofeng_wu
et al.). We hypothesize this may happen too late in the model to process the information, and fix it with back-patching.
3. Data sub-circuits, however, are modality-specific; Swapping them significantly degrades performance. This is critical - this highlights that the differences in data processing are a key factor in the performance gap.
2. Structure is only half the story: different circuits can still implement similar logic. We swap sub-circuits between modalities to measure cross-modal faithfulness.
Turns out, query and generation sub-circuits are functionally equivalent, retaining faithfulness when swapped!
1. Circuits for the same task are mostly structurally disjoint, with an average of only 18% components shared between modalities!
The overlap is extremely low in data and query positions, and moderate in the generation (last) position only.
We identify circuits (task-specific computational sub-graphs composed of attention heads and MLP neurons) used by VLMs to solve both variants.
What did we find? >>
Consider object counting: we can ask a VLM βhow many books are there?β given either an image or a sequence of words. Like Kaduri et al., we consider three types of positions within the input - data (image or word sequence), query ("how many..."), and generation (last token).
VLMs perform better on questions about text than when answering the same questions about images - but why? and how can we fix it?
In a new project led by Yaniv (@YNikankin on the other app), we investigate this gap from an mechanistic perspective, and use our findings to close a third of it! π§΅
Working on circuit discovery in LMs?
Consider submitting your work to the MIB Shared Task, part of #BlackboxNLP at @emnlpmeeting.bsky.social 2025!
The goal: benchmark existing MI methods and identify promising directions to precisely and concisely recover causal pathways in LMs >>
Have you heard about this year's shared task? π’
Mechanistic Interpretability (MI) is quickly advancing, but comparing methods remains a challenge. This year at #BlackboxNLP, we're introducing a shared task to rigorously evaluate MI methods in language models π§΅
SAEs have been found to massively underperform supervised methods for steering neural networks.
In new work led by @danaarad.bsky.social, we find that this problem largely disappears if you select the right features!
Thank you! Added to my reading list βΊοΈ
Should work now!
SAEs have sparked a debate over their utility; we hope to add another perspective. Would love to hear your thoughts!
Paper: arxiv.org/abs/2505.20063
Code: github.com/technion-cs-...
Huge thanks to βͺ@boknilev.bsky.socialβ¬, βͺ@amuuueller.bsky.socialβ¬, itβs been great working on this project with you!
These findings have practical implications: after filtering out features with low output scores, we see 2-3x improvements for steering with SAEs, making them competitive with supervised methods on AxBench, a recent steering benchmark ( Wu and βͺ@aryaman.ioβ¬ et al.)
We show that high scores rarely co-occur, and emerge at different layers: features in earlier layers primarily detect input patterns, while features in later layers are more likely to drive the modelβs outputs, consistent with prior analyses of LLM neuron functionality.
These differences were previously noted (e.g.,
Durmus et al., see image), but had not been systematically analyzed.
We take an additional step by introducing two simple, efficient metrics to characterize features: the input score and the output score.
In this work we characterize two feature roles: Input features, which mainly capture patterns in the model's input, and output features, those with a human-understandable effect on the model's output.
Steering with each yields very different effects!
Tried steering with SAEs and found that not all features behave as expected?
Check out our new preprint - "SAEs Are Good for Steering - If You Select the Right Features" π§΅
Logo for MIB: A Mechanistic Interpretability Benchmark
Lots of progress in mech interp (MI) lately! But how can we measure when new mech interp methods yield real improvements over prior work?
We propose π π ππ: a π echanistic πnterpretability πenchmark!
π¨π¨ New preprint π¨π¨
Ever wonder whether verbalized CoTs correspond to the internal reasoning process of the model?
We propose a novel parametric faithfulness approach, which erases information contained in CoT steps from the model parameters to assess CoT faithfulness.
arxiv.org/abs/2502.14829