You’ve generated 10k concepts with your favorite XAI method -- now what? Many concepts you’ve found are fairly obvious and uninteresting. What if you could 𝑠𝑢𝑏𝑡𝑟𝑎𝑐𝑡 obvious concepts away and focus on the more complex ones? We tackle this in our latest preprint!
Deadline Extended!
Submit to the LM4Sci Workshop @ COLM 2025 in Montreal 🇨🇦
🧠 Large Language Modeling for Scientific Discovery (LM4Sci)
📅 New Deadline: June 30
📢 Notification: July 24
📍 Workshop: Oct 10, 2025
📝 Non-archival short (2–4p) & full (up to 8p) papers welcome!
🚨 Call for Papers: LM4Sci @COLM_conf 2025 🚨
Excited to announce the Large Language Modeling for Scientific Discovery (LM4Sci) workshop at COLM 2025 in Montreal, Canada!
Submission Deadline: June 23
Notification: July 24
Workshop: October 10, 2025
Check out the full paper for the mathematical formulation, experiments, and our methodology: arxiv.org/abs/2504.00185 Code and other artifacts are available here: trishullab.github.io/escher-web/ Thank you for following along!
How it works:
1️⃣ LLM proposes concepts per class
2️⃣ CLIP-style VLM scores them
3️⃣ Escher spots confused classes
4️⃣ Escher stores this in a history bank
5️⃣ LLM proposes better concepts and stores them → repeat
The loop is self-amplifying: better concepts ➡️ better feedback ➡️ an even better concept library.
Escher solves this problem using feedback from a vision language model to improve the reasoning, specifically for fine-grained image classification.
Our hypothesis: the failure arises from the program synthesizers treating the vision model as a deterministic function. Reality is messy and the VLM outputs are stochastic. The LLMs assumptions of how the VLM will behave and how it actually behaves are decoupled. We need to overcome this decoupling.
Even with visual programming, the LLM proposing the program has no idea about the execution semantics of the underlying VLM. Things still don't work.
A visual program decomposes complex perceptual reasoning problems into a logical combination of simpler perceptual tasks that can be solved using off-the-shelf vision foundation models. This provides a modular and robust framework, but finding the correct decomposition is still extremely hard.
gpt-o3, which has probably seen this image before, reasons incorrectly about the type of lizard and gets it wrong. Visual feedback is extremely important here!
Reasoning about these images is pretty hard. o3 – even with web access – can’t do this for us out of the box. In such a situation, writing programs provides a mechanism for dividing up a complex reasoning task into solvable subtasks. This motivates most of the visual programming literature.
An example from inaturalist of two scientist deliberating how to classify a rare lizard. The first scientists gets it wrong because they aren't trained as a herpetologist. The second scientist is a trained herpetologist, and reasons in natural language how to correctly identify the image.
In many vision tasks, perceptual reasoning does not come naturally. Experts still have to deeply study an image, deduce relevant concepts, and reason about them in natural language (www.inaturalist.org/observations...). Our goal is to automate this process – with no human oversight.
Massive thanks to my co-authors Patrick Yuan, Ziniu Hu, @yisongyue.bsky.social, Jennifer J. Sun & @swarat.bsky.social for making this possible!
I’m presenting Escher (trishullab.github.io/escher-web) at #cvpr2025 Saturday morning (Poster Session #3). Escher builds a visual concept library with a vision‑language critic (no human labels needed). Swing by if you’d like to chat about program synthesis & multimodal reasoning!
Just julia things.
Happy to announce the PySR v1.0 release!
github.com/MilesCranmer...
PySR lets you do high-performance symbolic regression from Python.
Now, you can learn multiple symbolic expressions simultaneously!
Also:
+ Parametric expressions
+ TensorBoard support
+ Improved search
+ Julia-based inference
Missing NeurIPS this year but wanted to highlight our new paper on LLM-guided genetic programming: trishullab.github.io/lasr-web/
Our method, LaSR, conditions mutation/crossover operators on (1) an LLM's general domain knowledge, and (2) LLM-generated abstractions of high-performing programs. (1/2)
Check out the full paper for the mathematical formulation, llm scaling law experiments, and our methodology: arxiv.org/abs/2409.09359
More context here: x.com/atharva_sehg...
Thank you to all my coauthors: Arya, Omar, @milescranmer.bsky.social, and @swarat.bsky.social!
Arya and I'll be at #NeurIPS presenting LaSR (trishullab.github.io/lasr-web/) on Wednesday morning 11AM PST to 2PM PST (East Exhibit Hall A-C #4003). Drop by and say Hi!
