I wrote something about building systems people actually want as an academic in ML. It's pretty much an open letter to 6-months-ago me.
magnusross.github.io/posts/moms-m...
❗️ We just expanded our capacity of B200 SXM6 180GB servers – available in the DataCrunch Cloud Platform.
The best thing is…
You can deploy the Blackwell platform without approvals.
Just sign in, select the instance type, and start your deployment:
cloud.datacrunch.io?utm_source=b...
Also pretty cool to see open source community building on top of each other!
The paper also suggests Group Tied Attention (GTA), which works in the opposite direction and draws inspiration from MLA, incorporating those techniques into GQA.
The technique called Grouped Latent Attention (GLA) can now be split across devices according to the group, providing higher throughput without a drop in performance by maintaining high arithmetic intensity and achieving better parallelism.
Well, the paper suggests a hybrid method. What about using MLA and adding groups?
Instead, one must make a copy of the latent component across GPUs, which feels wasteful.
This is where MLA is somewhat awkward, and GQA scores some points back. MLA uses a single large latent head that must be replicated across all tensor-parallel GPUs, which means that sharding the attention computations across GPUs cannot be done.
First of all, a confession! In the blog titled 'Multi-Head Latent Attention: Benefits in Memory and Computation', we didn't tell the whole story—the benchmarking on a single GPU. In reality, for DeepSeek V3-style models, parallelization is needed.
The paper focuses on designing more effective decoding attention for inference in light of Multi-head Latent Attention (MLA) and Group Query Attention (GQA).
A new paper just dropped from Tri Dao(🐐)'s lab!
arxiv.org/abs/2505.21487
Here is my hot take!
🆕 Inference API for FLUX.1 Kontext [max] & [pro] are now available on DataCrunch!
We are an infrastructure partner of Black Forest Labs for Kontext, a suite of generative flow matching models for text-to-image and image-to-image editing.
Learn more: datacrunch.io/managed-endp...
🚨 Summer Inference by Symposium AI is happening next Wednesday, June 4, at 16:00-22:00.
🇫🇮 This event will bring together 250 AI engineers, researchers, and founders under one roof in Helsinki.
🔗 You can still grab one of the last remaining seats: lu.ma/x5hhj79x
Some links that helped me to understand the roofline model.
jax-ml.github.io/scaling-book...
kipp.ly/transformer-...
datacrunch.io/blog/multi-h...
The blog post explains these terms and how they relate to algorithm intensity. Let us know if you have any questions or spot errors.
#MLSky
However, more is at play; revisiting Kipply's infamous Transformer Inference Arithmetic article shows that the MLA mechanism used during inference is now compute-bound 🖥️ and not memory-bound 💾.
Looking at the projections involved in DeepSeeek's attention (MLA) of the KV cache automatically makes one think it means less memory needed in HBM, preventing dreaded out-of-memory errors 👿 .
Algorithm hardware co-design was a big reason the whale 🐋(DeepSeek) made such a splash 💦 with its V3 and R1 releases.
"Cost-aware simulation-based inference" is accepted at AISTATS 2025.
Check out our poster #205 on Sunday May 4th in Hall A-E if you are in Phuket. Finland's rising star @huangdaolang.bsky.social will be there to assist you :D
arxiv.org/abs/2410.07930
@fxbriol.bsky.social @samikaski.bsky.social
This is very true! Go and speak to people in more old-school businesses and you quickly realize that with current models you could already do so much.
I don’t mean to be a broken record but AI development could stop at the o3/Gemini 2.5 level and we would have a decade of major changes across entire professions & industries (medicine, law, education, coding…) as we figure out how to actually use it & adapt our systems.
AI disruption is baked in.
1/ If you are at ICLR / AABI / AISTATS, check out work from our lab and collaborators on *inference everywhere anytime all at once*!
Go talk to my incredible PhD students @huangdaolang.bsky.social & @chengkunli.bsky.social + amazing collaborator Severi Rissanen.
@univhelsinkics.bsky.social FCAI
I wrote something up for AI people who want to get into bluesky and either couldn't assemble an exciting feed or gave up doomscrolling when their Following feed switched to talking politics 24/7.
1/10🔥 New paper alert in #AABI2025 Proceedings!
Normalizing Flow Regression (NFR) — an offline Bayesian inference method.
What if you could get a full posterior using *only* the evaluations you *already* have, maybe from optimization runs?
@aidanscannell.bsky.social
Tired of your open-source ML work not getting the academic recognition it deserves? 🤔 Submit to the first-ever CodeML workshop at #ICML2025! It focuses on new libraries, improvements to established ones, best practices, retrospectives, and more.
codeml-workshop.github.io/codeml2025/
Average cost for a student is 86,000$ a year just saying 😜
Congrats Pierre!
This is so true!
Sounds fun! I want to hear about it when you are back!
