Howdy all. I'm unfortunately not going to be with my employer for much longer due to team relocation. If anyone has any info on roles that would allow me to continue my Rust compiler work (in New York City), they'd be greatly appreciated.
I'm writing an article series about creating tensors from scratch in Rust. #tensors #machine-learning #ml #ai
huggingface.co/blog/KeighBe...
🦀 Hello World!
The Rust project now has an official presence on Bluesky! ✨
We'll be posting the same on our Mastodon and Bluesky accounts, so you won't miss anything on either platform.
Want an in depth exploration of the different hardware architectures within AI?
Of course you do :)
Another great article by Chris Fleetwood:
fleetwood.dev/posts/domain...
Performance leap: TGI v3 is out. Processes 3x more tokens, 13x faster than vLLM on long prompts. Zero config !
True, but at the same time my man JJB famously said «Ooh! Ooh! Mooie! Woohoo! Aah!»
So yeah
Chart Title: Model Hardware vs Energy per GigaFLOP. Vertical Axis: mJ/GFLOP(Log) Horizontal Axis: Hardware Type(CPU, CPU + GPU, CPU + ANE) CPU: min 6.9 1st quartile 11.7 median 13.4 3rd quartile 35.6 max 53.1 CPU + GPU: 4.6 4.6 4.7 6.2 9.6 CPU + ANE: 0.9 1.0 1.1 1.4 1.8
Preliminary data shows the Apple Neural Engine uses ~94% less energy than the CPU and ~75% less than the GPU 🤯
On the On-Device team at Hugging Face, we've been profiling energy usage for CoreML models. Here’s some data I collected:
I, for one, don’t immediately see anything wrong with what you’ve said here.
There are perhaps some exaggerations here and there to drive home your points, but the best thread/rant on the subject (from the side of outraged bluesky users) that I’ve seen
It's Sunday morning so taking a minute for a nerdy thread (on math, tokenizers and LLMs) of the work of our intern Garreth
By adding a few lines of code to the base Llama 3 tokenizer, he got a free boost in arithmetic performance 😮
[thread]
I guess you’ll have to engage fervently with that content to bring it back. Good luck 🫡
Sky tweet
I was just about to tag you hehe
If you want to dive into async allocators a bit more:
open.spotify.com/episode/2YGI...
Our hardware is usually async in many different ways, but our default programming approach usually isn’t.
For example we approach allocating memory as a sync operation, but it usually isn’t. We could be doing stuff while allocating. Async allocators has a host of fun problems though :)
when you try to convert your text into smaller pieces but all it gives you is Elvish, that’s a tolkienizer
RoPE can be confusing, so here’s a great write up by my buddy Chris Fleetwood on the topic:
fleetwood.dev/posts/you-co...
I think you’re applying your own (better) logic and improving on what he actually means. Your point has merit, his does not.
Hold people accountable to their exact phrasing.
He specifically said to stop worrying about climate goals and instead funnel money into AI, no?
That’s what you should either agree with or not.
Applying AI in various fields is something else. Sounds good.
Generalisations outside what exists/can be inferred in the training set?
That is unfortunately impossible simply by how training works.
I think continuing to fund ML research is essential. But there are a limited amount of geniuses out there. Wild spending will not improve anything.
Hah I thought you blocked me for agreeing with you because my comments disappeared. Phew.
I guess they’re gone because the parent comments were removed.
MLE here.
I don’t want to put you down or anything, but I think you should look into the specifics a little closer.
It is correct that it can only solve the types of problems it has seen.
If the model is able to generalise beyond that then we’ve achieved AGE. Which we have not.
I work with AI. Specifically the actual implementation of them.
The way LLMs approach to a problem is the exact same approach as finishing a poem.
In other words it does not have the concept of problem solving, it is simply finishing text to the best of its abilities.
Why would I follow you if I didn’t want rants and tangents?
Go ahead :)
It is!
When he first shared the findings some time back I spent some time thinking about how to extract something valuable from it but came up short. All I’m left with is that it’s fascinating.
I feel like maybe it could tell us something about how to choose optimal precision, but 🤷
