Finally, it makes me a lot less worried about the AI-doomer narrative. This is still tractable. The basics are not that different from what we knew about DL 8-10 years ago. The scale is different, of course! And as I am working on gamedev, I was reassured that I could deal with it, if need be!
But I'm glad I did this end-to-end. Learnt a lot about the theory of deep-learning (refreshed my 10-year old knowledge as well on SGD, validation metrics etc) and the practical difficulties of doing this on a larger scale with distributed compute.
...but CUDA-based workflows are just way shorter time-to-get-to-training. On CUDA, I don't have to debug my sharding of data across workers, think too deeply about my compute graph yada yada - all of which I do have to take care of on TPUs. Configuring for deployment becomes a pain!
4. You can't get NVIDIA GPUs on AWS or GCP at all (I don't have a long billing history with those services)! I wanted to push context lengths higher with higher VRAM or more TPUs but the higher end configs have no availability (and you get quota limited). Had to do with v5 TPU pods, which work...
3. With an RTX 5090, it is eminently possible to train a model with 1024 embedding dimensions, 512 token context size, 32ish feedforward layers, 4 MTP depth and let it run on 17-20GB of tokenized English Wikipedia with a 131k vocabulary. Let it run for a couple of days. No latent attention.
Some key takeaways are:
1. Clean datasets matter a LOT for training good models. More than model-size/architecture. A 40-50M parameter model might outperform a 10x bigger model if trained on a nicer dataset.
2. Pre-compile your layers! Use inductor (GPUs) or openxla (TPUs etc). 6-15x speedups!
Spent the last 20 days obsessively learning about Transformers and LLMs. The bulk of which was learning about Pytorch, Lightning, OpenXLA. And parsing (and cleaning) wikipedia as a training dataset. Was finally able to train models of 10M-500M parameters and 32-1024 token context on a GPU and TPUs.
This is such a powerful message on AI and art from Brandon Sanderson: www.youtube.com/watch?v=mb3u...
More progress made on the Hikari engine. After implementing per-frame BVH-updates, I now have collision-detection (and some minor physics) in place. 1000 balls bouncing around a terrain with 5 animated meshes. Pretty decent realtime perf. #gamedev #gameengine #gamephysics youtu.be/UGjnM480xtI
Switching to Release mode, the thing now refuses to go below 60FPS even with 80 such meshes, even when I switch to battery mode on my laptop! Good lesson as I am now going to optimize things in Debug mode at "low power" profiles so that the real thing has large safety factors built-in. #gamedev
I have a stress test for the game-engine I'm writing in which I have 20 animated meshes with 1m vertices with per-frame BVH calcs 3-levels deep on the CPU. Was at 1-2 FPS and a couple of months of optimizations led to ~20FPS. Then I realized I was compiling in Debug mode.🤦♂️ #gamedev
...into them motivated by human reasoning. It is not clear whether the training of these models would have led to CoT reasoning being "discovered" spontaneously by these models.
Repeat after me: Transformers are just glorified statistical distribution learners and still represent memory rather than logic. What they have demonstrated that a lot of what we do with text and images might be similar to this. But even the "reasoning" models have gating criteria built...
Sometimes I wonder why people keep making the claims about LLM intelligence that they do (Gemini 3 Pro used here). This is well within the reach of an 8-9 year old (with some patience!) whose energy budget, even while calculating this, is far below that of the hardware that runs these AI models.
Also, finished the Golden Kamuy manga (the final season of the anime is airing). Was pretty good till the middle but then stretched the story for far too long, imho. Nevertheless, very good overall and a very unique setting which educated me a lot about the history of Hokkaido and nearby regions.
Back from the holidays and my codebase feels completely alien to me.
I believe that, in 2025, gaming on Linux has taken an irreversible leap. I am not an early adopter by any means (I like stable platforms as a dev), but this is my Steam replay for 2025. #linuxgaming #steamonlinux
It dawned upon me last evening that cities tend to to outlast countries, kingdoms and even civilisations. And they are able to preserve part of their existing identity while amalgamating new ones. There is something very resilient about cities as a construct.
Finished the remaster of Myst (2021), my first time ever playing the franchise - I admit I had to use walkthroughs on more than one occasion! But what a game and quite a reminder of how minimalism can still lead to a thoroughly engrossing game! #myst #riven
Perhaps there will be drastically different stuff down the line, but the more I use AIs, the more I become convinced that the current crop will not replace us. The capabilities of the SotA LLMs seem to be saturating very quickly and they are becoming (very useful) tools rather than replacements.
Implemented real-time collision-detection in my engine using per-frame BVH calculation for animated meshes. The colliding objects have their BVH drawn in orange with the intersecting nodes highlighted in cyan. (Also some basic heightmaps and fog). Next step: actual physics! #gamedev #gameengine
Finished playing #Exit8 - very innovative and very inspiring if you are an indie gamedev.
Finally, glTF morph and skeletal animations implemented within the engine. And now a dynamic bounding volume hierarchy for both static and animated meshes being calculated in realtime. #gamedev #gameengine
Mt. Fuji while landing into Tokyo Haneda airport today. #tokyo #mtfuji
My hacky solution for this has been to generate single-pixel dummy textures even for meshes that do not need them and slotting them in manually (eg, an object may not need an emissive or occlusion texture but the shader has those samplers declared). Not a huge performance/memory cost but ugly.
And, as is the case for most shader bugs, zero feedback from the GPU, especially my NVidia one. Surprisingly, it was on an Apple M4 Pro GPU that I got some feedback from the driver alerting me to the root cause (ie, the shader expecting a slotted texture because a sampler uniform was declared).
Had a lot of trouble figuring out a weird bug where, if you declare a sampler2D texture-sampler uniform var in GLSL and there are no textures slotted in, the shader goes kaput even if the sampler itself is never used (gated by conditionals). And that was my main PBR shader for most meshes!
Added Apple Neural engine support as well through python bindings to whisper.cpp (via pywhispercpp). Feels like it helps battery life more as compared to using the MPS backend on pytorch. #pytorch #whispercpp
Got real-time bounding-volume hierarchy creation in place in my engine. The left mesh with the cans and drums has 300k triangles. #gamedev
Just saw on HN that ffmpeg added whisper support! Darn it! But kudos to them!: code.ffmpeg.org/FFmpeg/FFmpe...
