There will also be continued interest in methods that allow more controllable manipulation of generated images (e.g. UIP2P arxiv.org/abs/2412.15216, Imagic arxiv.org/abs/2210.09276).
But maybe the pending copyright lawsuits will have major impacts on GenAI.
For GenAI, improvements to generation quality are going to come from better data curation and value fns to drive the model toward high-quality outputs. Standard model training results in outputs representative of the training distribution, but users don't want average—we want the best quality.
This is crucial for AGI, and will pose serious safety concerns. Models better at thinking outside the box and coming up with creative solutions will have broader implications than the prompter anticipated arxiv.org/abs/2412.04984.
Humans solve novel problems on-the-fly using System 2 reasoning: AI needs this too. By learning reasoning steps at training time, at deployment the model can build new sequences of reasoning steps, enabling it to extrapolate.
Reasoning capabilities are essential to reach robust perf in key ML products e.g. full self-driving. The distribution of driving scenarios is long-tailed, so even a model that covers most situations well may be faced with a novel situation outside its training data, but needs to respond correctly.
One way to do this is hierarchical LLMs like the Large Concept Model arxiv.org/abs/2412.08821, Byte Latent Transformer arxiv.org/abs/2412.09871, and Block Transformer arxiv.org/abs/2406.02657.
For agentic models, the focus is shifting to System 2-like reasoning. OpenAI’s o1/o3 models demonstrate reasoning step-by-step can improve output quality by leveraging test-time compute. But impressive results on ARC are expensive, hence there will be a focus on improving test-compute efficiency.
But the LLM training corpus is now the majority of worthwhile text humanity has ever written, and can't be meaningfully scaled further. As Ilya Sutskever put it at NeurIPS, "big data is the fossil fuel of AI".
With this in mind, what will be the next stage of AI development?
The turn of the year is a good time for reflection. Here's my thoughts on where ML is headed.
Advances have been driven by scaling—bigger compute, bigger data: bigger models. Moreover, larger data also gives a solution to OOD generalization—just increase the train set until everything is in domain!
This has some serious AI safety implications. Having an AI model able to classify what is in an image better than a human doesn't pose an existential threat. But when an AI model can perform long-term planning better than a human, "just unplug it" ceases to be a reliable solution
In "System 2 Reasoning Capabilities Are Nigh", I lay out comparisons between human reasoning and reasoning in AI models, and argue that all the components needed to create AI models that can perform human-like reasoning already exist.
arxiv.org/abs/2410.03662
It's very easy to get started with using the dataset. The commands to download it and load it for PyTorch training fit in less than half a tweet:
!pip install bioscan-dataset
from bioscan_dataset import BIOSCAN5M
ds = BIOSCAN5M("~/Datasets/bioscan-5m", download=True)
The dataset should be useful for a variety of research topics:
- multimodal learning
- fine-grained classification
- hierarchical labelling
- open-world classification/clustering
- semi- and self-supervised learning
BIOSCAN-5M is a multimodal dataset for insect biodiversity monitoring. It consists of 5 million insect specimens from around the world, with a high-res microscopy image, DNA barcode, taxonomic labels, size, and geolocation info for each sample.
arxiv.org/abs/2406.127...
I'm looking forward to NeurIPS this week! I'll be presenting two papers there. In the main conference, our new dataset BIOSCAN-5M, and in the System 2 Reasoning At Scale workshop my position paper "System 2 Reasoning Capabilities Are Nigh".
