I've argued for some time that AI models are at an interesting cul-de-sac. There are least three reasons:
- Increasingly exhausted pre-training data
- The architectural limits of large language models themselves
- The sharply worsening economics of model training
Granted, it is a nice cul-de-sac to have ended up in. Model gains were tremendous in the 2018-2023 period, and current models, prudently used, are highly useful.
But, it felt for a while to many that scaling laws were up there with F=ma in terms of describing a basic law of the computational universe. Throwing more data and compute at model variants would produce predictable engineered-in, just-add-capex, board-friendly gains.
This has turned out to not be the case. Instead, model releases have become more expensive and incremental, often indistinguishable from their predecessors, outside of esoteric and often gamed benchmarks.
With that in mind, a weekend Bloomberg piece tries to make the counter-argument, at least implicitly. It hints, heroically, and mostly citing the claims of insiders, that breakthroughs are ahead, in particular in continuous learning.
Continuous learning is one of the main problems with current models, which have specific end dates wrt training, and subsequent gaps must be backfilled by web search. Perhaps worse, they do not learn from what they are exposed to or retrieve, and attempts to make them do so often lead to catastrophic forgetting, wherein they not only fail to learn but also forget what they previously knew. It remains an unsolved research problem.
This passage has some choice quotes on how that problem is ... still not solved:
Google DeepMind considers continual learning to be among a handful of research areas that, if solved, could help achieve artificial general intelligence, according to a person familiar with the company who spoke on condition of anonymity to discuss internal matters. Google Research recently published a paper on a new method for continual learning that has attracted attention in the field.
OpenAI’s Sam Altman has also publicly hinted that continuous learning is key to AGI, a term that refers to AI systems that match or exceed humans on many tasks.
“I think we’ll start to get to a place where we put these systems together in new ways and we have kind of continuous learning where the systems just run forever and get smarter and smarter,” said Altman during a talk with venture capitalist Vinod Khosla earlier this year.
In the past few weeks, prominent figures such as former OpenAI chief scientist Ilya Sutskever and AI pioneer Richard Sutton have also stressed the importance of this approach to build smarter artificial intelligence systems. The heightened interest in the technique reflects the industry’s push to find new architectures and algorithms to advance the field, as some labs have confronted diminishing returns from existing methods.
Note what's going on here. Everyone agrees that models have reached a kind of pre-training dead end, even if they don't say that out loud, and even if the continuing utility of massive training runs underlies much of current capex, and they swap in an unsolved problem as a solution.
Granted, there are currently some workarounds. For example, retrieval augmented generation lets models access external databases, but it doesn't make the underlying model smarter.
In the interim, leading AI developers are pushing out relatively trivial updates to their models at a faster pace. Anthropic has said it's doing "more incremental improvements rather than only shipping the really big upgrades." OpenAI's GPT 5.2 came out this week to a mostly meh response. The pace of releases creates the impression of momentum through frequency rather than the magnitude of change.
What This Reveals
The technical reality:
- Pre-training gives broad knowledge but with a hard cutoff date
- Fine-tuning improves specific tasks but doesn't expand general knowledge
- Neither allows for open-ended learning as humans do
The AI industry spent years betting that scaling—more data, more compute, bigger models—would produce AGI. That bet has not paid off. The improvements are real, but increasingly incremental and slowing, while costs soar. The systems are impressive but bounded.
Now the labs are returning to older, harder problems. Continual learning. New architectures. Different training methods. These are necessary research directions, but they are a reminder that the next five years will be nothing like the last five. They're the work of an industry recalibrating after hitting a wall.
The article cheerfully frames this as labs "eyeing new breakthroughs." The reality: engineered-in gains via expensive scaling have run their course, and they are trying to figure out what to do next. There is no eyeing, contrary to the piece's headline, just hoping.