Will further AI capability gains come from incremental scaling, or do they require a fundamental research breakthrough?

Arthur Mensch · Apr 29, 2024 · hedged

He believes substantially better models are still achievable at a given size, though it remains an open research question

It is as open a question as whether bigger models trained longer on the same data get much better; you can predict performance but ultimately have to run the research

Scope: explicitly framed as an open question; requires empirical trial

10:22 20VC: Mistral's Arthur Mensch: Are Foundation Models Commoditising | How Do We Solve the Problem of Compute | Is There Value in the Application Layer | Open vs Closed: Who Wins and Mistral's Position

Aidan Gomez · Aug 19, 2024

Scaling compute reliably improves models and is the most trustworthy path, but it is also the dumbest and most inefficient one — better methods exist.

In roughly the year and a half since GPT-4, models around 13B parameters have surpassed a reported 1.7T-parameter MoE, showing quality can be reached far more cheaply than by scale alone.

Scope: scaling remains low-risk and compelling for those with a lot of money

7:24 20VC: Chips, Models or Applications; Where is the Value in AI | Is Compute the Answer to All Model Performance Questions | Why Open AI Shelved AGI & Is There Any Value in Models with OpenAI Price Dumping with Aidan, Gomez, Co-Founder @ Cohere

Steeve Morin · Feb 24, 2025

Scaling continues but with enormous waste and excess spend on the engineering side; you can do more with less, as DeepSeek showed

DeepSeek changed a few things and got multiples of effective compute capacity by being more efficient; he aligns himself with the efficiency and 'need something better' camps rather than brute force

Scope: capital is still available so brute force will continue in parallel

48:32 20VC: Why Google Will Win the AI Arms Race & OpenAI Will Not | NVIDIA vs AMD: Who Wins and Why | The Future of Inference vs Training | The Economics of Compute & Why To Win You Must Have Product, Data & Compute with Steeve Morin @ ZML

Bucky Moore · May 5, 2025

There will always be new scaling dimensions for AI progress, so running out of ideas entirely is very unlikely at least in the near term

Pre-training looked exhausted and many rationally predicted a slowdown, then test-time compute, post-training and RL opened new scaling dimensions; the best and brightest are concentrated inside the big labs trying new things daily, and he has learned never to bet against human ingenuity

Scope: whether the next scaling dimension is as steep as the last is hard to say; 'at least in the near term'; concedes it could eventually happen

55:11 20VC Exclusive: Why Mega Platforms Will Win in VC | Why You Cannot Do VC If You Do Not Do Pre-Seed | Why Market Sizing is BS | Where Will Foundation Models Build/Buy Apps vs Where Will They Not with Bucky Moore

Joelle Pineau · Nov 3, 2025

Compute and data contribute to AI progress roughly linearly, while algorithms are the ingredient that produces nonlinear, paradigm-changing jumps.

More compute and more data reliably yield better performance, whereas ideas like the transformer, the Adam optimizer, and reasoning-in-the-loop each changed the paradigm outright.

Scope: data quality and diversity matter, not just quantity

11:12 20VC: Cohere's Chief AI Officer on Why Scaling Laws Will Continue | Whether You Can Buy Success in AI with Talent Acquisitions | The Future of Synthetic Data & What It Means for Models | Why AI Coding is Akin to Image Generation in 2015 with Joelle Pineau

Joelle Pineau · Nov 3, 2025

Scaling laws have proven remarkably robust and she would not bet against them, though they do not deliver progress alone without algorithmic innovation.

Many people have bet against scaling laws in the past and the effect has held up, even if it doesn't play out exactly as expected.

Scope: they don't play exactly as expected; require accompanying algorithmic innovation

13:14 20VC: Cohere's Chief AI Officer on Why Scaling Laws Will Continue | Whether You Can Buy Success in AI with Talent Acquisitions | The Future of Synthetic Data & What It Means for Models | Why AI Coding is Akin to Image Generation in 2015 with Joelle Pineau

Richard Socher · Aug 18, 2023 · hedged

The claim that no model used today will be used in a year is only technically right: exact weights get updated constantly, but a lot of the same general model architecture, including transformers, will still be running in production

We update our own model every other week, so weights change; but the architecture persists

Scope: hunch

18:14 20VC: Does Value Accrue to Incumbents or Startups in the AI Race, Why Model Size Matters More Than Data Size, Why Artificial General Intelligence is Far Away, Why Carpenters Will Be Paid More Than Software Engineers & Future of Jobs with Richard Socher

David Luan · Jun 24, 2024

The transformer was the first model architecture generally applicable to any machine learning task, becoming the universal base element of AI and largely removing the need for low-level modeling breakthroughs

Previously each task needed its own architecture — CNNs for images, RNNs for text, tree search or RL for Go — whereas the transformer just worked for everything, freeing researchers to attack big problems instead

6:33 20VC: Why Foundation Model Performance is Not Diminishing But Models Are Commoditising, Why Nvidia Will Enter the Model Space and Models Will Enter the Chip Space & The Right Business Model for AI Software with David Luan, Co-Founder @ Adept

Max Levchin · Feb 5, 2025

We are reaching the upper limit of LLM efficiency, but progress will continue by expanding the systems built around LLMs rather than staying with pure LLMs

Reasoning models are already not pure LLMs because they contain iterative processes, so more interesting composite systems will be invented

47:11 20VC: Affirm Max Levchin on Why Grading Talent by Letter (A or B) is Total BS | How to Create a Culture of Post Mortems and Writing | Why You Should Only Study Failure Not Success & The Biggest Surprises Scaling to $18.7BN Market Cap

Nick Frosst · Sep 1, 2025

Despite the constant stream of new model releases, the underlying technology hasn't fundamentally changed since the 2017 transformer

Models are still sequence models predicting the next word; what's been added are training stages like supervised fine-tuning and reinforcement learning, and nobody has trained something fundamentally different from a transformer

Scope: on the one hand models do iterate very quickly; training methods have genuinely improved

19:53 20VC: Cohere Founder on How Cohere Compete with OpenAI and Anthropic $BNs | Why Counties Should Fund Their Own Models & the Need for Model Sovereignty | How Sam Altman Has Done a Disservice to AI with Nick Frosst

Joelle Pineau · Nov 3, 2025

Neural networks are here to stay as the dominant machine learning paradigm, contrary to her earlier belief that each jump in data scale would eventually surface a better alternative

In prior generations neural nets were the first thing tried as universal function approximators and were then beaten by something else (e.g. SVMs in the early 2000s), but that pattern has broken — backpropagation and gradient descent have proven to be a genuinely powerful way to learn

Scope: framed as her own changed mind, offered as a belief she was wrong about

41:31 20VC: Cohere's Chief AI Officer on Why Scaling Laws Will Continue | Whether You Can Buy Success in AI with Talent Acquisitions | The Future of Synthetic Data & What It Means for Models | Why AI Coding is Akin to Image Generation in 2015 with Joelle Pineau

Demis Hassabis · Apr 7, 2026

LLM foundation models will not be replaced; AGI will be built on top of them, and the only open question is whether they are the key component or the total system

Foundation models have proven they can do incredibly impressive things and returns from scaling laws are still coming, so the sensible bet is that anything else needed gets layered on top rather than substituted

Scope: his 'betting', alongside a 50/50 chance something is still missing

12:46 20VC: DeepMind's Demis Hassabis on Why AGI is Bigger than the Industrial Revolution | Why LLMs Will Not Commoditise & We Have Not Hit Scaling Laws | Bottlenecks in AI & The Energy Crisis Caused By AI | Whether AI Will Do More to Harm or Help Inequality

Clem Delangue · May 12, 2023 · hedged

It is unclear that raising vastly more money for compute is the right approach, because returns on training ever-larger models are starting to decline and compute is weakening as a moat.

We are starting to realize more compute is not necessarily the right thing, or at least not enough, and the return on investment on training larger and larger models is going down.

Scope: explicitly 'not really sure'

26:06 20VC: Why The Future of AI Is Open Not Closed, Why We Are Years Away From AI Being Autonomous, Why AI Founders Do Not Need to Move to the Valley & Why Founders Should Not Meet Investors in Between Rounds with Clem Delangue @ Hugging Face

Harry Stebbings · Jun 24, 2024

There is a lot more room for AI improvement from the increases in compute availability that are coming

11:46 20VC: Why Foundation Model Performance is Not Diminishing But Models Are Commoditising, Why Nvidia Will Enter the Model Space and Models Will Enter the Chip Space & The Right Business Model for AI Software with David Luan, Co-Founder @ Adept

Ethan Mollick · Jul 31, 2024

The scaling curve still has lots of juice left and the exponential will continue for a while — he has changed his mind back to this view

Researchers who weren't talking about scaling solving everything six to eight months ago are now confident again, which suggests a new generation of models has shown them something and 'everyone at the labs has that haunted look again'

Scope: He has gone back and forth on this; he doesn't know when the models will be released

61:56 20VC: Is More Compute the Answer to Model Performance | Why OpenAI Abandons Products, The Biggest Opportunities They Have Not Taken & Analysing Their Race for AGI | What Companies, AI Labs and Startups Get Wrong About AI with Ethan Mollick

Mati Staniszewski · Sep 8, 2025 · hedged

Scaling laws are still holding and AI progress is only scratching the surface — the adoption escape curve is just getting started

Scope: self-described as a biased view

12:56 20VC: ElevenLabs Hits $200M ARR: The Untold Story of Europe's Fastest Growing AI Startup | The Real Cost of AI from Talent to Data Centres | How US VCs are in a Different League to Europeans | The Future of Foundation Models with Mati Staniszewski

Demis Hassabis · Apr 7, 2026

Scaling laws have not plateaued; returns to scaling existing systems are still very substantial, though smaller than at the start of the scaling era

Early generation-over-generation jumps were near-doublings that inevitably had to slow, but Google DeepMind and other frontier labs are still getting a lot of return on compute expansion

Scope: progress is no longer exponential; returns are somewhat less than at the beginning

6:55 20VC: DeepMind's Demis Hassabis on Why AGI is Bigger than the Industrial Revolution | Why LLMs Will Not Commoditise & We Have Not Hit Scaling Laws | Bottlenecks in AI & The Energy Crisis Caused By AI | Whether AI Will Do More to Harm or Help Inequality

Harry Stebbings · Apr 14, 2026 · hedged

Scaling laws may be weakening, with increased scale producing slightly diminishing returns in performance

Relayed from a prior conversation with Demis Hassabis of DeepMind

Scope: framed as uncertain about whether scaling laws still hold

3:59 20VC: Anj Midha on Investing $300M into Anthropic | The Early Days of Anthropic & How 21 of 22 VCs Turned it Down | The Four Bottlenecks to Compute | What the China Has Smashed and Why We Should Be Worried

David Luan · Jun 24, 2024

Foundation model performance is not hitting diminishing returns from compute, because a second lever for improving models is only now starting to be tapped and will absorb enormous amounts of compute

A second method of improving model performance is just beginning to be exploited and will itself consume a boatload of compute

0:00 20VC: Why Foundation Model Performance is Not Diminishing But Models Are Commoditising, Why Nvidia Will Enter the Model Space and Models Will Enter the Chip Space & The Right Business Model for AI Software with David Luan, Co-Founder @ Adept

David Luan · Jun 24, 2024

There is a second, barely-tapped axis for making models smarter — having the model collect its own positive and negative data by experimenting in simulated environments and tools — and it will absorb enormous amounts of compute, so diminishing returns to compute are not a worry.

Instead of only feeding in human-written solutions, you can give a model a theorem prover or Jupyter notebook, let it attempt problems and reflect on whether it succeeded, generating its own training signal.

Scope: holds even if base-model scaling itself plateaus

12:04 20VC: Why Foundation Model Performance is Not Diminishing But Models Are Commoditising, Why Nvidia Will Enter the Model Space and Models Will Enter the Chip Space & The Right Business Model for AI Software with David Luan, Co-Founder @ Adept

David Luan · Jun 24, 2024

The critical path for model improvement is now shifting from base-model scaling to simulation, synthetic data and RL loops, because base-model scaling has passed the steepest part of its S-curve.

Doubling spend from $100M to $200M used to be the fastest way to ship a smarter model, but at billion-to-four-billion-dollar training runs it becomes very hard to raise the money to make the base model bigger.

14:15 20VC: Why Foundation Model Performance is Not Diminishing But Models Are Commoditising, Why Nvidia Will Enter the Model Space and Models Will Enter the Chip Space & The Right Business Model for AI Software with David Luan, Co-Founder @ Adept

Alex Schultz · Sep 5, 2025

The biggest remaining algorithmic upside is in deep research (AI debating with itself) and in reinforcement learning for post-training combined with synthetic and high-quality expert data — these are public breakthroughs whose potential is nowhere near maximized

These techniques are already out in public and drove much of 2025's model success, but nobody is doing them maximally yet, so there's large headroom

Scope: not novel or groundbreaking observations to specialists

21:11 20Growth: Meta CMO Alex Schultz on How All Founders Have to Change Their Marketing Playbook in a World of AI | Is AI Plateauing, What it Means if China Wins the AI Race and Why Zuck is a Generational Leader

Harry Stebbings · Aug 18, 2025

GPT-5 is not a step-function capability improvement over what existed before — it is essentially model optimization and routing, judged against an AGI-level expectation

The bar was set at AGI, and what shipped is model routing rather than new capability

19:09 20VC: Lovable CEO Anton Osika on $120M in ARR in 7 Months | The Honest Truth About Defensibility and Unit Economics for AI Startups | The State of Foundation Models: Long Grok, Short OpenAI, Why | Replit vs Lovable vs Bolt: What Happens

Nick Frosst · Sep 1, 2025

GPT-5 was worse than GPT-4, which tells you something about the limits of just throwing more compute at the problem

10:47 20VC: Cohere Founder on How Cohere Compete with OpenAI and Anthropic $BNs | Why Counties Should Fund Their Own Models & the Need for Model Sovereignty | How Sam Altman Has Done a Disservice to AI with Nick Frosst

Harry Stebbings · Sep 8, 2025 · hedged

Model progress may be plateauing into incrementalism, as suggested by GPT-5 being pitched on cost and efficiency rather than new capabilities

GPT-5's launch shifted from describing incredible new capabilities to describing cost and efficiency, which signals the capability ceiling has been hit

11:59 20VC: ElevenLabs Hits $200M ARR: The Untold Story of Europe's Fastest Growing AI Startup | The Real Cost of AI from Talent to Data Centres | How US VCs are in a Different League to Europeans | The Future of Foundation Models with Mati Staniszewski

Demis Hassabis · Apr 7, 2026 · hedged

Solving continual learning may require a brain-like consolidation mechanism that replays and elegantly incorporates new information into the existing knowledge base

The brain does this very elegantly, probably via sleep-based replay and consolidation of the day's memories, whereas labs haven't figured out how to integrate new learning into models that took months to train

Scope: all leading labs are working on the problem; a long-held intuition rather than a result

8:20 20VC: DeepMind's Demis Hassabis on Why AGI is Bigger than the Industrial Revolution | Why LLMs Will Not Commoditise & We Have Not Hit Scaling Laws | Bottlenecks in AI & The Energy Crisis Caused By AI | Whether AI Will Do More to Harm or Help Inequality

Jerry Murdock · Aug 22, 2026

Continuous learning cannot be bolted onto an existing frontier model; it will require a new architecture and completely new training

Models today are trained statically and then deployed; continuous learning is architecturally different

Scope: Labs will try, and early attempts will look like bolt-ons

49:44 20VC: The AI Bubble Will Burst: Half the Neoclouds Will Die | China: Should We Ban Chip Exports & Be Fearful of Chinese Open-Source | Mag7: Who Dies and Who Thrives: Why Meta is Meh and Microsoft is Mega

Jerry Murdock · Aug 22, 2026 · hedged

Continuous learning models are probably two to three years away but could be ten, and getting there will require a genuine breakthrough rather than incremental progress

Sample-efficient models are starting to show small successes but aren't robust enough to be useful yet, and the complexity being tackled is huge — like cancer, where fifteen years of 'we're on the cusp' produced management, not a cure

Scope: timing genuinely unknown; won't bet against the labs

53:00 20VC: The AI Bubble Will Burst: Half the Neoclouds Will Die | China: Should We Ban Chip Exports & Be Fearful of Chinese Open-Source | Mag7: Who Dies and Who Thrives: Why Meta is Meh and Microsoft is Mega

Aravind Srinivas · Jun 5, 2024

The next generation of models will produce an output, reason over it, elicit feedback from the world, and improve their reasoning — marking the start of a real reasoning era

Today's models only give you the output, with no loop back through feedback and revised reasoning

0:00 20VC: Perplexity's Aravind Srinivas on Will Foundation Models Commoditise, Diminishing Returns in Model Performance, OpenAI vs Anthropic: Who Wins & Why the Next Breakthrough in Model Performance will be in Reasoning

Aravind Srinivas · Jun 5, 2024

The next breakthrough in reasoning will come from bootstrapped self-improvement — models that produce an output, explain their reasoning, get feedback, and iterate until convergence — and that iterative loop is exactly what today's models lack.

Training on the explanation as well as the output teaches the model to think, check, and re-reason rather than just emit an answer; today's models only give you the output.

Scope: the secret sauce isn't known yet

15:02 20VC: Perplexity's Aravind Srinivas on Will Foundation Models Commoditise, Diminishing Returns in Model Performance, OpenAI vs Anthropic: Who Wins & Why the Next Breakthrough in Model Performance will be in Reasoning

Aidan Gomez · Aug 19, 2024

The meme that AI progress has plateaued is wrong — a major capability step change is coming from methods, not just more compute

Reasoners, planners and models that can try, fail, recover and carry out long-horizon tasks simply don't exist in today's technology because nobody had turned their focus there; teams have been working on it for over a year and it will be production-ready

Scope: next 12–24 months

44:10 20VC: Chips, Models or Applications; Where is the Value in AI | Is Compute the Answer to All Model Performance Questions | Why Open AI Shelved AGI & Is There Any Value in Models with OpenAI Price Dumping with Aidan, Gomez, Co-Founder @ Cohere

Joelle Pineau · Nov 3, 2025

AI code generation today is at the stage image generation was in 2015, and in another ten years the quality of generated code will be excellent

Image generation was bad in resolution and composition in 2015 and improved enormously by around 2022; code generation is on the same trajectory, so today's bad and discarded output is a phase

Scope: concedes a lot of currently generated code is bad and will be thrown away

37:31 20VC: Cohere's Chief AI Officer on Why Scaling Laws Will Continue | Whether You Can Buy Success in AI with Talent Acquisitions | The Future of Synthetic Data & What It Means for Models | Why AI Coding is Akin to Image Generation in 2015 with Joelle Pineau

Andrew Ng · Nov 17, 2025 · hedged

AI coding assistants are further along in maturity than image generation was in 2016-2017, because coding assistants are already delivering real value

In 2016 image generation wasn't super valuable, whereas today engineers refuse to give up coding tools and he would not want to code without them

Scope: 'I don't know' preface; still a lot of headroom for improvement

8:31 20VC: Andrew NG on The Biggest Bottlenecks in AI | How LLMs Can Be Used as a Geopolitical Weapon | Do Margins Matter in a World of AI? | Is Defensibility Dead in a World of AI? | Will AI Deliver Masa Son's Predictions of 5% GDP Growth?

Douwe Kiela · Jun 30, 2023

He was wrong to dismiss scaling: throwing an order of magnitude more compute and data at AI systems makes them dramatically better, and this is what OpenAI excelled at.

The scaling laws we now know about show continued scaling keeps improving systems; he used to mock OpenAI researchers for scaling instead of inventing new algorithms.

36:27 20VC: Why Data Size Matters More Than Model Size, Why The Google Employee Was Wrong; OpenAI and Google Have the Advantage & Why Open Source is Not Going to Win with Douwe Kiela, Co-Founder @ Contextual AI

David Cahn · Aug 5, 2024 · hedged

Current evidence favours the scaling-laws school over the research-breakthrough school on where AI progress comes from.

Two camps exist — pure scale versus breakthroughs in reasoning, data use and efficiency — and today's evidence points more strongly toward scale.

Scope: he hopes for more evidence of non-scale breakthroughs; having both would be better than one

19:47 20VC: Sequoia's David Cahn on AI's $600BN Question | Why the Data Centre is the Most Important Asset | Servers, Steel and Power: The Core Pillars Powering the Future of AI

Alexandr Wang · Jun 12, 2024

No model jaw-droppingly better than GPT-4 has appeared since GPT-4, despite vastly more compute expenditure across the industry

NVIDIA data center revenue inflected from ~$5B to north of $20B a quarter after ChatGPT, meaning well over $100B of GPU spend, yet the industry is still waiting for the next great model — and GPT-4 itself predated that spending inflection

3:56 20VC: Scale's Alex Wang on Why Data Not Compute is the Bottleneck to Foundation Model Performance, Why AI is the Greatest Military Asset Ever, Is China Really Two Years Behind the US in AI and Why the CCPs Industrial Approach is Better than Anyone Else's

Matt Clifford · Jul 1, 2024

The value of simply adding more compute and data to language models is flattening off — we are at the top of that S curve.

Most technologies follow S curves of slow, then fast, then slow progress, and the incremental value of continued scaling is now levelling off.

7:25 20VC: LLMs Are Reaching a Stage of Diminishing Returns: What is the Next S Curve | The Bull & Bear Case for China's Ability to Challenge the US' AI Capabilities | How AI Changes the Future of War & How Agents Will Reshape Society with Matt Clifford @ EF

Christian Kleinerman · Sep 22, 2023 · speculative

The next big innovation is the commoditization of computer vision and images following the same path as language models, plus truly seamless multimodal interfaces intersecting images, speech and text

What has already happened for language models is coming for vision and images, and the richer human-computer interface comes from seamlessly intersecting modalities

17:23 20VC: Are Foundation Models Becoming Commoditised? Do OpenAI and Anthropic of the World Have a Sustaining Moat? Why Smaller Models May Work Better? Why Incumbents with Data Power Win the AI War with Christian Kleinerman, SVP Product @ Snowflake

Matt Clifford · Jul 1, 2024 · hedged

Search techniques of the kind that made AlphaGo work, combined with LLMs, and better handling of non-text modalities are the most promising candidates for the next S curve.

Scope: 'could be'; 'might be another S curve'

9:12 20VC: LLMs Are Reaching a Stage of Diminishing Returns: What is the Next S Curve | The Bull & Bear Case for China's Ability to Challenge the US' AI Capabilities | How AI Changes the Future of War & How Agents Will Reshape Society with Matt Clifford @ EF

Matt Clifford · Jul 1, 2024

GPT-5 will not just be a bigger language model and Llama 4 will not just be a bigger Llama 3; the next generation will have to be architecturally or productively different

Simply adding compute and data has already been exhausted as the source of gains

13:42 20VC: LLMs Are Reaching a Stage of Diminishing Returns: What is the Next S Curve | The Bull & Bear Case for China's Ability to Challenge the US' AI Capabilities | How AI Changes the Future of War & How Agents Will Reshape Society with Matt Clifford @ EF

Ethan Mollick · Jul 31, 2024

Moore's law stayed on a sustained exponential only because the underlying technologies were repeatedly swapped out, so the real question for AI is what top-line intelligence maxes out at

Sustained exponentials in technology come from substituting underlying technologies, not from one technology improving forever

11:23 20VC: Is More Compute the Answer to Model Performance | Why OpenAI Abandons Products, The Biggest Opportunities They Have Not Taken & Analysing Their Race for AGI | What Companies, AI Labs and Startups Get Wrong About AI with Ethan Mollick

Avishai Abrahami · Jul 13, 2026

LLMs cannot do frontier science like astrophysics research

They may produce an answer but they don't run the underlying models and simulations; that requires different architectures

41:06 20VC: Wix's Founder on What Wall St Gets Wrong About AI and Wix | Will Base44 Win the Vibe Coding Wars | The Truth About the Economics of Vibe-Coding | The Buyback Disaster: Lessons Learned with Avishai Abrahami

Avishai Abrahami · Jul 13, 2026 · hedged

Model providers could reach scientific capability only by connecting different model types and architectures to their LLMs, which is not trivial

Breakthroughs like DeepMind's protein folding came from non-LLM architectures, not from LLMs alone

Scope: 'maybe' on whether providers do it within three years

41:29 20VC: Wix's Founder on What Wall St Gets Wrong About AI and Wix | Will Base44 Win the Vibe Coding Wars | The Truth About the Economics of Vibe-Coding | The Buyback Disaster: Lessons Learned with Avishai Abrahami

Matt Clifford · Jul 1, 2024 · hedged

The enormous capital and talent aggregated in a small number of AI labs will likely be enough to unlock the next S curve beyond simply scaling LLMs

The hype has concentrated so much capital and talent in a few companies that the returns to good ideas over the next few years will be very high

12:40 Concentrated capital and talent in labs will likely unlock the next s curve

Jerry Murdock · Aug 22, 2026

Continuous learning models will arrive within ten years and will replace every model that exists today, followed by lifelong learning models

Continuous learning is a top goal of frontier labs so models can handle more complicated tasks; physical AI and robots need persistent memory to learn and handle dynamic events, which requires fundamentally different models than anything trained to date

48:15 Continuous learning arrives within a decade

Aidan Gomez · Aug 19, 2024 · hedged

Scaling keeps making models smarter for a very long time without plateauing technically, but it requires exponentially more compute for linear intelligence gains, so the real binding limit is economic rather than technical.

You must keep doubling compute to sustain linear gains, and buyers won't pay for models that are too costly to serve relative to their intelligence.

8:35 Scaling continues technically indefinitely but economic serving costs bind first

Aidan Gomez · Aug 19, 2024

Almost all of the major gains in the open source model space have come from data improvements rather than scale.

Better scraping and parsing, upweighting the knowledge-rich parts of the internet over repetition and junk, and scalable synthetic data generated by models have driven most recent quality gains.

10:55 Open source gains come from data improvements not scale

Daniel Dines · Dec 18, 2024

Models will not innovate in a material way within a reasonable amount of time; LLMs have reached a kind of maturity

He is already satisfied with what both frontier models and smaller open models deliver for his use cases

7:18 Llms have reached a kind of maturity with no material innovation expected soon

Daniel Dines · Dec 18, 2024

Reaching AGI will require a new giant leap, not more of the current LLM paradigm, because today's LLMs do not actually reason — they are stochastic engines and represent a different kind of intelligence

He is worse than LLMs at math olympiad problems yet better at things they fail at, and LLMs make logical mistakes he never would — that asymmetry shows the essence of their intelligence is different, and it is not equipped for business operations that require reliability

33:05 Reaching agi requires a new paradigm since llms are stochastic not reasoning engines

Des Traynor · Nov 15, 2023 · hedged

AI is still in the steep spike of an S-curve rather than a plateau, so a company all-in on AI has no option but to keep adapting to OpenAI's rapid new capability releases.

The change is very big for customer service, technology and society generally, and only a government-sanctioned monopoly could afford to wait three years.

18:00 Ai remains in a steep s curve forcing continuous adaptation to new releases

David Luan · Jun 24, 2024

Model scaling is not showing diminishing returns; while each incremental GPU yields less, every doubling of compute delivers predictable and consistent gains in model intelligence

The apparent diminishing returns are an artifact of which axis you plot — the relationship is logarithmic, so returns look linear when measured per doubling of compute, as seen from GPT-2 to GPT-3 to GPT-4

11:06 Diminishing returns are an artifact of per gpu framing not per doubling of compute

David Luan · Jun 24, 2024

AI model progress will not plateau the way self-driving did, because the field is still making visible scientific breakthroughs rather than playing whack-a-mole on reliability

Self-driving got something working 60% of the time and then spent years chasing 99.99999% reliability; by contrast, in model building there are genuinely new scientific bets each day that dramatically improve performance — reasoning and GPT-4o's universal multimodality being examples — and more such shoes are yet to drop

41:59 Unlike self driving ai progress continues via genuine scientific breakthroughs not reliability grinding

Amjad Masad · Apr 25, 2026

Coding models are approaching a plateau in how good they can get

0:00 Coding model gains are flattening into a plateau

Nick Frosst · Sep 1, 2025

Belief in scaling laws as the path to AGI is not as widespread as the discourse suggests

If you ask computer science students at a university whether throwing more compute gets us to AGI, most say no

13:41 Scaling to agi belief is overstated by media not widely held by practitioners

Reid Hoffman · Jun 10, 2024

We are already in the equivalent of a resumption of Moore's Law acceleration, even though transistor-level Moore's Law did decelerate

Builders realized they could construct much bigger data centers and tune new chips like GPUs to the specific mathematical and graphics functions that matter

12:33 Compute scaling via bigger datacenters and tuned chips sustains moores law pace despite transistor slowdown

Reid Hoffman · Jun 10, 2024 · hedged

Attention transformers are unlikely to be the only path to scale learning systems, so alternative architectures may still produce interesting new frontier models

There is no reason to think one architecture is the single true path

18:19 Attention transformers unlikely to be the sole path to scaled learning systems

Demis Hassabis · Apr 7, 2026 · hedged

Long context windows are a brute-force approach to memory and there are better memory architectures still to be invented, alongside long-horizon hierarchical planning which current systems do poorly

Humans can plan many years into the future, and current systems just stuff everything into the context window

10:38 Long context is brute force new memory and planning architectures are needed

Winston Weinberg · Jan 19, 2026

Model performance is plateauing for consumer use cases but not for enterprise, and the consumer plateau doesn't matter because consumer needs are context and app connections rather than better reasoning

Consumer problems were largely solved at GPT-4-level reasoning; what counts as improvement there is calendar and app integrations, whereas enterprise capability keeps advancing

23:41 Plateau is consumer side only enterprise capability still advancing

Anjney Midha · Apr 14, 2026 · hedged

In materials science, throwing more compute at the problem is currently producing super-exponential gains per iteration

Periodic Labs runs a loop where LLMs predict new materials, robots synthesize them, physical machines like x-ray diffraction validate the predicted properties, and that verification data is piped back into training

4:19 Physical verification loops give super exponential returns to compute

Aravind Srinivas · Jun 5, 2024 · hedged

There will be another genuinely great model after GPT-4, and whether GPT-5 leapfrogs 4 the way 4 leapfrogged 3.5 is the real test of whether frontier models commoditize.

GPT-4o is more reliable, faster and cheaper but not much smarter than GPT-4 Turbo, so the question of a true generational jump is still open.

19:48 Whether gpt 5 leapfrogs gpt 4 as much as gpt 4 leapfrogged 3 5 is the test of continued scaling returns

Jonathan Ross · Feb 17, 2025

Training and test-time reasoning must be paired: better training makes a model more intuitive, and adding system-two reasoning on top yields polylinear (geometrically increasing) improvement.

Training gives system-one, stream-of-consciousness answers; the reasoning algorithm on top is the Big O complexity portion, and combining improved training with improved run-time compute compounds the gains

9:01 Pairing improved training with test time reasoning yields polylinear gains

Andrew Feldman · Mar 24, 2025

We will be much less dependent on transformers in three to five years, though what replaces them is unknown

Innovation doesn't stop, and the transformer has weaknesses people are desperate to overcome, such as the quadratic effect in the attention head; you run with the best you have and drop it the minute it isn't

29:37 Transformers will be substantially replaced within three to five years despite unclear successor