Does the cost of building each successive generation of frontier AI models rise or fall over time, unlike traditional software development?

Aidan Gomez · Aug 19, 2024

Falling compute costs (rising FLOPs per dollar) offset the rising expense of squeezing out incremental model gains

Costs are falling super fast on everything, including dollars per FLOP

25:17 Falling compute costs offset rising cost of incremental gains

Aidan Gomez · Aug 19, 2024

The falling price per FLOP is what has unlocked training much larger models today than was possible in 2017 or even two years ago

Parameter counts translate roughly proportionally into required FLOPs, and the cost per FLOP drops very quickly over time

25:41 Falling price per flop enables larger model training over time

Aidan Gomez · Aug 19, 2024

Falling compute costs only lower the barrier to building last year's model, which is worthless because there is no market for a previous-generation model

Costs to rebuild the prior generation fall by 10-100x a year thanks to better data and cheaper compute, but any technological development makes the last generation obsolete very quickly and nobody wants it

26:33 Falling compute costs only cheapen obsolete previous generation models real frontier costs still rise

Harry Stebbings · Aug 19, 2024

Model building differs from software because each successive generation costs an order of magnitude more rather than an incremental amount

In software v1 might cost $10M and v2 another $1-2M, whereas in models v1 costs billions and v2 costs multiples of that

27:03 Model generations cost order of magnitude more unlike incremental software costs

Aidan Gomez · Aug 19, 2024 · hedged

It is not a universal pattern that the next generation of a technology is cheaper to build; for complex technologies like chips each generation gets more expensive and is still worth doing

Very complicated pieces of technology like chips get more expensive per generation and we still build them because the payoff justifies it

27:22 Complex technologies like chips get more expensive each generation and are still worth building

Christian Kleinerman · Sep 22, 2023

The cost of training models will fall, driven both by cheaper compute and by reinvention of the training approach

Foundation models share many of the same datasets and near-identical processes, so taking a common subset and fine-tuning on top avoids duplicated cost — and results down that path have been reasonably good

22:31 Training cost falls via cheaper compute and shared dataset reuse across model families