Latent.Space
@latentspacepod · 5d ago"The cost of intelligence has been falling dramatically over the last couple of years.
The best way to think about that is that the cost for each tera of intelligence has been dropping. One fact on that is that you can get intelligence at the level of GPT-4 for over 100x cheaper than GPT-4 was at launch."
Micah breaks down one of the big trends he has been seeing at Artificial Analysis on the cost of intelligence significantly dropping.
@_micah_h
00:27
Latent.Space
@latentspacepod · 1w agoFrom a scrappy side project built to solve their own LLM optimization problems to becoming the industry’s de-facto independent scoreboard, Micah Hill-Smith and George Cameron went through the arc of launching Artificial Analysis for free, paying benchmarking costs out of pocket, and growing it into what many now call the “new Gartner of AI” for enterprises, labs, and developers.
We sat down with Micah and George to unpack why truly independent benchmarking is so hard (prompt variance, eval saturation, mystery-shopper policies), how the Artificial Analysis Intelligence Index evolved as old benchmarks broke, and what new metrics actually matter now such as agentic evals (GDPVal-AA). We also dig into the economics behind the “smile curve” of AI: why intelligence is getting 100–1000× cheaper per unit while total spend explodes, how reasoning and agents change token efficiency, and their bet that evals must continuously evolve or risk training the industry to optimize for the wrong things.
@swyx @_micah_h @grmcameron
00:45
4
5
37
31K
1
0
6
2
2

