Is language a sufficient representation for agents to understand the world?

Nick Frosst · Sep 1, 2025 · hedged

Language will become a more important interface for computing, though not the whole of it — graphical interfaces remain better for some tasks — and nobody has yet built the right language-first device

The Rabbit R1 and Humane pin gestured at something interesting but didn't get it right

Scope: 'I don't know if it will' be done right; language is not the best interface for everything

52:30 20VC: Cohere Founder on How Cohere Compete with OpenAI and Anthropic $BNs | Why Counties Should Fund Their Own Models & the Need for Model Sovereignty | How Sam Altman Has Done a Disservice to AI with Nick Frosst

Joelle Pineau · Nov 3, 2025

Language will remain the core paradigm for communicating with machines even as the prompt box disappears

Words are symbols that encode enormous information very efficiently and so much of human communication is language-based

Scope: distinguishes language-as-paradigm from the text prompt box

40:28 20VC: Cohere's Chief AI Officer on Why Scaling Laws Will Continue | Whether You Can Buy Success in AI with Talent Acquisitions | The Future of Synthetic Data & What It Means for Models | Why AI Coding is Akin to Image Generation in 2015 with Joelle Pineau

Steeve Morin · Feb 24, 2025

He is very bullish on world models / the JEPA approach as the successor to LLMs

Language is only one form of communication and a very narrow, lossy window into the world; JEPA works by minimising the energy to move between states, from which physics and intuitive behaviour emerge, which matches how the brain actually operates; and it comes from Yann LeCun, who is no bullshit

Scope: 'very frontier' — early and unproven

49:58 20VC: Why Google Will Win the AI Arms Race & OpenAI Will Not | NVIDIA vs AMD: Who Wins and Why | The Future of Inference vs Training | The Economics of Compute & Why To Win You Must Have Product, Data & Compute with Steeve Morin @ ZML

Jerry Murdock · Feb 28, 2026

Language is a subpar descriptor of reality, so autonomous agents need a new visual recognition system to get an objective view of the world

Language can only give a feeling about the world, not enough detail — which is why we have poets

37:31 20VC: Why Cursor is Dead | An AI Tsunami is Coming & You Need to Prepare | Systems of Record Become Valueless Databases with Agents | Is This The End of Tech Private Equity with Jerry Murdock, Co-Founder of Insight Partners

Jonathan Ross · Sep 29, 2025

AI arriving as language-based rather than esoteric intelligence is what made it universally usable, which is why AI came later than he expected but is growing far faster

Language makes interaction trivial so anyone can use it, unlike an Alpha Go-style esoteric intelligence

40:05 Language based interface drove universal adoption explaining late but fast growth

Mati Staniszewski · Sep 8, 2025 · speculative

As the different modalities combine, models will gain a much deeper understanding of everything around them

13:18 Multimodal combination deepens world understanding

Alexander Embiricos · Feb 21, 2026

Agents that work with your computer through code, not video and audio multimodal interfaces, are the way AI benefits reach people

Progress in multimodal models (video/audio) was slower than he expected, while code-based agents advanced

54:23 Code based computer use beats multimodal video and audio agents