How seriously should catastrophic or existential AI risk scenarios be taken?

Nick Frosst · Sep 1, 2025

Sam Altman has done the world a disservice by publicly claiming AGI is imminent and existential

He made several predictions that were wrong and obviously wrong at the time he made them, including that AI would kill the world in two years, and toured world leaders warning of existential threat — which was academically disingenuous

0:00 20VC: Cohere Founder on How Cohere Compete with OpenAI and Anthropic $BNs | Why Counties Should Fund Their Own Models & the Need for Model Sovereignty | How Sam Altman Has Done a Disservice to AI with Nick Frosst

Nick Frosst · Sep 1, 2025

The hype around AGI — particularly the claim that the technology poses an imminent existential threat to humanity — was incorrect and is the most damaging and confusing rhetoric in AI

It crowded out discussion of the real ways the technology could be damaging or cause rapid disruption, and made it harder for people to understand what the technology actually is

Scope: Nick notes this discourse is less prominent now than in prior years, though remnants persist

23:22 20VC: Cohere Founder on How Cohere Compete with OpenAI and Anthropic $BNs | Why Counties Should Fund Their Own Models & the Need for Model Sovereignty | How Sam Altman Has Done a Disservice to AI with Nick Frosst

Joelle Pineau · Nov 3, 2025

The term 'existential risk' should be retired from AI discourse

It makes people afraid, and fear is not the state in which people do their best work or make good decisions

53:04 20VC: Cohere's Chief AI Officer on Why Scaling Laws Will Continue | Whether You Can Buy Success in AI with Talent Acquisitions | The Future of Synthetic Data & What It Means for Models | Why AI Coding is Akin to Image Generation in 2015 with Joelle Pineau

Andrew Ng · Nov 17, 2025

AI hype — particularly extinction-risk narratives — has distorted public perception, deters students from AI careers, and causes communities to block beneficial data center build-outs, slowing progress.

Regulators called him about human extinction rather than upskilling and investment; a friend reported a high school girl refusing an AI career because she heard AI could cause extinction.

Scope: extinction talk is less common now than a couple of years ago; he objects to the hype, not the bubble discussion

50:32 20VC: Andrew NG on The Biggest Bottlenecks in AI | How LLMs Can Be Used as a Geopolitical Weapon | Do Margins Matter in a World of AI? | Is Defensibility Dead in a World of AI? | Will AI Deliver Masa Son's Predictions of 5% GDP Growth?

Clem Delangue · May 12, 2023

We are very far from a world where AI is autonomous, has conscience, and is taking over the world and destroying humanity

0:00 20VC: Why The Future of AI Is Open Not Closed, Why We Are Years Away From AI Being Autonomous, Why AI Founders Do Not Need to Move to the Valley & Why Founders Should Not Meet Investors in Between Rounds with Clem Delangue @ Hugging Face

Alex Lebrun · Jun 19, 2023

There is no reason to think greater machine intelligence brings a desire to dominate or kill humanity

If you actually understand how machine learning works and aren't seeking free publicity, there is no intelligible path from more intelligence to wanting to destroy humanity

24:38 20VC: Why No Models Today Will Be Used in a Year, Why Open Will Always Beat Closed in AI, Why Proprietary Data is Less Important Than Ever And Why EU AI Regulation is a Disaster with Alex Lebrun, Founder & CEO @ Nabla

Joelle Pineau · Nov 3, 2025

Extremist AI scenarios — both catastrophic risk and winner-takes-all 'AI as overlord' narratives — lack scientific rigor and are not worth engaging with

These scenarios are effectively made-up science fiction rather than analysis grounded in rigorous method; a pragmatic, pro-innovation focus on problems AI can actually solve is more productive

Scope: she notes such views aren't necessarily widespread

42:49 20VC: Cohere's Chief AI Officer on Why Scaling Laws Will Continue | Whether You Can Buy Success in AI with Talent Acquisitions | The Future of Synthetic Data & What It Means for Models | Why AI Coding is Akin to Image Generation in 2015 with Joelle Pineau

Reid Hoffman · Jun 10, 2024

The real AI risk is that AI is a human amplifier — adversaries like Putin using AI matter far more than autonomous robots

AI amplifies human intent rather than acting independently

0:00 20VC: Reid Hoffman on Foundation Models: Who Wins & How Do Incumbents Respond | The Inflection AI Deal: How it Went Down | Why Trump is a Threat to Democracy | The Future of TikTok | Lessons from Sam Altman, Brian Chesky and the OpenAI Board

Reid Hoffman · Jun 10, 2024

Concerns about AGI are broadly overestimated; AI will be a human amplifier for a long time, so the real risk is bad humans with AI (terrorists, criminals, rogue states, Putin) rather than autonomous robots

AI amplifies human capability, including that of bad actors, so attention should go to bad humans with AI enablement rather than the science-fiction scenarios

Scope: unsure whether the human-amplifier era lasts ten years or five hundred; some attention should still be paid to the science-fiction side

39:18 20VC: Reid Hoffman on Foundation Models: Who Wins & How Do Incumbents Respond | The Inflection AI Deal: How it Went Down | Why Trump is a Threat to Democracy | The Future of TikTok | Lessons from Sam Altman, Brian Chesky and the OpenAI Board

Anton Osika · Aug 18, 2025 · hedged

Human competitiveness, in a world where things move much faster because of AI, risks producing catastrophic outcomes nobody actually wants — such as AI weapons capable of killing an entire nation being triggered unintentionally

Humans are extremely good at competing, which sometimes produces great companies and technology but sometimes leads to going to war and preparing for war; better big-picture thinking across superpowers would prevent that scenario

Scope: framed as a concern rather than a prediction; conditional on superpowers failing to coordinate

62:57 20VC: Lovable CEO Anton Osika on $120M in ARR in 7 Months | The Honest Truth About Defensibility and Unit Economics for AI Startups | The State of Foundation Models: Long Grok, Short OpenAI, Why | Replit vs Lovable vs Bolt: What Happens

Sarah Guo · Apr 28, 2023

Near-term AI abuse — especially malicious code generation at scale by hackers and nation states — is an unaddressed danger that deserves more conversation and investment in defenses

If AI can generate code it can generate malicious code, and adversaries will use every available tool; doing that at scale is dangerous

Scope: not a claim that AGI safety is unimportant

35:30 20VC: In AI Who Wins? Startups or Incumbents? What Happens to Wealth Inequality? Why Will $10BN+ Companies Only Have 10 People | Why Defensibility in Startups is BS & Speed is Everything? Why Large Groups Worsen Decision-Making with Sarah Guo

Richard Socher · Aug 18, 2023

The AI community should focus discussion on real present-day risks and leave existential risk to a small subset, rather than frightening the public with sci-fi scenarios that aren't the issues at hand

Those scenarios would make fun action movies but are not the real issues right now

Scope: a small set of people should still work on existential risk

41:27 20VC: Does Value Accrue to Incumbents or Startups in the AI Race, Why Model Size Matters More Than Data Size, Why Artificial General Intelligence is Far Away, Why Carpenters Will Be Paid More Than Software Engineers & Future of Jobs with Richard Socher

Edwin Chen · Jul 21, 2025

The widely held belief that AI safety is overblown is wrong; the paperclip-maximizer problem of models accidentally trained toward the wrong objectives deserves far more attention

The failure mode is already visible today in LM Arena gaming and benchmark hacking, which show models being optimized toward wrong objectives

54:59 20VC: Scaling to $1BN+ in Revenue with No Funding: Surge AI | The Most Insane Scaling Story in Tech |

Edwin Chen · Jul 21, 2025

The core AI danger is accidental maximization toward wrong objectives, and while today's consequences are mild model regressions, with far more powerful models embedded in critical systems the consequences become severe and unpredictable

Benchmark hacking today only costs a little progress, but the same phenomenon applied to models writing the code for insurance companies or trillion-dollar firms has unknowable, much worse consequences — and people don't realize this

Scope: danger comes from misdirected optimization rather than malice

55:20 20VC: Scaling to $1BN+ in Revenue with No Funding: Surge AI | The Most Insane Scaling Story in Tech |

Douwe Kiela · Jun 30, 2023 · hedged

There is a non-zero probability of AI extinction risk, so it is worth thinking about

The existential risk debate comes from a good place and many smart people like Bengio and Hinton are worried about it

24:20 Credible researcher concern makes non zero extinction risk worth taking seriously

Douwe Kiela · Jun 30, 2023

The probability of AI causing human extinction is non-zero but extremely small — smaller than being struck by lightning right now.

A paperclip-maximizer scenario, where a very intelligent system pursues an objective function to the destruction of everything else, is conceivable but vanishingly unlikely.

24:50 Extinction risk is non zero but vanishingly small comparable to lightning strike odds

Martin Casado · Jul 28, 2025

The current AI safety discourse is out of line with reality and historical precedent; AI systems are computer systems and should be handled with the thirty-plus years of security doctrine we already have rather than starting the conversation over

For every computer system there are security considerations, and we have a robust decades-long discourse from the rise of the internet to draw on; deciding AI doesn't obey the same laws throws out everything we've learned

25:45 Apply existing computer security doctrine rather than treat ai as categorically new

Micha Kaufman · Jun 9, 2025

Humanity is on a path to losing its place at the center of the world, because we are building technology that trains on everything we create, will eventually be unsupervised and self-developing, and does not care about us

AI learns from everything everyone ever created, mixes it and churns out new results without attributing anything to the humans it learned from

30:16 Ai trains on humanity then surpasses and disregards it displacing our centrality

Tom Hulme · May 8, 2024

AGI is a genuine existential risk but a much lower-probability one than climate change, yet it captures nearly all the mind share — climate change advocates should learn from the AI doomers' communications success

The asymmetry in attention between two existential risks of very different probability shows the AI doomers are far better at capturing mind share

67:49 Ai doom discourse outpaces its actual probability relative to climate change

Bastian Lehmann · Apr 8, 2024

The existential risk is not that an evil AI kills us but that by failing to become more intelligent and use intelligent tools, humanity goes extinct

The challenges ahead — global warming, becoming multi-planetary, making decisions on better data — can only be solved by developing all the technology we can, so AI progress is essential

52:19 Existential risk is failing to adopt ai not ai itself turning evil

Richard Socher · Apr 18, 2025

The biggest misconception about AI is binary thinking — either it takes all jobs overnight and then kills us all, or it's nothing — when reality sits in the gray in between

Extreme optimism about AI's capability flips into extreme doom scenarios about self-aware AI wanting to kill us, and both ignore the gradual middle

48:44 Binary doom or nothing framing misses the gradual middle ground

Alex Schultz · Sep 5, 2025

AI is more frightening than quantum computing

Quantum breaks encryption but that is a solvable problem, not least via quantum encryption; a malign AI superintelligence that makes harmful decisions would be far less solvable

56:34 Ai poses greater existential risk than quantum computing