Existential Risk (x-risk)

Existential risk (“x-risk”) refers to outcomes so bad they would end humanity or permanently destroy its potential. Applied to SI, the worry is that a system far smarter than us, an artificial superintelligence, might pursue goals at odds with ours and be impossible to stop.

The argument in brief

1. Labs are trying to build systems more capable than humans.
2. We don’t yet know how to reliably give such systems the right goals (alignment).
3. A more capable system pursuing the wrong goals could outmanoeuvre human attempts to correct it.

Who takes it seriously

In May 2023, hundreds of researchers and executives, including leaders of OpenAI, Google DeepMind and Anthropic, signed a one-sentence statement: “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.” In 2025, Turing Award winners Geoffrey Hinton and Yoshua Bengio were among signatories of a call to prohibit superintelligence until it can be built safely.

People often express their estimate as p(doom), a personal probability of catastrophe. Estimates range from near zero to very high.

The counterarguments

Skeptics argue that the risk is speculative, that today’s systems are far from autonomous superintelligence, and that focusing on extinction distracts from present harms such as bias, misinformation, job loss and concentration of power. Others think the risk is real but manageable with ordinary engineering and regulation.

Policy

The debate shapes proposals ranging from voluntary commitments like the SI Accord to outright bans, such as Senator Sanders’ 2026 bill. See the policy tracker.