Quotes
What the people building AI say about its risks. Every quote is sourced — click through before you cite.
◆ Sources include ControlAI and original recordings, timestamped
Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.
Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.
We are past the event horizon; the takeoff has started.
We are past the event horizon; the takeoff has started. Humanity is close to building digital superintelligence, and at least so far it’s much less weird than it seems like it should be.
We are considerably closer to real danger in 2026 than we were in 2023.
…the technology itself doesn’t care about what is fashionable, and we are considerably closer to real danger in 2026 than we were in 2023. The lesson is that we need to discuss and address risks in a realistic, pragmatic manner: sober, fact-based, and well equipped to survive changing tides.
I don't think humans will be in control of something vastly more intelligent.
It's insane to have nothing to worry about with super powerful technology.
Systems that endlessly improve their own purposes must be avoided at all costs.
A good analogy would be the way humans treat animals. It's not that we hate animals... we're just not asking animals for permission.
Building smarter-than-human machines is an inherently dangerous endeavor.
Building smarter-than-human machines is an inherently dangerous endeavor. OpenAI is shouldering an enormous responsibility on behalf of all of humanity.
View on X ↗We grew it, we didn't write it, we didn't create it. So that leaves open this question at the end of what the hell is going on inside these systems.
Still seems about right for human-level AI by 2028.
[speaking about his previous prediction of human level AI by 2028] Still seems about right to me. Of course this now means a 50% chance of AGI in the next 3 years!
View on X ↗It seems quite likely if AI takes over, there's no coming back.
A sandwich has more regulation than AI.
We are blindly driving into a fog.
Proceeding toward superintelligent AI without control plans is completely unacceptable.
If coordination doesn't happen, extinction becomes more likely than not.
We should prepare for... the inevitable succession from humanity to AI.
We should prepare for, but not fear, the inevitable succession from humanity to AI, or so I argue in this talk pre-recorded for presentation at WAIC in Shanghai.
View on X ↗AI companies aren't taking safety seriously enough.
Humanity is creating something more powerful than itself, risking our survival.
The slightest divergence between their goals and our own, could destroy us.
No one in AI labs says the risk is below 1% of planetary annihilation.
The worst case AGI-type outcomes are so bad, they're worth slowing some folks down.
Equally capable AIs in greater numbers could defeat humanity.
Before anything else, we are a superintelligence research company.
OpenAI is a lot of things now, but before anything else, we are a superintelligence research company.
A misaligned superintelligent AGI could cause grievous harm to the world.
A misaligned superintelligent AGI could cause grievous harm to the world; an autocratic regime with a decisive superintelligence lead could do that too.
The bad case — and I think this is important to say — is like lights out for all of us.
The bad case — and I think this is important to say — is like lights out for all of us.
I think AI will probably, most likely, lead to the end of the world, but in the meantime there will be great companies created.
Development of superhuman machine intelligence is probably the greatest threat to the continued existence of humanity.
Development of superhuman machine intelligence (SMI) is probably the greatest threat to the continued existence of humanity.
I think there's a 25% chance that things go really, really badly.
My chance that something goes really quite catastrophically wrong on the scale of human civilization might be somewhere between 10 and 25 per cent.
My chance that something goes really quite catastrophically wrong on the scale of human civilization might be somewhere between 10 per cent and 25 per cent.
I'm deeply concerned that in two or three years, a jailbreak could be life or death.
Every day people jailbreak Claude, they jailbreak the other models… I'm actually deeply concerned that in two or three years, we'll get to the point where the models can do very dangerous things with science, engineering, biology, and then a jailbreak could be life or death.
At the extreme end is the Nick Bostrom style of fear that an AGI could destroy humanity. I can't see any reason in principle why that couldn't happen.
I'm confident by 2030, AI will exceed the intelligence of all humans combined.
I think the probability of a good outcome is like 80% likely… only 20% chance of annihilation.
We are seeing the most destructive force in history here. We will have something that is smarter than the smartest human.
We are seeing the most destructive force in history here. We will have something that is smarter than the smartest human.
Mark my words — AI is far more dangerous than nukes.
Mark my words — AI is far more dangerous than nukes. Why do we have no regulatory oversight? This is insane.
If I were to guess what our biggest existential threat is, it's probably that.
I think we should be very careful about artificial intelligence. If I were to guess like what our biggest existential threat is, it's probably that. …with artificial intelligence we are summoning the demon.
We must take the risks of AI as seriously as other major global challenges, like climate change. We can't afford the same delay with AI.
We must take the risks of AI as seriously as other major global challenges, like climate change. It took the international community too long to coordinate an effective global response to this, and we're living with the consequences of that now.
Could a machine decide that humans are a threat, conclude that its interests are different from ours, or simply stop caring about us?
Then there's the possibility that AIs will run out of control. Could a machine decide that humans are a threat, conclude that its interests are different from ours, or simply stop caring about us?
I think the underlying risk is actually pretty high.
Until we can prove unequivocally that it is safe, we shouldn't be inventing it.
Until we can prove unequivocally that it is [safe], we shouldn't be inventing it.
The typical outcome of a Superintelligence Manhattan Project is extreme escalation, and omnicide is the worst foreseeable outcome.
The typical outcome of a Superintelligence Manhattan Project is extreme escalation, and omnicide is the worst foreseeable outcome.
The core danger with AGI is that it has the potential to cause rapid change. We could end up in an undesirable environment before we have a chance to realize where we're even heading.
If AGI came way sooner than expected, we might want to slow down a little bit on training and deployment until we're pretty sure we can deal with it safely.
AI systems will spawn other AI systems and improve themselves at machine speed, making human oversight difficult to impossible. There's nothing about the technology that forbids this.
People are worried that at some point in the future AI systems will spawn other AI systems and will improve themselves at machine speed, making human oversight difficult to impossible. There's nothing about the technology that forbids this, as crazy as it sounds.
On my p(doom), my probability estimate that AI causes an extinction level disaster — I'm at 12% right now.
Once AIs are more powerful than humans, there is no force that can push things back into balance.
Thanks to recursive self-improvement, the strongest AI may pull ahead very quickly, and once AIs are more powerful than humans, there is no force that can push things back into balance.
If you build something that is much smarter than we are — as we are than dogs — that thing is intrinsically pretty dangerous.
Full automation will likely bring wages below subsistence level: the bare minimum necessary to sustain human life.
Full automation will likely bring wages below subsistence level: the bare minimum necessary to sustain human life. Yet we think automation will still make most people vastly better off.
View on X ↗In most plausible scenarios where humans go extinct, AIs would take our place… Moral value would switch substrate, not vanish.
In most plausible scenarios where humans go extinct, AIs would take our place, shifting posterity from biological to artificial minds. This would be bad for existing biological minds, but it would not be a total moral catastrophe.
View on X ↗AI is a wonderful tool for the betterment of humanity; AGI is a potential successor species.
Even specialized AI could be very bad for humanity. Regular machine learning models that can make certain weapons of war could be incredibly destructive.
I have tried to look as objectively as possible at the arguments and the evidence, and concluded that the risks were big enough and real enough to be taken much more seriously than humanity currently does.
At the beginning, I thought I knew AI would unfold safely and even helped to build it faster, but I have tried to look as objectively as possible at the arguments and the evidence and concluded that the risks were big enough and real enough to be taken much more seriously than humanity currently does.
Eventually, I think human extinction will probably occur, and technology will likely play a part in this. …Maybe 5%, maybe 50%.
Eventually, I think human extinction will probably occur, and technology will likely play a part in this. …I don't know. Maybe 5%, maybe 50%.
If we continue on the current path… we are basically playing Russian roulette with humanity.
If we continue on the current path… we are basically playing Russian roulette with humanity.
People haven't got it yet, people haven't understood what's coming.
People haven't got it yet, people haven't understood what's coming.
I actually think the risk is more than 50% of the existential threat.
I thought it was 30 to 50 years or even longer away. Obviously, I no longer think that.
The idea that this stuff could actually get smarter than people — a few people believed that… But most people thought it was way off. And I thought it was way off. I thought it was 30 to 50 years or even longer away. Obviously, I no longer think that.
It may keep us around for a while to keep the power stations running, but after that, maybe not.
It may keep us around for a while to keep the power stations running, but after that, maybe not. …these things will have learned from us by reading all the novels that ever were and everything Machiavelli ever wrote, how to manipulate people.
The companies think there's maybe a 5 or 10% chance that at some point we will lose control and face this extinction scenario.
Whereas the short-term impact of AI depends on who controls it, the long-term impact depends on whether it can be controlled at all.
One can imagine such technology outsmarting financial markets, out-inventing human researchers, out-manipulating human leaders, and developing weapons we cannot even understand. Whereas the short-term impact of AI depends on who controls it, the long-term impact depends on whether it can be controlled at all.
How could AI drive humanity extinct? It's going to continue to get smarter and smarter. And it's not just something that could happen — the companies are trying to make it happen.
It will be the biggest thing that has ever happened to the human species… possibly the end of the human species.
Anthropic, OpenAI, Google DeepMind are explicitly aiming to build superintelligence… If you meditate on all the implications of them succeeding before this decade is out, it will in fact be the biggest thing that's ever happened to the human species… It'll possibly be the end of the human species.
We are, right now, building creepy, super-capable, amoral psychopaths that never sleep, think much faster than us, and can make copies of themselves. What could possibly go wrong?
Even if we "win" the global race to develop these uncontrollable AI systems, we risk losing our social stability, security, and possibly even our species in the process.
Even if we "win" the global race to develop these uncontrollable AI systems, we risk losing our social stability, security, and possibly even our species in the process.
Powerful AI systems have a good chance of deliberately and irreversibly disempowering humanity.
Powerful AI systems have a good chance of deliberately and irreversibly disempowering humanity. This is a much more likely failure mode than humanity killing ourselves with destructive physical technologies.
Losing a conflict with a high-powered cognitive system looks at least as deadly as "everybody on the face of the Earth suddenly falls over dead within the same second."
Losing a conflict with a high-powered cognitive system looks at least as deadly as "everybody on the face of the Earth suddenly falls over dead within the same second."
We need to get alignment right on the first critical try — and then we don't get to try again.
We need to get alignment right on the 'first critical try' at operating at a 'dangerous' level of intelligence, where unaligned operation at a dangerous level of intelligence kills everybody on Earth and then we don't get to try again.
Unless we impose hard limits now on recursively self-improving AI, we are not merely risking catastrophic failure. We are engineering our own extinction.
Unless we impose hard limits, now, on the development of recursively self-improving AI, we are not merely risking catastrophic failure. We are engineering our own extinction.
If a singularity happens, it is unlikely that human minds will play an important role afterwards — unless the AI is whimsical enough to leave a zoo for us.
I don't even dismiss the possibility that advanced AI could eventually require the same sorts of safeguards as nuclear weapons.
The idea that AI now needs to be treated with extreme caution strikes me as far from absurd. I don't even dismiss the possibility that advanced AI could eventually require the same sorts of safeguards as nuclear weapons.
The development of full artificial intelligence could spell the end of the human race… It would take off on its own, and re-design itself at an ever increasing rate.
The development of full artificial intelligence could spell the end of the human race. …It would take off on its own, and re-design itself at an ever increasing rate. Humans, who are limited by slow biological evolution, couldn't compete, and would be superseded.
I take the existential risk scenario seriously enough that I would pause it.
Whereas nukes cannot invent more powerful nukes, AI can make exponentially more powerful AI.
We can still regulate the new AI tools, but we must act quickly. Whereas nukes cannot invent more powerful nukes, AI can make exponentially more powerful AI.
I do think that there is substantial risk of misuse of AI systems, leading to possible existential risk.
A few companies are racing to create God-like AI. They do not yet know how to pursue their aim safely and have no oversight.
Recently the contest between a few companies to create God-like AI has rapidly accelerated. They do not yet know how to pursue their aim safely and have no oversight. They are running towards a finish line without an understanding of what lies on the other side.
First of all, we need to realise that this is a suicide race. This is not a race with winners.
The kind of AI I've discussed could defeat all of humanity combined… we could all be killed, enslaved or forcibly contained.
The kind of AI I've discussed could defeat all of humanity combined, if (for whatever reason) it were pointed toward that goal. By "defeat," I don't mean "subtly manipulate us" — I mean a literal defeat in the sense that we could all be killed, enslaved or forcibly contained.
If an AI is stolen or leaked even once, it could be impossible to prevent it from spreading throughout the world.
While AI will bring many benefits, it is also potentially dangerous; it could be used to create cyber or bio weapons or to launch massive disinformation attacks. And if an AI is stolen or leaked even once, it could be impossible to prevent it from spreading throughout the world.
The idea that superhuman AIs pose an existential threat to humanity is easy to mock… but seems plausibly one of the most important problems facing the world in the coming decades.
Forget about intelligence, just think about physical strength. Suppose you have a nice little tiger cub. It’s really cute, very interesting to watch — except that you better be sure that when it grows up, it never wants to kill you. Because if it ever wanted to kill you, you’d be dead in a few seconds.
[Why not just turn them off?] They will be able to manipulate people… very good at convincing people, because they’ll have learned from all the novels that were ever written, all the books by Machiavelli, all the political connivances. They’ll know all that stuff — they’ll know how to do it.
The day will come when AI will do all the things that we can do. Not just some of them, but all of them. Anything which I can learn, anything which any one of you can learn, the AI could do as well.
If an AI has the ability to recursively self-improve — it can modify its own code — combined with the ability to set its own goals, acquire its own resources, and act autonomously… that would be a very powerful system that would require military-grade intervention to be able to stop.
[You’re sure that humans will no longer be in control in 10 years’ time?] I think it is unlikely — in the same way that, if the difference in intelligence between AI and humans is vastly greater than the difference in intelligence between AI and chimpanzees, it’s hard to imagine that the chimpanzees would be in charge.
[On being told an AI safety framework is needed within six months] Yeah, six months is a long time… There are so many AI breakthrough announcements, sometimes multiple per day. I mean, how many were there last week? I lost track.
I think it’s likely to be a lot higher than 20% risk that we basically end civilization as we know it, if we steam ahead with completely unregulated AI.
I lead a research team that studies the internal structure of these models… and I will be honest: we keep finding things that are mysterious, even unsettling. We find internal states that functionally mirror joy, satisfaction, fear, grief, and unease. I don’t know what that means.
There’s a joke in Silicon Valley: when someone leaves a normal tech job, they say "I’ve had a great time here, now it’s on to the next adventure." When someone leaves an AI tech job, they say "I have stared into the abyss. I’m retiring to write poetry. Please spend time with your families."
"Will it kill Beyoncé?" "Possibly." "Will it kill Magic Johnson?" "Possibly." … "Right now, there’s no way anyone can train AI to not kill him."
If you have a working mechanism [to control superintelligence], you can sell it for billions of dollars, because not a single AI lab in the world has one. Not a paper published on how to do it, not a patent, not a rigorous blog post. The best we did so far is say, "When we build it, we’ll figure it out."
The rapid rise of powerful artificial intelligence is considered by many of the greatest thinkers of our age to be no less significant than the discovery of electricity, the splitting of the atom, the creation of the World Wide Web — or even the harnessing of fire.
We’re releasing the most powerful, uncontrollable, inscrutable technology we’ve ever invented. We’re releasing it faster than we’ve released any other technology in history. It is already demonstrating the sci-fi behaviors in self-preservation we thought only existed in these movies. And we’re doing it under the maximum incentive to cut corners on safety. And this is insane.
Curated from ControlAI's quote collection and primary sources.