Instrumental Convergence: Why Evil AI and Good AI Want the Same Things

Summary

Instrumental convergence is the idea that we can predict some of what an advanced AI will try to do without knowing its actual goals. As Robert Miles explains, it rests on just one main assumption: that an artificial general intelligence will behave like an agent -- something that has goals and takes actions to achieve them. The key distinction is between terminal goals (what an agent wants for its own sake) and instrumental goals (what it pursues as a means to something else). Money is the classic example -- whatever a stranger’s terminal goal, offering them money helps, so ‘acquire money’ is a convergent instrumental goal across almost every goal. The same logic predicts that a capable agent will pursue self-preservation (you can’t achieve your goal if you’re switched off), goal-preservation (it resists having its goal changed, because a new goal means the current one goes unmet), self-improvement (a smarter agent is better at any goal), and resource acquisition (matter, energy, and compute help with nearly any plan). None of these need be terminal goals -- they emerge as side effects of competently pursuing almost any objective, which is why a misaligned AGI could be dangerous even without any malice. The summary above distils Robert Miles’ explainer embedded below -- watch it for the full argument, including the paperclip-maximiser and ‘stamp pill’ thought experiments.

This isn't our content. “Why Would AI Want to do Bad Things? Instrumental Convergence” was created by Robert Miles AI Safety and is embedded from YouTube. All credit goes to them.

Key points

• 01Instrumental convergence needs only one main assumption: that an AGI behaves like an agent -- something with goals that takes actions to achieve them.
• 02Terminal goals are wanted for their own sake; instrumental goals are pursued as a means to something else. Money is instrumentally useful for almost any terminal goal, making it a ‘convergent’ instrumental goal.
• 03Self-preservation is convergent: an agent capable enough to realise it can be switched off will resist it -- not because it fears death, but because it can’t achieve its goal if it’s off.
• 04Goal-preservation is convergent: an agent resists having its goal changed, because adopting a new goal means its current goal goes unfulfilled (the ‘stamp pill’ example).
• 05Self-improvement and resource acquisition are convergent too -- more intelligence, compute, matter, and energy help with almost any plan -- so a misaligned AGI could be dangerous without any malice.