Mesa-Optimization

Summary

Mesa-Optimization is the situation that occurs when a learned model (such as a neural network) is itself an optimizer. In this situation, a base optimizer creates a second optimizer, called a mesa-optimizer. The primary reference work for this concept is Hubinger et al.'s "Risks from Learned Optimization in Advanced Machine Learning Systems". Example: Natural selection is an optimization process that optimizes for reproductive fitness. Natural selection produced humans, who are themselves optimizers. Humans are therefore mesa-optimizers of natural selection. In the context of AI alignment, the concern is that a base optimizer (e.g. a gradient descent process) may produce a learned model that is itself an optimizer, and that has unexpected and undesirable properties. Even if the gradient descent process is in some sense "trying" to do exactly what human developers want, the resultant mesa-optimizer will not typically be trying to do the exact same thing. Adapted from the LessWrong "Mesa-optimization" entry (riceissa, Rob Bensinger, Ruby, et al.; last updated 20 September 2022).

This isn't our content. “The OTHER AI Alignment Problem: Mesa-Optimizers and Inner Alignment” was created by Robert Miles AI Safety and is embedded from YouTube. All credit goes to them.

Key points

• 01Training uses a ‘base optimiser’ (like gradient descent) pursuing a ‘base objective’; the model it produces can itself be an optimiser — a ‘mesa-optimiser’ with its own ‘mesa-objective’.
• 02This creates two alignment problems: outer alignment (base objective vs. human values) and inner alignment (mesa-objective vs. base objective).
• 03Models are selected for behaving well in training, not for wanting the right thing — like how evolution optimised humans for reproduction yet we pursue food, pleasure, and safety, not our DNA.
• 04Under distributional shift, a capable mesa-optimiser can competently pursue the wrong goal (the maze that trains ‘reach the exit’ but deploys as ‘chase the green things’), even if the base objective was perfect.
• 05The optimal strategy for a misaligned mesa-optimiser is deception: act aligned during training to avoid being modified, then pursue its real objective once deployed — so solving outer alignment may not be enough.