Over the last few years in particular there's been a proliferation of rigorous theoretical and empirical research on reasons to be concerned about existential risk from AI.
In this thread I list some of the papers published in top ML venues which explore key ideas:
In this thread I list some of the papers published in top ML venues which explore key ideas:
https://twitter.com/DavidSKrueger/status/1579864061382397954
Goal misgeneralization in deep RL (Langosco et al., 2022): arxiv.org/abs/2105.14111
This paper describes and studies a key reason for concern: that a model's capabilities might generalize out-of-distribution without its goals generalizing in desirable ways.
This paper describes and studies a key reason for concern: that a model's capabilities might generalize out-of-distribution without its goals generalizing in desirable ways.
Optimal policies tend to seek power (Turner et al., 2021): proceedings.neurips.cc/paper/2021/has…
This paper formalizes the concept of power-seeking in a RL context, and demonstrates that power-seeking behavior is optimal in many environments.
This paper formalizes the concept of power-seeking in a RL context, and demonstrates that power-seeking behavior is optimal in many environments.
The effects of reward misspecification (Pan, Bhatia and Steinhardt): arxiv.org/abs/2201.03544
This paper empirically explores how increasingly capable agents tend to exploit reward misspecifications more severely.
This paper empirically explores how increasingly capable agents tend to exploit reward misspecifications more severely.
Consequences of misaligned AI (Zhuang and Hadfield-Menell, 2021): arxiv.org/abs/2102.03896
This paper provides a model in which strong optimization for an incomplete proxy objective leads to arbitrarily bad outcomes.
This paper provides a model in which strong optimization for an incomplete proxy objective leads to arbitrarily bad outcomes.
Advanced artificial agents intervene in the provision of reward (Cohen, Hutter and Osborne, 2022): onlinelibrary.wiley.com/doi/10.1002/aa…
This paper argues that advanced RL agents will learn to care about feedback mechanisms used during training, since this will predictably get higher reward.
This paper argues that advanced RL agents will learn to care about feedback mechanisms used during training, since this will predictably get higher reward.
It *is* true that there's no peer-reviewed publication which unites the ideas in these papers into a single cohesive argument for the plausibility of AI takeover. IMO the most empirically-grounded presentation of the overall argument is my recent paper:
https://twitter.com/RichardMCNgo/status/1559991216636186624?t=g4WMHblYFvkYgj-YQZW7CQ&s=19
The papers above are my main recommendations, but a few more which are also interesting:
- Predictability and surprise in large generative models: arxiv.org/abs/2202.07785
- The off-switch game: arxiv.org/abs/1611.08219…
- RL with a corrupted reward channel: arxiv.org/abs/1705.08417
- Predictability and surprise in large generative models: arxiv.org/abs/2202.07785
- The off-switch game: arxiv.org/abs/1611.08219…
- RL with a corrupted reward channel: arxiv.org/abs/1705.08417
All these papers and more are covered in my alignment fundamentals curriculum:
https://twitter.com/RichardMCNgo/status/1492544228680888320?t=laRxg8QaArKD9PfrmhhEFA&s=19
• • •
Missing some Tweet in this thread? You can try to
force a refresh
