AI safety from @MIRIBerkeley: Categorizing Variants of Goodhart's Law
arxiv.org/abs/1803.04585
Essential for understanding a class of failures for machine learning systems
Goodhart's law: "when a measure becomes a target, it ceases to be a good measure."
arxiv.org/abs/1803.04585
Essential for understanding a class of failures for machine learning systems
Goodhart's law: "when a measure becomes a target, it ceases to be a good measure."
2/ Example: A lazy lecturer notices that students' attendance correlates with their exam performance. He decides to assign grades based on this proxy to save time on marking.
3/ Example cont: Students realise this and start attending classes dutifully to get easy marks. Attendance no longer correlates with exam performance. It is no longer a good proxy.
4/ Examples of Goodhart's curse in AI: Surprising Creativity of Digital Evolution
arxiv.org/abs/1803.03453
Multiple examples of evo algos "hacking" their simulators or finding ways of maximising proxies that are not useful for the designers.
arxiv.org/abs/1803.03453
Multiple examples of evo algos "hacking" their simulators or finding ways of maximising proxies that are not useful for the designers.
5/ Examples of Goodhart's curse in AI: Deep reinforcement learning from human preferences
arxiv.org/abs/1706.03741
page 9: "poor performance of offline reward predictor training"
When proxy (predicted reward) is not updated online, it fails to teach the desired behaviour.
arxiv.org/abs/1706.03741
page 9: "poor performance of offline reward predictor training"
When proxy (predicted reward) is not updated online, it fails to teach the desired behaviour.
6/ Examples of Goodhart's curse in AI: World Models
arxiv.org/abs/1803.10122 @hardmaru
Section 4.5: Cheating When training in imagination, agent steers towards a buggy region of simulation where it's easy to reap rewards. Proxy (imagined consequences) doesn't match reality.
arxiv.org/abs/1803.10122 @hardmaru
Section 4.5: Cheating When training in imagination, agent steers towards a buggy region of simulation where it's easy to reap rewards. Proxy (imagined consequences) doesn't match reality.
