Reward model hacking as a challenge for reward learning

📖 Description

This post discusses an issue that could lead to catastrophically misaligned AI even when we have access to a perfect reward signal and there are no misaligned inner optimizers. Instead, the misalignment comes from the fact that our reward signal is too expensive to use directly for RL training, so we train a reward model, which is incorrect on some off-distribution transitions. The agent might then exploit these off-distribution deficiencies, which I'll refer to as *reward model hacking*.

📊 Game Impacts

💭 Reactions

🔗 Sources

• [1] https://www.alignmentforum.org/posts/yyoKYmfFx7zpPyD99/reward-model-hacking-as-a-challenge-for-reward-learning