Empirical Observations of Objective Robustness Failures

📖 Description

[Inner alignment](https://arxiv.org/abs/1906.01820) and [objective](https://www.alignmentforum.org/posts/2mhFMgtAjFJesaSYR/2-d-robustness) [robustness](https://www.alignmentforum.org/posts/SzecSPYxqRa5GCaSF/clarifying-inner-alignment-terminology) have been frequently discussed in the alignment community since the publication of "[Risks from Learned Optimization](https://arxiv.org/abs/1906.01820)" (RFLO). These concepts identify a problem beyond [outer alignment](https://www.alignmentforum.org/tag/outer-alignment)/reward specification: even if the reward or objective function is perfectly specified, there is a risk of a model pursuing a different objective than the one it was trained on when deployed out-of-distribution (OOD). They also point to a different type of robustness problem than the kind usually discussed in the OOD robustness literature; typically, when a model is deployed OOD, it either performs well or simply fails to take useful actions (a *capability robustness* failur...

📊 Game Impacts

💭 Reactions

🔗 Sources

• [1] https://www.alignmentforum.org/posts/iJDmL7HJtN5CYKReM/empirical-observations-of-objective-robustness-failures