Why Agent Foundations? An Overly Abstract Explanation

📖 Description

Let's say you're relatively new to the field of AI alignment. You notice a certain cluster of people in the field who claim that no substantive progress is likely to be made on alignment without first solving various foundational questions of agency. These sound like a bunch of weird pseudophilosophical questions, like "[what does it mean for some chunk of the world to do optimization?](https://www.lesswrong.com/posts/znfkdCoHMANwqc2WE/the-ground-of-optimization-1)", or "[how does an agent model a world bigger than itself?](https://www.lesswrong.com/posts/efWfvrWLgJmbBAs3m/embedded-world-models)", or "[how do we 'point' at things?](https://www.lesswrong.com/tag/the-pointers-problem)", or in my case "[how does abstraction work?](https://www.lesswrong.com/posts/cy3BhHrGinZCp3LXE/testing-the-natural-abstraction-hypothesis-project-intro)". You feel confused about why otherwise-smart-seeming people expect these weird pseudophilosophical questions to be unavoidable for engineering aligned...

📊 Game Impacts

💭 Reactions

🔗 Sources

• [1] https://www.alignmentforum.org/posts/FWvzwCDRgcjb9sigb/why-agent-foundations-an-overly-abstract-explanation