How an alien theory of mind might be unlearnable

📖 Description

**EDIT**: *This is a post about an alien mind being unlearnable in practice. As a reminder, theory of mind is unlearnable in theory, as stated [here](https://arxiv.org/abs/1712.05812) - there is more information in "preferences + (ir)rationality" than there is in "behaviour", "policy", or even "[complete internal brain structure](https://www.lesswrong.com/posts/9rjW9rhyhJijHTM92/learning-human-preferences-black-box-white-box-and)". This information gap must be covered by assumptions (or "labelled data", in CS terms) of one form or another - assumptions that cannot be deduced from observation. It is unclear whether we need only a few trivial assumptions or a lot of detailed and subtle ones. Hence posts like this one, looking at the practicality angle.*

📊 Game Impacts

💭 Reactions

🔗 Sources

• [1] https://www.alignmentforum.org/posts/kMJxwCZ4mc9w4ezbs/how-an-alien-theory-of-mind-might-be-unlearnable