A very crude deception eval is already passed

📖 Description

I was thinking about possible evals that would tell us when we're getting to models that are capable of deception. One not-very-good idea I had was just to measure zero-shot understanding of relevant deception scenarios in a language model. I don't think this tells us very much about whether the model is in question is actually trying to deceive us, but it's a tiny bit interesting. Anyway, it seems like large language models look like they can do decent enough deception + theory of mind in a story-like setting that this is mostly already passed.

📊 Game Impacts

💭 Reactions

🔗 Sources

• [1] https://www.alignmentforum.org/posts/22GrdspteQc8EonMn/a-very-crude-deception-eval-is-already-passed