Behavior Cloning is Miscalibrated

📖 Description

Behavior cloning (BC) is, put simply, when you have a bunch of human expert demonstrations and you train your policy to maximize likelihood over the human expert demonstrations. It's the simplest possible approach under the broader umbrella of Imitation Learning, which also includes more complicated things like Inverse Reinforcement Learning or Generative Adversarial Imitation Learning. Despite its simplicity, it's a fairly strong baseline. In fact, prompting GPT-3 to act agent-y is essentially also BC, just rather than cloning on a specific task, you're cloning against all of the task demonstration-like data in the training set--but fundamentally, it's a scaled up version of the exact same thing. The problem with BC that leads to miscalibration is that the human demonstrator may know more or less than the model, which would result in the model systematically being over/underconfident for its own knowledge and abilities.

📊 Game Impacts

💭 Reactions

🔗 Sources

• [1] https://www.alignmentforum.org/posts/BgoKdAzogxmgkuuAt/behavior-cloning-is-miscalibrated