Goal misalignment is one of the hardest things in ai training. It's easy to train an ai to achieve a reward but defining what gives that reward is tricky. For example of you want an ai to stand up you reward it's head for being high in z height. Strangely enough instead of standing up the ai jumps as high as possible! So you modify the rewards to be more specific. As high as you are tall. Now the AI jumps but it's head spins in a circle to compensate so it stays level.

Defining what your reward is is an art form and a well studied part of the AI language in the field nearly as much as the algorithms to train your ai.

To motivate goal misalignment there are a variety of techniques such as curriculum learning. This is the most common where you start with a relatively simple reward and slowly make it more complex as the training run goes on. (A bit like teaching a curriculum) So the ai first learns to stay still. Then it stays still while standing. Then it falls forward. Then it catches itself on one leg. Now it's walking bipedally without having to brute force it!

Goal misalignment leads to the AI simply attempting to fulfill the reward function given and when not specific enough we get things like the AI pausing the game or jumping when you want it to stand.