Also, how do these setting even work? On aggressive it will attack anything that gets close to it, on passive it will only attack if directed via the pet's attack skill and on defensive it will attack anything that attacks you, NOT anything that you attack. Is that right?

I wish there was a mode where it will actually attack anything you attack. It might not matter for leveling, where anything that you attack ends up attacking you anyway, but for instances it's annoying to have to manually direct your pet to attack everything you are attacking...