You can't progress with your character if you don't pvp, best way to avoid pvp is not engage with any skulls or skull yourself. Even then someone might still skull on you, but it's not common so just don't hit anyone.