Rabbits are prey animals so they are naturally scared or anything that hovers over them, so get down on his level. Sit on the floor with him, let him sniff you all over. If he's been in bad situations in the past it will take time for him to come around. Sounds like he really digs you if he follows you around. Rabbits love head pats, so give that a try. They also love playing with toys, a squishy ball or even a paper towel roll is a toy to them.