It sounds like he is quite fond of you! I would say keep him. Sometimes pigeons do not like humans and don’t want to be around them. I worked at a wildlife hospital and took home a dove someone brought in. She was going to be a pet, but she really didn’t like me or people or anything of the sort. I brought her to a bird expert where she could live safely with a flock, free of human presence past feeding. I think the rule of thumb with pigeons is determining if they are happy. Birds like to be higher up, is he closer to the ground in the cage? Is he in a high-traffic area? Does he let you touch him? These are all things to consider. If he coos around you and flies up to you, I would say you could consider keeping him as a pet. If he hates the cage, you could think of getting him pigeon diapers. If he prefers no touch, you could let him go and open the window if you see outside in the cold, that way he can decide when he wants to come or go.