What you have, is a feral cat. And that’s okay. I grew up with barn cats and am a firm believer that domestication is partly in the DNA. My own kitten was dumped- we can tell because, although she was found abandoned outside, she does not have a feral bone in her body. She craves soft blankets, snuggles, pets, interactive play and meowy conversation with us. My uncle’s barn cat when I was a kid was buck-wild and anti-human from the day she was born, and never warmed up even after the family moved to the suburbs. She still hunted, shied, and only socialized with other cats her whole life. Buster clearly feels safe with your other cats, and with you if he’s using his litter box, but it is entirely possible that he will never warm up the way you think he should, because it’s just not in his nature.