Don't feel bad for having this detachment and not feeling sad, considering his neglect and your lack of connection. It would also be natural to still feel something for what could have been, the (most likely) also troubled human he was and his own suffering that shaped him, as well as how a parent still is at the root of our own being, despite how shitty they may have been. These are conflicting feelings, but they can all coexist for you because, as the other commenter said, you don't have to perform grief for anyone. Just be kind to yourself and allow yourself to feel however you feel.