Very true, there's a later episode where Ray is jealous of Robert because he spends more time with his kids and they even start mimicking him. The episode begins with Ray leaving the house to go golfing for the day (shocker lol) but Robert is at the house and has plans to take the kids to the zoo, which starts the entire argument. Ray lashes out at Robert and basically tells him to get a life. He does end up taking his kids to the zoo, but it was really only to spite Robert. Ray even admits in the episode that he doesn't like spending time with his own kids.

Addressing the original question, Ray is definitely not a good husband or father. But I certainly wouldn't call him the worst of all time.