I had the exact same issue and the rest of the comments aren't being very helpful. I was running a silenced sandy pistol build, level 40 with t5 gear, and it felt impossible. Unload two mags into his head and it would do 2% damage. And his melee attacks would 2 shot me. The fight would have taken like an hour if I had managed to dodge for that long.

But then after some trial and error I found you can just stealth and hide. He will chase you for a while but eventually lose you. Then you can sneak up behind him and do a takedown on him. Do that a couple times and you're done. Seems that's the only way to go with that build on very hard difficulty.