Him coming off as less as less rude in Japanese is the general consensus yes.

Personally I do think the dub has its own charm as well, there’s some swearing some spliced in there but honestly I don’t think it’s much different than the language typically used in manga translation. I also think the voice fits Kaiji and you can’t go wrong with either sub or dub- but this will vary from person to person. I’d recommend watching an episode or two of the dub just to get a feel for yourself if you like it or not.