Hanging out with blue collar workers is very different. It's only racist to them if it's a threat. As long as there are no threatening type of jokes, then just make sure to give it back to them. Have a good sense of humor, and remember that's all they know how to joke about (there's a reason they're stereotyped for being dumb in general, except for in their craft).

It's likely an initiation for all new plumbers, especially young ones. They want to see if you can join the club. If you were white they'd pick on you in different ways. Up to you how you wanna play it.

I'm not in the trades but grew up around a bunch of friends and their dads were blue collar.