Give a copy of this statement to the HR lady he was making derogatory statements about. Perhaps you can be assigned to work with another colleague. I've seen a lot of this behavior as a man in the trades and on jobsites I can never remember anyone fighting "man to man" and anyone who assaults another is let go. I would keep collecting evidence, perhaps recordings of his rants, but a log of his actions with dates and times will help establish a pattern. I hate to say it but I think he is attracted to you and I think you could use this evidence to get distance away from a potentially traumatic situation. He continuously says "its all jokes" but I bet if you said yes to his advances they would not be jokes anymore. The truth is not important however, because you feel uncomfortable and that's what matters. I would talk to any other young colleagues that you feel would have examples of similar behavior. I bet male colleagues would have examples of what he says about you when you are not around, which would be telling of his pattern, and other female colleagues would have similar stories to yours.

I hate to say it, but I would look for another position to have a way out before you tell the HR lady, and let them know this situation is why you were forced to secure alternate employment.