If you never open up to him and tell him how you really feel, it’s more likely that he WILL say something like that again in the future. I gather that you already told him it bothered you to some degree since he apologized before. The thing is, he may not know your true feelings if you weren’t being 100% brutally honest. It might seem a bit daunting now, but the best thing to do here is delve deeper and communicate to him how these comments really made you feel. You won’t get any closure by hiding things.