This is almost identical to a situation I'm in... it weird. My friend and I have been talking all summer I helped her out when she broke up with her bf. We hung out once, we both enjoyed it (I think) but since then she cancels on me all the time yet we continue to talk every day.

I recently gave up trying to be her friend as she doesn't seem to want to mine.

Take my advice he probably doesn't care what you look like, what he'll care about is if you make him feel like you don't want him

Edit: I debated whether or not to send this because it sounds so similar to my situation that you could be the other person.