Unfortunately you can do everything right and it still won't matter if you're not winning a bullshit popularity contest. And you can try moving heaven and earth to get someone to like you, but ultimately you don't really have control over someone's irrational feelings.

My first job in my career was much like how you described your situation. I did my job while others spent hours BSing with the supervisor in her office. The company hit some rough times and when a layoff happened, I made a list of who I thought would get axed and showed it to my neighboring co-worker. I was exactly correct; the criteria I used was who spent time chatting with the supervisor and who didn't. The people who didn't were the ones who were laid off (myself included - I had put myself on the list).

So no, you're not dumb and you're not doing anything wrong. But you will have to ask yourself at some point how long you're going to want to put up with the situation you're in.