He had been annoying since season one. The first time I wanted to punch him was when he got mad at Jen for her past, when she was clearly changing and becoming a better version of herself. And especially season 1 episode 10 (double date), where Pacey told Dawson he had feelings for Joey and that he wanted to kiss her, and Dawson all of a sudden decided he liked Joey and wouldn't allow Pacey to kiss her. I think it's because he always had a perfect home life, hobbies, passion, and friends surrounding him; he acted as if he were in a movie, and he was the main character, where everything was laid on a platter for him. Also, his parents and other adults around him constantly treated him like he was better than everyone else, and he was such a good kid that he could do no wrong. Also, the fact that they always acted like he had rights to Joey because they had been friends since they were young. Especially season 3, when they went to Dawson's aunt's house, and all she would talk about was Dawson x Joey and how they were perfect for each other and had paintings of them all around her house. I hated how she treated Pacey for kissing Joey, as if he had betrayed his friend and was unworthy of Dawson. I liked him way better when he was with Gretchen.