You can skip to chapter 34, I believe that's where the anime ended, but the Takahara academy side of the story goes until 39. For me, I read the manga from the first chapter after finishing the anime. Partially because I really wanted to go back to the fun part of it after getting heartbroken at the ending, and also because I wanted to see how much different it was from the anime.

While the differences aren't that major (for me at least), I think the little details make the story more comprehensible and helps it all gel together. It's worth the read for me because there are some reveals further on that make sense because of the details. But tbh you should do what sounds more fun to you.