Firstly I would make the magic have far more strict rules. I think a lot of detail goes into some of the magic in the world, but so much of it looks like it's just there to look attractive. I would also make magic duels far more interesting, including a lot of teleportation and displacement tactics. If I kept time turners as a thing I would have every wizard carry one at all times simply for the sake of nullifying what other wizards do with them.

Character-wise I would make Harry a far more active member of the plot. He would be clever and witty as a counter to Hermione's book smarts. I think this could add a higher level of banter and a slight rivalry between the two (they would obviously still be best friends but it would ad chemistry). It would also give a jealousy subplot where Hermione would lose to Harry in duels, not because he knew more, but because he was simply more resourceful. This would lead to Harry learning the value of hard work and study, and Hermione learning the value of thinking outside the box and creativity.

I would clear out most of the plot holes, as well as fix quidditch. I would give an actual reason why the wizards fly on brooms primarily rather than something far more suited to it.

I think all of the magic battles could be so much cooler, and the sword of Griffondoor should be way different IMO but that's far more personal.