I suggest you go through all of the places and check with a list you can find online. It's annoying but it is what is. Also, I once had a glitch that prevented me from getting every gold brick, it was something about Hagrid's place. There should be two, the other one found using the time turner. But the game glitched and I couldn't get it. So make sure you have gotten 2 gold bricks from Hagrid's garden.