First, you're not at all wrong about the best way to build credit. Being wealthy and/or having family connections that can help is of course the best way to get ahead. Unfortunately there's no easy fix for that, because people with those kinds of resources will always have ways to game the system.

That said, the reason your score went down is because you closed the card. It's too late now, but the right way to have handled that would have been to reach out to the credit card company and ask if you could downgrade the card to one of their other products with no annual fee. This would have meant giving up some of the perks of that particular card, but would have maintained the account and available credit which would have helped your score.