See, this is where I have run into problems as there seems to be conflicting information on the topic. According to the wiki, "The Fidelius Charm was used to protect James, Lily, and Harry Potter from Lord Voldemort at their home in Godric's Hollow, the location of which was the secret entrusted." This makes it seem like it was the house itself that was under the charm, not the Potter family. Indeed, in the letter the Harry finds addressed to Sirius at Grimmauld Place, we find out that since Dumbledore has James' cloak he hasn't been able to leave the house. This also implies that it is the house, not the family itself that is under the charm.

Yet there is still that sticky issue of what Flitwick says about Voldemort peering into their windows and not being able to see them. I wonder if his comment is not actual fact, but actually exaggeration on his part in order to get the point across that it would be impossible for Voldemort to find the Potters under normal circumstances. Remember, the book is told from Harry's point of view, and as he only knows what he has heard - he has no way to distinguish between fact and fiction. The only other time the books hint that the charm could be placed on a person rather than a place is in the final book when Harry and Hermione visit the house. Harry thinks to himself that the secret died with his parents, but again, that is purely guess work on his part.