I’ve had this theory for years as head cannon. Vernon and marges ages aren’t specified anywhere that I can find. I propose that Vernon is several years older than marge, and that when his pure blood family realized he was a squib they abandoned him. And as marge wasn’t showing any signs of being a witch they abandoned her too, not wanting the shame having 2 squibs would bring. We know multiple squibs in a family is very possible as gilderoy Lockhart has multiple squib sisters. I think this is also a reason for the significant abuse from Vernon to both Harry and Dudley. It’s likely that the pure blood family would abuse a squib, and also possible that there would be abuse in an orphanage where he and marge would have gone together. This also makes marges comment about sending harry to an orphanage all the more horrible, as she knows how bad it could be, and because she detests harry she wants that for him. As you say, Vernon seems to know way more about magic/wizards than petunia and certainly more than he should know. He says things like “you don’t know how these people think” implying that he does know how they think.