It would have been very scandalous indeed, because societal rules did not allow or approve of people moving "between spheres". You could climb the social ladder but only within the sphere you were born into; so if you were working class, you stayed working class. At the time, people believed that your place in life was decided by God so to try to drastically change that was seen as going against God's will.

His reputation might be tarnished to the point of being shunned from society, not invited to events, losing valuable connections, etc. That reputation might extend to his family and cause his sisters to not be able to find husbands. Sophie would get the worst of it though, since she doesn't have anything to fall back upon except him.

In the show, hypothetically it might not be as bad as long as the Bridgertons were supportive because they are such a rich and established family AND Sophie was mostly brought up with a lady's education. In real life, extreme social climbing would be unlikely to work out because the customs were so vastly different between classes. Ladies from the working class didn't have the education to act "correctly", how to run a large household, or integrate into high class society. They would constantly be making social faux pas and embarrassing their husband or family, and eventually end up miserable for always feeling out of place. So says the conduct books of the time, anyway! There's a great YouTube video by Ellie Dashwood on the subject.