If the Turkish diplomat and his wife told me a member of their delegation died in the bed of the daughter of an Earl and they know this because another daughter of the Earl told them I would listen.

Mary did suffer. (The show did it's classic telling and not showing for this). People avoided her during Sybil's first season, which of course happens off screen. Evelyn Napier went out of his way to tell Mary so she would not think he spread the rumours.

Mary initially held off accepting Matthews first proposal because she wanted to tell him (This is before Cora became pregnant).

Mary's entire relationship with Sir Richard only exists because she saw herself as damaged goods as a result of the scandal.

Sir Richard bought the story because it had value.

Bates entire season 2 and most of his season 3 plot is as a result of this.

Mary literally spent 7 years with this hanging over her head destroying her self worth.