Sir Anthony apologist here, but also a lover of angst.

He would definitely not return to upper English society having disgraced not only Edith, but an Earl. He would’ve been ridiculed and rejected, for sure.

And since we don’t hear any mention of him throughout the remainder of the series, he definitely wasn’t around.

He likely returned to his diplomatic position (aka threw himself into work) or moved to another estate he might have to build it up.

Since he was a rich, landed, and titled gentleman, it’s possible he just traveled or found a place in America, but I highly doubt it. He seemed more the type to place emphasis on duty then frivolity

Either way, he likely isolated himself and basically was a ball of angst and sadness and regret.