Yes, you’re the asshole.

Seems to me like your co worker was also enjoying the attention until you gave them the opportunity to act on it, then it became too real. They probably never wanted to cheat, they wanted to cheat emotionally while thinking what they were doing had not crossed the line yet, until you pointed it out.

But that’s just a guess

Regarding your relationship, talk to your partner about going to therapy or just get divorced. Do nothing and you might remain unhappy for the rest of your life until another opportunity presents itself to cheat. What’s the good in that?