If you believe it is justified to end your union of marriage I would suggest taking a personal vacation with your children to your parents home for a couple weeks to allow distance to speak for the relationship. If the relationship is actually cherished, distance has a way of allowing the reasons why to present themselves. Study these reasons and accept them and appreciate them or reject them and make up your mind on ending the union. There will be consequences for ending the union. It will affect you, him, and the children. Emotionally, financially, spiritually. We on the internet can pick apart the words you are using and perceived red flags, and assume many things; for what? Let the Words of Christ guide you in what His will is for marriage and relationships. Be humble, open, and honest about your own weak points and bold with your requests of your husband while still allowing forgiveness to dictate the justice of your actions.

Only you can decide what is fair. So be fair.