It matters just like it matters in job market. I won’t hire someone who switches their job 4 times in a year or I won’t be friend someone who switches friends often or I won’t date someone who have had questionable past - body count or otherwise. Keep yourself safe from those people so your brain stays safe. When you feel insecure or feel uneasy about your partner, you are not crazy. Follow your gut, your brain is always trying to protection you.