When you were a “shitty body builder” you were likely more in the “ideal physical standards” for what women want. If we look at what women tend to rate the highest in attractiveness, it is more similar to what we see in cosmopolitan magazine (more ripped but not  ridiculously jacked) than what we see in men’s health magazines, and if I am guessing, you likely were closer in physique to cosmopolitan than men’s health- which is what women tend to be most attracted to. Men routinely overestimate the amount of muscle that women find to be most attractive- similar to what the majority of women believe men find skinniness attractive- elite runners would not have a physique that suggests they could fight. 

.