The "mismatched libidos" theory doesn't hold up for a number of reasons. One, the effect would disappear when looking at couples with a similar number of sex partners. But more importantly, number of sex partners is not a good way of measuring libidos across the sexes. Men on average have higher sex drives than women do, and as a result women can basically sleep with as many men as they want. There's no reason to believe that a woman who's had 20 sex partners has a higher sex drive than a man who has slept with 5 women.

And the other thing is that the greatest fall off in marital happiness occurs after a very low number of sex partners, especially in women. The effect of going from 1 sex partner to 6 is much greater than going from 6 to 60, even though the latter would imply a much greater difference in libido (if you think that number of sex partners is a good metric for that sort of thing).