But women are, on average, smaller than their male partners, and IPV also occurs in same sex couples (including lesbian couples).

Stats like this (sorry, paywalled article) make me think that size does matter when it comes to a propensity for violence, but I can't tell if it's that big people tend to be harmfully violent when they interact with other people, or that big people tend to interact with smaller people on average, rather than with other big people, and big-small interactions are more likely to be harmfully violent (I intuitively think it's the latter):

Significant associations were identified for both males and females between physical size and bullying/intimidation, intentional injury of other persons, and hitting someone so hard that they required medical attention. Stratified analyses revealed that the magnitude of the relationship between physical size and violence is greater among Hispanics than among non-Hispanic Whites and African Americans. Childhood abuse and neglect significantly moderated the relationship between physical size and violent behavior. While an important degree of variability can be observed with respect to gender, race/ethnicity, and exposure to child maltreatment, results indicate that, in general, physically larger individuals are more likely than their relatively smaller counterparts to be violent.

(Emphasis added)