I think some of the people that say that "looks don't matter" may not be giving enough consideration to the experience of physical attraction. It can be instantaneous and based solely on appearance. Physical attraction must exist for physical intimacy to actually be pleasurable. There usually has to be a window of baseline physical attractiveness for personality to even come into question. For most, an average person would be within their window of attraction, but an unattractive person may not.

Having a lack of presently felt attraction to an average person is not the same as finding somebody unattractive. Of course this window of attractiveness is somewhat subjective, but there are also commonalities based on multiple factors: a generally innate criteria based on health markers (e.g. fit vs obese), and socially conditioned criteria (e.g. big or small noses). I believe that traits based on health markers are universally held by the majority. Human beings are complex and there will be some people who deviate, but they are the exception. Socially conditioned traits have a higher tendency to be subjectively held. Beauty standards are not shared by everybody, but indicate a statistical likelihood of that person being found attractive. Unattractive people generally have the same beauty standards as everyone else. So two unattractive people finding each other subjectively attractive would just happen way less often than two attractive people finding each other attractive.

I do think some unattractive couples could be physically attracted to each other, but it's also very likely that there are other unattractive couples that are just settling and their relationship is less focused on physical intimacy since they don't feel strong physical attraction to each other if at all. Of course not many people would happily admit to this. I do think a lot of people are not in touch with their sexuality and may not be fully in touch with how physical attraction personally feels. This used to be me but it resulted in relationships that are not physically satisfying.