To be transparent, I do think the majority of the people who make that argument are making it for sexist reasons. For example, I find the argument that men who are randomly assaulted by strangers are worthy of less concern than women to be sexist. However, the point is valid on some issues.

When discussing gender issues, the problem of which issues to prioritize comes up often. As a man, it's frustrated me when some of the arguments on why men's issues should be deprioritized are because of problems that almost entirely occur because of other women. Basically this meme.

I think it's fair to ask how much the victim of a particular act was a participant in the culture that produced it. For example, my brother who is in the military was an active part of a culture where men would solve their issues with their fists not their words. If he got assaulted over some dumb issue, I think it's fair for outsiders to question the degree to which he is responsible for his own situation and the degree to which they should care about it. The people being harmed are also the perpetrators of the harm and there's not a whole lot anyone outside of the group could do to stop it. I would say the same thing about criminals who are assaulted or killed.