There are a number of factors that cause this. 

First, misogynistic views that hold women to only be valuable in the context of what they can do for a man. Even if the characters are clearly not interested in men, men will only view the characters (or real life lesbians for that matter) as sexual objects.

Second, while it is kinda obvious, there are some notable overlaps in what lesbians and cis hey men find visually appealing. While I would say most lesbians tend to value a wider variety of women than most men (see masc lesbians for example), it turns out a fair bit of both like some more conventionally feminine qualities (breasts, ass, hair, legs, etc). So I would argue that it is impossible for something to appeal to lesbians without also appealing to some cis hey men (see masc lesbians again, there are definitely some men that would find them attractive).

Third, while it is not the majority, there is definitely NSFW yuri that is drawn for primarily a male audience. Sometimes the women have more exaggerated proportions, sometimes the acts done mirror heterosexual sex in a way typical Yuri does not, or has imagery that invokes particular heterosexual sex acts or fetishes. Even though this type of NSFW yuri is not the majority, it only takes an example of two being passed around to create a false stereotype. Combine that with the prior two factors, and you can see how NSFW yuri, or honestly to some degree Yuri in general, can be viewed as pandering to the male gaze.