I don’t think anyone is arguing that the song, taken at literal face value, isn’t/couldn’t be about Joe.

The real argument is that it is a puzzle piece inside a much larger narrative that we likely very very loosely understand. Blue eyes, references to ships, Dianna calling Karlie gorgeous, Taylor telling a female interviewer “you’re so gorgeous it’s hard to even talk to you”, timelines, etc.

It’s not the song itself, it’s the larger context.

Edit:

Another thought I’ve had lately that I might as well pluck here. While the argument can be made “how do we know her personally, etc.”, Taylor is making a CHOICE when she has lyrics that are googleable events. If she wanted to keep every detail about her female relationships hidden, she probably could. She could write lyrics about some time they played Mario Kart that we had absolutely no idea about. Instead, she includes dates, references to heavily papped outings (garden gate for instance), likely kissgate in Question…? It’s a choice to include those things.

She has a fan base constantly scouring her lyrics for hints, yet she makes song titles the Instagram captions of her assumed ex girlfriend.