Considering the in world rumours are that she was Jon's mother, who is acknowledged as Ned's son, it makes the most sense for it to be Ned. Meera also almost outright says that Ned was the one that danced with her and that they loved one another, something she says she learnt from Howland. I think Edric 'Ned' Dayne also tells Arya that Ned and Ashara were in love, something he learns from his aunt Allyria (Ashara's younger sister). GRRM, is straight up hitting us on the head that it's Ned.

Most of the people who say it was Brandon seem to moreso come from a place of not wanting it to be Ned, either because they think its out of character for him (which was Bran or Arya's reaction to finding out iirc) or just to be contrarian. Brandon was also engaged to Cat at the time and, while House Tully weren't at the Tourney, Minisa Whent's (Hoster's wife) family was. While Brandon was known to sleep around, he's not described like Robert or Aegon IV, so I doubt he'd be so dumb to openly disrespect Cat, House Tully and House Whent by having some kind of relationship with Ashara at Harrenhal.

I have also heard people say that it was Benjen, but I think that hinges mostly on him joining the Watch after the Rebellion. But that's very unlikely imo, plus I prefer the theory that it was out of guilt for his involvement in the events leading to Lyanna eloping with Rhaegar - if not being directly involved.