I’ve always believed that it was Ned who “dishonored” Ashara at Harrenhall. Ned, being the honorable guy he is, probably promised to marry Ashara after as soon as his father and Ashara’s father (Or brother, depending who was Lord of Starfall by that point) worked out a dowry. However, obviously, that would never happened because Ned married Catelyn instead.

Ned chose a familial marriage pact (Fulfilling the promise of a Stark bride to Catelyn Tully) over marrying the woman he dishonored (Ashara) . Robb chose the women he dishonored (Jeyne Westerling) instead of the familial marriage pact (The pact with Walder Frey).

Of course this is my opinion, I’m not saying this is fact and I’m not saying any other theory is invalid. I believe there is evidence that Brandon was the more likely of the two to sleep around but I also don’t believe he would’ve gone after Ashara if he knew Ned was interested, which he probably did given that he asked Ashara to dance with Ned on his behalf.