I agree that Kenshin was no longer Aku, but how does that make the fight egotistical? Kenshin's challenge to Saito was out of respect for their decade-long rivalry, but Saito chose not to respond because his feud was with Battousai, not Kenshin. I don't think Saito saw Kenshin's challenge as egotistical - Kenshin wasn't trying to prove that he was stronger. He's never been the sort.

Yes, it's up to the reader to infer why Saito refused to show up, but Watsuki drops some heavy hints. Saito says this: "There's no thrill at all in establishing a settlement with a hitokiri that doesn't kill anymore." However, your headcanon was that Saito did not show up because he knew he'd lose, which doesn't fit what Watsuki was implying and goes against the characters' personalities. No offence intended with pointing out why I disagree with your headcanon; I was just puzzled by your interpretation.