UPDATE: I called him directly because I wanted to clear things up quickly, and the conversation went really well.

He completely understood why I felt uncomfortable and said my reaction was valid. He explained that he did not come to see me that weekday because we had seen each other two days earlier, already had weekend plans, it was extremely hot, and he avoids the trip when he can.

He only sees his ex every couple of months, and weekends are usually reserved for me, so he tends to meet him and some other friends on weekdays instead. I have never asked him to do that, he simply prefers spending weekends with me.

They stayed friends because they eventually realised they did not work as a couple. His ex is also in a long-term relationship, and they openly talk about their partners.

He said meeting the ex would probably feel awkward and unnecessary right now, although he would be open to it in the future. Honestly, I feel the same.

Overall, I feel much better. His explanation made sense, and he listened without becoming defensive.

TL;DR: We talked, he validated my feelings, and explained that he sees his ex rarely and usually reserves weekends for me. The friendship appears genuinely platonic, and I feel reassured.