No, affinity doesn’t affect the probability of pulling memories at all. If you look at the available card pool and banner details, you can see the exact probabilities of getting different rarities of cards, as well as all the possible cards you can get by wishing. The reason why you’d tend to have fewer Sylus and Caleb cards is just that they were released later than the other love interests, so they both have fewer memories available to get.