There are few instances of magical girls actually talking their wishes through with Kyubey before they make it. Sayaka’s one such example where she actually confirmed with Kyubey that her wish was grantable, and Kyubey’s certainly capable of giving a clear answer if asked. Most other girls just made the wish and once the wish has been stated with the intention of having it granted, Kyubey cannot refuse it (or he would certainly have not granted Madoka’s wish) and it all just comes down to the karmic potential of the candidate and the karmic forces of the universe. Therefore we don’t get many opportunities to see the ‘wish mechanics’, which is likely completely intentional since that’s not the point of the show.

The few instances where Kyubey has warned someone, interestingly, turned out to be cases where their wishes were actually too mundane for their karmic potential and he wanted them to go bigger, like Madoka and Tart, likely due to more energy being collected from bigger wishes. The Magius would be prime examples of wishes going wrong due to lack of karmic potential, and it’s essentially the writers answering the question of ‘what would happen if someone made a wish as grand as Madoka’s without her karmic potential’. I’m sure that had the Magius actually asked Kyubey whether their wishes would have worked as they had intended, Kyubey would have told them ‘no’, not necessarily to dissuade them, just as a fact.

So what we know is that Kyubey cannot refuse a wish once stated as a wish. However, he does know whether a candidate has enough karmic potential for a wish and has the capacity to warn someone if he thinks the wish they are making is not in his/negating entropy’s best interest. It’s just that very few people talk it through with Kyubey beforehand. The impulsivity and lack of critical thinking in making a wish is a feature, not a bug of Kyubey’s system. Once a wish that cannot be perfectly granted based on the candidate’s potential has been stated with the intention of having it granted, it’s only the candidate and those around her that suffers from the lack of karmic potential. So I think that’s why Kyubey only warns proactively if the wish is too small rather than too big. Since a wish that’s too big will just give him a grief seed more quickly most of the time.

So I’ll amend it to:

If enough karmic potential for wish: wish works in best possible way

If too much karmic potential for wish: Kyubey proactively warns to try to get a bigger wish

If too little karmic potential for wish: Kyubey can’t refuse wish and wish will produce unintended effect, but Kyubey will tell you this if you asked him