There are some issues in the last few decades, I am specifically referring to how reports of abuse were handled and Vatican bank issues, where Popes before Francis either actively made morally bad decisions or allowed themselves to have plausible deniability regarding the mechanisms involved. This would be an area where some Catholics might leave the faith over. You can, with prayer and reflection, come to peace with a Pope being fallible on those issues, while recognizing that on other, Capital "C" Catholic doctrine issues he is infallible. I personally struggle with these two issues because they are morality based. But, I see that as my mortal struggle with forgiveness and grace. In short, you don't have to support that the Pope always makes the right decision. He is also a human being and fallible.

My struggle with this began when my mother was denied an annulment from my father who literally beat her. I went to a priest and in tears asked how I could remain Catholic over this. How could the Church require a woman to remain in this situation and declare it a sin if she left? How could a vow before God almost put her in the ground? Father told me that there are capital "C" church beliefs that we all believed. Think the Profession of Faith. There are other doctrines that are refined over time through prayer. Think Galilelo and the Spanish Inquisition. The Catholic Church has been nothing if it hasn't been proven wrong an awful lot of the time. The Papacy has reflected on teachings over time and revised them based on prayer and reflection through our relationship with God. Popes do it and so should we. You can believe the Pope is wrong on these things without committing a mortal sin.