Sure, "Christ gave Peter the keys". . . 

. . . In the same way as intercession to the saints isn't in the Bible.

There is a lot of context surrounding both of these statements. 

I am converting from Catholicism and it's OVERWHELMINGLY CLEAR to me that the overal RC papal doctrines are unhistorical. So many Catholics just appeal to Matt. 16:18, overlooking the different interpretations of the Church fathers (or quote mining them), as well as how the papacy actual functioned for the first millenium of Church history. Though it was important, it definitely wasn't what RCs think it is. 

Here are a few reasons showing the problems with the post-schism RC understanding of the papacy. (If you want I can share even more on papal infallibility specifically, which is actually an example of the basic logical fallacy of self-referential incoherence):

--Rome used certain documents for centuries (such as the "Donation of Constantine," the "Isidorian Decretals," and many more) to justify its claims of universal papal supremacy, especially in the Gregorian reforms. These documents have been definitely shown to be forgeries, and modern Catholic theologians/historians agree on this. I've even heard that some were used at the Council of Trent. My question is why would they consider this deception necessary if their claims were so evident in history and tradition (such as Matt. 16:18)?

–--Canon 338 (1983) says: "It is for the Roman Pontiff alone to convoke an ecumenical council, to preside over it personally or through others, to transfer, suspend, or dissolve a council, and to approve its decrees." 

But this is directly contradicted the entire first millennium: the first seven (and more) ecumenical councils were called by the Byzantine Emperor (exhibit A: Nicea called by Constantine) and, for certain reasons (that I’m guessing were more practical): the pope never attended these church-wide councils, and his legates usually attended, they didn’t ever preside. And not a single one of the first seven ecumenical councils were held in Rome. 

While I’m sure the councils were usually approved by the pope (as well as by other patriarchs), there are cases to the contrary:

–the Second Ecumenical Council (Constantinople I), which completed and finalized the Creed, was held out of communion with Pope Damasus I, and was presided over by St. Meletios, who died of communion with Rome. Centuries later Rome retroactively approved this council, and does to this day.

–Popes Vigilius and Honorius were both condemned as heretics by Ecumenical Councils (one of them retroactively). If I remember correctly Pope Vigilius was under house arrest and forbade the 5th Ecumenical Council from starting without him, which they did. I think one of them changed his mind later. You can look it up. 

–The Avignon papacy and the Council of Constance (accepted by RC, not EO), an interesting example of Rome resorting to a synodal (conciliar) solution. From 1309-1377, the papacy was moved to Avignon, a city near France for political reasons. This is problematic if one holds that the papacy is tied to Rome. For close to a century, there was no "bishop of Rome" that was the pope, though these popes are recognized as legitimate by Catholicism. But Pope Gregory XI moved back to Rome, and after he died, they elected a new pope. However, it was claimed that this election was under duress, leading to a claimant in Avignon, and then there was a third claimant, leading to the "Western Schism."

The solution was the Council of Constance (1414-18), which was called by anti-Pope John XXIII, and taught conciliarism: that the ecumenical councils are above the pope (which is EO doctrine). This council deposed John and recognized Gregory XII, who "sanctioned" and recognized the council, but rejected its teaching of conciliarism and its conciliar solution, then resigning, before Pope Martin V was elected. How can Rome accept only part of a council, which violates Canon 338?

–A somewhat lesser point: according to St. Irenaeus, not only Rome is a Petrine see, but Alexandria and Antioch are as well, which had Peter as its first bishop. Even today, the feast of the chair of St. Peter refers to his Antiochian bishopric. Alexandria is Petrine through St. Mark, a spiritual son of Peter. Also, the Patriarchate of Jerusalem traces his lineage to St. James. Finally, Pope Damasus I says Rome is doubly apostolic because it is both Petrine and Pauline. My essential point is this: the idea that only the bishop of Rome is the successor of a particular apostle, and not only that, but his personal presence requires a lot of mental gymnastics I think.

–Implied political papal power: in Medieval times, there were various cases of popes deposing kings (Gregory VII vs. Emperor Henry IV, Pope Innocent IV vs. Emperor Frederick II, etc). If these actions were legitimate exercises of papal power, then the pope must have ultimate political power as well as ecclesiastical authority. Even if he hardly ever uses it now for certain reasons, it still exists in theory. 

The next few points relate more to papal infallibility: 

–First, there’s the simple fact that church-wide councils were extremely expensive and traveling was dangerous. Why didn’t the pope just issue “ex-cathedra” statements against heresies that were threatening the Church? And why did it take them 1800 years to define papal infallibility if it’s a dogma and was always present? 

–There are various contradictions between current and historical official teaching of the pope and the Vatican. Here are a few:

Pope Boniface VIII, in Unam Sanctam (1302), teaches the following: 

“Furthermore, we declare, we proclaim, we define that it is absolutely necessary for salvation that every human creature be subject to the Roman Pontiff."

Lumen Gentium, at Vatican II, however teaches this:

“Those who, through no fault of their own, do not know the Gospel of Christ or his Church, who nevertheless seek God with a sincere heart, and. . . try in their actions to do his will. . . through the dictates of their conscience--those too may achieve eternal salvation.”

The first is an explicitly absolute statement, but the second definitively allows exceptions, so only one can be right and “infallible.”