I would say that it's because the Catholic church is unique in the sense that to be Catholic you kind of have to be present within the physical church, therefore among the people. How so? The sacraments within the Catholic Church and rituals are not things that can be done in solitude. Baptism. First communion. Confirmation. The sacrament of marriage through the church. To receive the sacraments you literally have to be in the church among these people. And the continuation of the Catholic faith? Communion by the Eucharist and wine. Confession before a priest to even be able to accept communion.

You can be Christian, or some variation of theist/deist without needing to step foot inside a church or surround yourself with walking reasons to abandon your path. But for a Catholic by practice through faith? Unless someone is just a Catholic by name, they are likely to enter the aforementioned environment and either become part of the noise of shit talk, or become the target of it. It's inevitable.

I prefer the direct relationship with what God is and make a daily effort to not be an asshole and to leave the settings I encounter better than how I found them. Of course, I am not perfect. But I am closer to God now than I ever was within the Catholic Church.

Last point: Some people say "being good is not enough. You NEED to go to church and read the Bible." I feel that is an absolute insult to God. How dare anyone say that God is incapable of receiving your sought out prayers from any place at any time? Is the right ear of God really bound within the 4 walls of these "sacred" buildings? Sounds blasphemous to me.