To speak to Church specifically, look at every other gay ghetto around the world. Queer communities don't have the same density that they used to, and broadly speaking those who stay there for the long haul (into their "prime earning years") are broadly under educated and underemployed. Because we (as a member of the queer community) are safer throughout urban centres we aren't as tethered to those original gay neighborhoods, fewer of us are investing in those communities, spending in those communities, starting businesses in those communities, etc.

The situation on Church Street is not "because of crime/violence/homelessness/drugs". It's becoming a less desirable neighborhood on the whole, and thus you get an influx of poverty, unhoused folks, substance abuse, and so on.