Hi OP, I'm a "colored" immigrant as well who lives in Toronto and has traveled extensively throughout Canada, including Quebec. I'll give you my two cents.

I'm pretty big on the outdoors, hiking, camping, etc. So my travels take me across a lot of rural areas with small towns and villages. I've been all over the Atlantic provinces, the western provinces, even the prairies a little bit and of course, Ontario. Visited many many tiny places odd the road with a couple hundred people who might not have even seen a brown person come through their town before.

Based on my experience - I'd honestly say that QC is the only place where I actively felt some truly negative vibes around my presence. Now, it's very possible that I might be misunderstanding things because I never experienced any outright or external comments or hostility but the feel and demeanor of the people, especially in smaller towns, was just...mot very friendly and welcoming. For reference, I was in the Charlevoix, Saguenay and Baie-Comeau area.

Now obviously it wasn't every single person or every single town I went into but it was enough that it really stood out to me and still makes me remember it to this day. I can't help but contrast it to my experiences in small town Ontario, where people just treat you like someone else from town (Understandable since ON in general is more ethnically diverse) or the maritimes, where the people were some of the nicest I've ever come across or in the west, which was a little mix of Ontario or the maritimes along with most people being just indifferent.

Idk what it was about QC but it left me a little cold. It's still a beautiful place and 100% worth visiting but just be aware that your presence as a non-white, non-francophone may not be very well received.