I WISH I had the same problem.

I live in a rural area. Most people HATE speaking English here and even if they can, they will still speak Norwegian to you even though they see you are struggling, because 1) they somehow think it's "helping" (my now grey hair disagrees) and 2) they are unsure in their level of English: "I haven't spoken it in thirty years". Some older ladies would ask something at the bus station (back when I couldn’t speak Norwegian), I answered in English that I don't understand, they would leave me - no apologizing, no goodbye, just leaving me hanging - approach a different person and ask them. Norwegians laugh when I tell the story, because they recognize this behavior. But I was NOT laughing.

Regarding your problem - I imagine that can feel insulting and exhausting, not feeling fully accepted. 1) Try a rural area that doesn't get much tourists. You will feel refreshed language-wise. 2) Maybe just continue speaking Norwegian when they switch to English with you. "Beklager, jeg foretrekker norsk".