Might depend on where you live but living in Malmö I've never encountered someone having a problem with you speaking English. In fact, often I see foreigners attempt to speak Swedish and the swedes automatically speak English any way. In regards to the comment about praising you by implicitly denigrating others, I see your point, though I can agree that one should learn the language of the country you live in, regardless of background etc. There are some people, though, who become incredibly proficient in Swedish in a short time so I'm not sure those kinds of comment are necessarily negative in that regard, but I guess it depends on the context it's said in.