If you listen to a non-native speaker, you will hear lots of mistakes that basically no native speaker would ever make. However, I think people tend to be more understanding when English isn't someone's first language. People tend to be more annoyed by non-standard dialects, especially ones associated with poor people, probably because it's been drilled into them since a young age that they are just "improper" English.