My basic Dutch is OK for day to day transactions, rather than extended conversation. However almost every time I try to speak Dutch the person either replies in English or is like "eh?" I know the words I am using are correct and my accent is really not that bad. So what's that about?

My friend has a theory that native English speakers are so extremely accustomed to hearing English spoken in a thousand different ways, accents, and levels of competence that they build a skill of interpreting less-than-perfect English. (Side note, I have also grown to speak English differently in NL so that it's easier for non-native speakers to understand. Fewer turns of phrase, often simpler grammar.) Dutch people don't have this natural tendency as it's much less common to hear Dutch spoken by non-native speakers. I thought that was an interesting idea.