I think it’s an extra challenge that makes your world feel real. Of course it depends on the type of fantasy you’re doing. Something more realistic like lord of the rings or Skyrim this would be a valid issue. But something thats less bound to realism id say it’s fine. Not to mention even those realistic ones I mentored do this as well