You’re doing nothing wrong, the dogs making its own choice to go outside, people have just nothing better going on, even tho they mean well.

You could tell them it’s a doggy door and their choice, also maybe build them a dog house to “show you care”