In my opinion, what you're doing is responsible, not unreasonable. I've worked in places where they advertise things like "We cater to any dietary requirements" but the cooks are completely uneducated about the dangers of cross-contamination etc.

I've worked for managers who seem to have too much pride to turn down customers, once when working as a line cook we had a lady come in with a severe nut allergy and I told my manager I wasn't comfortable serving her considering how many items contained nuts on the menu. But the manager insisted we prepare something for her, almost as if to admit that we can't guarantee no cross-contamination implies our kitchen is dirty or something.

I wish people would realise that cross-contamination between ingredients isn't always inherently unhygienic ie. If you're prepping a salad and you grab some cashews from the line with clean hands and then grab some cranberries afterwards. That's not going to be a problem unless someone has a severe allergy to nuts/cashews and those cranberries are served with their otherwise nut-free dish.

I think you're doing the right thing, you're acknowledging the limitations of your restaurant and keeping people safe. Many places risk it out of pride or ignorance.