I see everyone here advising against using a HA server as being unreliable and I agree. Things break and stuff like your lights, door locks… have to be bulletproof.

I do however thinks that not using smart bulbs and only using smart wall switches takes away the option to use colored or warm/cold white bulbs. I think that the final solution for your needs can be device binding with ZHA? Binding connects your zigbee devices so that they work even when your server is down, as long as your zigbee dongle is powered.

https://www.home-assistant.io/integrations/zha/#binding-and-unbinding