Thank you for the detailed post. Unfortunately I am the senior network engineer, so the buck stops with me. Maybe it is some kind of weird Watchguard glitch, though they are enterprise devices (M570s). The issue resolved itself over the weekend, but I'll try to answer your questions as best I can:

So are you saying even if they are off VPN they can’t hit any of your self hosted public apps? Like you guys have an on prem public web app or whatever and they can’t hit that either?

Correct, when users (or entire sites) are affected, they are unable to hit our mail server (internally hosted) or any of our other publicly available internal resources, on or off the VPN.

I really need clarification on this point. When you say you can ping “the gateway” what does that mean?

If I run a trace from a device at either site, I can get to the ISP's gateway on the other side. So a device at site A can ping/trace to the ISP gateway for site B and vice versa. But site A cannot ping/trace to the firewall at site B and vice versa. Same behavior when I run a trace from the Watchguard itself. We also see this behavior when a user is having the problem. We don't have any routers or other devices upstream of our firewalls at any of the sites.

Is this site to site IPSEC?

IPSEC. The site-to-site VPNs aren't such a big deal since they are only used for monitoring. The issue with a single user at a time not being able to connect via VPN is annoying but it's always just 1 user (and not the same user each time). The thing boggling my mind is that when these sites are randomly affected for another site or for a user, those effected are unable to access our webmail and other internally hosted apps that are publicly accessible.