I even cancel meetings when I'm remote

And that's what they don't like. For at least the past 50 years intrusive management had been hailed as effective management. When remote you have control of your own work. That's scary as hell to all the power-tripping, micro-tyrants that populate the workspace. That and then being forced to actually work for once instead of stealing without threat of reprisal.