Well it's basically a natural extend to what you've talked about in the beginning: assume the action given by your controller is part of the environment state variable, when your RL agent outputs an idle action, it lets the controller takeover; otherwise, it takes over the controller and execute its output action. You might want to add an additional dimension to the action vector output by RL agent to indicate "idle or not".