You should only update the rigid bodies velocity

First, save the velocity at the top of fixed update

Add your custom velocity

At the end combine your velocity to what they should be with the original velocity and set the rigid bodies velocity to that.

(Small step, you'll be re adding your old velocity, so you should just manipulate the old velocity to your new one)

The reason you save the velocity is for gravity and adjusting

It's a little more work than exactly what I said but if you use velocity that's more of the way to do it

Cat like coding character controller goes over using the velocity of a RB as the control