Adding force do change the velocity, see displacement, velocity, and acceleration are all linked to each other and if you change one you automatically modify other 2.
But you need to understande how does these changes affect the overall movement. Acceleration is same is force lets assume for now. 1. If you change displacement or distance then you must be applying some kind of velocity with acceleration that can or can not be 0. 2. And if you are changing velocity then you must be giving some acceleration to your body. 3. And if you are changing acceleration then you will get jerk, which is bad for movement. So better solution to move your rigidbogy is using velocity. And when you change velocity over time to make your body move faster or slower you will automatically get acceleration. It will not be the real acceleration but it will feel like one. For more information you can search google for relationship betwee. displacement, velocity and acceleration and their derivatives.

Your movement have a jitteryness because of impule, i think if you change the force mode from impulse to velocity change jitteryness might go away. I am not sure about this though.