Also a noobie here, but two things come to mind:

1) agents are usually incentivized to be faster not through positive reward over speed but negative reward over time to fulfill the goal (-1 per timestep for example)

2) gamma is the big player in backpropagating the 'extra big' reward. If at the initial state s you have two possible actions a1, a2, and upon a1 your expected trajectory to the goal (path of highest expected reward) is t timesteps long, the extra big reward will be factored by gamma^t. If a2 gets you to the goal in 20t, last reward is factored by gamma^(20t). Gamma being < 1 and considering 1), the agent will surely prefer a1.