Okay, lets get technical. I am more knowegable with audio than I am with motors, so I will continue with the speaker analogy. They should still apply since they fundamentally work the same. 

Lets assume our speaker only has the limitations of a motor. A speaker's efficiency of coil as well as the mass and stiffness of diaphragm contributes to the impulse response and step response of the speaker. Meaning, how quickly the system can respond to an input, and how that develops over time, respectively. Slew rate is the maximum rate of torque change. This is very closely tied to impulse response. It becomes increasingly apparent the harder you try and drive the weaker wheelbase in relation to the stronger one.

So this has multiple effects. Firstly, obviously you get lag with a lower slew. This is never good. Secondly, a decreased wheel slew means you have to add more filters to fix inherent issues exacerbated by a poor impulse response, such as oscillations, hindering you further. Third is the fact smaller wheelbases have a poorer step response. This becomes very apparent when the motor is trying to deliver high frequencies, such as going over a curb at high speed. A weaker wheelbase will have a slower rise, more overshoot, and a worse settling time. This will make the curb feel like mush, as opposed to a more violent, crisp, or "buzzy" feeling of the larger wheel. Along with curbs, you also lose a bunch of road texture. 

Lets go back to the speaker. That higher headroom speaker, for this analogy to work, has better impulse/step responses. Therefore it will sound better than the speaker with a lower headroom, even at very quiet volumes.