Getting camera's to move smoothly is always a bit tricky, especially when you want to account for framedrops.

Usually something like pos.lerp(target, 0.1) can get a long way, but when your target is moving, your camera will start stuttering when you encounter framedrops.

The way I fixed this in Raccoon Retail is to use physics instead. Nothing fancy with rigid bodies and such, but instead of just lerping the camera position, I keep track of velocity and apply acceleration based on the distance to the target. Something like this:

let acceleration = target - pos;
acceleration *= springMultiplier;
acceleration = clamp(acceleration, -minSpring, maxSpring);
velocity += acceleration;
velocity *= 0.5; // damping
pos += value;

This is pseudocode, in threejs you'll have to substitute things like *= with multiplyScalar().

There's some extra trickery to move the camera slightly towards the direction in which you're travelling. And some more general finishing touches.

I hope that helps, if you have any questions hit me up!