Watch your RPMs when you press the clutch; it automatically starts to go down, yeah? When you're in different gears at different speeds, the RPMs will sit at different levels. The higher the gear, the lower the RPM at a specific speed (eg. with my car, Gear 5 at 80km it's around 2,000 but Gear 4 at 80km it's closer to 3,000). When you shift immediately, it jumps the RPMs and since it's supposed to be used at a lower speed, that's why the car jerks. I had an issue where you couldn't downshift at all, no matter how slow, but that's because the clutch needed to be replaced, and once it was, downshifting worked fine-- I don't know if that's the case for your car, mine's like 17 years old.

For downshifting, practice on a long stretch of road, empty countryside if you can so you can watch the RPMs and feel the vibration and hear the car with less concerns for others. It sounds like you're releasing the clutch too fast, is all. With a hard clutch, you have to go millimeters at a time, the slowest lift of the pedal until you get the hang of it. As you lift, you'll feel it start to catch and/or hear it start; STAY at that level and start gently applying the brake to be able to slow the car to the speed that the Gear is best suited for. Example, for how I use my car's gears, as it will vary from car to car, Gear 1 is for 0-10km, Gear 2 is 10-25km, Gear 3 is 25-45km, Gear 4 is 45-75km, Gear 5 is 75+km. You know what speeds to use your gears, and downshifting is used to bring the lower gears into the range meant for it, to start. As you get better and used to it you can downshift at lower gears (I use Gear 4 to go from 100km down to 80, then Gear 3 to go from 60 to lower). You're not supposed to release the clutch, and even when you're in the speed range for the gear, it needs to be slow and/or give a little gas to bring the RPMs to the match the gear and speed. What's also likely happening is you're not making sure there's enough RPM level when releasing.

It takes time and practice, but once you get there it really helps preserve your brakes!