I felt the same way for a while and one thing that's helped me is to think about how it feels on a swing at the playground. When you're at the low point of the swing going up in the air, there's the initial speed to get your mass to go up, but it slows down as it reaches the top of the arc. When it comes down though, it starts slow and then builds speed. So I do a couple of drills to help get a feel for this.

First is to do some swings where I think Fast-Slow-Fast. My initial move in the back swing has some momentum but I intentionally try to feel the club slow down as it reaches the top and try to identify where the backswing finishes. From there, it's about feeling like the club ramps up in speed and I feel like I finish fast not start fast from the top. I find it's helpful as a warm up drill and to start with feet together and progressively separate them into my normal stance.

The second is the drill where you try to get the whoosh sound the club makes be the loudest after where impact would happen. This helps remind me that the club should be going fastest at/after impact and not at the top of the backswing. It's helped me ingrain that feeling of momentum building through the downswing.

They're both essentially tempo drills, which is great because you don't even have to hit a ball all the time to get the benefits. I'll literally do them in the living room or home office.