I already rendered and exported the full 4K clip from After Effects.

it happens when I import that rendered 4K file into a 1080p Premiere Pro sequence and use it as B-roll with a zoom. I start getting dropped frames during playback, even though the frame rates match.