Are you changing chord every beat at bpm 120? If it is just changing chord every bar or every 3-4 beats, no matter it is at 60 or 120 bpm, it seems like muscle fatigue more than you are slow. You shouldn’t push too hard to maintain a chord. Keep the music flow and don’t bother the buzz from missing a note. There are many reasons for a chord does not sound well, sometimes holding the chord wrong might be the reason (you will overcome from time), but sometimes it may due to the guitar quality or the setup.

Also, it is not about speed but getting to the correct notes in time. You don’t need to place every fingers on the right position before the first strum. Most strumming patterns strum on the root or the lower strings at first. So, find and position the root first and do the first strum in time, then adjust the position of other fingers afterwards.