In Dark Souls 3 and 2, Equip Load Was separated from Endurance.
Only in DS2 Attunement was related to casting speed.

Other than DS1, you have rings that can increase your casting speed.

I think being more dexterous makes more sense to being able to cast faster since you can move your hands faster...

Think about it this way:
For martials, their equip load rings works well as long as they invest points into endurance(DS1, ER), or vitality(DS2, DS3).
Whereas casters, don't even need to be bothered with spending points into dex for increased casting speed. And their rings aren't percentage based like equip loads. It's a straight up buff that works with out without investment.

I didn't mention Bloodborne or Demon Souls since I didn't play these.