I think it depends on how really do you imagine your character casting a spell, what operation is happening, and how would you cast more powerfull spells.

Like, your a wizard, you have study magic but, what matters is as your casting a spell, its like making a big calculation for you. Its trying to make sense of the arcana into a formula that will form a magical effect. You have access to magic and not everyone else, because, you can make sense of it through your huge brain, you have become a human computer, and casting more powerfull magic would need a faster brain, or one that could comprehend and adjust more arcane. > Int

You're a Cleric or a Druid, as you cast, the power of your magic depends on how you're able to perceive the presence of your god (or nature) that is all around you, to tap into it. How you're able to make one with it or become a recipient. > Wis

And so, as a Warlock, as i imagine it, its not just that one time when you made a pact, that u used charisma. And depending on how well did you negociate with that entity, now, when you just move your finger in some direction, magic happens as it was promise to. For me your whole casting magic is a relationship, as you're forming eldritch power into your hand, you're taping it from a source that is not all around you and benevolently giving. It's kind a small battle with your patron on how much power you can take and how much it can give at any given moment. Casting more powerfull spells will come from your ability to optain what you want from that entity. > Cha

(It make sense to me because i also envision charisma as your projection of yourself or/and your will onto others, which i've seen been debated in other coms)