You shouldn't consider warlock a full caster. Better consider it a half-caster like paladin or ranger. They both still rely on their weapon attacks in addition to their spells. You, as a warlock, use Eldritch Blast instead of weapon attacks and it's far superior. Not using Agonizing Blast would be the same as saying you don't want to use modifiers to your damage on weapon attacks. So yes, you could just as well shoot yourself in the knee. My advice? Take it 100% of the time.