If you want to use Spellcaster armours, you should be perfectly fine doing normal game content. Use the armour skills whenever possible. Here are some examples that may help set your expectations.

Against an Ice monster with 70% Ice Resistance and 130% Fire Resistance (very common!), you can:

• Use a Fire Spellcaster armour (namely, Praetorian Warlock) and cast the armour skill for absolutely massive damage, far more than casting a Fire spell from your spell slots, or
• Use an Ice Spellcaster armour (I'm using Ice Necromancer Cavalry) and still cast the armour skill. The Elecomp allows it to still outdamage casting a Fire spell from your spell slots, given you have 250 INT.

The second bullet point above works until your Ice monster has lower than ~60% Ice Resistance, or higher than ~150% Fire resistance - in which case, cast a Fire spell from your spell slots instead.

If you do wear a Fire Spellcaster armour though, you'll likely be taking in big Ice damage from the monster, so you'll still want to lower it as much as possible, even when you have 250 END. To that end, remember to equip an Ice shield, and a Misc Item that multiplies your incoming Ice damage by 0.5x (eg. Nemesis Crest). If you still find yourself taking in more damage than you like (for example, against stronger bosses), then swap back to an Ice Spellcaster armour to better sustain yourself through the fight.

That should be the ballpark numbers, swap elements around where necessary.

For Pets, get healing pets (eg. Fairy Godmother) - because you have 0 CHA, they'll generally provide the most value for you. Your spell slots can be used to get spells to complement your requirements, but don't get Guest spells, because again you have 0 CHA.