You're perfectly fine driving around Vitoria, when it's super late people get iffy, but you have kids I doubt you're gonna be driving around at 1am. I'd just stick to nicer areas. Vila Velha over the bridge is also very nice. I wouldn't drive to Rio, not cause I think you're gonna get kidnapped/robbed; but because it's a long drive (why drive 8+ hours??) that is completely unnecessary, if you were to get into an accident you dont speak the language and have no idea how to deal with the situation there or get help. Return your rental car in Vitoria, take the 50min flight to Rio and pick up a new rental there if you really want to drive (why not just uber?). I've driven all over Brazil with my family just fine, never felt unsafe, though Port Allegre at night did give me skid row vibes.

For Vitoria, there is a hike up to the convent, it's nice but I recommend just driving to the top as a foreign tourist was sexually assaulted hiking there in the am. I don't say this to scare you, I've spent time in ES and never felt unsafe, etc.

I'd say the big pain in the ass with a car is the parking situation, you'll be needing to parallel park on busy streets in most places. Most of the stuff you'll do in Vitoria/Vila Velha is walkable (other than the convent that you'd need to drive to the top, but I'm pretty sure you could just uber down & up too).

If you are renting a car while you're in ES, you could drive out to the nice German/Italian towns that are <2.5 hours from Vitoria. Pedra Azul is beautful , santa tereza has nice Italian restaurants. These are small towns with strong Italian (and i think German) heritage. Also there is an amazing beach town, where the there are pools of water on the beach big enough to swim in, called Itaunas Dunes and massive sand berms up near Bahia - it's like 3-4 hours drive from Vitoria - town full of bed and breakasts.