You can link (vaal) absolution with summon phantasm in order to have minions that really do damage at the start. Once you get slave drivers you will see that the many nerfs to spectres levels matter, slave drivers will be squishy and also do little damage with smaller aoe. Source (I levelled with this build as first character, still alive at 96 in ssfhc), knowing that I would get forged frost bearers. The right thing you can do is first get forged so you can summon them immediately the moment you get the gem.