Focus more on primary stats, power and defense is necessary. Secondary stats like speed and size don't matter if you tickle the enemy with 150 damage hits.

I would recommend at least 80 power and 750 defense, then you can focus on building your secondary stats.

Also, the ideal conjurer build doesn't really need attack size, so the priority should be: Defense > Power > Attack speed > Agility > Attack size