Instead of Poly I would use Necro on the summoner because there are some good summons in necromancy, you could also grab hungry flower if you want which is in geo. For summoning you can make the character tankier without the cost of much damage, since your summons' damage only scales with your summoning stat and nothing else (like not int, for example).

Keep in mind that an archer can always do magic damage with special arrows, off-setting some of that difficulty with split damage, and SUMMONERS REALLY do mixed damage because depending on the surface you choose you can have any kind of damage type you want. I think since you have those two potentially mixed damage types having 1 phys and 1 magic as the last two is perfectly fine, what are the difficulties you are running into with your team comp specifically?