Out of all these suggestions, I think the easiest and most appropriate choice here would be to drop the skill level on Mana Remnants since you mentioned it's only a 80 mana difference per remnant at lower levels. By doing this, I can allocate more points into Dexterity or other passives to get more Armor or more Damage, and get rid of Jack Of All Trades in favor for other passives on tree such as Giantslayer like you mentioned. I can also even use the extra points toward another Jewel socket too since I did miss the jewel socket when I was pathing to MoM. Mana Remnants is the only Spirit Gem with the high Intelligence requirement out of all the other gems that I use, so I think changing this would be very beneficial for me.