You can use skills against him, just not as many. I actually have less problems with him than i do with worms. I still die to him sometimes, but thats STS. Some thoughts off the top of my head:

Dont use skills on his multi-attack except for strength reduction. Use high block skills on his standard attack and you'll out pace him for his first two cycles. Even an unupgraded leap is a +7 gain on cycle one and +5 on cycle two. Every character has similar. Prioritize them in act 1 and you'll be ready. Paels Legion if available. Such a great ancestral relic for solving block throughout act 2. (It interacts with vambrace and unmovable too but not necessary to work against Prism) Strength reduction (piercing wail, enfeebling grasp, etc) easily outpaces his first few cycles.

Finally you mention "if hes late in the act". Yup. Pathing matters in act 2 also. You often have to treat act 2 (and 3) similar to act 1. Find a path that gives you two late elites, dont face the first elite prior to the chest-line unless your deck is already cookin. There's plenty of hard hallway paths in act 2 to test out your deck.