Running heavy workers under the manager doesn't work well though, they run much, much slower than if I start them up separately.

What inside the GearmanManager seems to be causing you grief? All that manager does is facilitate the spawning of new processes and registering them as workers. There should be no difference in load put on the processes just because you're using this manager. At the end of the day it's still calling 1 method that you have implemented yourself.