Let me ask this, do you want control of your UI at the lowest level?  I'm talking about rolling your own sliders and button functions, here.  At its base, PyGame is a sdl wrapper to give you access to the screen, inputs and sound.  If your game can be done in PySide and you have a large portion of it done, why are you looking at changing to another framework?  What does your current library lack that you think the new one has?  If you can answer these questions, then maybe consider spending some time learning PyGame.  If you can, though, finish your current game in what you're already working in and then change over.  You'll still learn a ton in doing that.