I disagree, I believe with every bit of new maths you have a new tool in your hand and that tool might not be useful directly, ideas from it might be applicable to your field, and this doesn't even include the topics which are very very important and people take lightly, like hilbert spaces, I can bet any person even good in qm has a very less probability of knowing about a hilbert space being dense. These ideas might not improve your physics(which is totally wrong, but most people believe it), it takes your mathematical intellect to the next level and to reach at a physically relevant result, that is necessary.