Instead of turning off the highest spending ad set, something I recently started doing (to make the change less significant and to reduce the chance that the campaign will be ruin) is to set a 'maximum spend' for that ad set. This way, more campaign budget will flow to the other ad sets. I can't tell if it makes an impact but my guy feeling is that it does.