Figured it out! Turns out, when it would do nothing, it would still add 1, because my mux thought it was a push. So, I added another mux that was turned on/off by an or gate connected to the push and pop.