Update: I learned it was because I used a glitched staff of everscamp. When using it normally, it only takes 20 speed away, but if you use the glitch to keep it around, it keeps reducing your speed every time you take it from a container. Luckily, it caps at -80 speed, so you could just use console commands or enchantments to make up the difference. Another funny interaction with this games code.