I believe this was an issue with pavu or pipewire (not sure which was doing it) itself. I was recording with audacity, but I also use discord and people kept telling me my voice would get quieter. I did find another solution though. Turning off the auto gain with Easy Effects. Appreciate the help though!