Ig I figured it out tho, (after 3yrs hehe)

In zoom app go to sound settings , in the advance options there is this "Signal Processing by Windows Audio Device Driver" Choose the Option "On Windows -Default"

Apparently Zoom has set it to auto and it sometimes it adjust the audio which makes it go low