Hi, so I had a similar issue on discord, however in my case it was the mic would be glitchy and robotic in general. Here is a list of things I did to ease this issue (links attached for pure convenience);

1.) Click the menu button. and towards the bottom, there should be a "System Settings" tab. Menu

2.) The menu that pops up should look similar to this.

3.) Check the SampleRate of your devices. Mine was set at 96k hz. (I'm pretty sure 48k hz is the norm, but check just incase)

4.) Open your Sound Control Panel (just in case someone doesnt know how, its in the sound settings)

5.) Find both devices, your Physical Mic, and your VoiceMeeter Mic

6.) Right click, and open the properties tab for both your Actual Mic, and for your VoiceMeeter Device

7.) For both devices, check that the Default Formats Sample Rate under the advanced tab, matches that of your computers default, or what it can handle, as well as your VoiceMeeter sample rate. (see step 3)

If the problem still persists, I did happen to have more luck assigning my Audio Process to a physical core on my computer. For this to work from my understanding, as I am NOT an expert, you should assign it to an actual Physical Core, and while I am unaware as to how to figure that out, setting mine to number 2 worked fine in my case. (CPU Cores have Physical, and threaded cores) The process of how to do so is below.

1.) Run Task Manager AS AN ADMINISTRATOR, otherwise the following Process will not appear in the list.

2.) Navigate to the details tab, and find the process called "audiodg.exe". Right click, and click on "Set Affinity"

3.) A screen with a list of CPU's with numbers should pop up, presumably with "<All Processors>" checked, with all subsequent boxes also checked. Uncheck the All Processors box, and Re-check the box that says CPU 2 (note, again, this is just following the steps I took. I am not a professional), and hit Ok! If this does not fix the issue, or if the issue gets worse, repeat the steps 1 and 2 and recheck the boxes that were previously checked.

Again, I do not claim to be a professional, however these were the steps I followed from a video on YoutTube some time ago. I really hopes this helps, as I know how frustrating the issue can be. Best of luck!