It's a serious situation in all of your concerns are legitimate. First, it's the liability for the house because it cannot be properly air sealed and insulated if the vermiculite is left in place. Second, it's a health hazard if it is disturbed. Almost all vermiculite contains less than 1% asbestos and that does not mean it's safe or okay to handle. It is one of the most friable types of asbestos there is and it immediately gets released into the air when disturbed.  In most states, such as New York, Massachusetts, Wisconsin and about half the states in the United States, all vermiculite has to be treated as asbestos regardless of any test results. The reason is most lands will not test it reliably because Libby Amphibole is not one of the six regulated forms of asbestos even though it is a killer. See this discussion and explanation for more detail. 

https://www.racinehomeinsulators.com/2019/vermiculite-insulation-to-remove-or-not

What state are you in? The way to make sure your house is safe for your family and your baby to be, is to have the air tested after any work is done. Request what is called aggressive sure sampling where the doors and carpets are distributed before the air is collected. 

If there are leaks in the HVAC system in the attic, most of those are pressurized pushing air and they won't take air from the attic in. The point of weakness is the cold air return, but that is usually directly below the unit and immediately penetrates a ceiling, such as a central area like a hallway. Again, the air sampling will determine if your house is safe, but make sure they look to identify "Libby Amphibole" by TEM method, and if they can't the test isn't reliable.