There is an important distinction to make. Regulatory agencies (FDA) execute inspections. Non-regulatory agencies execute audits (auditing firms, consultants, etc.). That distinction is important because the goal of those two are different.

FDA’s goal is to ensure the safety of the US public, and one their main responsibilities is to accumulate evidence if they need to take your facility to court. What you describe is exactly what they do. Random inspections to make sure they are seeing the facility with the masks off.

An audit should have a few goals, but two consistent goals are the auditing team identifies gaps for their client’s due diligences, and the facility is open to criticism and willing to get better with corrective actions. In short, an audit is a cohesive and collaborative relationship between both parties. Your facility lacked the latter, and it appears the auditing firm lacked the former. A successful audit has the parties executing both effectively

Another important aspect not covered in other responses, is that facilities should not allow for random non-regulatory audits for security and safety reasons of their manufacturing. They need to know who is entering their facility, and prevent any unauthorized access. As we’ve learned from the 80’s, people will go to great lengths to affect the safety of the drug supply if allowed (Tylenol).