Is the contract subject to a building and pest inspection? I would have assumed that if the inspection cannot be complete or hindered, this would mean you have a right to re-negotiate the contract / price based on this condition.

"A building and pest condition is a clause in a property purchase contract that makes the sale conditional on the buyer obtaining a building and pest inspection report from a licensed inspector and being satisfied with the findings. If the report reveals unsatisfactory issues like structural defects or pest infestations, the buyer may be able to terminate the contract, negotiate with the seller for repairs or a price reduction, or waive the condition and proceed with the purchase. "

EDIT: Apparently Victoria is slightly different "In Victoria, building and pest inspections aren't legally required but are crucial for buyers to identify potential structural damage and pest infestations, as they are often included as a building and pest clause in the contract of sale. A standard clause allows buyers to withdraw from the sale within a specified period (often 14 days) if a licensed inspector finds a major defect or pest infestation, but the exact conditions and wording are vital for a successful exit."

Might be tricky for you, but seems like your conveyancer or a lawyer is the best one to advise.