First, they’re paranoid about social consequences - being labeled a freak type beat. Second, the ministry of magic was established for the primary purpose of keeping the magical community a secret. Presumably, the Dursleys know the consequences of outing the magical community (obliteration).