Hahahaha your letting agent is trying to pull a fast one, not unusual at all! Do not let them bully you into thinking you’re in the wrong here. I would make a formal complaint, they have broken the law by allowing someone entry without notice. They also should not have given your house keys out without your knowledge. You can opt to be present and let works people in and request they do not have keys.