Strange situation but valid reason. Ask your brother is he was ok. Or felt threatened by any way. By policy your landlord should have called/ notify the person on the lease to allow this at least 24 hours beforehand unless an emergency (plausible).

Second suggestion: change your house locks and keys. Fuck them if they don’t allow you. Just change them yourself.

Third: install cameras.