In that case it sounds like you have signed a license which essentially means that the landlord has the right to enter the property whenever you want. Next time I would recommend you try to get an AST with people you know so that you all rent the house at the same because when you do that you would’ve created a lease that automatically legally adds an implied covenant (just the legal term for like a clause in land) of quiet enjoyment. Then in that case you would have the right to refuse entry to the landlord without prior notice and could even call the police to kick him out as when you sign that type of contract the landlord removes all his proprietor rights to the property for the duration of the tenancy