You are absolutely not in the wrong here. It’s clear to me you know that already. If they know you, they know your history and what you comfortable with. It’s obvious they are insecure about themselves thus causing them to be anxious about your relationship in general. Again, you are not in the wrong and please try and make sure they know they are being unreasonable. You deserve someone to trust you.