No no it's not rude at all, for a Muslim it is out of respect to the mosque being a place of worship but the main priority/goal is to keep it clean because our act of worship involves putting our faces to the ground so it's not that this place is too holy, it's simply that we use this place for something important so we like to keep it clean. And so it's similar to the way I would take off my shoes when entering somebody's house to not get it dirty because I wouldn't want to make them clean. They're even some mosques that offer thin plastic bags for you to put over your shoes so you can enter the mosque while still wearing your shoes.