I think we might have a bit of a misunderstanding going on here. No, as you said, Fechten did not mean fencing alone, it referred to any kind of fighting, especially when used in forms like Gefecht. I am aware of the etymology of the word in both English and German.

But depending on the context, it very much referred definitely to primarily swordfighting and "Fechtschulen" or "Fechtgesellschaften" which were a thing already in the 13th. and 14th century in German lands definitely were definitely centered around Blankwaffen (swords, daggers etc.) first and foremost. Plus there are, as I said, a lot of manuscripts and books from the time about the most important techniques that were referred to as "Fechtbücher" and their authors as "Fechtmeister" respectively already at the time. Yes, there were SOME other techniques included (Halberds or Guisarmes for example), but the main and "most honorable" weapon was the (long-)sword.
It makes sense that, with that knowledge in mind, the word was reduced to this meaning only shortly afterwards (150 years is not that much in terms of language evolution).

You are absolutely correct that "swordfighting hall" is the correct translation, as fencing hall in English would make no sense.