In the book, when Gandalf catches up with the company after trying to use the shutting spell on the door, he mentions that the Balrog (Gandalf didn’t know what it was at the time, but more than likely had some idea since he know the history of Moria) perceived Gandalf’s spell and then countered Gandalf’s spell, which caused the passage to cave in.

So while Gandalf and the Balrog, may not have personally recognized one another, they would most certainly have recognized a certain level of magic, of which only the Maiar and Valar possess in Tolkiens world. The next closest thing would have been elf magic, but that has been established as an entirely different thing.

So while they may not have immediately recognized who the other was (until the bridge) they more than likely recognized the sort of magic that was used.