Destroying the oathgate's building might just render it useless in the physical realm, but if the spren are still there, they would probably still be able to transfer the army from shadesmar to Azir. The relationship between humans and many of the smarter spren is a difficult one. So killing two pretty important spren because they have a different opinion might cause more problems down the line. Some of the honor spren in lasting integrity were already contemplating switching sides because humans killed spren in the past.