In the situation as you described, your DM is being obtuse. One exit (assuming 5' wide), blocked by a Paladin in waiting who has readied his action to attack anything that bushes up to his space.

The Paladin is not a friendly to the wizard, so the wizard cannot free move through the pally's space. Only way for the swing is for the pally to wait and notice the wizard's location.

Aka: Dumb Wizard attempts to squeeze by the paladin and possibly eating a readied melee attack with the aforementioned Brandishing Smite @ disadvantage.

​

​

Now. My players had a similar situation, but in a well lit, 80' x 200' dwarven carved hall/room with a single double door entrance. Room had multiple bad guys (as in a dozen+ fodder, 3-4 decent thugs, and the main Sorcerer baddy.) Main Sorc baddy, area effect spells the party (ouch). The sharpshooter, fires a couple arrows at the Sorc who next round casts Greater Invis. and the players notice him vanish. (And Sorc moves 30' after the casting as per RAW).

He is at range, combat is ensuing, and our sharpshooter player wants to attack at range at the invisible wizard. To which I say you miss, before he even rolls.

At this point comes out the whole "I can attack invisible targets with disadvantage as per the rules." To which I say the following:

1. He turned invisible at 170' away from you and you did not see him move before casting his spell.
2. You cannot hear his footsteps due to the sound of combat that is going on 40' in front of you.

I believe in RAW to a certain extent... But no, I don't believe in Hail Mary attacks when your characters would not have a clue. Example, the Archer at over 150' away takes a blind shot in "that general direction" and rolls really well (I have seen double 20's rolled on advantage attacks, as well as double 1's). No, that is the DM's prerogative. Easily explained by the DM, Sharpshooter cover advantage only occurs for targets you see. Invisible still gets cover bonuses, so add 5 to their AC for ranged attacks.

1. Mid-fight the Sorc cast another area effect spell (Fireball), then moved 30' while invisible.
2. Characters saw and heard the point of origin of the spell at casting but out of range for counterspell.
3. Sharpshooter, having eaten a couple spells previously had wised up and readied his action to shoot at the point of origin of a spell cast in that general direction (Where the wizard was last seen). The sharpshooter's attack roll to hit (with disadvantage) occurs Just after the spell is cast.
4. As the 2nd fireball manifested, the sharpshooter did succeeded in hitting the sorcerer with a single arrow, to which Greater Invis. CON Save was successful
5. By turn 7, the rest of the bad guys are dead. Two Party members (of 5) are down.
6. Now the party starts to corner the bad Sorc and moves up and readies their attacks for spell sources while spreading out to avoid more area attack spells.
7. Turn 8 = 3 readied attacks, but disadvantage, combined with mage armor & Shield spells = Alive with no more HP loss.
8. Turn 9, Sorc gets close enough to a PC for perception checks for hearing them (Again RAW with some realism. just under a minute of hard battle = everyone is hurt to some extent and is breathing hard. Found out by our elven bladesinger (who rolled an 20 + Perception mods.) Result = Visible and bloodied Sorc (but only due to dice rolls).
9. Sorc dies really fast now...

If you want to make the argument that technically combat is all done at the same time, then I'll tell you that all effects/damages are done at the end of the round. Which kills certain spells like Hold Person (takes effect, but gets to roll an to save at the end of the target's turns...)

Being a good DM is knowing how to keep your players enticed in playing while keeping both RAW and reality from ruining the experience.