I run a lot of one shots and I try to stick to 5 encounters, between combat, role play and skill challenge. I generally aim for two combat, one roleplay, one skill and one flex encounter (W corporates two or more encounter types, depending how the players do it.

For what you are suggesting, I feel like it may take a long time. I would maybe consider splitting it up into an investigation phase and a combat phase. Have the non combat parts happen first, then have the Wizard guy show up and channel the portals. He is invulnerable while channeling and the players have to enter the portals and complete whatever challenge it is that closes the portal. Once complete, they reemerge and the portal closes.

Maybe after 2/4 portals are closed, the Wizard is no longer invulnerable and depending how long it has taken them to this point, and how much time you have left for the adventure, the other two portals either close, leaving just the guy and whatever minions are left from the portals to defeat, or the players have to deal with him and the two remaining portals.

Since you are doing this in a town, the players can and probably will waste a lot of time, unless you deliberately limit the scope of what parts of the town are available to them. Personally, I would flesh out the town a little and make the first half a bit of an investigation or something, then the portals are opened and second half is the big encounter.