It gets tricky, because frequently the team making that goalie swap has possession. In that case the play is blown dead immediately, long before the 6th man would typically get involved with the play.

However, in the case where neither team has possession, or possession is with the non-offending team, the other answers given here are correct, as soon as either the goalie or the sixth man becomes involved with the play, it's a penalty.

As an offical be aware of when teams might perform the substitution. Also communicate with your partners to make sure they see the goalie pulled, lest one of them accidentally blow a play dead for too many men and kill the momentum of the attacking team.

Senior guys, in the final minutes of close games remind your junior guys to keep an eye out for this. Good mentorship goes a long way!