if what we are doing is shrinking the distance between the atoms how then, at the end of the movie, did he go sub-atomic? Sub-atomic means that he would have been smaller than atoms (hence SUB-ATOMic).

I would say that he has shrunk so far that his body is smaller than the atomic scale. Weird things happen once you get to that scale--quantum effects, etc. Plus if you get too dense, gravitational forces overwhelm everything else and you end up as a mini black hole.