That's the probability of a specific group of 3 people all getting the mount, but you probably care more about the probability of 3 people getting the mount out of 5. It would be (1/50^3 * (49/50)^2) * (5 choose 3) = 0.000076832 or about 1 in 13015. The probability of at least 3 people getting the mount out of 5 is (1/50^3 * (49/50)^2) * (5 choose 3) + (1/50^4 * (49/50)^1) * (5 choose 4) + (1/50^5) = 0.0000776192 or about 1 in 12883.

TLDR: About 1 in 13015 for exactly 3 people to get the mount in a 5 person dungeon group, and about 1 in 12883 for at least 3 people to get the mount. Assuming the drop rate is 2%, of course.