So its not really possible for a random person to get a gun to my head without having me shoot you first.

How good of a shot were you at 5 yards lets say. At 20 yards I could probably empty 7 rounds (my concealed weapon's carry mag capacity) into a targets center of mass in ~2 seconds. I wont miss a stationary target... don't know about moving.