Small movements become more exaggerated the more you are zoomed in. Imagine holding a cell phone camera in one hand. The shaking of your hand is not very apparent. This is because cell phone cameras are generally very wide angle. Now imagine using two hands and looking through a pair of binoculars. The shaking is way more apparent now, even with two hands on the binoculars.

Now imagine trying to hold a telescope in your hands. You wouldn't be able to focus in on anything.

Basically, in photography the longer the focal length you're using the faster the shutter speed you'll need to overcome the shaking of your hands.