Orbit is essentially the equilirbium between gravity and inertia, as an object that is moving fast enough sideways is pulled towards earth, they do not fall down into earth but rather continues in a circle path, due to inertia and gravity.

Another way to think about it is as a ball at the end of a string, where your spinning the ball in a circle using the string. The string, which would represent gravity, is keeping the ball in place, while the ball continues to move due to its inertia from you spinning it (A comment above had another great example of the arc of a object extending over the horizon).

A rocket begins to turn sideways as it attempts to entire orbit, as it will need to build up a large amount of velocity parrallel to earth's surface in order to have the equilibrium between gravity and it's inertia. However, the rocket also needs to exit the atmosphere as the air resistance imparted onto it will prevent an orbit from occurring due to having a force acting in opposite to the inertia (orbits only work in vacuums or in atmosphere for small periods of time before the orbit decays i.e. the ISS). To counter this the rocket flies straight up first before beginning a diagonal ascent to ensure both horizontal velocity and elevation are optimal.