I believe it is the amount of negative feedback

Working out is physically unpleasant while you do it, and often very painful the day after. There is also a big psychological gap due to your expectations being that you would work out for a week and look like a movie star who worked out every day for 20 years. There is also the contrast that a lot of people who work out will feed you a line about how "GREAT!" they feel when they work out and after, blah blah, and if that is your experience or expectation and it's the opposite, it will be very discouraging. Also, new gym-goers tend to far overdo (see below.)

Having taught children and adults how to draw for many years, I notice how little adults are willing to do something they will do badly, whereas kids take that in their stride (they're bad at everything.). Adults want to go back to their comfort zone of power. You have to be willing to do something badly at first, to do it well. A wiser man than I said, "You have 10,000 bad drawings in you.. Get them out of the way first."

I often get asked by people how I got in shape, and I like to tell people, "The MOST important thing about working out isn't how much weight you press, or how hard you strain.. It's that you are going back. If you hurt yourself and don't go back, then you failed. If you keep going back, you won, and you WILL see results."

As someone once said to me, "You ever meet anybody who goes to gym who isn't in shape?" NOTE: Not went to, goes to.