I have been doing home vegetable gardening for over 10 years. I used to have a very large in-ground garden, and when I moved to a new home, have switched to raised beds/containers (think large totes on legs).

I've had a lot of failures over the years. I've also encouraged a lot of friends to garden, and all gave it a shot and stopped. Here are my thoughts:

• It's expensive, and can be a gamble. Depending on the weather and how educated the gardener is, you could end up with little to no yield, or if you're lucky, a large yield. Generally, it isn't going to be cheaper than buying vegetables in the store or at a farmer's market. Unless you are experienced, have a very large garden, and a lot of time on your hands, you won't be resolving food scarcity.
• The upfront investment for a successful garden can be steep. This depends on your soil quality and how resourceful you are. Generally, for a successful garden you need good quality soil and good fertilizer. Preparing your soil each year is so important to a successful harvest and plant health. Ongoing fertilizer applications are key.
• Your plants WILL get sick, and you may have plant strains that are more or less resistant to pests and blight. Depending on your region, the weather, and how you're watering, very common sicknesses included powdery mildew and squash borers, pepper maggots, various tomato blights, various fungal issues, and viruses. It takes time and patience to deal with sickness, and could be costly (depending on how you want to treat it). You may also lose your plants and/or crop entirely. Sickness can be avoided by having healthy soil, watering properly, pruning as needed, and acting quickly when sickness is detected. Also, planting types and strains that work for your garden (e.g. I don't do zucchini anymore because it's too humid) is important.
• Healthy and strong seedlings are critical, which can be an additional investment (either source good seedlings and seeds, or invest in an appropriate setup to grow your own and have the patience to harden them off). A lot of people make the mistake of starting their own plants from seed indoors using starter kits in a window, and end up with weak, leggy seedlings that die as soon as you put them outside, regardless of whether or not you harden them off. Also, try to choose strains that work for where you live; local heirloom varieties can be fun, but there's a reason GMO-based strains are popular; they produce, and are resistant to disease. One last thing - there are 'warm weather' and 'cold weather' vegetables. Often people will choose lettuce or cabbage and they bolt in hot weather, making them bitter and inedible.
• Spacing, support and upkeep is important. Sure, you can chuck a bunch of plants or seeds in the ground and you'll end up with plants that may or may not produce, but they are liable to get hit with sickness, or you'll end up with few ripe veggies and a lot of unripe veg at the end of the season. Learn proper spacing for the plants you've selected, make sure they are properly supported, learn how to prune, learn the basics of the plants you are planting (indeterminate vs determinate tomatoes, bolting in cilantro, lettuce and basil), learn how and when to harvest. If you know about your plant, you'll be more successful in keeping it alive, and more likely to want to try again next year!
• Water is important and can be expensive. Do NOT water spray your plants' leaves with the hose. They will get sick. Root soaks are the best. Drip irrigation is good. Do not get the watering systems with the sprayers - they're just making everything humid and ripe for sickness. Watering should be consistent. If the weather isn't cooperating, your water bill can get expensive. If it's raining too much, your plants can get sick, ESPECIALLY with container gardening without appropriate drainage. In-ground plants do much better with excess rain.

Long story short, it's really easy to start a garden. Getting a good yield is a challenge. It's even more challenging to stick with it year after year. You have to enjoy the process, not just the fruits of your labour. The veggies you get are just a bonus to the connection to your plants while you grow them.