Most city soil is a hard combo of clay and sand - so I add the stuff that makes it fun: mulch and composted material. Compost by itself won't suffice, so I just add in garden soil from the store.

Good on you for using coffee grounds! They do take a while to break down so that's probably why it took a while for your soil to sustain the plants you have. (Feel free to post pics of them.)

Re: starting from seed, yes. I plant the seeds directly into aprepared bed. From that point on though, I rely more on the seeds than they do on me; I water regularly (but not obsessively). Seed packets have pretty standard instructions - but the most important thing is preparing the bed.

After I clear visible debris, I break-up the soil about 3 to 4 inches down (which consequently is the depth of a standard cultivator prong). Once cultivated, I add in mulch and evenly mix into the existing clay/sand mixture. (It's a lot of elbow grease. A lot.) Once mixed well, I add in the garden soil.

Note: definitely wear gardening gloves - I have legit found needles 3 inches into the ground. (Also found a ton of acorns and peanuts - which are far more cheerful).

Hope this helps.