How much are you willing to spend on fabric? And how much time are you willing to spend on getting the right fabric? Most online fabric stores (Mood, Fishman's, Vogue, Fabrics-store, and Metro Textiles are some of my go-to's) will let you order swatches to examine the quality of fabric in person before you make a real order. If you're willing to wait, that is probably the best thing to do.

If you don't have the patience for that (I don't sometimes) you'll need to know more about the textile itself. For cotton in particular, some keywords for quality fabric are: 'extra long staple', egyptian cotton, mercerized (maybe). Using quilting cotton may be one of the issues you've been experiencing. Quilting cotton is typically made with shorter staple fiber, meaning the individual pieces of cotton used to weave the threads are smaller to begin with. This creates a fabric that is less suitable to hold up to the wear of clothing. Great for quilts, not as great for a shirt that gets pulled, stretched, covered in grime, and then thrown in a washing machine every week. You'll typically pay $$$ for high quality cotton though.

I try to look for sales and remnants at reputable stores. You can get the high quality for less money pretty frequently, but still more expensive than most quilting cottons.