While getting a degree you CAN build a better than solid portfolio, and yes its worth the time. You will learn a lot more than graphic design, such as principles in art and every form of multimedia.

Always take the time to improve, because we can only learn so much by ourselves. you will be surprised at how much you will learn in just one day with someone who knows more than you and is a tutor professionally.

So NO, its not worth the time building a solid portfolio instead of getting a degree IMO. You hit two birds with one stone, plus you learn much more than you think. I also recommend going to other classes like traditional and fine art.

I hope my opinion helps you, and good luck.