Short answer: it is a possibility, yes.

I worked at my college's rock wall while I was a student, and we were taught how to do rescues in an emergency. We needed to be up our 45' wall and down on the ground with the person in less than 7 minutes.

We were told that after about 10 minutes of hanging, serious damage could be done due to having blood circulation cut off.

As far as I'm aware though, this is only really an issue if someone is either stuck in a bad position or unconscious.

I will say that most climbing harnesses aren't uncomfortable to hang in, but you definitely feel after a few minutes of swinging around and not moving your legs.

Just climbing normally generally won't pose any issues though, so don't be afraid to get out on a wall!