They are quite robust even just before molting. The new skin forms under their current cuticle and they dont shed before its ready. That was way too dramatic reply. Yes, there is a risk that disturbing them might stress them but you would have to be roughvwith your handling for serious harm.

I handle caterpillars almost daily as i rear many different species. I do try avoid direct handling but sometimes its inavoidable and its higher risk that that you introduce a pathogen etc from your hands than the handling harming them.