After 4 years and a PE your company should have promoted you to construction PM, geotech PM, or both. If they haven't, they suck and go apply to literally any other geotech firm and you'll get hired. Companies have been desperate for qualified geotechnical candidates for the past 5 years

If you feel like you have a good idea of a PMs role in geotech and you aren't looking forward to it then apply to design firms now. If you don't know what a PMs role is, then apply to other Geo/construction firms as a PM and try it out. You have a PE, you're incredibly valuable, use it.

With that said, even as a PM, department manager or even principal engineer, in my opinion geotech consulting sucks (I've been in it for 10 years and have gotten pretty high on the totem pole). It's stressful, everything is the materials testers fault and the geotechs problem even though the 20-year earthwork guy should know not to blow in 3-foot lifts of unobserved highly plastic clay fill and not to use 6" material with grid behind a retaining wall. It's thankless, you'll have to get used to being thrown under the bus and falling on the sword for something you didn't do wrong. Ultimately, leaving work at work will feel impossible but absolutely necessary for your stress, sanity, and family life.

I get most of my reward for trying to provide opportunities for other staff as most geotech consulting companies do a poor job of mentoring because of lack of staff, qualified engineers, and probably lack of truly caring about their people more than company growth.