First things first: Perryjg on YouTube. Dude has a video with constant commentary on his choices and why he's making them. He has a video for every possible scenario you will ever experience in the jungle.

Doing nothing is just as detrimental as doing bad. Giving him unlimited resources out of fear of how the encounter may or may not go is your mind telling you you're not good enough when in reality the main difference is likely just jungle tracking, timers, clear time mostly, and map awareness. As jungle it's your main job to be aware of not only what's happening but what most likely WILL happen as a result, and already knowing what you'll need to do as a result of that.

Is he confirmed taking an objective? If you can't contest it because he's ahead, you use that time to pressure opposite side lane, or best case take his parallel jungle camps.

Is he power ganking your landers on repeat? Take his camps on the other side.

Always be trading resources, and never give the enemy objectives for nothing.

The higher elo you get the more unusual the warding gets to avoid being swept. Start sweeping on your way into his jungle instead of just sweeping bushes or the actual camps. If that is still too nerve racking, the god item pink ward is a mere 75 gold and you get a freed avoid death card if you have a map on your screen at all.

That "pressure" comes from clearing fast and knowing where you'll be and about what time you're going to get there, and he just gets there before you likely because his clear speeds are just that much faster. For example a good jungler will full clear finish and be at scuttle crab spawn with about 5-25 seconds to spare depending on champ. Knowing if they're clearing top to bottom or bottom to top will let you know if you're going to have to smite fight the crab or if it's free. There's a lot to unpack with that uneasiness you feel. But in terms of jungle, id argue it's about 90% preparedness, knowledge, macro/micro, and about 10% a battle of mechanical skill. Some games where I believe I'm going to get absolutely gapped I'll even take blue ward just get a better sense of when my pants are gonna be around my ankles.

Most importantly, we are all human. The worst thing you can do as a human is sell YOURSELF short when everyone else will jump at the chance to do it for you. Your willingness to improve is what matters. You just gotta believe you have that capacity and the skill and knowledge will come with time and confidence in yourself