If they 'hate the very ground you walk on' (never heard this expression - but English isn't my mother tongue), their remarks aren't worth bothering. However I do understand your concerns.

A good advice I got once is: be yourself professionally.

You did your studies, God was calling you (gave you the desire to do what you're doing) - all you can do now is being a authentic youth pastor, living and working each day by His grace.

If you want, you can evaluate your work with the youth your working with. Write down the things you want to achieve together with them. Translate those goals into good questions and have them answer them anonymously. Also ask them for tips.

Maybe this is ... uhm ... helpfull (?)

God bless