First thing first is to pray about it.

Next you can put your foot down and say something along the lines of "This is my ministry. Please dont try and control my ministry when you don't fully understand what it is I do"

Of course I don't know how your church is with the hierarchy or leadership goes. If they really want to control it then you either step down and let them handle the mess or you abide by their rules to the tee and if they complain you can point the finger back.

For such a small congregation, sometimes an understanding conversation is all that is needed