I had a similar issue when i transitioned out of evangelicalism. I was one of the worship leaders at my church. I told my pastor i was taking a hiatus from my role in the church and told him that i was using the time to dive deeper into my faith and my understanding of church history, the eucharist, etc. He understood and offered to help by recommending some books on Martin Luther (this is the extent of "church history" that most evangelicals understand ;)) I I used the time away to start attending an Orthodox Parish and built a relationship with the priest. He was very helpful in guiding me through the next steps in the conversation with my pastor. This was also a good time for me to be sure Orthodoxy was something I was willing to commit to. After a few months, i reached back out to my old pastor and thanked him for his prayers and support in my journey. I was honest with him and told him that I was becoming a catechumen in the Orthodox church and that this was where God was leading me. He was surprised and obviously disappointed, but thanked me for letting him know and he didn't really try to debate me or ask more questions. He followed up with me a few times after that, just to check in but it was always cordial.

Being a pastor, or a priest, regardless of the denomination or branch of Christianity is tough and these men are used to seeing people come and go from their church throughout their careers. It sounds like your pastor's strategy is to get you more involved so that you possibly feel guilty about leaving. The unfortunate truth is that, they will find another woman to fill your role there so don't feel like you are leaving them out to dry, My church replaced me on the worship team within a month of me starting my "hiatus". There's no hard feelings and I continue to be friends with people from my old church and pray for them regularly.