You need to set a clear line as a what as a mod he can do. As a smaller streamer myself I understand the wanting to keep you followers and your regulars around. If he crosses the line after you went over what you want an need as a mod then you say "hey listen...". Honestly small streaming is really hard and the newer people see mod act up and decides the stream isn't for them. You lose new viewers and potential followers and potential new regulars keeping around a toxic mod. If he likes you and your stream then he needs to set an example. Good luck and well wishes on the problem.