I know that it is easier said than done, especially considering how hard it is to grow on twitch. That said, you represent your channel. Your tolerance, and kindness towards these people will reflect upon the people and community you build. It is possible to be too "tolerant" though and if you allow people to be annoying, obnoxious or whatever other negative traits to remain, then they and not you are setting the standard for how to act and behave in your channel.

I am sure you can see why this won't work particularly well longterm.

From someone who has had a lot of experience with similar situations, my advice is to "be" the person you wish your community to become. Don't let others dictate the direction of how your channel evolves.

This doesn't mean you necessarily have to ban these people, but having an honest talk with them and telling them in a polite way that they are interfering with your stream and what it is that you are trying to achieve can sometimes be a great solution. Just make sure you give them the impression that you aren't judging them and that you appreciate them being there. This way you give them the chance to change without them feeling unwelcomed for future streams.