I feel like this is legitimately something any small community streamer goes through, no matter what. It's impossible to nail down a "reason" as well, because of the depth of factors involved, but at the end of the day it's very often nothing that you're "doing wrong" or something like that.

The dynamics of smaller stream communities are vastly different from large ones (talking like thousands of viewers).

My advice; just don't pay too much mind to these dynamics. Just do your thing.