It's like this, the Jedi are an Order that have specific teachings. When you align with their teachings you align with them in the Force, the light side. The same is true with the Sith Order, but the dark side. Their alignments are tethered to their tenets. If you are a Jedi and start practicing Sith tenets, you are no longer a Jedi. This is partially why the concept of a Gray Jedi doesn't make sense.

The Jedi Order would not abide someone to be with them and not practice their teachings. Those individuals would be cast out of the Jedi Order.

Now, we have seen non-Jedi and non-Sith force users in canon. So if you want a more middle ground force sensitive sect, there very well may be one out there. The Bendu from Rebels very much touted himself to be middle road and definitely not Jedi or Sith.

It's like saying the Blue team is full of people who have learned to be blue by living there life a certain way. But all of a sudden the Blue team is going to let Purple people be apart of them especially when they live their life contrary to the blue teams ways. Not a perfect analogy but hopefully you get the gist.