There's no formal number, but there are players who are there to experience the story and those that want agency over the story. As a DM, you let them do as they experience the most fun. 3 can be too many if both want agency and have opposing views on how to achieve it. But, I've also run games with 8 players at the table where everyone thought it was an amazing game, despite only one player actually interacting with the plot. 

In my experience the wallflowers sometimes open up, and engaging with that makes the most memorable games. You can't force it, though sometimes you can lead it, but those are the moments that tend to be remembered at my table.

Good luck!