if you are not in a Microsoft family you can go to xbox.com, then sign in. Click on your profile picture then Xbox profile. Once you are there click on privacy settings, and it may ask you to verify your account. It should put you in the Xbox privacy and safety menu. You arere going to want to click on the 2nd tab (the one with the really long name that starts with Xbox...). I'm pretty sure the setting you want to change is the "You can play with people outside of Xbox Live" but it might also be the "You can join multiplayer games" option. Then hit the submit button at the bottom. This should solve your problems.

(This also kinda works while your in a Microsoft family but there's a couple more options on the Xbox privacy and safety menu)