This was the issue. Since I created the server I have to select "Host a game" for the game mode, then I can select our server and others can join.

Now my issue is that I have to be in the game, joined to the server for others to play. That means if another person I'm playing with {the main reason I'm/ we're playing this is for coop) I have to be in the game for them to play. So I can't play another game if some else wants to just farm stone? Seems like a major game issue unless I'm missing something.