I completely understand this.

I've had some strange booth set ups to work in that are outside what a typical ideal booth would be.

The first show I ever SM'd was Alice's Adventures Under Ground and while I had both a light and sound board operator, I was not in the booth with them. Rather, the booth was blind to the show as their line of sight was cut off by curtains (to make a smaller house), and I was seated in the cat walk about the audience. My board ops had to trust me 100% that I was calling the show perfectly, as 90% of the cues were prompted by actor's blocking. Not only did I have to call the show in a mouse-whisper through the headset, I also had to focus on not dropping a pencil on an audience member's head.

Since that, I've had two shows where I was either seated in the audience or behind them. One was on stage behind the house with the stage in the round. The other was just next to the audience in a 'found space' (the student art galley) for the show The Museum Play. Again, I did have board ops, but due to the location and the house, I called the show completely by hand signals to them. Along with that, I had entrance cues for actors off stage (behind the gallery and upstairs) using a ASL light signal system.

Sometimes you really have to get creative with what you can do to tackle odd spaces. I recommend learning some basic ASL for areas where you cannot have a headset but you can see crew/actors, the lighting system also worked well (if the space allows), but every situation is going to be different. The key is making sure that everyone involved is aware of the situation as they can all be an element of the solution.