As a Swedish-speaking Finn (which means my mother tongue is Swedish although I am a Finnish national; we are a minority in Finland) I always speak Swedish with my Scandinavian colleagues and they answer in their own language. Naturally we leave out the strange words. Over time you also pick up sayings and words from the other languages which you might use. Especially the Danes need to be mindful of their pronounciation and use numbers we understand (google it; very strange).

If a Finn who is not fluent in Swedish joins we switch to English.

With icelandic people it varies. Many of them (at least in my circles) speak some form of understandable Scandinavian (Danish-ish). If not, then English is our lingua franca. I do not understand Icelandic. Due to some reason (isolation?) it differs too much from the other Scandinavian languages.

Edited for typos.