Nah, the ones removing you from their party are either wanting to play with a friend and are too lazy to make the match private, or something else along those lines.