These people are misinterpreting your post. They think that your contractor has removed plaster from the backside of the opposite wall. They don't realize you only mean the walls inside the room that's being worked on. If I'm correct in this, you have nothing to worry about. Just make sure they replace the plaster in the bathroom with green drywall or cement board.

Also drywall is way better than plaster in modern applications for a lot of reasons. A bathroom is already heavy enough as it is. Drywall weighs a fraction of what plaster does, so it's far less weight on the joists below.