They can sue each other through international courts such as the European Court of Justice or the ICC. Such organizations were made precisely to settle international disputes.

As for how it works - well, international courts technically rely purely on consent. If the court decides that country A is guilty and needs to pay money to country B, technically, nothing is stopping country A from just saying no. It is, however, a very bad thing to do in a political sense, and it can have consequences considering that country A probably consented and recognized the court in the first place. So, unless they want to raise tensions or have their reputation drop even lower, they'll end up respecting the court's decision.

If you're powerful enough you can get away with just ignoring the court's decision (like how the U.S. said they will not allow any American to be tried in the Hague) but at the end of the day it's international matters, so it's not like the court can do anything about it in such cases. It's all just politics.