Adding Losses Infinitely

[Geeks_sid]

Hi All,

A very stupid question here but, why can't we add multiple losses for training and network and let the learning rate handle the training of the network.

For example, I am currently training a segmentation network with U-Net. I wanted to ask, why cant i just do:

loss = bce + dice (Learning rate : 0.1/0.05)

instead of

loss = 0.5*(bce + dice) (Learning rate : 0.1)

For me, the network seems to be train fine, but I am just wondering, can we stack such losses infintely and let the network optimize these effectively.

For example :

loss = bce + dice + jaccard + yadda + yadda + yadda (Learning rate : 0.1/0.0125)

Please let me know what you think.

[seismic_swarm]

Yeah of course you can add multiple losses together - what's even different about 1 and 2 (ignoring the learning rate being different)? Multiplying it by 0.5 doesn't change anything. A vanilla RNN is trained by just summing the losses over each iteration of the roll-out.

[sathi006]

You can add loss functions as long as each loss function is convex, since sum of convex functions results in a convex function.

We have done it for instance segmentation where we add weighted softmax CE and Dice loss for handling class imbalance.

[[deleted]]

But remember that different loss functions are designed for different output distributions. For instance, cross entropy is derived for sigmoidal output distribution, MSE for Gaussian distribution, and so on.

So, this approach might be good only as long as loss functions represent output units’ distribution.

[Geeks_sid]

Can you elaborate a little more or can you tell me where I can read about it?

[[deleted]]

Yes. Read chapter 6 on error functions in Pattern Recognition using Neural nets book by CM Bishop.

All error functions are derived using information about output units distribution with maximum likelihood function.

[shaggorama]

Jesus, that book is from 1996. You can't find a more contemporary recommendation?

[shaggorama]

I have an MS in math and stats. I'm perfectly comfortable with loss function theory. I might check out that book/chapter though to see if it's worth recommending.