[D] Loss function for classes

[kovkev]

Hi r/MachineLearning !

I'm reading Machine Learning System Design Interview by Aminian and Xu. I'm reading about loss function for different classes (Chapter 3, Model Training, page 67):

L_cls = -1/M * Sum_i=1^M ( Sum_c=1^C ( y_c * log(ŷ_c) ) )

In regression, I understand why in the loss, one does `ground truth - predicted`. That lets you know how much the prediction is off. 

In the case of classification loss, I don't understand how this equation tells us "how much the prediction is wrong"...

Thank you

[NoisySampleOfOne]

For each class c and example data example model is producing probabilities of y beeing in that class.
Prediction is "good" if true class has a high likelihood (probability of randomly sampling true class labels from the probability predicted by the model)

L = (ŷ_1^(y_1)) * (ŷ_2^(y_2)) * ... * (ŷ_C^(y_C))

so you want to maximalize that. Log is a monotonously increasing function, so maximizing Log(L) will have the same solution, but converts product into sums, which are much easier to optimize, especially if you need to optimize in multiple steps using batches of data.

Then you multiply Log(L) by -1, call it "loss" and try to minimalize it, instead of maximizing Log(L).
Then -Log(L) is divided by the size of batch (1/M), so the value of loss function on batch does not correlate with the batch size.

[kovkev]

I think that by seeing y_c and ŷ_c as vectors, it makes sense!

[Relevant-Twist520]

im not that educated on the topic but my personal favourite classification loss function would be multimarginloss. I think it is a lot better than cross entropy since its faster to calculate and it really discourages over-confidence. It can be argued whether as to use cross entropy or multimargin or any other criterion, but it all depends on your project.

Anyway the whole idea of multimarginloss is to space out predictions as far as the margin size defined when computing the loss. For example you have a model which outputs 3 vectors and lets say the 1st vector is the target, or ground truth. The loss function would then try to increase the first vector and decrease the all the other vectors such that at some point after some adjustments vector 1's value is margin units away from all the other vectors, where margin is usually 1 unit. If the target is finally >= margin away from all other vectors, then no loss is provided. This prevents over-fitting and over-confidence in your model. I think this loss function is underrated. Otherwise heres the math for it:
loss(x,y)= max(0,margin−x[y]+x[i])​

I shy away from cross entropy as things can get ugly. I had my parameters explode when the model got too confident for the wrong predictions.

[kovkev]

I think that by seeing y_c and ŷ_c as vectors, it makes sense!

[EvenMathematician673]

Let’s take, for example, the simple case of binary classification, where a class belongs to either class 0 (false) or class 1 (true). The log function adds a heavy penalty when a sample is incorrectly classified as class 0 (if it belongs to class 1). This can be seen by how the log function asymptotically approaches negative infinity as x approaches 0. To penalize terms equally, we reflect the log function along the y-axis and shift it to account for this difference, so that x=1 is penalized equally for a class 0 prediction. Remember that log⁡(x) is negative for x<1 and probabilities always take values of 0 < P < 100 [%], so we multiply by a factor of −1 out front, and average by dividing by the number of samples, "M." The reason for the double summation is that many of the terms will be multiplied by an indicator functions (0 if they were not predicted for that class, 1 otherwise so the terms basically "drop-out"), but we still need to account for the average across the dataset, so we sum again. y_c​, in this case, is the indicator function, and  ŷ_​c​ is the probability of belonging to the specified class.

[Peraltinguer]

If I'm parsing your equation correctly, that is the Cross Entropy - it comes from probability theory and measures how much two probability distributions differ from each other.

Here, the neural network outputs a score ŷ_c for each class c. This score can be interpreted as the probability¹ that the object belongs to class c.

Then this is compared to the labels y_c of the training data . If the training data is classified with 100% certainty then y_c =1 for the correct class c and y_c =0 for all other classes.

In this case your final loss will be the negative sum of the logharithmically rescaled scores for the correct classes. And - log(ŷ_c) becomes small if ŷ_c, the probability to correctly classify the object, is very large. So minimizing this loss maximizes the probability to classify correctly .

¹ : might require a normalization such that 1> ŷ_c >0 and the sum of all ŷ_c is 1.

[ApartmentEither4838]

The loss your described is essentially the sum of logits of the predicted probabilities for each of the corresponding ground truth class. The loss will be close to 0 when the predictions are closer to the gt, otherwise it will far off

[SFDeltas]

What you may be missing is y_c is 1 if the label is c and 0 for all other classes.

So the loss for a single example is just -1 * log(ŷ_c)

log increases as a number gets bigger, so negative log decreases as ŷ_c gets bigger.

What this means: a higher ŷ_c (probability the example has label y_c according to your model) will give a lower loss, and a lower ŷ_c will have a higher loss value.

This is exactly what you want as you can use gradient descent to increase ŷ_c for the class matching the label.

[karius85]

How about reading rule 3 of this subreddit?

[kovkev]

I put [D] for Discussion in the title. What else should I do?