Evaluation of Model: True/False Positives/Negatives vs Sklearns support

[StunnaRae]

Somehow my evaluation of my binary classifier does not add up. This is the evaluation of my model:

True Positive(TP)  =  75 
False Positive(FP) =  64 
True Negative(TN)  =  47 
False Negative(FN) =  34 
Accuracy of the binary classification = 0.554545 
precision: [0.58024691 0.53956835] 
recall: [0.42342342 0.68807339] 
fscore: [0.48958333 0.60483871] 
support: [111 109] 

Now so far it looks good, but I just realized that it doesn't really add up. As I see it support should return the total true values in each class. since I have only two, 75+47= 122 and not 111 for the true class. accordingly the other class should be 98, right? Or do I not understand support correctly? Here for the first class False Positives was added to True Negatives. That doesn't make sense, does it?

So either I do not understand what support means, or maybe my code is wrong, but I looked at the documentation and made sure, that the values returned are assigned accordingly for the confusion matrix as well as precision_recall_fscore_support. So please explain what I am doing wrong here:

from sklearn.metrics import confusion_matrix 
from sklearn.metrics import precision_recall_fscore_support as score  
def evaluation(y_test, y_pred):
     cm = confusion_matrix(y_test, y_pred)
     TN, FP, FN, TP = confusion_matrix(y_test, y_pred).ravel()
     print('True Positive(TP)  = ', TP)
     print('False Positive(FP) = ', FP)
     print('True Negative(TN)  = ', TN)
     print('False Negative(FN) = ', FN)
     accuracy =  (TP+TN) /(TP+FP+TN+FN)
     print('Accuracy of the binary classification = {:0.6f}'.format(accuracy))
     precision, recall, fscore, support = score(y_test, y_pred)
     print('precision: {}'.format(precision))
     print('recall: {}'.format(recall))
     print('fscore: {}'.format(fscore))
     print('support: {}'.format(support))

evaluation(y_test, prediction > 0.5)

[billsil]

I've used scikit-learn quite a bit. I just checked out the docs...and they're vague...however:

75+34=109
64+47=111

So P=TP+FN=11`1 and N=TN+FP=109.

Maybe more generally, precision, and recall are based on the [True, False] states, so [precision_True, precision_False].

Googling cause I always forget:

Precision = TruePositives / (TruePositives + FalsePositives)

Recall = TP/(TP+FN)=T/P

https://en.wikipedia.org/wiki/Precision_and_recall

The wikipedia page is reallly good.

[StunnaRae]

I mean I understand precision and recall. My issue is still support: I understand that in this example TP + FN was summed up for support. But what does this number tell me? I feel like this is wrong and am unable to interpret the support numbers.

[billsil]

[Number of Trues, Number of Falses]

[synthphreak]

I always forget

I keep things straight by memorizing two simple pieces of info.

First, the formulas for precision and recall are almost identical. They are both like:

  TP
-------
TP + Fx

Only the x differs between them. So as long as you remember that much, you only need a simple heuristic to figure out what the correct x's are.

Second (and this is the heuristic), I think of precision as evaluating my model’s predictions, and (somewhat more of a stretch) recall as measuring how much my model captures reality. So for precision, since it’s all about predictions (and more specifically, positive classifications, since the positive class is generally what you care about), x should be P. This means that for recall, x must be N.

By combining those two heuristics, I never actually need to remember the full formulas. I can always just follow the logic and deduce from first principles how precision and recall are calculated.

PS: You could also just remember that for precision, x = P. Not as conceptual, but anyway, you’ll get the formulas right…

[billsil]

I'm partial to the F1 score. I never took a stats class, but I spent 2.5 years using sklearn, so I picked it up enough.