How do you read a confusion matrix?

1 Basic terms

A confusion matrix is a four-celled structure that indicates the outcome of a prediction. The rows represent the actual labels, while the columns show the predicted labels. These four cells are: true positive (TP), true negative (TN), false positive (FP), and false negative (FN). TP is when the model correctly predicts the positive class, TN is when the model correctly predicts the negative class, FP is when the model incorrectly predicts the positive class, and FN is when the model incorrectly predicts the negative class. This can be visualized as a table with two columns: one for predicted positive and one for predicted negative; and two rows: one for actual positive and one for actual negative.

2 Accuracy and error rate

One of the simplest metrics to measure the overall performance of a classifier is accuracy, which is the proportion of correct predictions out of all predictions. It can be calculated as: Accuracy = (TP + TN) / (TP + TN + FP + FN) Accuracy tells you how often your model is right, but it does not tell you how well it can differentiate between the classes. For example, if your model always predicts the positive class, it will have a high accuracy if the positive class is very common, but it will fail to identify the negative class. Therefore, accuracy can be misleading if the classes are imbalanced or if you care more about one type of error than another. The opposite of accuracy is error rate, which is the proportion of incorrect predictions out of all predictions. It can be calculated as: Error rate = (FP + FN) / (TP + TN + FP + FN) Error rate tells you how often your model is wrong, but it does not tell you what kind of errors it makes. For example, if your model always predicts the negative class, it will have a low error rate if the negative class is very common, but it will miss all the positive cases. Therefore, error rate can also be misleading if the classes are imbalanced or if you care more about one type of error than another.

4 Precision and F1-score

Another way to evaluate a classifier is to look at how well it can avoid false positives and false negatives simultaneously. This can be done by using two metrics: precision and F1-score. Precision, also known as positive predictive value, is the proportion of predicted positive cases that are actually positive. It can be calculated as: Precision = TP / (TP + FP) Precision tells you how reliable your model is when it predicts the positive class, but it does not tell you how many positive cases it misses. For example, if your model only predicts the positive class when it is very confident, it will have a high precision, but it will also have a low sensitivity. Therefore, precision can be misleading if you want to find all the positive cases or if the positive class is rare. The opposite of precision is F1-score, which is the harmonic mean of precision and sensitivity. It can be calculated as: F1-score = 2 * (Precision * Sensitivity) / (Precision + Sensitivity) F1-score tells you how well your model balances precision and sensitivity, but it does not tell you how well it performs on the negative class. For example, if your model is very good at finding the positive cases and avoiding false positives, it will have a high F1-score, but it may also have a low specificity. Therefore, F1-score can be misleading if you care about the negative class or if the classes are imbalanced.

5 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?