Table of Contents
- Accuracy
- Confusion Matrix
- Precision
- Recall
- F1 Score
- False Positive Rate (FPR)
- False Negative Rate (FNR)
- Summary of Metrics
- Conclusion
When evaluating the performance of a classification model, particularly one trained using Logistic Regression, various evaluation metrics help us understand the model’s behavior in detail. These metrics provide insights into how well the model is performing, especially in terms of how it handles positive and negative predictions. Let’s go over the most common evaluation metrics used in classification tasks:
1. Accuracy
Accuracy is the simplest and most commonly used metric to evaluate the performance of a classification model. It measures the proportion of correct predictions made by the model.
- Formula:
- $$Accuracy = \frac{\text{TP} + \text{TN}}{\text{TP} + \text{TN} + \text{FP} + \text{FN}}$$
While accuracy gives us an overall performance estimate, it may be misleading, especially in imbalanced datasets. For example, if the dataset has a majority of negative instances, the model might predict all instances as negative, achieving a high accuracy even though it performs poorly on the positive class.
2. Confusion Matrix
The Confusion Matrix is a detailed breakdown of the model’s predictions, showing how many instances were correctly or incorrectly classified across each category.
A Confusion Matrix looks like this:
| Predicted Positive | Predicted Negative | |
|---|---|---|
| Actual Positive | True Positive (TP) | False Negative (FN) |
| Actual Negative | False Positive (FP) | True Negative (TN) |
- TP (True Positive): Instances that are actually positive and predicted as positive.
- TN (True Negative): Instances that are actually negative and predicted as negative.
- FP (False Positive): Instances that are actually negative but predicted as positive.
- FN (False Negative): Instances that are actually positive but predicted as negative.
The confusion matrix is the foundation for many other important metrics.
3. Precision
Precision is the ratio of correctly predicted positive instances to the total number of instances predicted as positive. It answers the question: “Of all the positive predictions made by the model, how many were correct?”
- Formula:
- $$Precision = \frac{\text{TP}}{\text{TP} + \text{FP}}$$
- Use case: Precision is particularly important when we want to minimize the number of false positives. For instance, in spam email detection, you don’t want to misclassify a legitimate email as spam.
4. Recall (Sensitivity)
Recall, also called Sensitivity or True Positive Rate (TPR), measures the proportion of actual positive instances that are correctly identified by the model. It answers the question: “Of all the actual positives, how many did the model correctly identify?”
- Formula:
- $$Recall = \frac{\text{TP}}{\text{TP} + \text{FN}}$$
- Use case: Recall is crucial when we want to minimize false negatives. For example, in medical diagnoses, it’s important to correctly identify patients with a disease (minimizing FN), even if it means allowing some false positives (misclassifying healthy individuals as sick).
5. F1 Score
The F1 Score is the harmonic mean of Precision and Recall. It is particularly useful when we need a balance between Precision and Recall, especially when we are dealing with an imbalanced dataset where one class is underrepresented.
- Formula:
- $$F1 Score = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}$$
- Use case: F1 score is ideal when both false positives and false negatives are equally important. For example, in fraud detection, both failing to identify fraud and falsely accusing a transaction can have significant consequences, so we aim for a balance between precision and recall.
6. False Positive Rate (FPR)
The False Positive Rate (FPR) tells us the percentage of negative instances that are incorrectly classified as positive. It is the ratio of false positives to the total actual negatives.
- Formula:
- $$FPR = \frac{\text{FP}}{\text{FP} + \text{TN}}$$
- Use case: The FPR is crucial in situations where false positives can have serious consequences. For example, in email filtering, if legitimate emails are falsely marked as spam (false positives), important emails might be missed, which can lead to significant issues.
7. False Negative Rate (FNR)
The False Negative Rate (FNR) is the percentage of positive instances that the model incorrectly classifies as negative. It is the ratio of false negatives to the total actual positives.
- Formula:
- $$FNR = \frac{\text{FN}}{\text{FN} + \text{TP}}$$
- Use case: FNR is important when we need to avoid missing actual positive cases. For example, in medical diagnostics, a high FNR means that patients who have a disease might be wrongly classified as healthy, leading to severe consequences.
Summary of Metrics for Logistic Regression
| Metric | Formula | Use Case |
|---|---|---|
| Accuracy | TP+TNTP+TN+FP+FN\frac{TP + TN}{TP + TN + FP + FN} | General performance evaluation |
| Confusion Matrix | — | Breakdown of predictions (TP, TN, FP, FN) |
| Precision | TPTP+FP\frac{TP}{TP + FP} | Minimizing false positives |
| Recall | TPTP+FN\frac{TP}{TP + FN} | Minimizing false negatives |
| F1 Score | 2×Precision×RecallPrecision+Recall2 \times \frac{Precision \times Recall}{Precision + Recall} | Balance between precision and recall |
| FPR | FPFP+TN\frac{FP}{FP + TN} | Minimizing false positives |
| FNR | FNFN+TP\frac{FN}{FN + TP} | Minimizing false negatives |
Conclusion
Choosing the right metric for model evaluation depends on the specific problem you’re trying to solve. While accuracy might be enough for some cases, it is essential to delve into other metrics like precision, recall, and F1 score to get a better understanding of how the model is performing, especially when dealing with imbalanced datasets or when false positives or false negatives have significant consequences.