6.7: Evaluating Model Performance

Evaluation Metrics Calculations

From the confusion matrix:

• Sensitivity (True Positive Rate) = TP / (TP + FN) = 50 / (50 + 10) = 0.833 or 83.3%
• Specificity (True Negative Rate) = TN / (TN + FP) = 35 / (35 + 5) = 0.875 or 87.5%
• Overall Accuracy = (TP + TN) / Total = (50 + 35) / 100 = 0.85 or 85%

ROC Curve and AUC

The ROC curve is a graphical representation of a model’s diagnostic ability. It plots the True Positive Rate (Sensitivity) against the False Positive Rate at various threshold settings. The area under the curve (AUC) gives a single metric that reflects the model's ability to distinguish between classes. A perfect model has an AUC of 1.0, while a purely random model has an AUC of 0.5.

In the ROC graph shown below, the solid curve represents the performance of our logistic regression model at different threshold levels. The diagonal dashed line represents a baseline model making random guesses. The farther above the diagonal our curve lies, the better the model performs.

Understanding model performance is essential for making informed decisions based on logistic regression output. While accuracy is often the first metric business professionals look for, it can be misleading in situations where the classes are imbalanced—for instance, when only a small percentage of customers make a purchase or default on a loan. That’s why more nuanced metrics such as sensitivity (true positive rate) and specificity (true negative rate) are crucial. These help decision-makers understand not only how often the model gets it right, but also which kinds of predictions it gets right—and which ones it misses.

The confusion matrix provides a simple but powerful summary of a model’s strengths and weaknesses by breaking down correct and incorrect predictions. It allows analysts to evaluate trade-offs, such as whether it's more important to catch all potential buyers (sensitivity) or to avoid targeting unlikely customers (specificity).

The ROC curve takes this a step further by showing how model performance changes at different probability thresholds, giving users the flexibility to fine-tune the model depending on business priorities. A high AUC value indicates that the model is generally good at distinguishing between the two classes.

Together, these tools help assess not just whether a logistic regression model is effective, but how and where it is effective. This enables better alignment between model predictions and real-world business strategies—whether in marketing, risk management, or customer retention planning. In the next section, we’ll look at how to interpret and communicate logistic regression output in a way that supports business storytelling and decision-making.