How to Measure the Performance of a Machine Learning Classifier with Python and Scikit-Learn?

Classifier evaluation needs both thresholded errors and probability-ranking metrics. This example uses scikit-learn’s breast-cancer dataset, a fixed stratified 70/30 holdout, train-only standardization, logistic regression, and a prevalence-only baseline.

Confusion matrix and class metrics

The positive class in scikit-learn’s dataset is benign. On 171 holdout samples, the model produced accuracy 0.988, benign precision 0.991, benign recall 0.991, and benign F1 0.991.

Accuracy reports the total fraction correct. Precision answers how many predicted benign cases were benign. Recall answers how many benign cases were found. F1 is their harmonic mean. The confusion matrix keeps the underlying false-positive and false-negative counts visible.

ROC AUC and average precision

ROC AUC uses probability scores, never hard class labels. The model scored ROC AUC 0.998 versus 0.500 for the prevalence baseline. Average precision was 0.999.

Coefficients

Because scaling is fitted inside the pipeline on training rows, coefficient magnitudes are comparable across standardized inputs. Signs refer to the benign class; they are associations, not clinical causal effects.

Limitations

This benchmark is not a clinical validation. A random split does not test another hospital, instrument, population, or future period. ROC AUC does not choose a deployment threshold and does not encode the greater cost of missing malignancy. Clinical use requires external prospective validation, calibration, subgroup analysis, and expert governance.