Customer Churn Prediction - Understanding Models with Feature Permutation Importance using Python
Customer churn is an imbalanced classification problem: only 14.5% of the 3,333 telecom customers in this dataset churned. This updated tutorial uses a fixed seed, a stratified 70/30 holdout, stratified cross-validation on training rows, and a prevalence-only baseline. The final model is evaluated once on the untouched 1,000-row holdout.
Data and reproducible split
The notebook downloads the public Kaggle dataset barun2104/telecom-churn when a local copy is unavailable. It uses nine account and usage fields. DayMins remains excluded to preserve the tutorial’s compact feature set.
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.30, random_state=42, stratify=y
)
Training and baseline
The random forest uses balanced class weights. A five-fold StratifiedKFold grid search selects depth, tree count, and minimum split size by ROC AUC. The best training-fold ROC AUC was 0.916; its parameters were max_depth=16, min_samples_split=20, and n_estimators=256. Hyperparameter selection never sees the holdout labels.
Verified holdout results
The prevalence-only dummy classifier scored ROC AUC 0.500. The tuned forest scored ROC AUC 0.874 and accuracy 0.879. For the churn class, precision was 0.558, recall 0.793, and F1 0.655. Accuracy alone would conceal the minority-class trade-off.
Permutation importance
Permutation importance is computed on the holdout with ROC AUC as the scoring function. Monthly charge, data usage, and customer-service calls produced the largest mean decreases in AUC when shuffled.
Limitations
This is a small observational dataset with no time field, acquisition channel, intervention history, or cost model. A random split cannot test temporal drift. Balanced class weights and a 0.5 threshold encode one operating trade-off; a provider should select a threshold from retention costs and capacity. Permutation importance measures predictive dependence in this fitted model and does not establish that changing a feature will prevent churn.
