Using Fairlearn to Build Fair Machine Learning Models with Python: Step-by-Step Towards More Responsible AI

A model can have acceptable overall accuracy while producing very different outcomes across groups. This updated tutorial uses current Fairlearn APIs to measure group metrics and demonstrate one post-processing mitigation on a deterministic synthetic fixture.

Audit the baseline

baseline_frame = MetricFrame(
    metrics={
        "accuracy": accuracy_score,
        "selection_rate": selection_rate,
        "true_positive_rate": true_positive_rate,
    },
    y_true=y_test,
    y_pred=baseline_predictions,
    sensitive_features=A_test,
)

Sensitive group membership is not supplied to the baseline classifier, but disparities can still arise through labels and correlated features.

Apply ThresholdOptimizer

mitigator = ThresholdOptimizer(
    estimator=model,
    constraints="demographic_parity",
    objective="accuracy_score",
    predict_method="predict_proba",
    prefit=True,
)
mitigator.fit(X_train, y_train, sensitive_features=A_train)

The mitigator learns group-specific randomized thresholds and therefore needs sensitive features at prediction time.

Demographic parity is only one definition of fairness. It can trade off accuracy, true-positive-rate parity, calibration, and individual outcomes. Production use requires affected-stakeholder input, legal review, careful definition of groups and labels, uncertainty analysis, monitoring, and an appeals process.