Skip to content

model_tests.FEAT.Perturbation

Perturbation Objects

@dataclass
class Perturbation(ModelTest)

Test if the specified metric of specified attribute subgroups of original dataset is worse than that of perturbed dataset by a specified threshold.

To pass, if ratio is used, the ratio (with metric of perturbed data as denominator) of the respective subgroups metrics of the datasets should not exceed the threshold.

If diff is used, the difference (with metric of perturbed data as subtrahend) of the respective subgroups metric of the datasets should not exceed the threshold.

The test also stores a dataframe showing the results of each groups.

Arguments:

  • attr - Column name of the protected attribute.
  • metric - Type of performance metric for the test, For classification problem, choose from 'fpr' - false positive rate, 'fnr' - false negative rate, 'pr' - positive rate. For regression problem, choose from 'mse' - mean squared error, 'mae' - mean absolute error.
  • method - Type of method for the test, choose from 'ratio' or 'diff'.
  • threshold - Threshold for the test. To pass, ratio/difference of chosen metric has to be lower than the threshold.
  • proba_threshold - Arg for classification problem, probability threshold for the output to be classified as 1. By default, it is 0.5.
  • test_name - Name of the test, default is 'Subgroup Perturbation Test'.
  • test_desc - Description of the test. If none is provided, an automatic description will be generated based on the rest of the arguments passed in.

add_predictions_to_df

def add_predictions_to_df(df: pd.DataFrame, model, encoder) -> pd.DataFrame

Predict a set of dataset using the given model, and output the predictions together with the df. Before predicting, encode the categorical features in the dataset with the encoder object.

Arguments:

  • df - Dataset to be predicted by the model, protected attributes not to be encoded
  • model - Model class object, preferably Sklearn class.
  • encoder - One hot encoder class object for protected, preferably Sklearn class, must contain transform() function.

get_metric_dict

def get_metric_dict(df: pd.DataFrame) -> Tuple[dict, list]

Output a dictionary containing the metrics and a list of the metric's sample size for each subgroup of protected attribute, from a dataframe containing 'truth' and 'prediction' columns.

Arguments:

  • df - Dataframe containing 'truth' and 'prediction' columns.

perturb_df

@staticmethod
def perturb_df(attr: str, df: pd.DataFrame) -> pd.DataFrame

Perturb (by shuffling) the protected attribute column values of a given df and and output a new dataframe.

Arguments:

  • attr - Column name of the protected attribute to be perturbed.
  • df - Dataframe containing the protected attribute.

get_metric_dict_original

def get_metric_dict_original(x_test: pd.DataFrame, y_test: pd.Series, model, encoder) -> dict[str]

Get metric dict for original dataset.

Arguments:

  • x_test - Test dataset to be predicted by the model, protected attributes not to be encoded
  • y_test - Series/array/list containing the truth outcome of x_test
  • model - Model class object, preferably Sklearn class.
  • encoder - One hot encoder class object for protected, preferably Sklearn class, must contain transform() function.

get_metric_dict_perturbed

def get_metric_dict_perturbed(x_test: pd.DataFrame, y_test: pd.Series, model, encoder) -> dict[str]

Get metric dict for perturbed dataset.

Arguments:

  • x_test - Test dataset to be perturbed and predicted by the model, protected attributes not to be encoded
  • y_test - Series/array/list containing the truth outcome of x_test
  • model - Model class object, preferably Sklearn class.
  • encoder - One hot encoder class object for protected, preferably Sklearn class, must contain transform() function.

get_result

def get_result(x_test: pd.DataFrame, y_test: pd.Series, model, encoder) -> pd.DataFrame

Output a dataframe showing the test result of each groups. For an example in 'gender' attribute, male subgroup fail the test if FPR of male group in original data - (or division) FPR of male group in perturbed gender data > threshold.

Arguments:

  • x_test - Test df to be inputted into the model, protected attributes not to be encoded
  • y_test - Series/array/list containing the truth of x_test
  • model - Model class object, preferably Sklearn class
  • encoder - One hot encoder class object, preferably Sklearn class attributes, must contain transform() function

plot

def plot(alpha: float = 0.05, save_plots: bool = True)

Plot the metrics of the attribute subgroups resulting from the original and perturbed data respectively, also include the confidence interval bands.

Arguments:

  • alpha - Significance level for confidence interval.
  • save_plots - If True, saves the plots to the class instance.

run

def run(x_test: pd.DataFrame, y_test: pd.Series, model, encoder) -> bool

Runs test by calculating result and evaluating if it passes a defined condition.

Arguments:

  • x_test - Test df to be inputted into the model, protected attributes not to be encoded
  • y_test - Series/array/list containing the truth of x_test
  • model - Model class object, preferably Sklearn class
  • encoder - One hot encoder class object, preferably Sklearn class attributes, must contain transform() function