Adversarial Validation Overview

In the event you had been to review a few of the competition-winning options on Kaggle, you may discover references to “adversarial validation” (like this one). What’s it?

In brief, we construct a classifier to attempt to predict which knowledge rows are from the coaching set, and that are from the take a look at set. If the 2 datasets got here from the identical distribution, this needs to be unimaginable. But when there are systematic variations within the function values of your coaching and take a look at datasets, then a classifier will be capable of efficiently be taught to differentiate between them. The higher a mannequin you may be taught to differentiate them, the larger the issue you could have.

However the excellent news is that you may analyze the discovered mannequin that will help you diagnose the issue. And when you perceive the issue, you may go about fixing it.

This put up is supposed to accompany a YouTube video I made to elucidate the instinct of Adversarial Validation. This weblog put up walks by means of the code implementation of the instance offered on this video however is full sufficient to be self-contained. You could find the entire code for this put up on GitHub.


Studying the Adversarial Validation mannequin

First, some boilerplate import statements to keep away from confusion:

import pandas as pd
from catboost import Pool, CatBoostClassifier


Knowledge Preparation

For this tutorial, we will be utilizing the IEEE-CIS Credit Card Fraud Detection dataset from Kaggle. First, I will assume you’ve got loaded the coaching and take a look at knowledge into pandas DataFrames and referred to as them df_train and df_test, respectively. Then we’ll do some fundamental cleansing by changing lacking values.

# Exchange lacking categoricals with ""
df_train.loc[:,cat_cols] = df_train[cat_cols].fillna('')
df_test.loc[:,cat_cols] = df_test[cat_cols].fillna('')

# Exchange lacking numeric with -999
df_train = df_train.fillna(-999)
df_test = df_test.fillna(-999)

For adversarial validation, we wish to be taught a mannequin that predicts which rows are within the coaching dataset, and that are within the take a look at set. We, subsequently, create a brand new goal column through which the take a look at samples are labeled with 1 and the prepare samples with zero, like this:

df_train['dataset_label'] = zero
df_test['dataset_label'] = 1
goal = 'dataset_label'


That is the goal that we’ll prepare a mannequin to foretell. Proper now, the prepare and take a look at datasets are separate, and every dataset has just one label for the goal worth. If we educated a mannequin on this coaching set, it might simply be taught that every little thing was zero. We wish to as an alternative shuffle the prepare and take a look at datasets, after which create new datasets for becoming and evaluating the adversarial validation mannequin. I outline a operate for combining, shuffling, and re-splitting:

def create_adversarial_data(df_train, df_test, cols, N_val=50000):
    df_master = pd.concat([df_train[cols], df_test[cols]], axis=zero)
    adversarial_val = df_master.pattern(N_val, substitute=False)
    adversarial_train = df_master[~df_master.index.isin(adversarial_val.index)]
    return adversarial_train, adversarial_val

options = cat_cols + numeric_cols + ['TransactionDT']
all_cols = options + [target]
adversarial_train, adversarial_test = create_adversarial_data(df_train, df_test, all_cols)


The brand new datasets, adversarial_train and adversarial_test, embody a mixture of the unique coaching and take a look at units, and the goal signifies the unique dataset. Notice: I added TransactionDT to the function record. The rationale for this may develop into obvious.

For modeling, I will be utilizing Catboost. I end knowledge preparation by placing the DataFrames into Catboost Pool objects.

train_data = Pool(
holdout_data = Pool(



This half is straightforward: we simply instantiate a Catboost Classifier and match it on our knowledge:

params = 

mannequin = CatBoostClassifier(**params)
_ = mannequin.match(train_data, eval_set=holdout_data)


Let’s go forward and plot the ROC curve on the holdout dataset:

It is a good mannequin, which implies there is a clear approach to inform whether or not any given document is within the coaching or take a look at units. It is a violation of the belief that our coaching and take a look at units are identically distributed.


Diagnosing the issue and iterating

To know how the mannequin was ready to do that, let’s take a look at an important options:

The TransactionDT is by far an important function. And that makes complete sense on condition that the unique coaching and take a look at datasets got here from completely different intervals (the take a look at set happens in the way forward for the coaching set). The mannequin has simply discovered that if the TransactionDT is bigger than the final coaching pattern, it is within the take a look at set.

I included the TransactionDT simply to make this level–it isn’t suggested to throw a uncooked date in as a mannequin function usually. Nevertheless it’s excellent news that this system discovered it in such a dramatic trend. This evaluation would clearly aid you determine such an error.

Let’s get rid of TransactionDT, and run this evaluation once more.

params2 = dict(params)
params2.replace("ignored_features": ['TransactionDT'])
model2 = CatBoostClassifier(**params2)
_ = model2.match(train_data, eval_set=holdout_data)


Now the ROC curve seems like this:

It is nonetheless a reasonably robust mannequin with AUC > zero.91, however a lot weaker than earlier than. Let’s take a look at the function importances for this mannequin:

Now, id_31 is an important function. Let’s take a look at some values to know what it’s.

    '', 'samsung browser 6.2', 'mobile safari 11.0',
    'chrome 62.0', 'chrome 62.0 for android', 'edge 15.0',
    'mobile safari generic', 'chrome 49.0', 'chrome 61.0', 'edge 16.0'


This column comprises software program model numbers. Clearly, that is comparable in idea to together with a uncooked date, as a result of the primary prevalence of a selected software program model will correspond to its launch date.

Let’s get round this downside by dropping any characters that aren’t letters from the column:

def remove_numbers(df_train, df_test, function):
    df_train.loc[:, feature] = df_train[feature].str.substitute(r'[^A-Za-z]', '', regex=True)
    df_test.loc[:, feature] = df_test[feature].str.substitute(r'[^A-Za-z]', '', regex=True)

remove_numbers(df_train, df_test, 'id_31')


Now the values of our column appear like this:

    'UNK', 'samsungbrowser', 'mobilesafari',
    'chrome', 'chromeforandroid', 'edge',
    'mobilesafarigeneric', 'safarigeneric',


Let’s prepare a brand new adversarial validation mannequin utilizing this cleaned column:

adversarial_train_scrub, adversarial_test_scrub = create_adversarial_data(

train_data_scrub = Pool(

holdout_data_scrub = Pool(

model_scrub = CatBoostClassifier(**params2)
_ = model_scrub.match(train_data_scrub, eval_set=holdout_data_scrub)


The ROC plot now seems like this:

The efficiency has dropped from an AUC of zero.917 to zero.906. Which means that we have made it just a little more durable for a mannequin to differentiate between our coaching and take a look at datasets, however it’s nonetheless fairly succesful.



After we naively tossed the transaction date into the function set, the adversarial validation course of helped to obviously diagnose the issue. Extra iterations gave us extra clues that a column containing software program model info had clear variations between the coaching and take a look at units.

However what the method is just not capable of do is inform us find out how to repair it. We nonetheless want to use our creativity right here. On this instance, we merely eliminated all numbers from the software program model info, however that is throwing away probably helpful info and may finally damage our fraud modeling process, which is our actual aim. The concept is that you wish to take away info that isn’t essential for predicting fraud however is essential for separating your coaching and take a look at units.

A greater strategy might need been to discover a dataset that gave the software program launch dates for every software program model, after which created a “days since release” column that changed the uncooked model quantity. This may make for a greater match for the prepare and take a look at distributions whereas additionally sustaining the predictive energy that software program model info encodes.



About the Author

Leave a Reply

Your email address will not be published. Required fields are marked *