Credit Card Fraud Detection¶

First, I'd like to thank Janio Martinez & Pavan Sanagapati for inspiring me to take on this project. Both have provided excellent, in-depth analyses of this Kaggle Dataset, and I've learned a great deal from both.

I'd also like to thank Marco Altini for his article Dealing with Imbalanced Data: Undersampling, Oversampling, and Proper Cross-Validation, which helped me better understand many of the concepts in this notebook.

If you have any questions, suggestions for improving upon my approach, or just like my work, please don't hesitate to reach out at keilordykengilbert@gmail.com. Having a conversation is the best way for everyone involved to learn & improve 😊

Introduction¶

In this kernel, our overall goal is to develop the best model for classifying both fraud and safe transactions correctly, working with Kaggle's Credit Card Fraud Detection dataset.

The Data¶

We are given a dataset of roughly 285K transactions to work with, for which we have the following features: Time, Amount, and 28 anonymized features (V1-V28) which are the result of a previously-performed PCA. We know that these anonymized features have already been scaled, but Time and Amount have not. We also have our target variable Class, which tells us if transactions are fraudulent.

Goals¶

• Perform initial Exploratory Data Analysis.
• Balance the data through random undersampling, and using this balanced data to determine which features are most highly correlated with fraud, attempt to address the most influential extreme outliers.
• Train, cross-validate & test a variety of different baseline classifiers on our undersampled balanced data to determine which best recognizes fraud.
• Cross-validate and test our best baseline model on the original data to determine its true performance.
• Investigate if we can improve our baseline model by training it on oversampled (SMOTE) balanced data, rather than undersampled balanced data.
• Train Neural Networks with both undersampling and oversampling approaches to test which peforms better.
• Identify which model performed best overall.

Our approach includes:

1. Exploratory Data Analysis & Preprocessing
   • View imbalance between fraud & safe transactions
   • Identify null values
   • View feature distributions
   • Perform feature scaling where needed
2. Random Undersampling
   • Identify true correlations
   • Remove extreme outliers
3. Dimensionality Reduction Gutcheck
   • Perform PCA, T-SNE, Truncated SVD
   • Infer how well our classifiers might performance
4. Baseline Classification Models
   • Training & cross-validation on undersampled data
   • Optimize hyperparameters with GridSearchCV
   • View learning curves to check if models are under/overfitting
   • Check ROC curve performance
   • Test models with undersampled data, view confusion matrices
   • Determine which model best classifies fraud (logistic regression)
5. Logistic Regression: Undersampling (NearMiss) Performance
   • Train logistic regression using undersampled (NearMiss) data
   • Cross-Validate on the original data, compare performance to cross-validation on undersampled data
   • Test performance on final test data, look at precision-recall score
6. Logistic Regression: Oversampling (SMOTE) Performance
   • Train logistic regression using oversampled (SMOTE) data
   • Cross-Validate on the original data, compare performance to undersampled model
   • Cross-Validate on the undersampled test data, compare performance to undersampled model
   • Test performance on final test data, compare both under & oversampled logistic regression models
   • Using accuracy, precision-recall score, and confusion matrices, determine whether undersampling (NearMiss) or oversampling (SMOTE) gives better results for logistic regression
7. Neural Networks
   • Create two simple neural network classifiers
   • Train the first using NearMiss undersampling, test performance with original test data
   • Train the second using SMOTE oversampling, test performance with original test data
   • Using confusion matrices, compare results
   • Determine which model performed best

Let's begin by loading the following libraries:

Exploratory Data Analysis & Preprocessing¶

First, let's take a high-level view of our data to get a good sense of what we're working with. In particular, let's look at summary statistics for each feature, the balance between fraud & safe transactions, and null values.

No nulls!

Awesome! This gives us a helpful insight: with only 0.17% fraud rate, we have a very imbalanced dataset, which we will need to account for when developing our predictive models.

Let's take a closer look at the distribution of each feature.

The anonymous V__ features tend to exhibit various distributions and ranges. Overall, values across all features fall within the range (-114, 121). Note we're using a logarithmic y-axis to help visualize the data more clearly.

Let's also look at the distributions for Time & Amount:

Scaling¶

Our next step in preparing the data for predictive modeling is to scale our features appropriately. We can infer that the V__ features of this dataset are already scaled, as they have already undergone PCA (in which scaling would have occurred). However, we still need to scale Amount and Time. To do this, we'll use RobustScaler(), which performs better on datasets with significant outliers (note the outliers in the Amount distribution above).

Success! (Note the change in scale on the x-axis)

Now let's split our data into our final train and test populations. While we will train & cross-validate our models on various samples of the the training data (final_Xtrain, final_ytrain - more on this later), we will preserve this test data (final_Xtest, final_ytest) to evaluate the final performance of our models at the end of our analysis. Note that we make sure to stratify by our target variable, Class, in order to ensure both populations are representative.

Awesome! Per our output, both our train & test populations have 99.83% safe and 0.17% fraud transactions.

Random Undersampling¶

Now that we've explored and scaled our data, we can focus on "balancing" our data.

Balancing our data means resampling our data such that we have a 50/50 split between fraud and safe transactions (as we just saw, we're pretty far from that at the moment with 99.83% safe vs 0.17% fraud). It is import that we create this 50/50 split before training our models because otherwise our models will overfit to the majority class, assuming that essentially all transactions are safe. We want to train our model to recognize the characteristics of fraud, not assume that most transactions aren't fraud. Thus we must create a 50/50 split to remove the effect of the imbalance during training.

Let's consider two ways to create this 50/50 split:

Random Undersampling: Here, we reduce the number of transactions in the majority class (i.e. safe transactions) randomly, until the counts of both the majority and minority classes are equal.

SMOTE Oversampling: Here, we increase the number of transactions in the minority class (i.e. fraud transactions) through creating synthetic transactions in between existing ones that are in close proximity. Synthetic transactions continue to be created until the counts of both the majority and minority classes are equal.

We will perform both sampling approaches to determine which performs better. Let's start with random undersampling:

Awesome! Through random undersampling, we now have a balanced dataset, txn_under, with which we can predict fraudulent transactions without overfitting.

Next, let's create a correlation matrix to get a clear view of which features are most heavily correlated with fraud. Note that we should only use a balanced dataset (i.e. txn_under) to determine correlations, as per our discussion above, an imbalanced dataset (i.e. txn) overfits to the majority class. Let's create correlation matrices for both to illustrate the difference (but make sure you only reference the balanced matrix to determine correlations!).

Great! As we can see, balancing our data gives very different (true) correlation values. Referencing the balanced output, we see fraud is most highly correlated with the following features:

Negative Correlations: V14, V12, V10

Positive Correlations: V11, V4

Let's view these features a bit more closely...

Awesome! Just as we should expect -- fraud transactions have lower values for features that are negatively correlated (V14, V12, V10) and higher values for features that are positively correlated (V11, V4).

Note the presence of outliers in some of these features. We want to remove extreme fraud outliers from the most highly correlated features. This is an important preprocessing step for maximizing model performance on recognizing fraud. Addressing outliers in features with the highest correlations will help ensure we are addressing only the most impactful outliers, and removing only the most extreme outliers will help us prevent unnecessary loss of information.

We will use the IQR to identify outliers. But first, let's get a better view of the distributions for each feature, to ensure they are roughly normal, and thus our outlier removal approach makes sense.

These all look normal enough to proceed. We're ready to remove outliers:

Great! We successfully removed the most extreme outliers from our most highly-correlated features.

Gutcheck - Dimensionality Reduction¶

Before we proceed with creating classification models, let's first get a sense of how effective our models might be through performing dimensionality reduction on our data.

Specifically, we'll use three dimensionality reduction methods (t-SNE, PCA & Truncated SVD) to reduce the number of our features to just two. We will then graph the results on the xy-plane, highlighting fraud and safe transactions differently. If we're able to see a clear separation between classes in the graphs, that will give us an indication that further predictive models may perform well at classifying fraud.

Fantastic! We see a clear separation between fraud & safe transactions in all 3 graphs, especially in PCA and Truncated SVD! This gives us a good indication that our predictive models will be able to effectively classify fraud.

Baseline Classification Models¶

We're finally ready to create our classification models! We'll start with four types of classifiers: Logistic Regression, K Nearest Neighbors, SVC, and Decision Tree.

First, we'll train and cross-validate our models on the txn_under dataset to get a sense of which model does the best job of recognizing fraud. Once we have an opinion on which model does best, we'll cross-validate that model on the unsampled (original) data to get an objective view of its true performance.

A Note on Cross-Validating Imbalanced Data:¶

Before we proceed further, let's digress for a moment to talk about a common mistake made when dealing with imbalanced datasets. While we should train our models on balanced data, we should cross-validate on imbalanced (i.e. original) data to best evaluate the objective performance of our trained models. Cross-validating on the original data gives us the most accurate view of our model's performance in production, because the original data preserves and is representative of the imbalance we would expect to see in production.

Our initial goal here, however, isn't to determine objective performance, but rather to determine which model does the best job of identifying the characteristics of fraud. In this case, it is okay for us to cross-validate with our undersampled data, as long as we are consistent in our approach across all models. We will still be able to rank models' abilities to identify fraud by cross-validating on undersampled data, even if the scores we derive are not representative of what we would expect to see in production.

Let's start by splitting our undersampled data, txn_under, into train and test sets:

Despite the fact that these scores are not representative of the true production accuracy, we do see that all four models do a very good job (~90% or more) of identifying fraud... which is good, and inline with what we might expect given what we saw from our dimensionality reduction earlier. Let's see how much we can improve these scores further by optimizing our hyperparameters with GridSearchCV. Again, we'll perform cross-validation on the balanced data so that we can compare to the baseline scores above:

Awesome! We were able to improve our cross-validation scores.

Let's plot learning curves to get a sense of whether our models are over/underfitting. Note that the wider the gap between our training and cross-validation scores, the more likely we are overfitting:

Great! We don't appear to be overfitting given the lack of a large gap between curves as training size increases, and we don't appear to be underfitting given how high our scores are.

Let's take a look at the Receiver Operating Characteristic (ROC) curve for each of our models, to get a better sense of their performance. ROC curves are used to show in a graphical way the connection/trade-off between False Positive Rate (the percent of safe transactions our model classifies incorrectly) and True Positive Rate (the percentage of fraud transactions our model classifies correctly) for every possible decision boundary in our models. In other words, it shows the tradeoff between our incorrect classification of safe transactions and our correct classification of fraud transactions as we move through every possible decision boundary.

To evaluate model performance, we'll calculate the area under the ROC curve (AUC). This tells us which model does the best job of distinguishing between fraud and safe transactions overall.

Awesome! With the highest ROC_AUC score and undersampled cross-validation accuracy, logistic regression looks like it could be our best model...

We're finally ready to test our models with our X_test/y_test data to determine which model does the best at recognizing fraud (keep in mind, this is still not the original/imbalanced data).

Awesome! Based on our undersampled test data, logistic regression performed the best at recognizing fraud. We will continue our analysis with just Logistic Regression.

Logistic Regression: Undersampling (NearMiss) Performance¶

First, let's come back to our previous conversation on the proper way to cross-validate for estimating true performance. So far, we've only cross-validated using balanced/undersampled data. Let's now cross-validate logistic regression using the original, imbalanced data.

Recall that we still have to train our model on balanced data. To do this, we'll implement the NearMiss() algorithm, which is an undersampling technique that works by randomly removing samples of the majority class that are close to samples of the minority class, thus clarifying the boundary between both classes and improving model classification.

Now let's print our performance metrics for both cross-validation approaches (balanced vs imbalanced data) to see how they compare.

Awesome! While we tend to see lower scores when cross-validating on the original/imbalanced data, these differences make sense.

For example, we would expect to see much lower precision when cross-validating on the original/imbalanced data than the undersampled/balanced data, simply because the imbalanced data consists of far more safe transactions. Thus, we would expect a higher number to be misclassified as fraud, lowering precision.

Note, however, that recall remains relatively unchanged. This also makes sense, as we're cross-validating on the same (i.e. unsampled) set of fraud transactions in both scenarios.

Now we're ready to test our logistic regression model (trained on NearMiss undersampled data) with our final_Xtest and final_ytest data. We'll use both precision and recall to evaluate performance:

Our average precision-recall score on our test data is objectively low. This may seem suspiciously low after seeing our ROC curves above, but remember, we're now evaluating our model on the original/imbalanced data. As we can see in the graph, our model allows many more safe transactions in (i.e. missclassifying them) as we relax our decision boundary.

Logistic Regression: Oversampling (SMOTE) Performance¶

Now let's proceed with our oversampling approach! We'll be using the SMOTE technique to oversample.

SMOTE¶

Unlike with the undersampling approach in which we randomly remove data from the majority class, with SMOTE we create new synthetic points for the minority class to achieve class balance. Specifically, SMOTE creates synthetic points between closest-neighbors of the minority class. This approach retains more information than undersampling, as we aren't removing any data from the original dataset. However, because we have to train on more data with SMOTE, the approach is more time/resource-intensive.

Cross-Validation¶

Again, while we should train our model on the oversampled/balanced data, it's important that we cross-validate on the original data. Overfitting can result if we cross-validate on the oversampled data, as this population contains additional synthetic samples and does not represent the true imbalance between classes. As with undersampling, we want to cross-validate on the original/imbalanced data to get an objective view of model performance.

Interesting! We tend to see higher cross-validation scores (precision, accuracy, f1) when training with oversampled data than when training with undersampled data from earlier in our analysis... which makes sense.

For example, we see significantly higher precision when training on the oversampled data. The reason for this is that we don't lose any information on the safe transaction set during training (e.g. by randomly removing data) in our oversampling approach, which allows our model to more thoroughly learn the characteristics of safe transactions, and thus missclassify them less often, increasing precision.

Note, however, that recall remained relatively unchanged. This is a testament to how well SMOTE is able to add new synthetic fraud transactions to the training data while maintaining the original distribution/characteristics of fraud.

Out of curiosity, let's compare how our logistic regression models trained on undersampled data (log_reg) vs oversampled data (log_reg_sm) perform on the undersampled test data (X_test & y_test).

Awesome, SMOTE slightly improved logistic regression's performance on the undersampled test data!

Let's now see which model performs best on the original test data (final_Xtest, final_ytest). We'll start with accuracy:

Wow! Oversampling with SMOTE greatly improved accuracy on the original test data. Let's compare precision and recall as well:

Awesome! SMOTE oversampling gives much higher accuracy overall, as well as a higher precision and f1 score for fraud transactions. Additionally, as we can see in our precision-recall graph directly above, training with SMOTE resulted in a much better precision-recall score overall. In comparison to our undersampling precision-recall curve, we now let far fewer safe transactions in (i.e. missclassifying them) as we relax our decision boundary. Clearly, the information loss through undersampling negatively affected the model's ability to correctly classify safe transactions.

Neural Networks¶

Now let's implement two simple neural networks (NNs) to see how they perform!

To create our NNs, let's have one input layer with the same number of nodes as features plus a bias node, a second hidden layer with 32 nodes, and one output node classifying the transaction as 0 (safe) or 1 (fraud).

First, we'll build our undersampled model.

Undersampled NN¶

Great! Now, let's train this network on the undersampled data, then we'll predict on the final test data.

Now, let's create a function to plot the confusion matrix...

Awesome! The undersampled NN performed pretty well, with recall and precision similar to what we saw from our oversampled logistic regression model earlier.

Let's see how our performance changes when we train our NN using oversampled data.

Oversampled NN (SMOTE)¶

Awesome! The oversampled NN performed very well, with high f1 scores on both safe and fraud transactions, and exceptionally high fraud precision.

While the oversampled NN miscalssified a handful more fraud transactions than the undersampled NN, the undersampled NN missclassified almost 200x more safe transactions. It's important to recognize that the cost of missclassifying a safe transaction is non-zero. Missclassifying a safe transaction blocks the cardholder from making additional transactions until that cardholder can verfiy their account, which costs the cardholder and the financial institution. Overall, it's probably safe to assume that the cost of missclassifying 1 fraud transactions does not outweight the cost of blocking ~300 safe transactions. Thus, we assume the oversampled NN is the best model overall.

Findings / Conclusion¶

While the plan is to continue to iterate on this project, below is a summary of what we've learned so far.

EDA & Preprocessing¶

It's important to always explore the distributions of features in a dataset before beginning any predictive modeling. This helps with many things, including identifying which features need to be scaled, uncovering imabalances in our data, and locating outliers, nulls, and other important values.

When determining feature correlations in a highly imbalanced dataset, it's important to first balance our data through a method like random sampling. If we don't first balance our data, any correlations we derive will overfit to the majority class, and we won't have a clear view of which features most influence the minoirty class.

In our analysis, we balanced our data through random undersampling of safe transactions. We were then able to accurately determine correlations and identify which features would most influence our predictions. From those highly correlated features, we were able to identify and remove the most impactful extreme outliers.

Anticipating Model Effectiveness with Dimensionality Reduction¶

We can use dimensionality reduction techniques like T-SNE, PCA & Truncated SVD to get a sense of how well our classifiers might perform.

To do this, we first reduced the features of our dataset to just two, and visualized the results in the xy-plane. Highlighting the different classes in this visualization, we saw how easily seperable or "clustered" each class was, which gave us an indication that our classification models would perform well down the line.

Cross-Validation & Imbalanced Data¶

In dealing with this imbalanced dataset, it was important that we first balance our data before training our classification models. Otherwise, our models would have overfit to the majority class, assuming that practically all transactions are safe.

When cross-validating, however, we took two approaches:

1. First, we cross-validated on the undersampled (balanced) dataset. This allowed us to see which model best recognized fraud transactions. The performance metrics from this cross-validation approach (accuracy, precision, recall, etc.) were not representative of what we would expect to see in production, however, because the balanced data did not represent the imbalance we would expect to see in production.

2. Second, to get a true view of our model's performances, we cross-validated on the original imbalanced data. This dataset was representative of the imbalance we would expect to see in production, thus cross-validating on this dataset gave an accurate view of our performance metrics.

Baseline Models - Random Undersampling¶

We began by evaluating the performance of four different models: Logistic Regression, K Nearest Neighbors Classifier, Support Vector Classifier, and the Decision Tree Classifier. We made sure to optimize the hyperparameters for each model using GridSearchCV, trained and cross-validated our models on the undersampled (balanced) data, and checked learning curves to confirm our models were not over/underfitting.

To evaluate which model performed best at identifying fraud, we looked at both cross-validation accuracy and ROC_AUC on our undersampled (balanced) training data. We further evaluated each model on our undersampled test data, creating confusion matrices for each.

In the end, logistic regression performed the best at identifying fraud, thus we chose to focus on this model moving forward.

Logistic Regression Performance - Random Undersampling¶

Proceeding with our logistic regression model trained on undersampled data, we cross-validated the model using the original (imbalanced) training data to get a true view of its performance. We saw that our true performance metrics were lower than when we cross-validated on the undersampled data, especially precision (which made sense, as the imbalanced data consists of far more safe transactions with the potential to be misclassified as fraud).

Using our final test data, we looked at our model's precision-recall curve. We found relatively low scores overall, further indicating that our model did not do a great job of keeping safe transactions from being missclassified as the decision boundary was relaxed.

Logistic Regression Performance - Oversampling (SMOTE)¶

Next, we went back to the drawing board and trained logistic regression on oversampled data using the SMOTE technique. We optimized hyperparameters with RandomizedSearchCV this time, due to the large size of the training data. Upon cross-validating with the original (imbalanced) training data, we saw significantly higher precision and accuracy than from our previous logistic regression model trained on undersampled data. The reason for this is that we don't lose information about safe transction when oversampling (SMOTE), thus we can better classify them.

We saw that while both models performed similarly on the undersampled (balanced) test data, the oversampled logistic regression model produced much higher accuracy on the final (imbalanced) test data. Furthermore, the oversampled logistic regression model far outperformed the undersampled model in terms of precision-recall.

Neural Network¶

Finally, we put our baseline model aside, and investigated the performance of neural networks. We built two simple NNs: the first was trained on the undersampled data, the second was trained on the oversampled data.

After evaluating both NNs on our original test data, we found that the oversampled NN also outperformed the undersampled NN (assuming 1 fraud is less important than ~300 blocked accounts).

Key Takeaways / Next Steps¶

Balancing our dataset using oversampling (SMOTE) helped us improve our logistic regression and Neural Network models over random undersampling. While it's possible that undersampling could outperform oversampling in certain scenarios (e.g. depending on the tradeoff between false positives and false negatives), in general oversampling preserves more of the original data, and thus models trained with oversampling perform more accurate classification. Of course, the tradeoff with oversampling is that it uses more time/resources, given it consists of more training data.

Next Steps: Recall that we removed outliers from our undersampled data before training our models. We should also do the same for our oversampled data to see if our performance improves further.