Personal Loans & Tree-Based Models 🌲¶

In this project, we'll be investigating historical loan data, with the ultimate goal of developing the best tree-based classification model to separate defaults from safe loans. We'll compare the performances of Decision Trees, Random Forests, Gradient Boosted Trees, and Neural Networks.

The Data¶

We'll be using historical personal loan data from LendingClub.com. LendingClub is a P2P lending platform that connects people who need money (borrowers) with people who have money (investors). As an investor, you would want to invest in borrowers who show a profile with a low probability of defaulting.

The data is from 2010-2013. I chose loans from this time range for two reasons. First, the maximum term for LendingClub loans is 5 years, so we would have realized all defaults by the date this data was pulled (early 2020). Second, I wanted to avoid including loans heavily affected by the Great Recession (2007-2009).

Project Inspiration¶

There are a few reasons why I decided to do this project.

1. I used to work as a risk analyst for a marketplace lender that refinanced student loans. In my job, I built the company's underwriting and pricing models. While I evaluated applicants on factors like FICO score, DTI, free cash flow, and delinquencies to approve these loans, many of the cutoffs were predetermined by our financing partners, and I never actually built a predictive ML model.

2. I'm eager to get a bit more experience working with different tree classifiers. I'd like to improve my understanding of the strengths and weaknesses of each model, and see how each performs under different hyperparameters. Finally, I'd like to create a strategy that identifies the best model for different cost assumptions (cost of misclassifying defaults, cost of misclassifying safe loans).

3. I'm considering becoming an investor on LendingClub's platform. I'd like to have a perspective on whether a loan will default, and building a classification model with LendingClub's historical loan data will help me do that.

Approach¶

We'll take a systematic approach to exploring our data and developing our models.

1. Import Data
2. Data Preprocessing
   • Remove Unnecessary Columns
   • Null Values
   • Feature Engineering
   • Exploring Our Target Variable
3. Correlations
4. Visualizations
   • Feature Distributions
   • Nominal & Correlated Features
   • Additional Views
5. Additional Preprocessing
   • Box-Cox
   • Dummy Variables
   • Scaling
   • Confusion Matrix Function
6. Tree-Based Classifiers
   • Decision Tree
   • Random Forest
   • XGBoost
7. Neural Network
8. Conclusion
   • Final Thoughts

If you have any questions, suggestions for improving upon my approach, or just like my work, please don't hesitate to reach out at keilordykengilbert@gmail.com. Having a conversation is the best way for everyone involved to learn & improve 😊

Alright! I'm excited to revisit my old lending stomping grounds, gain experience working with tree-based models, and build the best predictive strategy for my investing needs. Let's get started!!!

Import Data¶

Data Preprocessing¶

Remove Unnecessary Columns¶

The dataset has over 152 columns, many with duplicative and missing information. Let's just use the following for now:

• loan_status (target variable): Whether the loan was paid off or charged off (defaulted).
• int_rate: The interest rate of the loan, as a proportion (a rate of 11% would be stored as 0.11). Borrowers judged by LendingClub.com to be more risky are assigned higher interest rates.
• grade: Rating for the quality/riskiness of the loan (riskiness increases alphabetically).
• sub_grade: Sub-rating for the quality/riskiness of the loan (riskiness increases numerically).
• term: Length of time to payoff loan.
• dti: The debt-to-income ratio of the borrower (amount of debt divided by annual income).
• loan_amnt: Total amount lended.
• home_ownership: Applicant's housing type.
• installment: The monthly installments owed by the borrower if the loan is funded.
• purpose: The purpose of the loan (takes values "credit_card", "debt_consolidation", "educational", "major_purchase", "small_business", and "all_other").
• emp_length: Applicant's length of employment.
• delinq_2yrs: The number of times the borrower had been 30+ days past due on a payment in the past 2 years.
• earliest_cr_line: Start date of earliest credit line.
• issue_d: Date loan was issued.
• annual_inc: The self-reported annual income of the borrower.
• fico_range_high, fico_range_low: upper and lower bound of borrower's fico score.

Null Values¶

Now let's see how many null values we have and how best to address them.

Great! Given employment length is categorical, and there may be a meaningful reason for why this information is missing, we will create the 'N/A' type for nulls.

Feature Engineering¶

Using the data in our table, we'll create the following features:

• fico: Average of fico_range_low and fico_range_high.
• cr_hist_days: The number of days since borrower first opened a credit line.

Exploring Our Target Variable¶

Let's take a look at the different classes in our target variable, loan_status.

Great! We're only interested in loans that were either Fully Paid or Charged Off (i.e. Defaulted). Let's drop the other loan statuses (which represent <0.5% of the data) and change the values of Fully Paid to 1 and Charged Off to 0.

Correlations¶

Let's take a look at correlations between our features.

• Correlations with Target Variable: Identify which features best predict defaults.
• Correlations between Other Features: Identify which feautres are highly correlated and may be duplicative.

We'll assign numerical values to our non-numeric features using .astype('category').cat.codes. This converts the feature data to a categorical data type, identifies the unique values in lexicographical order, and assigns the integers [0,1,2,...], respectively. For ordinal features like grade, sub_grade, and most of empl_length, this preserves the information contained in the feature's order. This blog post does a good job of explaining how this works in more detail.

Correlations with Target Variable

• While our features are not highly correlated with loan_status in general, we will visualize how loan_status varies with some of its more correlated features (as well as nominal features) later in this notebook.

Correlations between Features

• int_rate, grade, and sub_grade are all highly correlated with each other (.95+) and provide duplicative information. Our tree algorithms will be most precise with a continuous numerical variable, so let's keep int_rate and drop grade and sub_grade.
• loan_amnt is highly correlated with installment. Given installment is less correlation with loan_status, let's remove this feature as well.

Visualizations¶

Time to explore our data with some visualizations!

Let's start by looking at the distribution of each feature individually. We'll finish up by investigating how loan_status varies across both its most-correlated features as well as nominal features (e.g. purpose).

Feature Distributions¶

Observations¶

• Interest Rate: Roughly normally distributed between 5-25%, mean around 13%.
• Debt to Income: Roughly normally distributed between 0-35%, mean around 15%.
• Fico Score: Lognormally distributed between 660-850. It would appear LendingClub's underwriting cutoff is 660.
• Length of Credit History (Days): While the distribution does tail off to the right, it's fairly normally distributed within 0-10,000 days around a mean of about 5,000 days.
• Loan Amount: Ranges between \$0-\\$35K with a slight right skew. Appears LendingClub did not issue loans above $35K.
• Annual Income: Heavily skewed to the right.
• Employment Length: No clear pattern, about a third of the population has 10+ years of employment.
• Home Ownership: About 50% of borrowers have a mortgage, roughly 40% rent, and most of the rest own.
• Purpose: More than half the borrowers are using their loans for debt consolidation, next most popular is to pay off a credit card balance.

Nominal & Correlated Features¶

Now let's take a look at how loan_status varies across our nominal and correlated features.

Let's start with the nominal features: purpose, term, and home_ownership

Observations¶

• Term: The probability of default on a 5-year loan is twice as high as the probability of default on a 3-year loan. This makes intuitive sense, as a longer term consists of more payments and thus more opportunities to default. More about this on Investopedia.
• Home Ownership: There do not appear to be any significant trends between home_ownership and loan_status. While the average default rate is high for the OTHER category, there are not enough samples here for us to have confidence this default rate is higher.
• Purpose: Clearly, certain loan purposes are riskier than others. For example, a small business loan (26% default rate) is much more risky than an auto loan (11% default rate).

Let's take a look at the most correlated features: fico & int_rate

Observations¶

• Fico: In general, the default rate decrease with fico. Note in the histogram how the ratio of defaulted loans to fully-paid loans decreases as fico increases. The lmplot also shows how defaults and fico are negatively correlated.
• Interest Rate: In general, the default rate increases with interest rate. Note in the histogram how the ratio of defaulted loans to fully-paid loans increases as interest rate increases. The lmplot also shows how defaults and interest rate are positively correlated.

Additional Views¶

In this last section of visualizations, we'll explore relationships between a few different features, and we'll compare the behavior of defaults to fully paid loans across different subpopulations of the data.

let's start by investigating the trend between int_rate and fico. In general, we would expect interest rate to decrease as fico increases. Not only does this make intuitive sense, but we see these features are negatively correlated (-0.54) in our correlation plot above. Let's see if this trend waivers at all within the four term by loan_status subpopulations.

Observations

Okay cool, nothing out of the ordinary here. Across all term and loan status subpopulations, we see a consistent decrease in interest rate as fico increases.

Now, let's investigate the trend between int_rate and annual_inc. Assuming that borrowers who make more money are less likely to default, we would expect interest rate to decrease as income increases. However, per the correlation plot above, it appears that interest rate has almost no correlation with annual income whatsoever (-0.01). This seems odd, given it makes intuitive sense that higher income means less risk... Let's see if this lack of correlation is consistent within term by loan_status subpopulations as well.

Note: annual_inc is highly skewed. To best visualize trends in the data, we will use the log of annual income (log_annual_inc).

Observations

• For 36-month loans, the interest rate tends to decrease as income increases.
• For 60-month loans the interest rate tends to increase as income increases.

I'm not sure why this would be... putting a pin in this for now.

Finally, let's take a look at loan_amnt by term.

Observations

• Overall, the loan amount for 60-month loans tends to be much higher than for 36-month loans. The average 60-month loan is for about $20K, whereas the average 36-month loan is about half that much.

• Few 36-month loans exceed $25K.

• The distribution of defaulted loans is very similar to fully paid loans, regardless of term.

Additional Preprocessing¶

Note: Tree-based algorithms are not sensitive to feature magnitudes. So standardizing our data (e.g. scaling and normalizing) is not necessary before fitting our three models. Here's a helpful article that explains this in more detail: When and Why to Standardize Your Data. However, at the end of this notebook I'd like to compare the results of my tree-based models to a neural network, for which standardizing data is recommended (link). We'll standardize the data now to keep everything consistent for all models.

Box-Cox¶

First, we'll normalize our features with a Box-Cox transformation.

Dummy Variables¶

Next, let's create dummy variables for our categorical features.

Scaling¶

Great! Now, let's define our train and test populations and scale our data.

Confusion Matrix Function¶

Before we proceed with training & testing our classifiers, let's create a function to cleanly plot a confusion matrix.

And last, let's check our class imbalance.

Tree-Based Classifiers¶

Alright, we're ready to build our tree-based classifiers! We'll build a simple Decision Tree, a Random Forest, and an XGBoosted Tree.

Let's start with our Decision Tree.

Decision Tree¶

First, a few implementation details:

SMOTE Oversampling: There is a significant class imbalance in our data (84% Fully Paid, 16% Defaulted). It is important that we create a 50/50 split in our training data because otherwise we risk the model overfitting to the majority class (i.e. always predicting "Fully Paid" would give 84% accuracy). Another way to think about this is that we want to train our model to recognize the characteristics of defaults, rather than assume that most loans will pay in full.

To create this 50/50 split in our training data, we'll oversample the defaulted loans using the SMOTE technique. With SMOTE, we increase the number of loans in the minority class (i.e. Defaulted) through creating synthetic samples in between existing ones that are in close proximity. Synthetic loans continue to be created until the counts of both the majority class (Fully Paid) and the minority class (Defaulted) are equal.

GridSearchCV: To optimize our model's performance, we'll tune our hyperparameters by looking at all combinations with GridSearchCV.

make_pipeline: We'll create a pipeline with make_pipeline to train our model on SMOTE oversampled data and cross-validate our results using GridSearchCV to ensure that we choose the best hyperparameters.

Alright, looks like our Decision Tree model was able to recognize 51% of defaults and 72% of fully paid loans. Another way to look at this is that our model sacrificed 15,702 safe loans to identify 5,337 defaults.

Let's take a look at the optimal hyperparameters chosen by GridSearchCV:

Looks like entropy outperformed gini for measuring impurity. The model also chose our highest max_depth value (4), our lowest min_samples_leaf value (2), and our lowest min_samples_split value (2).

Let's see which features were most important in our model. To do this, we'll use the attribute .featureimportances, which returns the importance of features by computing their (normalized) total reduction in impurity. Here's more detail on how feature importances are measured%20calculates%20each,number%20of%20samples%20it%20splits.).

Interest rate was by far the most important feature for predicting defaults... which makes sense. Term also played a role in prediction, which seems consistent with the stark difference in default rate we observed between 36 and 60 month loans earlier in our analysis.

Let's visualize the decision tree to see exactly how our model is making decisions.

Great! We can see what a significant role interest rate plays in reducing entropy throughout our model. For more on how to read/visualize decision trees, checkout this article.

Finally, let's create a dataframe to track the results of our models to compare their performances later on.

Random Forest¶

Awesome! Now let's train and test our Random Forest model.

A Random Forest classification model consists of many Decision Trees. It performs classification by choosing the most common class predicted by its trees. Understanding Random Forests

RandomizedSearchCV: To optimize the performance of our Random Forest, we'll tune our hyperparameters with RandomizedSearchCV instead of GridSearchCV. With GridSearchCV, every combination of hyperparameters are tried. RandomizedSearchCV, on the other hand, tries only a predefined number (n_iter) of random combinations of hyperparameters. Using RandomizedSearchCV is preferable here because testing every combination of hyperparameters would become too costly / take too much time. RandomizedSearchCV will also be preferable for training our XGBoost model, where we'll want to define certain hyperparamters as continuous distributions. This article gives a good, high-level comparison of GridSearch and RandomizedSearch.

Good, looks like our Random Forest model was able to recognize 61% of defaults and 65% of fully paid loans. Another way to look at this is that our model sacrificed 19,532 safe loans to identify 6,323 defaults.

Let's take a look at the optimal hyperparameters chosen by RandomizedSearchCV:

Looks like bootstrapping performed best, and that the log2 method of selecting features to consider at each split also performed best. The model chose our highest max_depth value (4), our highest min_samples_leaf value (4), and a min_samples_split value of (3). Finally, our Random Forest performed best when including 600 decision trees.

Let's see which features were most important in our model.

Once again, we see that interest rate and loan term are the two most important features, respectively. In our Random Forest, however, term seems to play a larger role than it did in our simple Decision Tree model. We also see a handful of additional features with lower importance (loan_amount, home_ownership, dti etc.)

XGBoost¶

Great! Now we're ready to train and test our XGBoost Tree model.

XGBoosted Trees are similar to Random Forests, in that both models combine the results of a set of Decision Trees. XGBoost differs from Random Forest in the way it builds those Decision Trees. Random Forests build each tree independently. XGBoost builds one tree at a time in a forward stage-wise manner, introducing a weak learner after each tree is built to improve the shortcomings of existing weak learners. More on this.

By carefully tuning parameters, gradient boosting can result in better performance than random forests. However, gradient boosting may not be a good choice if you have a lot of noise, as it can result in overfitting. It also tends to be harder to tune than random forests.

Great, looks like our XGBoost model was able to recognize 57% of defaults and 68% of fully paid loans. Another way to look at this is that our model sacrificed 18,125 safe loans to identify 5,919 defaults.

Let's take a look at the optimal hyperparameters chosen by RandomizedSearchCV, as well as the features that were most important in our model:

Interesting! Term is now at the top of the list, followed by type of homeownership, followed by interest rate. Purpose = debt consolidation also played a significant role.

Neural Network¶

Now let's implement a simple Neural Network to see how it performs against our tree-based models!

To create our Neural Network, let's have one input layer with the same number of nodes as features plus a bias node, a second hidden layer with 38 nodes, and one output node classifying the loan as 0 (Fully Paid) or 1 (Defaulted).

Awesome, looks like our Neural Network was able to recognize 39% of defaults and 80% of fully paid loans. Another way to look at this is that our model sacrificed 11,183 safe loans to identify 4,025 defaults.

Conclusion¶

We're ready to compare the results of our models.

Evaluating Performance: A perfect model would be able to predict all loans correctly, allowing us to avoid all future defaults and only accept loans that will pay in full. While that's a nice thought, none of our models are perfect. Instead, we want to find which of our models does the best job of minimizing total loss. Loss comes from misclassifications. There are two types:

1. Misclassifying loans that will default: When our model approves a loan that will default in the future, we lose the remaining balance plus interest to be paid. These are false_negatives.

2. Misclassifying loans that will pay in full: When our model denies loans that will pay in full, we lose the interest we could have made on that loan. These are false_positives.

We can calculate our total loss as:

Total Loss = (false_negatives * false_negative_avg_cost) + (false_positives * false_positive_avg_cost)

While we don't know the average cost of false negatives or the average cost of false positives, we can look at the ratio between the two to evalute the above equation. Using this ratio allows us to simplify the above equation as follows:

loss_ratio = false_positive_avg_cost / false_negative_avg_cost

=>    Total Loss = (false_negatives * false_negative_avg_cost) +
                   (false_positives * false_negative_avg_cost * loss_ratio)

=>    Total Loss / false_negative_avg_cost = false_negatives + false_positives * loss_ratio

The best model is the one that minimizes false_negatives + false_positives * loss_ratio for a given loss_ratio.

It makes sense that false negatives are more costly than false positives (i.e. we lose more misclassifying defaults than safe loans). Thus, we'll just look at loss_ratios between 0 and 1 to evaluate model performance.

Great! Looking at the minimum values in the graph above, we should choose the following model under each loss_ratio condition.

• loss_ratio < 27% -- Random Forest: Choose the Random model if the expected loss from misclassifying a safe loan is less than 28% the expected loss from misclassifying a default. Note that XGBoost closes the gap on the Random Forest as we approach the 27% loss_ratio, and outperforms it above this level.
• 27% <= loss_ratio < 36% -- Neural Network: Choose the Neural Network model if the expected loss from misclassifying a safe loan is between 27% and 36% the expected loss from misclassifying a default.
• 36% <= loss_ratio -- Baseline: Choose the Baseline model (assume all loans are safe) if the expected loss from misclassifying a safe loan is at least 36% the expected loss from misclassifying a default.

For no loss_ratio was the Decision Tree model the best choice. Why Random Forests Outperform Decision Trees

Final Thoughts¶

So what's a reasonable loss_ratio to assume? Well, that depends on how much we expect to make on a safe loan vs how much we expect to lose on a default. While both of these values depend on a wide range of factors, we can (grossly) oversimplify our estimation as follows

• Assumptions

  • $1000 loan amount
  • 5-yr loan
  • Assume 5% incremental annual return (LendingClub takes a cut, and considering alternative investment strategies like a 5-yr CD)
  • Assume we only make half our money back if a default occurs.

• Amount we expect to make if loan is fully paid (link to calculator) = $132.27

• Amount we expect to lose on default = $566.14

loss_ratio = 132.27 / 566.14 = 23.4%

In this case, our best model would be the Random Forest.

Thank you!¶

Thanks for going on this journey with me! If you have any questions, suggestions for improving upon my approach, or just like my work, please don't hesitate to reach out at keilordykengilbert@gmail.com. Having a conversation is the best way for everyone involved to learn & improve 🌲😊🌲