This article is part of a series where we explore, preprocess, and run several machine learning methods on the Taiwan Credit Default dataset. The dataset can be found here: dataset .

In a previous article, we used XGBoost as a way to identify the most important features in the dataset, in terms of

the KFold algorithm to help discover the relative merits of several machine learning algorithms for a binary classification task. Specifically, the task is to predict whether a given credit customer will default on their payments or not. KFold allowed us to quickly run several bare-bones models on the data and plot the results to help narrow down one or two promising candidates for further development.

First, let’s import some of the packages we will need.

Next, we will import the dataset we prepared in the last article. It has one-hot encoding applied to the categorical features and standardized scaling applied to the numeric features. The code below will read the CSV file into a Pandas dataframe. We set the index_col parameter to 'ID' to retain the index from the original file.

The dataframe must be separated so that the features are together and the labels, or targets are alone. Below, we use Pandas to achieve this.

Now, we will split our data into training and test datasets. Each pair has the X, for the features, and y for the labels. The random_state is set to ensure reproducibility of our results if others want to simulate our steps.

The next step we are going to take is to produce a baseline model for this binary classification task. To do this, we choose the Logistic Regression algorithm, which performed near the top of our results last time. The parameters are what we saw as most successful in the last article.

We look at the confusion matrix for the Logistic Regression classifier and see that the majority class, 0, has the most accurate predictions by far. As it turns out, there are many more 0’s in the dataset than 1’s. This kind of class imbalance can lead to problems with modeling. In a future article, we will look at some potential solutions for class imbalance.

The classification_report gives details about the precision, recall, and f1-score for the model. We can see the accuracy is about 82%.

Hyperparameters can be notoriously difficult, and time-consuming, to tune. With GridSearchCV we can compare the performance of several versions of the same base classifier on the same task. We instantiate a classifier object, then hand the parameter grid to it.

Each line of the param_grid has a set of hyperparameter settings that defines the base model further. GridSearchCV runs each model on the dataset, and produces a set of attributes. We can access the attributes, like best_params, to show which configuration works best.

Below, we experiment with different regularization settings and tolerances on the Logistic Regression model. Also, the upcoming XGBoost Classifier parameter grid is defined.

Here, we define the GridsearchCV instance for the Logistic Regression classifier and run the fit method with the training data.

Instead of evaluating the models on the basis of simple accuracy, we have decided to use the roc_auc score. Looks like the best configuration for Logistic Regression is the following:

Now, we discover the best configuration for XGBoost below:

We know better now the best version of each model for this dataset, given the parameters we decided to evaluate. In order to evaluate a classifier’s performance in this case, we could settle for the simple accuracy score. However, we have already observed a significant imbalance between the two classes in the dataset. In cases like this, roc_auc will allow us to assess performance in a more robust way.

The ROC-AUC Curve plot below compares the Logistic Regression classifier to the XGBoost Classifier in terms of the the area under the curve of the ROC, or Receiver Operator Characteristic. For imbalanced datasets in binary classification tasks, this is generally a better measure of a model’s performance than standard accuracy.

The red dashed line represents the XGBoost classifier and the orange line shows the Logistic Regression classifier. The diagonal blue line is the 50% mark, below which the model performs worse than a random guess.

We can see that the XGBoost classifier has the edge, as it covers a bit more area of the whole. To see a detailed, numeric representation, we can generate the actual label predictions for each classifier with their respective predict methods. Scikit-learn has the classification_report which displays a breakdown of of precision, recall, f1-score, and accuracy.

precision    recall  f1-score   support

           0       0.84      0.95      0.89      4687
           1       0.66      0.35      0.46      1313

    accuracy                           0.82      6000
   macro avg       0.75      0.65      0.67      6000
weighted avg       0.80      0.82      0.80      6000

For a visually appealing display of the ability of the classifier to predict each class, the confusion_matrix is useful. We make the graph from the data provided by the Scikit-learn confusion_matrix module. Then, we use seaborn and its heatmap graph to make the point visually, with the actual numbers for each class presented. This makes it easy to see the general point and also be able to analyze more in terms of specific data.

precision    recall  f1-score   support

           0       0.84      0.95      0.89      4687
           1       0.68      0.35      0.47      1313

    accuracy                           0.82      6000
   macro avg       0.76      0.65      0.68      6000
weighted avg       0.81      0.82      0.80      6000

Evaluated from these standpoints, the two classifiers are almost identical. Both have similar scores for each class and accuracy is identical. However, the XGBoost classifier shows slightly more promise. First, it has more correct predictions of both the 0 and 1 classes. Also, the false positive and false negative rates are lower. In the case of assessing default risk, it is better to keep the false positive rate as low as possible.

These gains in accuracy and performance are admittedly modest. Earlier, we noted the sizable imbalance in the classes. There are several strategies for dealing with class imbalance and it seems worthwhile to explore some options to improve our predictive power.

In the next article, we will look into using two techniques called SMOTE and Neighborhood Cleaning Rule to apply some sampling methods to address the imbalances in the dataset and hopefully improve performance.

This article is part of a series where we explore, preprocess, and run several machine learning methods on the Taiwan Credit Default dataset. The dataset can be found here: dataset .

Two important questions usually come up when we are trying to figure out how best to model a dataset, and hopefully make predictions based on it:

· Are all the given features relevant, and if not, which subset will be best?

· Given the best subset of features, which machine learning model will yield the best results?

Because of the limitations of computing power, even now in 2022, it is important to reduce the number of dimensions to as few as possible, without sacrificing significant amounts of information. In this article we will look at one strategy for selecting the best subset of features, using the XGBoost algorithm. Then, we will use the resulting subset of data to evaluate several basic classifiers to decide which deserves more time tuning for optimal performance. Let’s get started!

The first step is to import some initial packages and import the data from Excel. Below, we have imported the data into Pandas with a helper application, xlrd. This will allow us to import .xls files, and even select the specific sheet in the workbook. This produces a dataframe with 24 columns with 30,000 instances. For a more detailed look at this dataset, see this article .

Now, we will import most of the packages, most from Scikit Learn, that we will use going forward.

This short piece of code will separate the features from the target variables.

Our next step is to use train-test-split to give us subsets of the data to train and test our models. We will use an 80/20 split (20% is testing data) and set a random_state of 42, which seems to be somewhat of a tradition.

There are several ways to identify the best subset of features to select for training a model. The basic concept is that we want to retain those features with the most importance for predicting the target variable and leave off the rest of the features that do not benefit the prediction task enough to warrant the added complexity. Complexity can entail overfitting, difficulty of interpretation, and excessive resource needs. Therefore, we will want to minimize it.

For this example, we will use the popular XGBoost algorithm in classifier mode. Here, we fit the model to the training data and then use the feature_importances_ attribute to discover the features that retain the most explanatory variance, while minimizing the number of features. At a certain point, some features yield less explanatory power than they are worth in terms of complexity.

The plot below shows the index numbers of each feature in the dataframe and what relative importance each has in predicting the target variable.

According to this analysis, two features appear to hold a larger share of the predictive weight. Now, it should be noted that by adjusting the n_estimators, learning_rate and/or the max_depth in the XGBClassifier, we may see some features become slightly more prominent. Altogether, for this example, this configuration worked well.

Instead of just picking the top features by sight, we can use a formula of some kind to pick them for us. In this example, we chose to select all features that were greater than the median of all feature importances.

Now, we will fetch the feature names, to make it easier to refer to them in the next steps.

Of those features we decided to retain, some are categorical while others are continuous numeric features. Therefore, we will want to apply a different type of transformation to each so that the models can use the data well. To do this, we will use the Scikit Learn ColumnTransformer to apply the transformations appropriately.

There are several ways to prepare categorical data for modeling. In this case, we will apply the OneHotEncoder to all categorical features. Since we decided to be selective about the features we retained, the complexity burden will thereby be lessened. For each categorical feature, the OneHotEncoder will create a new feature for every binary outcome within the original feature. For instance, if a column has 4 possible categories, the OHE creates 4 columns and then puts either 0 or 1 in each row to declare whether that category is absent or present in that row. We end up with a sparse data representation that accounts for the categories but is more easily used by models.

For the numeric features, we will keep it simple and apply the StandardScaler to them. This standardizes each feature to remove the mean and scale to unit variance. With this transformation, the data will not be as susceptible to outliers. There are other types of scaling, such as the MinMaxScaler, but we will use the StandardScaler.

We apply the column transformations to both the training and test data for X with the fit method.

We can see the new feature names of the transformed dataset below:

With the data pared down to more essential features and transformed for use in machine learning models, we move on to create the apparatus to test a series of classifiers side-by-side, to get a better idea as to which one(s) will be worth investigating further. This step is often called model selection. The models, for the most part, do not have any hyperparameter changes. We run a vanilla version of each to attempt to get a ballpark estimate of the relative merits of each.

The steps below were borrowed from, and inspired by, the book, Machine Learning and Data Science Blueprints for Finance by Hariom Tatsat, Sahil Puri, and Brad Lookabaugh (O’Reilly, 2021), 978–1–492–07305–5.

I recommend reading it. It has succinct and useful descriptions of common techniques in ML, especially for the financial domain.

The first step is to create a Python list, and then append each of our candidate classifiers to it.

Next, we set parameter values, like n_folds and scoring for the KFold cross-validator. Here, the choice of roc_auc as a scoring metric is important. Our task is supervised binary classification, which means we have access to the target variable, and it is a binary choice, 0 or 1.

The Area Under the Curve of the Receiver Operating Characteristic graph is well-suited to this classification task. It plots the true positive rate against the false positive rate while varying a discrimination threshold. The score is the area of the graph that is under the line produced. The perfect score is 1.0 and a not-so-good score is anywhere below 0.5. The lower the score, the worse the classifier performs. Also, a score below 0.5 indicates that the classifier shows no real ability to predict the target.

Finally, we will import the Python time module to be able to compute the seconds that elapse for running each classifier. We print the results to the console.

Finally, we will make a series of boxplots to help us compare the different classifiers.

Compared side-by-side, we can see that the XGBoost classifier performed better than the others. Sharing second place, Logistic Regression and Linear Discriminant Analysis show promise. With this information, we can go on to focus on one or two of these classifiers and tune their respective hyperparameters. By using this method upfront, we save time and energy by focusing on developing models that show genuine potential for performing the task at hand.

In another article, we will do just that and see how well one of these non-neural network approaches can classify credit default instances.

This article is part of a series where we explore, preprocess, and run several machine learning methods on the Taiwan Credit Default dataset. The dataset can be found here: dataset .

For any dataset we work with, it must be cleaned up and transformed so we can use it for building models and/or deriving insights from visualizations. This step, often called “data preprocessing”, or “data wrangling”, goes together with the overall process of EDA, or exploratory data analysis. Ultimately, to gain optimal value from our data, we need to know what is in the dataset: details about the features, such as data type, and encoding; and as much as possible about the potential of the data to teach us.

The example we will use in this article is a dataset from 2005 (link is above) which shows details on credit card default in Taiwan over the span of a few months. Twenty-three features were collected, with a mixture of categorical and numeric variables. A single binary column gives the target variable, for which 0 stands for ‘no default’ and 1 for ‘default. I suggest reading the brief on the dataset at the link above to gain more knowledge of the dataset, which is from the well-known UCI Machine Learning data repository.

Let’s get started by importing the dataset into Pandas:

Notice here we have imported the xlrd package. This package helps importing an older Microsoft Excel file type, the xls file. It may have to be installed in your Python environment; in which case we should be able to install it with pip.

The dataframe column names are below:

We know from the dataset description that the last column, ‘default payment next month’, is our target variable. Before we transform the data, it may be simpler to separate the ‘X’ data, that is the features, from the ‘Y’, target data. Below, we use the Pandas indexer, iloc, to make two dataframes, ‘X’ and ‘Y’:

We will operate on ‘X’ and leave ‘Y’ aside for now.

Next, we will prepare the data for general machine learning and deep learning model building. When we read the dataset description, we gathered which features are categorical, and which are numeric.

The categorical features in this dataset use integers to encode the categories. Even though these are technically numbers, if we want the categorical nature of each to be represented, it will help to encode them. There are several different methods for encoding categorical features. In this instance, I have chosen one-hot encoding. This method will take a single categorical feature and create a new set of features where each represents the presence or absence of a given category.

For example, the ‘MARRIAGE’ feature in this set has three possible values: (1, 2, 3). The OneHotEncoder from SciKit-Learn will create three new features that can only have two values, 0 or 1. These indicate absence or presence of that specific category within the instance, or row, of the dataset. The overall result is a sparser representation of the same information. Later, this will likely help any machine learning models or neural networks we use to understand the data better.

To handle the numeric features, we will apply the StandardScaler to them. This is designed to standardize the features so that the overall dataset has a relatively similar range. It will find the mean and standard deviation of each feature, subtract the mean, and then divide by the standard deviation. This operation is simple and common when preparing data; it retains the variance of the features but places it within a common scale.

In the code below, we have created two lists, one for the categorical features and one for the numeric (continuous) feature names. Doing this beforehand will make running the transformations easier. Now, we can use the SciKit-Learn ColumnTransformer to construct a dual-pronged transformer to apply the chosen transformations to each subset of columns. The use of the “remainder=’drop’” parameter within the ColumnTransformer will exclude the original features from the resulting dataframe. It should be noted as well that the string placed in the first position of each transformer definition will be the prefix for each of the new features created.

After defining the ColumnTransformer, we run the fit method on the ‘X’ dataframe:

Below, the configuration of the ColumnTransformer object is shown:

A call to the get_feature_names_out method will display all the new feature names. A truncated list is shown below for brevity’s sake.

Now, we need to create a new transformed dataframe. We use Pandas to create a new ‘X_prep’ dataframe. Notice we can simply call the transform(X).toarray() method to give the new dataset and the .get_feature_names_out() method to feed in the new column names. The prior index is retained by specifying the index=X.index . Calling info() on the new dataframe reveals all of the columns and their data types. When we do this, we will see that every column is a floating-point number.

The next to last step is to concatenate the new X_prep features to the target column, Y. The code below will achieve this, and we will have a dataframe of features combined with the target variable.

For easy use with most of the upcoming work at creating machine learning models and so forth, it makes sense to write the new standardized and one-hot encoded dataset into a CSV file. Then, we will have the same dataset to start with for each of our models with no need to repeat the initial processing.

In the next episode, we will load this dataset from the file and continue to get acquainted with the data and see how some basic machine learning models perform on it.