Research and application of XGBoost in imbalanced data

As a classic over-sampling algorithm with universal applicability, SMOTE²⁸ realizes data synthesis based on random interpolation by selecting the nearest neighbor distance between samples, expands the feature decision area of minority samples, and can effectively balance data. However, it also has problems such as low quality of synthetic samples, fuzzy class boundary, and uneven distribution of a few samples.²⁹

The subsequent improved algorithms mainly include Borderline-SMOTE, ADASYN, and SVM–SMOTE. Among them, Borderline-SMOTE and ADASYN mainly solve the quality problem of the generated samples, while SVM–SMOTE divides the minority samples into safe (more than half of the nearest neighbor samples belong to the minority class), dangerous (more than half of the nearest neighbor samples belong to the majority class), and noise (all the nearest neighbor samples belong to the majority class), and then training SVMs based on dangerous samples, using the support vectors found by SVM to generate new samples close to the boundary of most classes and a few classes, giving full play to the advantages of SVM algorithm in boundary decision-making, and can solve the problems of generated sample quality and class boundary ambiguity at the same time. It is an over-sampling method with good effect.

Easyensemble³⁰ is a hybrid ensemble under-sampling algorithm. It uses Bagging to fuse random down sampling and AdaBoost algorithm, which makes up for the defect that the general under-sampling algorithm may lose important classification information. The AdaBoost algorithm selected by the base classifier also improves the classification accuracy and generalization ability. Different from the supervised combination of another representative algorithm, BalanceCascade, EasyEnsemble is based on unsupervised under-sampling. It has the advantages of low-time complexity and high utilization of data, which can greatly avoid the waste of limited data resources. It is an effective and more expansive method. The principle steps of EasyEnsemble are shown in Table 1.

For a given training set with $n$ examples and $m$ features, $D=\{(x_i,y_i){\}}_{i=1}^n$$(|D|=n,x_i\in {\mathbb{R}}^m,y_i\in \mathbb{R})$. XGBoost can be regarded as an additive model ${\hat{y}}_i=\sum _{k=1}^K{f}_k(x_i),f_k\in F$ which composed of $K$ CART trees. Among them, $f_k(x_i)$ represents the predicted value obtained after inputting the ith sample $x_i$ into the kth tree, ${\hat{y}}_i$ represents the final prediction result of $x_i$, and $F$ is the set space of all regression trees. The objective function of XGBoost can be defined as $\mathrm{Obj}=\sum _{i=1}^{n}l(y_i,{\hat{y}}_i)+\sum _{k=1}^K\mathrm{\Omega }(f_k)$. Among them, $y_i$ is the real result and $\sum _{i=1}^{n}l(y_i,{\hat{y}}_i)$ is the loss function, which can measure the prediction ability of the model; $\sum _{k=1}^K\mathrm{\Omega }(f_k)$ is the regularization term of the model, used to control the complexity and avoid overfitting.

The modeling process of XGBoost is to leave the original model unchanged, and set the input of the next tree as the residual of ${\hat{y}}_i$ and $y_i$. The general steps are as follows:

In the above formula, ${\hat{y}}_i^{\left(t\right)}$ and ${\hat{y}}_i^{(t-1)}$, respectively, represent the predicted value of the model during the tth and previous $t-1$ iterations of $x_i$, and $f_t(x_i)$ is the newly added prediction function in each round, so the objective function of the tth iteration is

Since ${\hat{y}}_i^{(t-1)}$ and $l(y_i,{\hat{y}}_i^{(t-1)})$ are both constants, the second-order Taylor expansion can be used for the loss function to approximate the objective function. Let $g_i={\partial }_{{\hat{y}}_i^{(t-1)}}l(y_i,{\hat{y}}_i^{(t-1)})$ and $h_i={\partial }_{{\hat{y}}_i^{(t-1)}}^{2}l(y_i,{\hat{y}}_i^{(t-1)})$, omitting the constant term, we have

In XGBoost, the number of leaf nodes $T$ and the weight $w$ of the tree are considered to define the complexity of the tree, that is, $\mathrm{\Omega }(f_t)=\gamma T+\frac{1}{2}\lambda \sum _{j=1}^T{w}_j^2$. Among them, $\gamma$ and $\lambda$ are the parameters controlling complexity. The larger the value, the more complex the structure of the tree.

In general, formula (1) can be rewritten as the following format

where $I_j=\{i|q(x_i)=j\}$ is the sample on the jth leaf node, and $w_j$ is the weight of the jth node. Let $G_j=\sum _{i\in I_j}{g}_i$ and $H_j=\sum _{i\in I_j}{h}_i$, then there is

Assuming that the tree structure has been determined, the optimal solution obtained by directly deriving $w_j$ is $w_j^{*}=-(G_j/H_j+\lambda )$. Substituting this solution into equation (2), the optimal value of the objective function is

XGBoost calculates the gain value through the formula

and selects the feature with the largest corresponding gain value and the split point under this feature for splitting.

Among them, $(G_L^2/H_L+\lambda )$ and $(G_R^2/H_R+\lambda )$, respectively, represent the structure score of the left and right subtrees, and $((G_L+G_R{)}^2/H_L+H_R+\lambda )$ is the structure score when the current node is not split. The rest of the detailed algorithm flow can be found in the literature.⁸

Because the $L_1$ regularization term has stronger anti-noise ability and robustness, but there may be multiple optimal solutions, and the weight coefficients will be sparse, while the $L_2$ regularization term has lower computational complexity and faster speed. Therefore, this article refers to the idea of ElasticNet,³¹ combines $L_1$ and $L_2$ regularization terms, and redefines the complexity of the tree model, which is

then formula (2) becomes

The optimal solution of the derivative solution is $w{\prime }_j=-(G_j-\lambda /H_j+\lambda )$, then the optimal value of the objective function is

and the gain calculation formula becomes

It can be seen from the above formula that the new definition makes $|w{\prime }_j|<|w_j^{*}|$, that is, compared with the original regularization term, the parameters become smaller after the change, so overfitting can be avoided and the model variance can be reduced.

Since mixed sampling can take into account both under-sampling and over-sampling, it can often produce better results. Therefore, this article considers the integration of mixed sampling method and ensemble learning method, and designs an XGBoost imbalanced data classification algorithm which combines the SVM–SMOTE algorithm and the EasyEnsemble algorithm. Its strategy is as follows:

The flow framework of the algorithm is shown in Figure 1.

The experimental platform environment of this article is Window 10×64 operating system, 8 GB memory, Intel(R)Core(TM)i7-7500 [email protected] GHz; the experimental tools are mainly Python3.9, including xgboost 1.3.3, jupyter1.0.0, seaborn0.11.1, sklearn0.0, pandas 1.2.2, numpy1.20.1, matplotlib3.3.4, imblearn0.0, and other packages.

In order to prove the classification performance of the proposed algorithm in imbalanced data sets, two public imbalanced data sets are used for experiments. The first data set is the credit card data set in Taiwan publicly provided on UCI website. The data set records the history of bank customers’ arrears, credit data, statistical characteristics, billing statements, and other information from April to September 2005. It has 24 characteristic attributes and one category identification, which can predict the customer’s default situation. The specific meaning of features is shown in Table 2. The second data set is the credit fraud data set provided by ULB machine learning laboratories. The data set contains the credit card transactions of a bank in Europe in September 2013, including 492 frauds in 284,807 transactions, and the data categories are extremely imbalanced. For security reasons, the original data set has been desensitized and principal component analysis (PCA) dimensionality reduction, with a total of 29 feature attributes and 1 category identification. Among them, the features “V1”–“V28” are the principal components obtained by PCA, and the features not processed by PCA are “Time” and “Amount.”“Time” represents the time interval between all transactions and the first transaction, and “Amount” represents the transaction amount. For the class label “Class,” 1 represents the fraud and 0 represents the normal.

The first data set is mainly shown and analyzed below. The data in the second data set do not need cleaning, and there is no significant correlation between characteristic variables. The processing flow is similar to that in the first data set.

First, by loading and viewing the data set, and performing missing value testing and descriptive statistical analysis, it is found that the data have a total of 30,000 rows and 25 columns, all features have no missing values, and all the data are integers. Due to the large differences between the values, standardization is required. Part of the data and test results are shown in Figures 2 and 3.

Second, by looking at the sample distribution of each feature, the outliers were tested, and it was found that the features EDUCATION and MARRIAGE were abnormal. The value of DUCATION is 0, 5, and 6 more than in the introduction, and the number of outliers is 345, which is relatively small compared to the total sample size, so it is classified as one with 4. The value of MARRIAGE is 0 more than in the introduction, and the number of outliers is 54. It is classified as one with 3 for correction and filling.

At the same time, analysis shows that there will be 6636 customers defaulting next month, which is much lower than the number of non-defaulting customers (23,364), so the data are obviously imbalanced. The information of the two experimental data sets is summarized in Table 3.

Moreover, by analyzing the variables in the credit card data set, it can be seen that among all customers, the default ratio of males is 24.2%, and that of females is 20.8%. The number of customers who repay on time between 30 and 40 is the largest, and the older the age, the higher the default rate. The unmarried customer group has the largest number of repayments on time. The number of defaults is similar to that of married groups. High school customers have the highest default rate, and the higher the education level, the lower the default rate.

Finally, through comprehensive feature analysis, it is found that there is only PAY_0 has the most complete information in the repayment status, and the two types of features BILL_AMT and PAY_AMT are highly correlated, so features with large correlation coefficient with default.payment.next.month can be selected for modeling, respectively. Therefore, delete ID, PAY_2–PAY_6, BILL_AMT2, BILL_AMT4–BILL_AMT6, PAY_AMT3–PAY_AMT 6, and default.payment. next.month. The specific feature correlation heatmap is shown in Figure 4.

In traditional classification problems, single evaluation indexes such as recall rate and accuracy rate are often used to better reflect the performance of the algorithm.

However, in the face of the imbalanced classification problem of data tilt, only using single index no longer has good reference value. Therefore, this article selects the typical comprehensive evaluation index G-mean and AUC value to compare and analyze the prediction effect of the experiment.

In the confusion matrix, $\mathrm{TP}$, $\mathrm{FP}$, $\mathrm{FN}$, and $\mathrm{TN}$, respectively, represent the sample sizes of true positive, false positive, false negative, and true negative cases. From this, the definitions of the above-mentioned types of evaluation indexes can be obtained

Therefore, G-mean contains two single evaluation indexes $\mathrm{Sensitivity}$ and $\mathrm{Specificity}$, which improve and perfect the overall accuracy and can better measure the classification effect. In addition, the receiver operating characteristic (ROC) curve is drawn with $\mathrm{FPR}$ and $\mathrm{TPR}$ as the horizontal and vertical axes, respectively. AUC is the coverage area below the ROC curve. The larger the value, the closer the ROC curve will be to the upper left corner, representing the more accurate the classification effect of the model.