The Study And Application Of Ensemble Of Trees Based On Boosting

The ensemble of trees algorithm is a popular algorithm in the field of machine learning in recent years,especially the XGBoost algorithm.In 2014,Chen Tianqi released the first public version of XGBoost.And then,among the 29 challenge winning solutions published at kaggle's blog during 2015,17 solutions used XGBoost.Among these solutions,eight solely used XGBoost train the model.Compared with decision tree algorithm,the ensemble of trees algorithm are a kind of state-of-the art learning approach,and can overcome some problems that are difficult to solve for decision tree.Ensemble of trees algorithm train a set of decision trees and then combine them for use.With a decision tree as a base learner,different ensemble methods can be used to build different ensemble of tree algorithms.The ensemble algorithm is mainly divided into two major family algorithms,namely Boosting and Bagging.This article mainly discusses Boosting-based ensemble algorithm.First,the artical introduced some basic theory of the ensemble of trees algorithm,including the basic concept of the decision tree and ensemble algorithm.Second,three ensemble algorithms are introduced,including AdaBoost,GBDT and XGBoost.The AdaBoost algorithm is based on the reconstruction of the sample weights,while the GBDT and XGBoost algorithms are both based on learning to the gradient of loss function.Andthe XGBoost is an improvement of GBDT.Then the regularization methodsthat are used in Boosting algorithm are explained and tested.Finally,the algorithmsare applied to the lithology identification dataset.The accuracy,computational speed and anti-noise performance of the algorithm are compared.The XGBoost method is leading in many aspects.Then the XGBoost algorithm is applied to image classification,and a CNN-XGBoost algorithm model is proposed,which can effectively improve the accuracy of image classification.