Forestnet: A Learning Architecture Combining Deep Networks And Decision Forest

Recently, deep learning methods have drawn a great attention in academic world, and have been widely used in industry. Deep learning models, especially convolutional neural network, has achieved state-of-the-art performance on a wide variety of tasks whose data is image or some other data with a similar feature of image, such as speech and natural language. The success of convolutional neural network is closely related to the great ability of convolutional layer in feature learning. However, when it comes to the tasks whose data has a very different feature compared to image, deep learning models fail to outperform traditional models remarkably. Decision tree ensemble models are still the most widely used models and achieve the state-of-the-art performance in these tasks. It probably indicates that decision tree is more suitable for the ?non-image-like? data than deep learning models.We propose a novel model called Forestnet, which combines deep learning and decision tree ensemble. Forestnet embeds regression tree to extract feature from the input data, which is similar to the usage of convolutional layers in convolutional neural network. Regression tree can learn good features from the non-image-like data, thus improves the performance of Forestnet on these tasks. We design a algorithm based on backpropagation algorithm to learn Forestnet model, which updates the regression trees in back propagate process by gradient boost method.We evaluate our model on classification, regression and ranking tasks. The experiments on several datasets show that Forestnet can achieve better performance than both multilayer perception and gradient boost decision tree. We also study the relationship between the structure of Forestnet and the performance and find that we can improve the performance of Forestnet by increase the depth(the number of fully connect layer), or the width(the number of nodes of forest layer).