Research On Algorithms For Ensemble Multi-label Learning

With the trend in social demands,more and more AI applications involve multi-label learning tasks,such as text classification,semantic annotation,social networks,gene prediction,drug diagnosis and treatment,etc.,which are different from traditional single-label learning tasks(binary classification or multi-class classification).Multi-label learning allows one instance to be associated with multiple labels,and multiple label information needs to be output.Due to the complex dependencies between labels,the traditional single-label learning methods cannot be applied well.Therefore,multi-label learning has become one of the current research hotspots in the field of artificial intelligence.This paper mainly focuses on multi-label classification tasks.Currently,multi-label learning algorithms are mainly based on the idea of ensemble learning,aiming to improve or extend the single-label learning algorithm to adapt to multi-label learning scenarios,including traditional ensemble multi-label learning and deeply ensemble multi-label learning.Although the research of multi-label learning has achieved great success,it still suffers from some key problems and challenges as follows.Firstly,traditional ensemble multi-label learning fails to deal well with paired local dependencies between labels.How to improve the performance of traditional ensemble multi-label learning methods by using local label dependencies is a key problem worth solving.Secondly,in multi-label learning,missing and incomplete labels are a common phenomenon.How to complete labels based on existing theories and carry out incomplete multi-label learning is a key problem worth solving.Thirdly,traditional multi-label learning only focuses on a relatively small number of labels(such as 1000 or fewer labels),but with the increase of Internet data,the number of labels has broken through to tens of thousands or millions.How to propose a solution to the extreme multi-label problems is a key problem to be solved urgently.Fourthly,as the number of labels continues to expand,labels show a trend of long-tail distribution,that is,the head classes occupy most of the data samples(many-shot Learning),while the Few-shot learning occupies most of the tail classes.How to propose multi-label learning suitable for long-tail distribution is a key problem to be solved urgently.Therefore,this paper mainly focuses on the above four problems to carry out research on multi-label learning.At the bottom,ensemble learning is adopted as the theory to study multi-label learning based on ensemble learning.The main work includes the following:1.For traditional multi-label ensemble learning scenarios,most of the ensemble modes are based on Bagging and Boosting,while Stacking is rarely considered.However,most of the existing ensemble methods have the following two limitations:First,most of the existing integration strategies adopt voting or weighted voting,and seldom consider the selective ensemble of classifiers;Secondly,the pairs of local dependencies between labels are not considered.Therefore,a weighted stacking selection ensemble algorithm is proposed in this paper,called MLWSE,which not only extends multi-label stacking ensemble mode,but also has achieved good performance in robustness,parameter sensitivity,and convergence on three different datasets,including2-D simulation datasets,13 Benchmark datasets,and real cardiovascular and cerebrovascular disease datasets.2.For incomplete multi-label learning scenarios,most of the existing multi-label learning methods based on matrix completion are currently used to complete missing data.However,most of the existing multi-label learning methods based on matrix completion have the following two limitations: First,they do not make good use of feature auxiliary information,that is,different feature structures of manifold subspaces;Secondly,most matrix completion methods only consider the missing of label set,not the missing of feature,which makes the application of the method limited.Therefore,this paper proposes incomplete multi-label learning based on a manifold subspace ensemble,called BDMC-EMR,which not only makes good use of feature auxiliary information but also achieves good performance in transductive multi-label learning and inductive incomplete multi-label learning.3.For extreme multi-label text classification scenarios,the current deep network structure is mostly used for extreme multi-label learning.However,the existing deep network structure for extreme multi-label text classification mainly has the following two limitations: first,the dependence among words,phrases,and labels is not taken into account simultaneity;Secondly,the huge label space brings data sparsity and scalability problems.Therefore,extreme multi-label learning is divided into intermediate quantity level(100-30000)and extreme quantity level(millions).For intermediate quantity level,hybrid CNN structure of adaptive space-time representation ensemble network based on CNN and RNN is proposed in this paper.This method integrates the interactive attention of word,phrase,and label,and effectively improves the discriminant ability of the classifier to extreme multi-label.However,this method can only adapt to the intermediate level.For extremely quantity level,this paper proposes the integrated Transformer multi-view representation framework multi-V-Transformer based on different Transformer characterization capabilities.This method alleviates the scalability problems caused by a large number of labels by clustering them.Finally,the generalization performance of the model is improved by multi-view attention representation,extreme multi-label clustering learning,and reduced label embedding learning.4.For long-tail multi-label learning,traditional methods are mainly based on resampling strategy and re-weighting strategy.Recently,the best methods to deal with long-tail distribution are as follows: one is based on a two-stage decoupling training method,namely representation learning stage and classifier training stage;The second is based on multi-stage knowledge transfer training.Although these methods have achieved good success,they have the following two limitations: First,these methods need multi-stage training,and when dealing with large multi-label data sets,the model will bring huge training cost and time overhead;Second,due to the co-occurrence of labels in multi-label scenes and the dominance of a large number of negative tags,the multi-label long-tail distribution is more frequent and tedious,and there are fewer methods to consider the knowledge interaction between the head class and tail class,making the problem of multi-tag long-tail distribution difficult.Therefore,this paper proposes OLSD,a long-tail learning framework based on a self-distillation network structure.This method only requires one stage of training and can achieve training accuracy compared with the multi-stage model,but this method is limited to supervised learning tasks.In order to further consider self-supervised learning tasks,a self-supervised representational distillation framework DS-SED based on comparative learning was proposed in this paper.This method not only made up for the application limitations of OLSD but The effectiveness of our method is also verified in downstream many-shot and few-shot tasks,long-tail visual recognition tasks,object detection,and semantic segmentation tasks.