Research Of Deep Ensemble Method Based On Stacking Structure

How to mine potential key information from data and utilize the learned knowledge reasonably to realize efficient information processing is one of the hot topics in machine learning.Due to the restriction of shallow learning,shallow neural network can hardly discover the value hidden in data to the maximum extent.Well-known deep learning algorithms often use plenty of hidden layers to construct deep neural networks and perform training process in a manner of iterative parameter updating.However,a single large deep network often suffers from the time-consuming iterative tuning since the complicated structure and many parameters to be adjusted are involved.These shortcomings result in the limited further development of the traditional deep learning.In order to tackle the problem that shallow ELM cannot fully mine the implied knowledge of samples,this paper proposes two fast training deep networks based on stacking structure,including DELM and DS-A-ELM.Deep extreme learning machine(DELM)regards ELM as the basic learning unit to realize hierarchical feature representation learning.With knowledge augmentation strategy,DELM cascades multiple ELM sub-models and expands data space layer by layer in a feedforward way to improve the overall classification performance.In each sub-model,input weights and hidden biases are randomly assigned while the activation function of hidden unit transforms the input instances into the hidden layer space.Then,the output weight is analytically calculated.Several modules can be introduced into stacking structure to integrate the knowledge of multiple modules.Finally,it ceases the subsequent stacking immediately when the prescribed depth requirements is met or the performance threshold is reached.In this paper,DELM algorithm is introduced in detail and its effectiveness is verified in EEG recognition of epilepsy.Based on DELM,this paper further proposes a deep stacking extreme learning machine based on adversarial sample learning(DS-A-ELM)according to another idea of knowledge augmentation.Before each module starts training,it needs to first implement intentionally adversarial attack on training samples to generate adversarial samples.Then,the input data space in current layer is updated to capture better augmented knowledge progressively.So DS-A-ELM can still well preserve the enhanced classification performance,excellent time efficiency and favorable generalization performance.In this paper,DS-A-ELM is also described detailedly,and the performance of the deep stacking structure model is validated on a large number of UCI datasets and motor imagination classification datasets.In response to difficulties in parallelization of a single large-scale deep neural network and the limited capability of feature extraction of modules in the above work,this paper further proposes a deep ensemble learning model(PH-E-DNN)combined with FCM clustering method.Before each hierarchy,FCM partition is firstly used to separate all samples into several local classification models,and classification knowledge on regional subsets can be learnt by parallel multiple DNN modules.Local knowledge is integrated and expanded to the current regional subset.The knowledge of multiple modules in triple hierarchies is deeply fused so that the augmented data could be consequently generated.The final classification is carried out on the global augmented samples,thus,the comprehensive output is obtained.Experimental results on a large number of UCI benchmark datasets show that PH-E-DNN has excellent model performance in classification tasks.