Research And Application On Broad Learning System Under Different Modal Problems

Thanks to the development of sensor technology,the form of data storage mostly exists in various modalities such as image,text,sound and texture.Traditional robotic perception learning only includes tasks such as image classification,pixel segmentation,and object detection of visual modality.With the increasing demands for robot flexibility,multi-modal machine learning methods have become an effective tool for analyzing,mining and applying massive multi-modal data.Classic machine learning methods generally introduce deep structures which are time consuming,difficult to converge,and costly.Recently,C.L.Philip Chen,et al.proposed a novel alternative neural network: broad learning system(BLS),which possesses the characteristics of powerful mathematical theory,simple flat-layer network structure,and rapid incremental modeling process,etc.,have been successfully applied to various classification and regression tasks.Experimental validations show that BLS can learn incrementally and quickly while maintaining high classification accuracy and stability,which greatly saves training time.However,BLS is a classifier mainly for learning single modality without the ability of feature fusion.As a traditional fusion framework,Canonical Correlation Analysis(CCA)can use covariance for feature mapping and joint dimensionality reduction.Moreover,for the further cross-modal fusion,it is difficult for CCA to solve the circumstances of data missing and lack of one-to-one correspondence.Therefore,it is necessary to consider learning weak correlations for pairing and fusion.This paper studies the problems above based on BLS,aiming at finding a simple and efficient fusion technology,to refine the underlying input data,learn high-dimensional abstract representations,fuse different modalities information,and improve recognition accuracy.The related algorithms can greatly shorten the calculation time while greatly ensuring the efficiency of classification and recognition,also applied to multiple fields such as multi-modal image classification and cross-modal identity authentication.The contributions of this work are summarized as follows:(1).A double broad learning structure(DBL)is proposed to solve the problem that BLS cannot handle the multi-modal information.DBL is an extended BLS structure comes from two multi-modal inputs.The subsequent research in this paper is based on DBL.(2).The CCA algorithm is introduced to realize multi-modal feature fusion.A novel fusion model is proposed based on DBL.The effective features fusion problem between different image modalities(RGB,Depth)is successfully solved.Experimental results on the proposed model have verified the good classification performance.(3).Aiming at the cross-modal feature fusion problem,a weakly paired fusion framework based on cluster canonical correlation analysis(Cluster-CCA)is proposed.With feature representations,paired mapping and joint dimensionality reduction,the weak correlation between different modalities is learned to construct a strong correspondence.Experimental verifications have proved the generalization and effectiveness of proposed fusion frame.