Research And Application Of Representation Learning Based On Sum-Product Networks

In recent years,deep learning technology has achieved rapid development and great success in computer vision,natural language processing and speech processing.The continuous progress of representation learning research has greatly promoted the rise of this deep learning technology.Learning perfect embedding of data and feature representation makes it easier to extract useful feature information from the data when building predictive models or classifiers.However,the current mainstream deep learning model has black box attributes and lacks theoretical guidance for the corresponding neuroscience.The mainstream deep learning model cannot perform inference tasks.The structure of mainstream deep learning models is often fixed,and there is no corresponding structure learning algorithm.The study of deep learning mainly focuses on supervised deep learning.The training of mainstream deep learning models is inseparable from large-scale human labeled data.However,the acquisition of large-scale human labeled data is very difficult,even impossible in some tasks.Therefore,the use of semi-supervised or unsupervised deep learning has become an urgent research need.In recent years,in the field of unsupervised deep learning,auto-encoder have achieved great success.But traditional deep autoencoders have many limitations,such as: Firstly,there is a structural duality between the encoder and the decoder in the autoencoder model,but this is ignored in traditional autoencoder model applications.Second,the training process of the encoder and the decoder is separated,and the feedback signal between the encoder and the decoder is not shared.To solve the above problems,this paper studies a new type of probabilistic deep learning model: Sum-Product Networks.The SPNs model has recursive probability semantics and has strong theoretical support.The SPNs can perform exact inference,and their structure can use structural learning algorithms to generate network structures from data.Based on these advantages of SPNs,this paper proposes an Layered Sum-Product Networks(LSPNs)as a feature extractor and designs an improved SPNs structure learning algorithm.Inspired by dual learning,this paper also proposes an auto-encoder architecture based on Dual Sum-Product Networks.The main work of this paper is summarized as follows:(1)This article introduces the basic knowledge of Sum-Product Networks.Second,the inference method of SPNs was explained.Then,the generative and discriminate parameter learning of SPNs were introduced.Finally,the main structure learning method of SPNs was introduced.(2)The improvement of the SPNs is mainly reflected in two aspects,LSPNs and Learn LSPNs learning algorithm were proposed.First,the LSPNs was proposed,the structure of SPNs was reconstructed.Then,the SPN nodes were arranged in a hierarchical structure,the same layer contains the same nodes,and different layers contain different nodes.Last,increasing the output of the input layer to the hidden layer of each SPNs.Second,an improved SPNs structure learning algorithm Learn LSPNs is proposed,spliting the data matrix slices always into two.Then introduce the tree distribution as the leaf node distribution.Lastly,proposed variable splitting method is based on the concept of entropy.(3)In order to verify the ability of LSPNs as a feature extractor which a large number of comparative experiments were conducted in this paper.Firstly,visualization the sample generation in LSPNs.Then,visualization the feature extraction in LSPNs.Lastly,the LSPNs are used for image classification tasks.The comparison models are RBM,DBN,MADEs and VAE,which proves the superior performance of LSPNs in feature extraction.The filtering embedding of LSPNs is designed to verify the accuracy of its internal nodes in image classification experiments.Then,the improved SPNs structure learning algorithm is compared with the traditional SPNs structure learning algorithm Learn SPNs.The improved SPNs structure learning algorithm is validated in the model parameter statistics and testaverage log likelihood function.(4)The Dual Sum-Product Networks autoencoder is proposed.Firstly,designing the structure of Dual Sum-Product Networks autoencoder as a dual form.Then,explicitly exploiting the structural duality between them to enhance the training process.Finally,training the SPNs encoder and MPNs decoder simultaneously in Dual Sum-Product Networks autoencoder,making the feedback signal shared in both SPNs encoder and MPNs decoder.(5)In order to verify the representation power in Dual Sum-Product Networks autoencoder,this paper designs a datasets reconstruction and multi-label classification task experiment.The experimental results show that Dual Sum-Product Networks autoencoder has better reconstruction ability than traditional Sum-Product Networks autoencoder.The Dual Sum-Product Networks autoencoder has better JACCARD,HAMMING and EXACT MATCH scores compared with the traditional Sum-Product Networks autoencoder,MADEs,RBM,SAE,CAE and DAE models.