Research On Construction Method And Application Of Deep Probabilistic Models Integrating Text Structure Information

As an important support for contemporary artificial intelligence technology,and also as a classic model in the field of machine learning,deep neural networks have made key breakthroughs in image recognition and detection,natural language processing,biological information representation and other fields,showing the superiority of deep learning algorithms in performance.However,due to the large number of non-linear activation functions in the network,existing deep neural networks generally have the problem of poor interpretability,leading to that users can not understand or even trust the decisions made by the network in many practical applications.Compared with these black-box deep neural networks,benefiting from the introduction of deep learning thoughts into traditional topic models with practical-meaning model parameters,deep probabilistic topic models exhibit excellent model interpretability while retaining hierarchical network structures,which have also received extensive attention from industry and academia.However,existing deep probabilistic topic models are limited by bag-of-words data representations and Gibbs samplingbased training and testing methods,leading to a series of problems such as ignoring the temporal relationship between words and the semantic connection between documents,confined to single-modality observations,unable to perform real-time prediction,and difficult to process large-scale datasets.To this end,this thesis is dedicated to constructing deep probabilistic topic models according to different actual scenarios,such as text analysis,document relational network modeling and image-text multimodal representation learning,proposing corresponding accurate and fast model inference methods,and trying to combine them with deep neural networks to improve the model performance on the premise of interpretability.The main contributions of this thesis are listed as follows:1.In the field of text analysis,focusing on dealing with the problem of ignoring the temporal relationship between words in the existing probabilistic topic models,this thesis proposes the first probabilistic convolutional topic model that can directly process sparse text sequences through probabilistically modeling the documents represented as one-hot sequences,namely convolutional Poisson factor analysis(CPFA).Compared with traditional probabilistic topic models,CPFA explores and visualizes phrase-level topics composed of phrases with similar semantics or structures on the basis of extracting implicit representations of documents with richer semantic information.Further,this thesis combines CPFA with existing deep topic models to alleviate the information loss caused by ignoring word orders,and retains the hierarchical network structures of deep topic models to extract latent document representations under various semantic levels,which further improve the model performance in downstream text analysis tasks.2.Considering that the developed probabilistic convolutional topic model CPFA is limited by exploring phrase structure information under a single semantic level,this thesis extends the shallow CPFA to a deep convolutional topic model,named Poisson gamma convolutional belief network(PGCBN),to extract hierarchial textual information form the sparse text sequences with a multi-layer convolutional structure.To handle with large-scale corpus,this thesis constructs a Weibull-based hierarchial convolutional inference network to achieve outof-sample prediction and introduces the shortcut module to enhance the connection between adjacent latent document representations,which can be further combined with PGCBN as a hierarchical Weibull convolutional variational autoencoder(HWCVAE).Through introducing the side information of document labels and layer-wise attention mechanisms into the hierarchical latent document representations of HWCVAE,this thesis develops several supervised variants to improve the model performance in document classification task,and evaluates the robustness and generalization ability of these developed models with a series of semi-supervised experiments.3.Aiming at dealing with the problem of ignoring the connection between documents in the existing probabilistic topic models,this thesis develops a graph Poisson factor analysis(GPFA)to jointly model the link structure and document content in the document relational network.Moving beyond sophisticated approximate assumptions of existing relational topic models,the developed GPFA can integrate the relation information of documents into their latent representations with closed-form Gibbs sampling update equations.To explore hierarchial semantic relations between documents,this thesis extends the shallow GPFA to a graph Poisson gamma belief network(GPGBN)with multiple hidden layers,and explores the underlying reason for the generation of document relations at a specific semantic level with specific visualization techniques for deep probabilistic topic models.For more efficient model inference and network information aggregation,this thesis constructs two different Weibull graph variational autoencodings(WGVAE)with powerful graph neural networks to extract high-quality hierarchical latent document representations,leading to improved performance over baselines on various graph analytic tasks.4.Considering that traditional probabilistic topic models are limited to single-modality inputs,this thesis focuses on modeling multimodal observations composed of image-text pairs and develops a novel multimodal Poisson gamma belief network(m PGBN)that tightly couples the data of different modalities via imposing sparse connections between their modality-specific hidden layers.Compared with traditional black-box multimodal representation learning methods based on deep neural networks,the developed m PGBN can not only extract interpretable hierarchical multimodal latent representations,but also provide a novel solution to visualize the generative process and the relationships between modalities at different levels of abstraction.To make our model both scalable to large-scale datasets,this thesis constructs a Weibull-based multimodal inference network for fast in out-of-sample prediction,and develops a task-driven supervised extension to extract more discriminative multimodal latent representations via introducing side information.