Research On Topic Models Based On Generative Neural Networks

With the expansion of the Internet community,documents from media such as news,blogs,and social networks have exploded,and the need to discover specific information from them has become stronger.Fortunately,topic models that can automatically discover underlying themes from documents provide a way to organize,explore,and understand such large-scale text collections.In recent years,the development of neural networks offers a flexible learning framework for topic modeling.Therefore,neural topic models,i.e.,topic models based on neural networks,have attracted a wide range of research interests in this field.Nevertheless,the existing neural topic models still have many limitations.For example,they did not consider the relationship between documents in the topic inference process,and they cannot effectively and efficiently utilize knowledge contained in large-scale corpora.Consequently,this thesis proposes solutions to these challenges,and the main contributions of this thesis include:1.Toovercome the limitation of ATM(Adversarial-neuralTopic Model)that it cannot directly infer topic distributions from documents,this thesis proposesToMCAT,a neural topic model based on cycle-consistent adversarial training.In addition to using a generator to capture semantic patterns in topics,ToMCAT also employs an encoder to infer document topics.Toencourage the generator/encoder to produce more realistic target samples,discriminators for word/topic distributions are introduced for adversarial training.Additional cycle-consistency constraints are utilized to align the learning of the encoder and the generator to prevent them from contradicting each other.Furthermore,for documents with labels,this thesis proposes sToMCAT that extendsToMCAT with an extra classifier.The classifier is jointly trained withToMCAT so that the training ofToMCAT could be guided and regularized by document labels.Experimental results on unsupervised/supervised topic modeling and text classification demonstrate the effectiveness ofToMCAT and sToMCAT.2.Toovercome the limitation of existing neural topic models that they did not consider the relationship between documents,this thesis proposes GTM,a neural topic model utilizing document relationship graphs.In GTM,the topic inference process of a document uses not only its bag-of-words representation but also more global information that is captured by building a corpus-level document relationship graph.The graph contains document nodes and word nodes,and the connections between document and word nodes are determined based on document-word co-occurrences.By connecting different types of nodes,it becomes possible to propagate information across nodes.In order to effectively use the document relationship graph during inferring the topics of a document,GTM uses a multi-layer graph convolutional encoder to aggregate information from its neighboring document and word nodes in the graph.Extensive experiments have been conducted on three datasets to compare the performance of GTM with several state-of-the-art topic models and the results demonstrate the effectiveness of the proposed approach.3.Toovercome the limitation of existing neural topic models that they cannot effectively and efficiently utilize knowledge contained in large-scale corpora,this thesis proposes a novel training strategy based on pre-training and fine-tuning.In specific,a topic model is firstly trained on a large corpus only once,which is called pre-training.Afterward,it can be fine-tuned on any other dataset,which is called fine-tuning.As the model architecture used in pre-training and fine-tuning is the same,it incurs little computational overhead to any subsequent training.Experiments have been conducted on three datasets and the results show that the proposed approach significantly outperforms not only some state-of-the-art neural topic models but also topic modeling approaches using pre-trained language models.