Statistical Analysis Of Literature In Artificial Intelligence Based On Text Mining Technology

Under the background of the continuous promotion of the digital economy,artificial intelligence develops rapidly and is deeply integrated with a variety of application scenarios.It is of great significance for researchers in this field to obtain high-quality literature information in artificial intelligence in a short time,quickly find hidden contents and relationships,and accurately grasp the recent development trend and hot direction.This paper uses crawler technology to obtain high-level literature information in artificial intelligence,uses text mining technology,combines keyword co-word analysis and abstract LDA topic model,and draws the knowledge graph in artificial intelligence by using the visualization technology of Gephi and Python.So as to obtain the development status and hot topics of literature in artificial intelligence,which has a certain reference significance for other researchers to grasp the latest trends of literature research.Firstly,use Python software Request toolkit to obtain web page data,Lxml toolkit to analyze text data,and crawl the high-level literature of CNKI and WOS databases with "artificial intelligence" as the keyword from 2019 to 2021.After de-duplication and de-blank processing,a special word segmentation database and stop word list in artificial intelligence are constructed.Jieba and nltk databases are used to segment Chinese and English text data respectively,remove stop words,and obtain text data convenient for subsequent analysis.Secondly,through the traditional literature measurement method,this paper makes a statistical analysis on the data of the number distribution of literature,the degree awarding unit of the author of master’s and doctoral thesis,the source journals of WOS foreign literature and the core author group of literature,so as to describe the development status of research in artificial intelligence.Thirdly,co-word analysis is carried out based on keywords,including using word frequency analysis to show the basic situation of literature keywords,and using co-word cluster analysis to analyze the hot trends in the research field.This paper constructs the highfrequency keyword co-occurrence matrix through Python software,uses Gephi software to discover the community based on the Louvain algorithm,analyzes the macro and micro indicators of the network,and draws the network co-occurrence map of the literature in artificial intelligence,so as to reveal the research hotspots in artificial intelligence according to the visualization results,and compare the similarities and differences of the hot trends of the literature at home and abroad.Finally,LDA topic model is carried out based on the summary.This paper constructs the LDA topic model,optimizes the number of topics based on the confusion degree topic variance index,and uses the pyLDAvis visual tool to dynamically display the topic extraction results.Analyze the topic results by integrating the topic feature words,topic distribution map and co-word analysis content,and get the hot topics and their evolution trend in artificial intelligence,so as to help researchers in this field quickly understand the research status and grasp the emerging research direction.