| With the development of the social science research knowledge level,the social science knowledge is also rapidly updated and iterated,and the thesis library is becoming larger and larger.In the face of an increasingly large thesis library,when researchers want to fully understand relevant research in a certain field and continue to further their studies on this basis,they often spend a lot of time and energy to find relevant papers in the early stage,and even miss some important development directions.Therefore,improving the retrieval efficiency of papers to improve the productivity of scientific research is a very worthwhile content.On the other hand,with the advent of the era of big data,data mining and machine learning algorithms are becoming more and more mature.Natural language processing technology is also widely used in text extraction,topic extraction and other related aspects,providing technical support for the use of automation to process a large number of papers.Retrieval and analysis of papers through automation rather than manpower can save a lot of manpower and material resources and improve production efficiency.This article analyzes the current problems faced by the domestic research institute of social science thesis analysis,and combines the current development of processing technology for big data text,using the Citation-LDA(Citation Latent Dirichlet Allocation)and BERT(Bidirectional Encoder Representation from Transformers)to perform calculations,combined with data analysis,mines,integrates and displays relevant information of social science papers.The Citation-LDA model is a citation-based LDA model.Because the citation information of the paper contains more information and is smaller in length,using the citation information to perform model calculations can greatly speed up the calculation and reduce the impact of noise.Through the CitationLDA model,not only the thesis topic can be found,but also the topic changes and the milestone papers under the topic can be summarized based on the citation information.The BERT model mines the topic of the article from another angle.Through the BERT model,all papers can be represented as a word vector,and by clustering the word vectors,a paper cluster with similar topics can be obtained.By subject extraction of the paper clusters,you can get all the topics and the papers and the ranking of the papers under each topic.The above two models are combined with each other,which can more accurately summarize the relationship between the paper and the topic.Combined with the processing of other relevant information of the paper,a series of deep-level information such as the subject of the paper,thematic changes and development,the research area of the author of the paper,and milestone papers related to the research direction of the paper can be shown to the researchers,which is convenient for the researchers to make relevant Exploration of the development direction of the thesis.In terms of structure,the project is a web project.Spring-boot is mainly used to build the project.Elasticsearch is used as the storage engine to facilitate the rapid search of information.The model is written using a python script,and the model calculation results are stored in Elasticsearch and the file system.On the front-end side,the BootStrap framework and Vue.js framework are used to realize the visualization of related operation results and data.All data sources used in this article are from Nanjing University Data Center and full-text PDF file in the social science.In the project,I undertook the author analysis and data statistics part of the data analysis,word segmentation training,and the design and implementation of the front-end part of the project. |
PDF Full Text Request