| In the Internet era,massive web page information emerges endlessly,especially in the field of science and technology.A large number of academic journals publish papers every year,and lots of researchers publish information on the Internet at the same time.If someone wants to get information about academic journals or researchers,it is not easy to make it.He or she needs to click on a series of hyperlinks on the Internet.Meanwhile,It is necessary for researchers to get academic information quickly.In this background,this paper studies the extraction and aggregation of heterogeneous academic information on the Internet,and proposes an automated algorithm framework to help researchers quickly mine the required information from a large number of heterogeneous web pages on the Internet.This paper mainly does the following works:1.Aiming at the problem of information extraction and aggregation of web-based academic journals.In this paper,C-HMM algorithm framework is proposed.The content-extraction algorithm in the framework achieves the effect of noise reduction.And the HMM can extract multiple websites simultaneously.These improve the generalization ability of the model compared to existing heuristic algorithms.The C-HMM algorithm framework is divided into three steps.Firstly,the crawler technology is used to get the home page of academic journals.Then,after the step of data preprocessing,the content-extraction algorithm is used to extract the important information in the homepage.Finally,the HMM model is used to extract and aggregate the information of academic journals.2.Aiming at the problem of information extraction and aggregation of web-based academic researchers,this paper proposes a F-HMM algorithm framework.The fastText algorithm in the framework pre-labels the webpage information block,which solves the problem that the keyword dictionary cannot pre-label multiple information blocks of the character.Based on the fastText algorithm,the HMM model is used to describe the timing information of the information blocks,which improves the effect of the model.The F-HMM algorithm framework is different from the C-HMM framework in the three aspects: First,SVM is adopted to select the homepage of academic researchers,and replace the keywordmatching strategy.Second,due to the complex structure of the academic researchers' homepage,the content extraction algorithm may filter useful information,so this algorithm is abandoned.Third,using the fastText algorithm can replace the original keyword matching method to pre-label the information blocks.3.The above two tasks are an important part of the research,development and application of the rapid knowledge sharing system in the era of big data and mobile internet in Jilin Province.The above work,the automatic crawler system of papers,news and CFP information are added into the development of the "Academic Headline" APP,which also facilitates researchers to quickly get academic information.At present,APP has more than7,000 users,4 million papers,6,000 journals and 6.7 million academic researchers,the effects show that the work of this paper has achieved good results. |
