Design And Implementation Of Words Correlation Query System

With the spread and development of the Internet, The generation and spread of network information is also growing fast, Age of Big Data comes into being. In the face of the impact of big data, journalism and media industry took measures according to its features. Different from the pure text report in the past, it find the meaningful information from unstructured data, and then analyze the information. Finally, it present the analysis results of data to users in the form of charts and other visual interpretation. This visualized presentation of news data, making it more convenient to get and master the useful information for users, received widespread high praise.To meet the visualized presentation of news data, we designed and implemented a words correlation query system based on LNMP(Linux+Nginx+MySQL+PHP) and multi-process task scheduling. This system is divided into two parts, the first part is offline data production. We selects People’s Daily, the official media of news, as a data source, and crawled its data from 1945 to 2015, which were stored in the database as the original data. Then we did the data preprocessing on the original data, which contains Chinese words segmentation, table sharding and multi-process data analysis. In the end, we created words index data and the corresponding relation data of words and articles, storing in the sharding tables. The second part is the real-time search query that consists of front-end and back-end services two parts. The two parts are the web interface of the system based on the LNMP and the words correlation data query backend services based on multi-process task scheduling with the Celery framework. The web interface use Http requests to communicate with the backend services.In this paper, to meet the demands of news data visualization, we introduce the words correlation query system with design, implementation, deployment and test. This system can provide words in People’s Daily from 1945 to 2015 correlation data query service for researchers in some specific fields, such as psychology.At present, the platform is still in beta version. We have invited some psychologists to use our system, and won theirs recognition.