Design And Implementation Of Hadoop Based HSK Test System

Hsk is a national exam,its official purpose is for testing of the non-native Chinese official Chinese level and establish a national standardized test.Our group developed the HSK examination system which was used in European Confucius institute.The Speed of generating paper gradually slow down under the condition of high concurrency,since the system has accumulated a large amount of historical data users.We did not consider the personalized feature of candidates when we developed the system at that times.In addition,the system could not construct paper in a dynamic way according to the reflection of candiates.In order to solve the existing problems,we decide to use collaborative filtering recommendation algorithm which is implemented on Hadoop ecosystem in a parallel way to recommend personalized questions according to candiates’ s response to questions.In this paper,the main research contents and work is as follws:Ⅰ.Analysing the demand and function of the HSK examination,the system adopts B/S structure of the hierarchical design model architecture,planning the overall design goal,architecture design,logical structure,the key technology.Designing and implementing the core modules of the system.Ⅱ.Aiming at the deficiency of HSK Chinese test system,a collaborative algorithm based on project is proposed,which is recommended for candidates.Design and implement a collaborative filtering plan based on the Hadoop platform.Decompose the computational task into a series of MapReduce workflows and perform distributed processing on the Hadoop platform.Ⅲ.In order to reduce the logical coupling between the test module and other modules of the system,a data collection system based on message queue and Spark Streaming framework is designed and implemented.Through the Kafka message queue to complete the candidates real-time score record conversion,and through the Streaming real-time stream processing module to complete the collection of candidates to collect the score.