Design And Implementation Of College Entrance Examination Recommendation System Based On Spark

In order to solve user's dilemma that the user can not get valuable information and information can not be used,people put forward the concept of recommended system.With the arrival of large data age,the recommendation system began to face difficulties in dealing with massive amounts of data.In order to get out of the woods,combination of large data processing technology is an inevitable trend.As a leader in large data processing technology,Spark put forward the RDD data model and memory-based computing model,and has been widely used in e-commerce,video,social and other fields.But in the field of education,whether it is the recommended system or large data processing techniques involve less.College entrance examination as a big event in the field of education,its volunteer to fill is the focus of attention of candidates.The candidates volunteer admission information over years as the important reference data to fill in candidates volunteer,can be very low utilization because of the large and complex data characteristics.This paper will combine recommended system with Spark,a large data processing framework,applied to the field of education which Spark and recommended system are less involved in to help candidates solve volunteer enrollment link volunteer choice.The work done in this paper has the following:(1)Using HTML + CSS cascading style sheet + JSP front-end development technology,design and development of the system's Web front-end interface.Which includes user registration interface,user login interface,volunteer recommendation display interface and related college entrance examination information(policy,news,university information and professional information)browsing interface.Not only to ensure the system practicality and ease of use,but also to provide users with a good interactive experience.(2)The Web front end is used as the user log to design a good performance log collection module.First,Use the Flume log collection tool to collect log information;then the collected information is transmitted by the Sink assembly to Kafka,to uniformly deliver the log information using the function;finally,Use Spark Streaming to clean up and extract the log information collected in Kafka and store it in the HDFS file system for later use.(3)Design recommendation engine in the real situation of college entrance examination.First,select the appropriate user attributes by reading a lot of literature about Application of College Entrance Examination,and it is then used to calculate the similarity of user attributes,establishing similar matrix,find similar users;Secondly,this paper analyzes several of the most common recommendation algorithms,combined with the real scene of college entrance examination volunteer to fill the user-based collaborative filtering algorithm as the recommended algorithm of the system;Finally,the final recommendation list is generated by the parallel calculation of the Spark.(4)Build Spark distributed cluster development environment,to achieve the overall development of the system and related testing.First of all,according to the description of the relevant documents,in the actual environment of the laboratory to build a three-node Spark distributed cluster development environment;Secondly,write code in Scala language for system development;Finally,Test log collection tools and the performance of Spark related components,after the system is developed,at the same time,test the recommended results accuracy and overall system satisfaction,in order to ensure a good user experience.