| With the rapid development of the Internet,recommendation systems have become more and more widely used in the Internet.The recommendation system discovers the information that the user is interested in by analyzing the user’s historical data information,and then recommends this information to the target user.In academia,many recommendation algorithms have been proposed.Among them,the user-based collaborative filtering recommendation algorithm is one of the most widely used recommendation algorithms in the recommendation system.The user-based collaborative filtering recommendation algorithm does not require users to provide clear requirements.The algorithm finds neighbor users with similar interests of the target user,and then recommends items that the neighbor users like and that the target user has not followed to the target user.However,the user-based collaborative filtering recommendation algorithm is still affected by the popularity of items and the number of users’ common ratings when calculating user similarity,and further improvement is still needed.At the same time,with the gradual increase of the data scale,the running time of the recommendation algorithm also increases,and the existing recommendation algorithm is inefficient when processing massive amounts of data.In response to the above-mentioned problems,the user-based collaborative filtering recommendation algorithm is mainly improved in the following two aspects:When calculating user similarity,the user-based collaborative filtering recommendation algorithm did not consider the issue of project popularity and improved it.Usually,items that users pay too much attention to are all popular items.However,it is difficult for this type of item to reflect the similarity of users.On the contrary,unpopular items can better reflect the same hobbies among users,and better reflect the similarity between users.Considering the influence of item popularity on the recommendation algorithm,the popular item penalty factor is integrated into the user similarity calculation formula of the recommendation algorithm in this paper,and the user similarity calculation is improved.When the user-based collaborative filtering recommendation algorithm uses the adjusted cosine similarity formula to calculate the user similarity,it does not consider the effect of the number of common ratings on the similarity result,and calculates the user similarity with the penalty factor of the popular item The method is further improved.And on this basis,the calculation of user similarity is fully considered to be affected by the number of common ratings.Introduce JS divergence as a correction factor to correct the similarity results of users from the perspective of probability distribution.Design a parallel recommendation algorithm based on the Spark platform with the improved serial recommendation algorithm.This parallel recommendation algorithm reads data from the HDFS distributed file system on the Hadoop platform.Mainly use operators to perform data operations on RDDs and Data Frames,and convert user data into new RDDs that can be manipulated and Data Frames that can be directly processed by SQL instructions.The design process of the parallel recommendation algorithm in this paper is divided into four stages.In the first stage,parallel algorithms are designed for the data preprocessing process.The original data set RDD is divided into multiple nodes,and the data splitting operation is performed in parallel in different tasks of multiple nodes.In the second stage,a parallel algorithm is designed for the user similarity calculation process.In the user similarity stage,a parallel algorithm is designed for the Cartesian product calculation process between any users.The third stage is to design a parallel algorithm for the user rating prediction process,and design the parallel algorithm for the process of calculating the predicted rating by the target user using the rating prediction model.The fourth stage designs a parallel algorithm for the target user recommendation process.Parallel algorithm design is applied to the process of sorting all users’ score prediction results in descending order. |
