Research And Design Of Micro-Video Recommendation System Based On Spark Big Data

In today's era,science and technology are developing rapidly,and short videos have gradually moved into daily life with the characteristics of being short,concise,rich in content,and vivid.As an important way of leisure and entertainment today,short videos provide users with a lot of wonderful content.Faced with the explosive growth of short video resources,how to obtain short videos that users are interested in is a problem that needs to be solved,and a recommendation system came into being.Most of the short video industries,such as Douyin,Kuaishou,Toutiao,Weibo,and Miaopai,have adopted various recommendation strategies in their operations.The function of the recommendation system is to present the content that the user is interested in to the user based on the user's basic information,preferences,browsing history,and scoring records.However,short video recommendation systems often have to deal with massive calculations and data sparsity problems faced by recommendation algorithms.Based on this background,this paper proposes a method using the Spark data platform and a short video hybrid recommendation algorithm that improves collaborative filtering and content recommendation.The main work done is as follows:To deal with the problem of massive computing,this article uses the Spark big data platform to deal with large-scale data processing.Spark is an open source cluster computing environment similar to Hadoop,but unlike Hadoop,Spark can increase the running speed of applications in the Hadoop cluster by 100 times in memory,and can even increase the running speed on disk by 10 times,which is effective It alleviates the massive calculation problem faced by the short video recommendation system.To solve the problem of data sparsity,some scholars have proposed a short video hybrid recommendation algorithm based on collaborative filtering and content recommendation.Based on this,this paper will propose a short video hybrid recommendation algorithm that improves collaborative filtering and content recommendation.Calculate the similarity between short videos through the classification tags of the short videos and the user's rating history,and then predict and fill in the missing rating values in the user-short video rating matrix,effectively alleviating the rating data faced by the short video recommendation system Sparse problem.