Research On Randomized Singular Value Decomposition Based On Hadoop

With the development of mass data in life,the concept of big data has become major topic of discussion in recent years.In the era of big data,data is everywhere,including Internet,telecommunications,etc.,have been integrated into the imprint of big data.However,most of these huge amounts of data are irrelevant to ordinary families.It is these irrelevant useless data that seriously interfere with the audience's choice of the relevant useful information.In fact,it is seeming that this is an era of information overload,but in fact overload is spam,and quality information is always scarce.The recommendation system is proposed to solve such a problem.Matrix decomposition is a common technique used in current recommendation systems.Compared to other recommended algorithms,matrix decomposition can bring better results,and can fully consider the impact of various factors,and has very good scalability.Singular value decomposition is a mature technology in the recommend system.The traditional singular value decomposition can only decompose the dense matrix.However,the actual user and item matrix are sparse and the singular value decomposition has high time.The complexity,when the size increases,the decomposition efficiency is unbearable.It is effective to use the random algorithm to solve the problem that the decomposition speed of the singular value decomposition is time-consuming.This paper uses Count Sketch,an algorithm that solves the problems of finding frequent items in the data stream,to accelerate the singular value decomposition of the matrix.Experimental analysis,this method can play a very good acceleration effect.Then a single random algorithm has its advantages,but it also has its shortcomings.Following this problem,this paper proposes randomized singular value decomposition algorithm based on two random schemes.This algorithm can compensate for the insufficiency of single random algorithm and complement the merits of the two random schemes to improve the speed of singular value decomposition further.The randomized matrix decomposition technique in the thesis,combine random algorithm with the traditional matrix decomposition and running in distributed environment,compares different aspects of the random singular value decomposition algorithm by experimental means.The experimental results show that the algorithm is effective and can greatly improve the computational efficiency under the premise of sacrificing less accuracy.