Design And Implementation Of Big Data Recommendation System Based On Multi-GPU Computing

With the rapid development of Internet Ecology,the amount of data needed for data analysis in recommender system has increased exponentially in recent years,which puts forward higher and higher requirements for the computing power of recommender system.Based on multi-GPU parallel computing technology,we build a parallellized high-performance big data recommendation system.This paper mainly completes the following innovative work:First,we proposed a GPU parallellized recommendation algorithm based on matrix factorization.The algorithm use hash table to initialize sparse feature matrix and avoids the defects of traditional algorithms in dealing with a large number of blank data blocks in high sparsity data set,and solves the access conflict problem of the algorithm in multi GPU architecture.It can complete multiple rounds of training of 1GB data in a few seconds,and is suitable for the application scenarios of training algorithms requiring high real-time performance.Experiments on several datasets show that the performance and training speed of the algorithm are better than all the existing algorithms,and there is good scalability in multi-GPUs architecture.Second,we propose a GPU-Based parallelized recommendation algorithm based on GAN.The algorithm transforms the traditional GAN architecture into a multi-GPUs computing architecture,allocates discriminators and generators to the GPU computing grid for parall-el computing,and uses an efficient parallel feature extraction algorithm to solve the problem of feature extraction.It improves the computing speed while maintaining the accuracy of the original Gan recommendation algorithm,and is suitable for the application fields tha-t require high-precision recommendation algorithm Scenery.Experiments on several experimental datasets show that the algorithm can effectively improve the training speed of Gan network and maintain the original calculation accuracy.Third,Considering the defects of the existing big data analysis system and the actual needs of the system,we build a versatile and scalable parallel recommendation system for big databased on K8S technology and microservice technology to overcome some defects of existing systems.At the same time,we provide a variety of other recommendation algorithms to form an algorithm library,provide a container computing model and convenient result analysis and visualization module for the whole system.It has good robustness and security,and supports multi-user concurrent operation,and allows users to customize data types,algorithm types and new system functions.A large number of module tests and system tests reflect the usability of the system.