Design And Implementation Of Machine Learning Supporting Platform For Big Data

Machine learning is a multidisciplinary interdisciplinary,and its initial motivation was to give computer systems the ability to learn.Early machine learning data sources were single and relatively simple to handle,but the business scenarios that could be addressed were limited.Under the current wave of big data,machine learning faces new challenges and opportunities.Using machine learning methods for common voice,picture,and text directions has brought about major breakthroughs.However,the development process of machine learning applications is still relatively complex,often requiring device developers and algorithm developers to work together,algorithm developers can not directly optimize platform computing efficiency,developers can not understand the specific workflow of the algorithm,making machine learning applications The development has become more difficult.For the difficulty of machine learning application development,this topic focuses on the machine learning algorithm support platform to simplify the development process of algorithm application and accelerate development efficiency.Traditional machine learning algorithm development processes often include data preparation,model modeling,model training,and model optimization and deployment.Aiming at the model optimization,this paper proposes a deep learning model distillation algorithm based on the countermeasure strategy to optimize the problem of high performance loss of the traditional model distillation.The experiment is carried out on the classification problem to verify that the scheme is superior to the traditional model distillation.For the model training,this paper proposes a low-bandwidth distributed gradient descent algorithm to solve the distributed network bottleneck of gradient descent.The reliability of the method is verified by experiments on the classification problem.In order to realize the system,this paper fir-st expounds the research background and significance of the platform,and analyzes the resear-ch of related technologies in academia and industry.Then,the requirements of the system are analyzed.According to the main process of developing a machine learning algorithm,it is divided into four processes:data preparation,model modeling,model training and model deployment,and the key problems of system requirements are studied.The corresponding solutions are given,including the model distillation using the countermeasure strategy to improve deep learning,and the reduction of the bandwidth loss of the distributed gradient by compressing the gradient.Then based on the understanding of the requirements and key issues,this paper designs the overall architecture and functional modules of the algorithm platform,analyzes the typical scenarios in the key modules,and illustrates the implementation of key algorithms.Finally,the deployment and testing of the platform are explained to verify the correctness and integrity of the system.At the same time,the shortcomings in the research work are expounded,and the future research of the proj ect is prospected.