Design And Implementation Of Deep Learning Platform Based On Containerization Technology

With the continuous development of deep learning,more and more business scenarios use deep learning algorithms to solve practical problems.However,the iterative cycle of deep learning algorithm applications is generally long,including necessary steps such as data collection and labeling,algorithm training and deployment.Due to the complexity of its processes,numerous people with different professional backgrounds are needed to participate,which increases the communication cost and reduces the algorithm iteration efficiency.In addition,there are many challenges in all steps of the algorithm iteration cycle.For example,in the data preparation stage,it's costly to collect and label data.Imperfect data management methods may cause the loss of data assets;In the algorithm training stage,algorithm depends on complex environments,and unscientific model management methods may cause loss of models and log files;In the algorithm deployment stage,algorithm service development,deployment and maintenance also face many difficulties.To solve above problems,this paper includes the following three aspects:(1)This paper builds a "data-training-deployment" closed-loop production deep learning platform that provides one-stop deep learning services,mainly including four business function modules:dataset management,algorithm training,online algorithm services,and self-service image algorithm tasks.(2)In the "data" stage,aiming at the inefficiency of traditional data labeling methods,this paper designs and implements an algorithm-assisted labeling system.Using image clustering and label prediction algorithms,the image labeling efficiency is increased by nearly three times.(3)In the "deployment" stage,the implementation of traditional algorithm deployment is complicated,which requires a lot of costs on development,deployment and maintenance.In view of this problem,this paper proposes a set of unified computing framework for online algorithm services based on kubernetes.Algorithm developers do not need to write service-oriented deployment code,they only need to focus on the algorithm and deploy algorithm services in one click.This paper realizes the automation and standardization of algorithm service deployment and maintenance,while providing algorithm service with high concurrency and high availability.