The Design And Implementation Of Service-Oriented One-Stop Machine Learning Algorithm System

With the development of machine learning technology,machine learning model has been used more and more frequently in software industry,and its application scope is more and more extensive.However,new technologies usually bring problems and challenges.There are two thorny problems in the application of machine learning technology in new bee network company:one is that the threshold of developing machine learning model is high.Accordingly,building model for a small-scale requirement still needs the participation of senior algorithm engineers,which leads to high development costs;the other is that machine learning model is tightly coupled with the traditional software system,resulting in poor scalability of the system.Every time a new model needs to be online,it will cause a greater invasion of the original system code.To solve these problems,this paper designs and implements a service-oriented one-stop machine learning algorithm system.The system provides the one-stop machine learning service from data acquisition to model online.It aims to enable ordinary developers who have a little machine learning knowledge background to skillfully build common machine learning models according to business,so as to make the threshold of using machine learning models lower and ultimately reduce the cost of developing algorithmic models.In addition,the system combines the process of model online with the idea of service-oriented,which decouples the machine learning model from the business system and improves the extensibility of adding new models in the system.The system is designed with modularization idea.It has five core modules:the first one is data acquisition and storage module.This module is mainly responsible for training data access.It encapsulates different data source access and provides a unified data access API to the rest modules.The second module is model parsing module.It is mainly responsible for processing the model description transmitted from the front end.It transforms the model description into the computational graph object,which can be recognized by system.The next module is data preprocessing module.This module mainly receives the source data information,then cleans and standardizes the source data.After data cleaning and standardizing,it performs feature engineering.The forth module is algorithms engine module.It is based on the Tensorflow framework to implement the algorithms in the algorithm library.In this module,system uses parameter server as default to do distributed model training work.The last core module is the algorithm service module.This module is mainly responsible for adding the trained model to the model services provided by the system.At present,the system has been used stably in the production environment.Compared with the previous situation without the one-stop algorithm system,the cost of model development and model online has been greatly reduced.What's more,the correct rate of the model created by the system can meet the business standards.The number of online models keeps increasing monthly,so as to achieve the expected goal of developing this system.