Design And Implementation Of Machine Learning Algorithm Toolbox In Eole System

With the rapid development of computer technology and network technology,the amount of data grow with each passing day,we ushered in the era of big data.As an effective means to extract value from the massive data,machine learning techniques have made great progress.But it's getting harder and harder to use.The business personnel using machine learning techniques to solve massive data processing problem,often need to learn distributed related knowledge,besides knowledge of data modeling and data analysis.And the existing solution to the problem of using machine learning with massive data,or provides only the implementation of the algorithm and does not provide a complete execution environment,or has very high learning cost on other aspects.Aiming at these problems,this paper presents a solution that combines computational ability and machine learning algorithm,namely in the existing distributed massive data processing platform increases a machine learning algorithm toolbox to provide distributed implementation of some classical algorithm in major field of machine learning.In order to solve the problem of the execution environment,the algorithm toolbox relies on the massive data processing platform Eole system,and provides unified algorithm executive function.In view of the learning cost,algorithm toolbox using browser/server architecture,provides friendly graphical interface with high fault tolerance,makes it very simple for users who do not have computer knowledge and experience of using professional tools to use machine learning algorithms to analyze massive data.The algorithm toolbox introduced in this paper is designed to provide easy-to-use machine learning algorithm to the user,including algorithms which belong to machine learning classification and regression,cluster analysis and association rules.In the aspect of algorithm execution,the algorithm toolbox uses HDFS(Hadoop distributed file system)as data storage environment,yarn(Hadoop computing framework)as the underlying computational framework,implements an algorithm execution module based on Actor model.In the aspect of user interaction,for different functions,the algorithm toolbox uses algorithm task management module and dataset processing module to analyze and respond to the request sending from browser side.At the same time,as part of the Eole system,the algorithm toolbox provides Web services through the Web display module of Eole,and accesses database through the persistent module of Eole.