Research And Implementation Of Algorithms Automatic Integration Mechanism In Massive Data Analysis Platform

With the development of data analysis techniques, data analysis has been widely applied. Data analysis has played a more and more important role in the corporate decision-making and product design. And because of the development of enterprises, the number of products, number of product types, and number of users surge, the amount of data within the enterprise also shows explosive growth, which are has brought great challenges for data storage, operation, and management. The mining of the potential value of data becomes more difficult. Faced with the challenges of data storage, data processing capabilities, performance and other aspects, traditional data analysis system has been incapable of analyzing massive data. With the cloud computing technology becomes more mature, massive data analysis platform based on cloud computing emerges.In cloud-based massive data analysis platform, the algorithm for processing data is the core part of the data analysis. The richness and diversity of algorithms is an important criterion to measure the usefulness of a data analysis platform. Therefore, how to integrate the different types of third-party data analysis algorithms efficiently and conveniently has become a very critical issue.To solve these problems, this paper presents an algorithms automatic integration mechanism in massive data analysis platform. With this mechanism, the third-party algorithms can be integrated into the existing platform efficiently and conveniently. And these algorithms can also operate in conjunction with other algorithms to perform more complex analysis process. In summary, the main content of the paper includes:1. The establishment of the algorithm model:Study the features of algorithms in massive data analysis platform, and set up a unified algorithm model, to achieve a unified description of different algorithms.2. Automatic modeling of algorithms:Study on how to convert different algorithms to modeling algorithms that are in line with the standard algorithm model.3. The automatic integration of modeling algorithm in the platform: Study on how to make the data analysis platform be able to automatically recognize the modeling algorithm, and integrates it into the resource library of the platform.4. The management of algorithms:including the deletion, modify, update and other operation on algorithm.5. Finally, this paper applies the mechanism to the real case. The algorithm can be fast and easily integrated into the platform in this case. This experiment verifies the feasibility of the mechanism in this paper.