| With the rapid expansion of enterprise data and the emergence of various kinds of. parallel computing framework, traditional BI platform has been incapable of rapid analysis of the data. Because of the limitation of memory, it has been unable to process these data. Therefore, the parallelization of algorithms has become a general trend. And to develop BI platform, which supports distributed deployment and parallel algorithm, has been one of key exploitation. This paper is based on a research issue which raised by a communications enterprise. The goal is to develop a management platform for the needs of analysis of big data and parallel algorithms model library.Firstly, a parallelization scheme for a specific algorithm HeteSim, which proposed to measure the similarity in information networks, is researched in this paper. Then a design scheme of parallel algorithms library management system based on cloud computing platform is proposed and achieved. This system aims at the management of BI models and business applications. Specifically, the main contributions of this paper include the following two parts:1. To parallelize the HeteSim algorithm. First, we make a brief introduction of some similarity measures based on network structure, and describe the algorithm principle, mechanism and advantage of HeteSim in detail, as well as analyze its shortcomings in solving large-scale problems. Considering the characteristics of matrix block and the advantages of parallel computing framework MapReduce, we design a parallel method combining the feature based on meta-path of HeteSim, and then validate the parallelized algorithm. 2. To design and implement parallel algorithm management system that can deploy in the cloud platform. Firstly, it uses the layered architecture to complete the design of the system, and the background of core based on the runtime environment and execution engine of Pentaho platform, as well as uses Flex to develop front end to make it easy for users to use the Web interface. Then the system core function module detailed design scheme is given. The core features of the system include: dragging and dropping the workflow and parameter configuration, computing parallel algorithms, managing users, algorithm and business application effectively and conveniently. Based on the above design, it realizes the implementation of parallel algorithm management system based on Pentaho. |
