Optimization Of Distributed Machine Learning Programming Model Based On Dependency Of Data Access

In the era of big data, the value of the data is becoming more and more important. The main task of machine learning is to infer a model. As the data volume presents explosive increasing, it becomes more difficult to conduct such machine learning tasks as no single compute node can solve such large scale machine learning problems. Thus, both the dataset and the CPU-intensive workload should be distributed over compute nodes. Recently, several distributed machine learning systems which leverage the parameter server architecture have demonstrated state-of-art performance. However, those systems expose the(key, value) access methods to programmers. This burdened programmers with the system level optimization and automatic task-partition and task-parallelization in order to achieve good performance.We propose a new distributed machine learning framework, which also leverage parameter server architecture, but provide high-level interface to developers. The interface decouple the detailed accessing methods of parameter and tensors from application core logic. We also break down the process of machines learning task as a series of stage. The high level interfaces not only greatly simply the process of writing machine learning programs but also provide opportunities for system designers to perform system-level optimization, such as task-partition and task-parallelization.We build a document analysis application which based on the Latent Dirichlet Allocation(LDA) technology, a kind of topic model in text-mining area, to verify our prototyping system. Results demonstrate that our system is scalable in single node or in distribute setting and the time spent at barrier is not big than 15% of the time spent at total training. In one word, our system is scalable and incur low overhead while providing simplified interface and modified building block for large scale machine learning applications.