Distributed Learning on Large-scale Dat

Big data helps machine learning to get better models, but also brings computational challenges. Conventional machine learning algorithms are usually designed for small-scale data. When faced with large-scale data, especially one whose volume exceeds the capacity of a single computer, these algorithms become infeasible. Nowadays, people increasingly rely on distributed computing clusters to process large-scale data. Designing effective algorithms for distributed systems becomes an important topic in machine learning area. In this thesis, we focus on two specific algorithms, exploring their extension in distributed computing environment.;In the first part of this work, we studied the problem of estimating Dirichlet process mixture models. We designed sampling algorithms that are suitable for distributed systems. They allow processors to discover new clusters independently while maintaining their consistency via consolidation schemes. The developed algorithms require low communication cost and can be easily applied to asynchronous settings. This part of the work has been accepted in 26th International Joint Conference on Artificial Intelligence (IJCAI 2017).;In the second part, we studied batch normalization techniques within deep neural networks. We reformulated the classical batch normalization method by treating batch statistics as model parameters and introducing proximal optimization procedures to update them through iterations. This design eliminates the communication required by accumulating batch statistics thus scales well in distributed computing environments. In the meantime, it effectively mitigates the performance issue of small mini-batches. This part of the work has been submitted to 31th Annual Conference on Neural Information Processing Systems (NIPS 2017).