Scalable And Highly Efcient System For Deep Learning On Heterogenous Environment

Deep learning has claimed ground-breaking results in the domains of speech, vi-sion and document understanding. Its success owes a great deal to its ability of extract-ing complex features in a deep hierarchy of neural network. Learning a huge amountof parameters well critically depends on computing over large volume of training data,making training such networks both computational and I/O intensive. Existing ap-proaches have explored the power of GPU and large-scale clusters. Unfortunately, theyoftenexposeplatform-dependentAPIsdirectly, leadingtoalgorithmdevelopmentsthatare platform-dependent, tedious to implement and optimize, and difcult to maintain.Minervaproposesamatrix-basedprogrammingmodelthatallowsstraightforwardimplementation of deep learning algorithms. It preserves the imperative and procedu-ral coding style that is familiar to algorithm researchers, resulting in compact codes.The Minerva runtime dynamically converts the user code into a datafow representa-tion that can be executed by diferent processes in parallel and on heterogenous hard-wares. When running under a distributed setting, the system automatically infers near-optimal placement of majority of the data, and efciently hides communication over-head. Without changing a line of code, our training algorithms run on top of modernlapton/workstation, high-end server and server cluster, with and without GPU acceler-ation. Our evaluations show that this layered design strategy exploits many forms ofparallelism, scales out efciently, and outperforms many existing alternatives, some-times substantially.