Design And Implementation Of A Big Data Intelligent Analysis Platform For Heterogeneous Computing

This thesis designs and implements a big data intelligent analysis platform for heterogeneous computing.The platform manages computing resources of CPU and GPU in a cluster,providing an integrated pipeline for big data and artificial intelligence.At the heart of this platform is a custom module named DeepMaster,coordinating the function of other component,such as distributed computing engine,file system,resource negotiator and container virtualization.DeepMaster supports all mainstream Deep Learning framework and distributed training of neural network.Performance model of the platform is also presented in this thesis as well as the proof of higher GPU utilization.In addition,the benchmark of training ResNet on ImageNet shows that there is little performance degradation on platform.This thesis has developed friendly user interface for application submission,with which applications can be migrated to the platform with little modification.Realtime log viewing interface and graphic monitoring interface are also provided.And a migration plan to cloud infrastructure is proposed,based on cloud orchestration and dynamic scaling techniques.In practical deployment,GPU is used more frequently than CPU.Analytics-Zoo is integrated to the platform to leveraging CPU as a complement of GPU.A comparison between it and CPU version of tensorflow on programing model and performance is also presented.