Study And Inplemtation Of Big Data Processing And Management Platform Based On Open Source Software

With the coming of the Internet, more and more data are producing,passing and sharing in the Internet. There is a lot of benefits behind the big data, so many Internet company are moving their focus on the big data processing. In order to handle this magnificent data, there is a lot of open source project coming out, such as Hadoop, Spark, Storm etc.However, these open source projects are developed independently, does not monitor the whole group and utilities very well, HDFS does not has a good servitization and visualization, and there is no specific solution for different situation by different schedulers of Yarn, the usability of job management and monitoring is not easy to use.For the problems, we solved them in two aspects. Firstly, we did a deep studying on schedulers of Yarn, which is a popular resource management in big data. We did some experiments for different schedulers in different situations, and give some advices to how choose a suitable scheduler in different situations. We investigated and analyzed the main big data projects, with the features and demands of the upper business and app, designed and implemented a big data processing and management platform. This platform consists of five sub system, group monitor sub system, file management sub system, resource and application management platform, API and Web Service sub system,visual Web platform. Finally, we tested the functions of the whole system and deployed the platform and it has supported some third-party apps.The result of this paper is to have a deep studying on the scheduling of Yarn and build a united big data processing and management platform,and investigate and study on some key technology, supported the related platforms and needs efficiently. Except this,also can give some advices to resource management, file management, monitoring nodes and visualization of the platform.