Design And Implementation Of A Multi-online Big Data Management System

As human society from the IT era which control as a starting point towards the DT era which for the purpose of productivity, more and more enterprises realize the huge value behind the data.They invest a lot of money and talent to promote the construction of big data infrastructure.This project is developed by college of computer of Huazhong university of science and technology and entrusted by some mechanical and electronic enterprise.This System needs to complete the acquisition, transport, storage, indexing and retrieval task for the huge amounts of data.The Article abstracts the general problem of each module needs to solve based on the system and construct a big data management system model suitable for traditional manufacturing.Firstly, the paper introduces the theoretical basis and key technology related to system design, which includes a non-blocking communication, load balancing technology, the ZooKeeper and HBase, etc.Then the paper has carried on the demand analysis to system and has been clear about the design goal of the system and the function need to be implemented.It introduces the system architecture and business process as a whole and then partition the system according to the function module, which includes the GPRS module, the server-side data receiving module, the server-side data parsing module, soft load center module, data storage module and data query module.Then the paper introduces the scheme of realization of each module.It uses non-blocking IO for data transmission module, uses ZooKeeper building load balancing module, uses distributed database HBase building data storage and query module.Finally, it completed the test work related to system.The system designed by the paper is fully functional, performance, easy deployment, and simplicity of use.More importantly, it meets the needs of the enterprise managing massive data.