Research Of Massive Data Processing In The Vessel Monitoring System

Currently, the Vessel Monitoring System is developing along the direction of automation,intelligent and digital, which leads to the increasing in the amount of monitoring data and a reducedability to process the data, so the response performance of system is decreasing dramatically. Thus, inthe Vessel Monitoring System, how to query and obtain the required monitoring information from themassive monitoring data quickly and easily has been one of questions of great concern to the users.Hadoop platform is a free and open source cloud computing development platform which is widelyused and simple operation, Hadoop can improve the capability of large-scale data processing by theway of cluster. Therefore, this article brings the Hadoop platform to the realizing of Vessel MonitoringSystem, which focuses on the research of massive data processing base on Hadoop platform, theprimary research of paper as follows:(1) Giving the framework of Vessel Monitoring System base on Hadoop platform, thisframework is composed of the system application layer, the middleware layer and the data storagelayer, the middleware layer is the core layer of system.(2) Proposing a distributed query algorithm based on Hadoop platform. This paper adds the ideasof histogram to the Top-k query algorithm and proposes a more efficient ITPUT query algorithmwhich is put into use in the Hadoop platform based on the concepts and structure related to thedistributed query technology.(3) Designing a Job scheduling algorithm based on the Hadoop platform. This paper designs aThree-Queue Job scheduling algorithm for the original lack of Job scheduling algorithms on theHadoop platform, which builds models from the judgment of fast and slow nodes, the judgment ofpriorities and the load balancing dynamically. The algorithm combines the MapReduce model to solvethe data local issues on massive data processing and reduce system consumption better.(4) Completing the design of Vessel Monitoring System and the development of main functionbased on Hadoop platform and describes the implementation details about the ITPUT distributedquery algorithm and the Three-Queue scheduling algorithm based on Hadoop platform. At last, thischapter gives some primary interfaces of system.