Research Of Coal Mine Equipment Data Standardization And Cleaning Based On Hadoop

As the network information technology rapidly development and the increasingly, complicated mechanical equipment, Coal mine equipment stored data scale extend continuously, the amount of data from gigabytes to terabytes even petabytes, and the differences of data is becoming soaring enormous. For this kind of huge amounts of heterogeneous data of coal mine equipment, Traditional data processing architecture have not the ability to meet the requirements of data processing and mining depth. Yet, Cloud computing technology can achieve efficiently to handle these problems.According to the actual situation and demand of the existing system of coal mine machinery, analysis of cloud computing technology used wide effectively in industrial production. Set up equipment of coal mine data processing system based on the Hadoop platform, using the platform mass storage HDFS graphs and distributed computing technology. The level of these massive amounts of data processing, analysis and deep mining achieve. This paper first analysis comprehensive the coal mining equipment system, aiming at the problem which coal mine equipment mass heterogeneous format data needs to be standardized description, designed with XML data format diversification as an application of the huge amounts of data standards. Through metadata extraction and relationships mapping model to implement the diversification of huge amount data, fulfill the heterogeneous data description standardization of coal mine equipment; Then under the laboratory environment, set up a platform for the Linux cluster and Hadoop cluster. As data processing equipment of coal mine system experimental platform, analyzed in detail using HDFS graphs and distributed computing technology; Finally extracted the characteristic value of coal mining equipment vibration data and the whole process of design to clean abnormal points. Using experimental verification algorithm can process effectively to the coal mine equipment vibration data. on the Hadoop platformBy testing the performance of the whole system platform, carried on the detailed analysis, the system run normal and efficient. This article, based on the coal mine equipment of Hadoop data processing system,can effectively adapted in coal mine machinery equipment applications maintenance, provides a good technical support. for the coal mining enterprises to analyze...