Huge Amounts Of Sensing Data Management System Based On Hadoop

In recent years, along with the growing of the production or distribution data carrier, such as PC, mobile phones, car embedded devices, the number of global data is rapidly increasing, data types are also increasingly rich. According to the statistics by the year2012, global data amount will reach2.7ZB. The transaction data represented by electronic commerce, the interactive data represented by social network, the sensing data represented of mobile terminal these three dominant unstructured types of data will make up the total number of80%-90%, the growth is far higher than structured data, the global has full access to the big data age. In order to effectively manage and high efficiency analysis of the mass data, the traditional database system architecture has ragged, and it gradually cannot be competent for the current demand.Since2003, Google has announced three papers that have a far-reaching influence on later development of cloud computing:GFS, MapReduce and BigTable. Apache Hadoop open source project cloned the framework, introduced a Hadoop system, and realized its own HDFS, MapReduce and HBase. This system has been recognized and widely used by the academia and industry, it increasingly forms a system which contains the advantages of easy deployment, easy development, complete functions and good performance.In our study, we want to manage the data collected by sensors.Because the sensors will collect data in every certain time, it will produce a large number of sensor data. These data is half structured data and stored according to the records, therefore we decide to use a scalable and high concurrent Hadoop cloud platform. However Hadoop is suitable to write once and read many times, the real-time reading and writing support is bad, so we choose the HBase distributed database system for support of real-time reading and writing. This paper will use the combination of HBase and Hadoop to make up the management system of mass sensor data, realize the efficient management, and finally query the sensor data in time through the web interface.