Design Of Data Processing System Based On Hadoop

Nowadays, with the development of information technology, a large number ofdata carriers are applied by people, such as computers, mobile phones and varioussensing devices, which results in doubling the amount of data, and the data types areincreasingly complicated. The amount of data to be processed is about500TB one dayin Facebook. According to statistics, Up until2015the amount of data worldwide willreach8ZB, and every18months, the amount of data will be doubled. In whiche-commerce transaction data, social networks and mobile data terminal data representedthree leading unstructured data will account for90%of total global data, so the world isalready in the era of big data. The arrival of big data is a challenge for many companies,how to deal with large data restricts development of these enterprises, which hasbecome an important bottleneck.The appearance of the cloud computing provides the storage and processing of thedata with a reliable tool. The proceeding data is derived from the temperature sensor.Sensors collect data in real time so it can produce a very large amount of data. Thepresent widely used cloud platform-Hadoop is uesd as the data storage system, and thedistributed database-HBase is used to store data. Both has the very good extensibility,therefore they can meet the growing demands of the amount of data. In order tofacilitate users to manage and check the temperature data, the Java Web technology isused to design the visual management interface, allowing users to check the data via theWeb anywhere by useing the protable devices like the mobile phones, the tablets and soon.The practical operation results of the system shows that the data processing systemcan analysis and processing the sensor data very well. It provides a basic platform, andit can be applied to other systems, such as the transportation systems, the weathersystems and many other practical applications, so it has good scalability and applicationvalue.