Research And Application Of Telecom Big Data Processing Based On Hadoop

With the progress of science and technology,communications equipment has also been a great development,followed by the communication equipment is a lot of data.Most of these data have been unstructured form of presentation.People need to quickly query the data and accurate analysis to obtain the necessary information.But the traditional relational database of these data is difficult to quickly deal with.There is an urgent need for a new data analysis system to meet current needs.In addition,the traditional relational database in dealing with large amounts of data,you need to deploy a lot of high-configuration computer,an increase of cost.Hadoop distributed system can be deployed in a large number of inexpensive computers,the hardware requirements are not high.Hadoop has two core elements: MapReduce and HDFS.MapReduce can be summarized as the data in Hadoop in the running process,the first data in the Map function in the form of key-value split,and then the Reduce function key to the same key-value reorganization.HDFS is a distributed file system of Hadoop,which is different from many distributed file systems nowadays.It is a good way to deal with the hardware failure problem,detect the problem hardware and return to normal.Hadoop is an open source implementation of Google's big data,so this paper analyzes Google's big data theory and studies the running process and characteristics of GFS,MapReduce and BigTable.The Hadoop HDFS,MapReduce and HBase operation of the process and the basic principles of doing a thorough study.This paper analyzes the shortage of traditional relational database in querying massive data,designs and implements the query model of Hbase communication data based on Hadoop and its application.According to the different amount of data of the traditional database query based on the running time of Hadoop and Hbase database query based on the comparison analysis of the causes;and a comparative study of Hadoop in different task granularity running time,has made the corresponding analysis.