Design And Implementation Of Massive Log Data Quasi-Real-Time Query System Based On Hadoop

With the development of science and technology,we have stepped into an era of information explosion.At the same time,the data generated worldwide is also growing at a geometrically rapid ratio.Humanity has entered the era of big data.Companies in various industries around the world have also quickly recognized the importance of data.Data has also been incorporated into the scope of assets by various companies,resulting in the concept of data assets.Data assets have become the company's core competitiveness.How to build a low-cost,high-efficiency big data ecosystem around the company's big data information has become an important task for all companies.The purpose of this paper was to build a big data analysis prototype system based on the user's mobile Internet log,which is one of the telecom operators' operating domain data.This prototype system designed the architecture of Hadoop-based quasi-real-time query system for massive log data and implements and validates the main modules.Its designing principle was to build a high-performance,complete,open platform.The relevant technologies for system functions and application characteristics,for example,the principles and characteristics of Hadoop ecosystem such as HDFS,MapReduce and Hbase,were also studied and discussed in detail in this paper.According to the system implementation and final test verification results,it shows that the Hadoop ecosystem technology can support the construction of big data analysis platform and complete the data collection,analysis and processing to the final application process support.The Hadoop-based massive log data quasi-real-time query system designed and implemented in this paper has certain practicability,strong system versatility and scalability.