| With the rapid development of the Internet,this had resulted in a massive data more than ever before.The implementation of the traditional data warehouse technology is gradually losing its ability to satisfy the ever increasing requirement,thus different methodology of collecting,processing and utilizing data needs to be developed in order to keep up with today?s application.This article aimed for the breakthrough of the new technology by adapting framework of Hadoop system which considered the collaboration between HDFS and MapReduce for the purpose of improving efficiency on data storage plus its applications.The system also utilizes the cluster competitive advantage and had enabled itself quicken the timeframe for concurrent processing,thus supplemented the lack of ability that traditional data warehouse provides.Further to points above,this research article also break down on the pros and cons of the traditional data warehouse and Hadoop system respectively then came up with collaborative method by extract the advantages from both that enhanced the efficiency for data application and apply this to the mobile advertising platform;To elaborate this,the PostgreSQL technology from traditional data warehouse is used to process the structural tabular data which does not require massive storage and adapted Hive technology from Hadoop system to process the rather larger semi-structured data like web log data.The practicality of such syndicated technology was tested within five different functional segments.They are namely data environment deployment,data synchronization,customization of statistical analyses of task,task scheduling and query reporting.From the experimental results,the enhancements were found on below1.Synchronization of heterogeneous data.Using the Kettle tool from ETL to extract,filter,transform and delete the incongruent data within MySQL database from multiple servers,then apply service logic to load the data into designated PostgreSQL database.2.Efficiency in processing the data within massive data warehouse.Collaboratively using Hive application to process massive web log data as it is built on top of Hadoop and will automatically run the its tasks using MapReduce,thus achieving the purpose of processing data simultaneously with Hadoop clusters that resulted in improvement of efficiency.3.Improved efficiency on data exchange between PostgreSQL and Hadoop cluster at a more cost effective manner.Adapting Sqoop as crossover tool for data transmission which was specifically designed for massive data exchange.Sqoop breaks down the related data into clusters and creates Hadoop task to process each segment.This dramatically improved the efficiency of data exchange between PostgreSQL and Hadoop clusters.4.Access authorization and data security.Except for the direct access of system administrator,only granted access for the end users will be by the querying portal from the mobile advertising platform website.Based on their respective individual needs,system would return applicable data that are sensitive to them only.Design of such system not only maximized the data manipulation but also increased the security of the sensitive data.This article takes into account the increasing needs for a more effective data processing system that is no longer satisfied by traditional data warehouse technology.The research examined the benefits of existing data warehouse system and applied Hadoop technology synergistically.Throughout the proper design and implementation of such data analysis system within designated mobile advertising platform,both functional and performance tests shown great improvement on function modules and also systematic functionality.This further proves the practicality of this synergetic technology to replace the deficient traditional data warehouse system. |
PDF Full Text Request