Design And Implementation Of Log Mining System Based On Cloud Computing

With the development of social informatizing, information quantity inevitablyincreases rapidly. An important device of dealing with the storage and calculation ofaccompanying massive data effectively is cloud computing. Log mining system based oncloud computing takes advantage of cloud computing method and analyses uses’ logs ofa search engine. Afterwards, statistical data from all dimensions and the platform of datamining have been achieved by complex multi-dimensional analysis and cross-overanalysis. The obtained thirteen specific flow indices of search engine website can be usedfor the observation of the changes of website traffic, lay foundation of systemcustomization and products, business and strategic decision.This article, according to the software engineering method, first analyzes theservice and requirements analysis of the system and find out four functional requirements.Then it gives the overall design of the system. It presents the system flow and framework,and puts forward the system is divided into three modules: log pretreatment, log analysisand statistics, on-line analytical processing. In the part of system designing, the paperdetailly analyzes the designing of each data model, the XML configuration, thedimension and fact tables, and the rules of dimensional and cross-over analysis. Then itproposes the realizations of the log loading and ETL process, the realizations of thedimension parser and algorithms of indices, and the solution of data warehouse andmulti-dimensional analysis, especially gives the detailed implementation process forindex algorithm based on Hadoop.This text spells out development instance of log mining system based on cloudcomputing by means of the application of related knowledge of cloud computingtechnology, Map/Reduce processing framework, and OLAP of data warehouse.