The Design And Implementation Of Log Parsing System Base On Hadoop

With the popularization of information intellectualization technology,the enterprise website is gradually integrated with E-commerce,enterprise management and other functions by traditional enterprise propaganda,which has become an important production tool.Enterprise Web site in the content management,the use of BigData technology to discover new business models and user portraits,for enterprises to bring huge profits;relative to the site content management,log management lags behind,currently usually with the use of Full-text search tools or manual search methods to manage,There are the following problems:First,the analysis is not comprehensive can not timely feedback dynamic information;the second is that the log analysis is biased to the site fault location,Web site optimization and information security data analysis is weak;three is unable to deal with the massive log timely analysis and results of real-time query,four is Visual module personalization and two Manage for current log.Aiming at the problems existing in current log management,this paper designs and develops a website log parsing system based on Hadoop platform,in which Hadoop platform provides off-line calculation and real-time query capability of petabyte log data;the application system adopts mainstream JavaEE architecture to design and develop,extend and maintain more easily;at the same time provides web application function based on HTML5 to facilitate parse logs and provide more flexible analysis of the results for the website data,and provide more flexible analysis of the data for the data analysis,and provide more customized data analysis of the data for the data analysis of the second dimension of the website.The main work of this paper is divided into two parts,part based on the Hadoop architecture of the Web site log resolution system design,the other part is based on the Hadoop architecture of the Web site log resolution system.In the design part,the paper expounds the requirements and business process of the log parsing system,puts forward the system architecture,and on this basis,designs the system in detail,including the MapReduce calculation model and the HBase storage model.In the implementation section,the application of virtualization software,the configuration and development of Hadoop cluster and hbase cluster are expounded in detail,and the integration of Hadoop and HBase is described in detail,and the application system of log parsing is coded and tested and deployed.The Web site log parsing system based on Hadoop platform provides a BigData processing solution for enterprise website Log Management,which has important application value and foreground.