Design And Implementation Of Web Log Analysis System Based On Storm

Under the Internet age.People benefit from the various types of Websites and applications,at the same time they make their contributions to the web log in anytime.These logs record the footprints of people browsing on the site,and also record plenty of information with much potential value.Real-time analysing web log not only helps users get more humane services,but also conducive to the site itself to improve operation and maintenance.Today,The advent of massive data real-time computing framework Storm has made it possible for large-scale real-time web log analysis.Based on Storm,aiming at the application scenarios and technical problems in Web log analysis,this paper designs and implements a set of high performance and low coupling combination system,which provides a practical solution for real-time analysis of Web logs.The work of this paper includes the following aspects:1)First described the background and significance of this research topic,and then outlined the relevant technologies involved in the system,and finally focused on the system's core computing tools Storm which is used for real-time computing.2)The system requirements are analyzed,the overall architecture is proposed,and the most appropriate tools are matched for each module designed in the architecture,which makes the system have the characteristics of low coupling,easy expansion and high availability.The system solves a series of problems such as real-time collection of Web logs,asynchronous reading,real-time computing,storage display,real-time monitoring and so on.This paper deeply integrates each function module so that they can operate normally under the overall framework.3)The process of computing web log through Storm is designed.The analysis process includes the practice of K-means++ algorithm and sliding window in Storm.After the practice is programmed,the system is functionally verified,the result is displayed by the web page.4)Carried out the status monitoring of the running system and proposed some optimization methods through comparative analysis based on Storm.