Design And Implementation Of Log Analysis Subsystem Based On An Operator System

The rapid development of the Internet has brought a huge amount of log data to enterprises.Data growth brought the higher requirements of calculation and storage,meanwhile,in it there is great value.The user behavior characteristics and fundamental demand contained in these data is an important source of information for enterprises to grasp market trends.However,the computing power and storage capacity encountered the bottlenecks when processing the massive log data with traditional technology.The scalability and computing speed are greatly limited due to the characteristics of traditional technique,which performs the task with a single-server and can lead to the risk of data loss.Therefore,this thesis designed and implemented the log analysis system based on the distributed architecture.First of all,this thesis introduces the background of the subject,and summarizes the related technologies of log analysis system,which includes Hadoop,Flume and Redis.And then according to the system design features,this thesis makes the functional and non-functional requirements analysis with use cases diagram.According to the requirement analysis,this thesis divide the system into four modules,including data collection module,data preprocessing module,data processing module and data representation module.Based on the architecture design,this thesis makes a detailed introduction about the specific design and implementation of each module combined with sequent diagram and class implementation.At last,it tests this system and shows the effect diagrams.The test results indicate that this system has achieved our goals.The log analysis system based on the distributed architecture is able to overcome the shortcomings of traditional technology,including slow calculation and small storage space.It can splits the large files into each node of the distributed file system server,dispersing the stress of server and providing good backups and fault-tolerant mechanisms.At present the log analysis system has been usered in a company.