Research And Implementation Of Efficient WEB Container Log Processing System Based On Spark

In the log of web container,it contains the behavior information that users access to the Internet.By analyzing the log data,we can understand and judge the user's behavior.At present,Spark based log analysis system is widely used in most enterprises.However,due to their own reasons and Spark log data skew and other reasons,in many cases,the system has low efficiency,even to the normal operation of the phenomenon,this paper puts forward optimization scheme,aimed at avoiding system failure phenomenon and improve the operation efficiency of the system.This system uses Spark to analyze the log data of electronic commerce website,and through the data preprocessing algorithm and data transmission technology research,and combined with the log analysis business,design the optimization scheme.The data preprocessing algorithm in the data processing of the calculation stage "for the calculation of the" two stage",the log data more evenly distributed to all nodes in the cluster,reduce the amount of data processing and computing tasks to improve the degree of parallelism calculation,so as to enhance the overall efficiency of data analysis.In a highly efficient data transmission technology,this paper adopts the pipeline communication technology,reduce the number of connections,reducing the amount of business time,compared with the traditional HTTP protocol,greatly enhance the efficiency of data transmission.Finally,after testing,through the optimization analysis of job execution failure probability is greatly reduced,the efficiency of the whole system has been nearly 15%increase,the results show that the performance of this system can effectively optimize the scheme.