Design And Implementation Of Realtime Log Analysis And Information Management System Based On Spark Streaming

With the explosive growth of Internet traffic,the continuous emergence of network applications makes the network environment become more and more complex.At the same time,it has brought much more difficulties to network security analysis.Compared with past network attacks,nowaday ones are more concealed,complex,difficult to detect,which makes it much less preventable.Also,a large number of targeted malicious network attacks are becoming more and more common.Internet companies operate lots of product servers to provide users with rich network services.In this case,once an attack of huge flow data stream causes server crash or a decline in performance in all aspects of the product server,there is no way to continue to provide efficient and stable network services.Therefore,Internet companies need a system that can detect DDo S(Distributed Denial of Service)attacks in real time to provide security detection for the product server.When a DDo S attack occurs,the response solution will be proposed timely to reduce the damage.In this paper,aiming at analyzing logs to detect DDo S attacks accurately in real time,a method based on Spark Streaming is proposed.At the same time,a DDo S detection method based on information entropy combined with clustering is proposed to detect whether a DDo S attack has occurred in real time.If an attack occurs,a warning message will be sent to customers.In this paper,the requirement analysis of real time log analysis system based on Spark Streaming is completed,and the system is designed and implemented,which provides a visual system for detecting DDo S.In terms of system design and implementation,this paper proposes an open source real-time computing platform design solution based on big data technology.The design consists of five levels,each of which provides services for a function entity upwards.This paper mainly realizes the functions of customer information management,server management,customer service management,server status monitoring,customer service alarm,customer service alarm and resource information management.Among the above functions,the most important modules are customer service alarm and resource information management.Resource information management module can provide services of cleaning server configuration and task distribution,and customer service alarm module can send messages to customers accurately in time.