The Design And Implementation Of The Log Audit System Based On The Big Data Platform Named Spark

With the development of computer and network technology, network security has gradually become a global problem. Network security is threatened by the following four aspects:forgery, modification, eavesdropping and interruption. In order to improve the security of the network, security threats are usually used in the system, such as encryption, authentication, authorization and audit. Audit is to track the customer’s access to the content and access methods and then find the system’s security risks, and thus develop the corresponding security measures to make up for the system’s security vulnerabilities.Log audit system is a kind of auditing tool widely used in computer and network. With the increasing complexity of enterprise network architecture, log audit system faces huge amount of data, data storage format and collection methods and data processing real-time diversity and many other issues. Big Data technology 4V features, Volume (large), Velocity (high speed), Variety (diversity), Value (value), show that the use of big data technology log audit system is bound to become the next generation of audit system development trend. Through in-depth analysis of large data Spark technology and log audit system, audit log Spark technology-based systems for a detailed design and implementation.In the requirements analysis phase, the requirements of the overall overview of the system and the development of the objectives and problems to be solved. Using the use case diagram and other UML tools in accordance with the functional and non functional division of the system requirements are described in detail. The functional requirement analysis of centralized management platform, alarm, query, statistics, audit investigations, log collection of the use case description, non functional requirements are given in the constrained project as well as the development and project operation period of quality requirements.In the system architecture part according to the system requirement analysis, the design objectives and principles are given and the physical structure and the whole structure of the system are also given. In which the physical structure of the system is located in the physical environment, and the system structure of the system includes the complete function module of the whole system.On the basis of the detailed design on the part of the job description of the contents of the specific implementation and testing. For example Flume acquisition module configuration file, XML standardization files, code and system operation analysis module main functions and the like. Finally, based on the application of big data platform Spark log audit system is briefly introduced, and a summary of the development group Spark Big Data platform log audit system in the future.On the basis of the detailed design, it describes the specific implementation and testing of some of the work content. For example, Flume acquisition module configuration file, the XML specification document, analyzes the module main function code and the system operation condition and so on. Finally, based on the application of the Spark data platform log audit system, the paper makes a brief introduction to the development of the Spark data platform log audit system.This paper is based on the Spark data platform log audit system, which has the advantages of fast speed, good scalability, large amount of processing data, and has achieved good results in the practical operation. And in this paper, the system has been accepted as a prototype for the invention patent. Text is a useful attempt to analyze and research the log audit system.