Design And Implementation Of Distributed Auditing System Based On OLAP

With the arrival of the era of cloud computing, massive structured data audit becomes hot issues along with the electronic commerce, the Internet and other emerging areas of the Internet and the gradual, the historical data of the audit is aimed at the threat or potential threat detection and prevention through the log files, and the recorded data, and the difficulty is how to realize the collection of data query extemporaneous, online and interactive. In the field of data audit, audit of data warehouse mainly has two methods, namely the accounts table reduction method and basic data verification method. To solve the above problems, this paper will use the data cube lattice model and distributed technology to design a distributed audit system in a high efficiency way. In order to solve the above problems, this paper has the following contributions::(1) By comparing the advantages and disadvantages of all kinds of distributed technology, choose out data distributed framework which is fit for large data audit and transplant the general audit data from traditional database migration to Hadoop MapReduce parallel framework Spark.(2) Compress aggregated data on a distributed platform. Firstly analyze the characteristics of distributed storage platform, and then the closed cube technology is used to compress the aggregated data, at last store the compressed data in the HDFS in the form of closed cubes for audit query.(3) After obtaining closed cube, query the cube in distributed system combined with the characteristics of the spark programming model RDD.For the sake of the characteristics of distributed system and query efficiency, the new query is in a different way.(4) Integrating technology mentioned before, this paper designs a kind of appropriate audit architecture based on distributed system. To solve the following two questions:l)the massive data storage and data aggregation query.2) the application of audit rules in big data platform. Combine closed cube technology, distributed technology and audit technology to design a set of practical data audit system, and verify its function through an experiment.