The Design And Implementation Of TDH's Data Security Audit System

In recent years,big data technology has gradually matured,and more and more industries use big data technology to improve the original business system of the enterprise and thereby enhance the core competitiveness of the enterprise.Transwarp Data Hub(TDH)is an enterprise-level big data platform developed by Transwarp Information Technology(Shanghai)Co.,Ltd.,designed to build enterpriselevel data warehouses and data marts for various industries,including commercial banks and other financial institutions.Because Chinese policies and regulations have strict regulatory requirements for financial institutions such as commercial banks,commercial banks need to monitor platform users when using big data platforms,so that all operations of platform users can be traced.The TDH's Data Security Audit System is designed and developed under this background.It's core goal is to monitor the data operations of users using the TDH platform in real time.The system uses the method of collecting TDH's logs to obtain the original data of the operation behavior of TDH users,and then uses this as a basis to restore the data operation behavior of TDH users.At the same time,the system will detect the operation behavior of TDH users according to the expert rules set by security auditors.When the operation behavior of TDH users is a violation operation,it is necessary to trigger an alarm and notify the personnel in related positions.The system is divided into two applications: Runtime and Configuration.Runtime monitors the data operation behavior of TDH users in real time;Configuration provides a security audit personnel management terminal page to control the process of each link in Runtime and view the details of related events.The TDH's Data Security Audit System uses the Filebeat to collect the logs of the related components of the TDH platform,and the Kafka message queue to transmit the log data.Log data processing and TDH user operation behavior monitoring and other functions are implemented using Spring Boot to build applications.The system storage uses Mysql relational database and Redis cache,where Mysql is used to save persistent information,and Redis is used as a cache to improve the operating efficiency of the system.TDH's Data Security Audit System proposes a data security audit solution for the big data platform of commercial banks.This system not only meets the needs of commercial banks for security audits,but also has the characteristics of big data platforms that are basically non-aware and responsive.