Design And Implementation Of Real-time Dependence-preserving Log Compaction System

Recently,the concern of APT(Advanced Persistent Threat)attacking large institutions has been increasing.When detecting APT attacks,forensic analysis is invoked to analyze dependencies between operating system events,find the intrusion point and determine the impact.Currently,in large organizations,petabyte-level event logs are required for forensic analysis,which not only brings huge storage overhead but also sharply increases the computation time.Therefore,how to compress the log data without affecting the results of forensic analysis becomes an important and urgent research topic.Existing research lack practicability.They are only applicable to either offline largescale log data or specific event types,which cannot meet the requirements of universality and real-time performance.To address the above limitations:1)We propose two real-time compression algorithms: global dependencepreserving and attack-related semantics-preserving.The former reduces events that do not affect global dependencies.The latter reduces events that are not related to the attack by analyzing the event context,given that most forensic analysis targets at restoring attack chain only.Both algorithms are practical in abundant scenarios.They are operating system independent due to the implementation at the log level without internal program analysis.Besides,they can handle various log types such as files and networks.2)We implement a prototype on Windows to verify the correctness,generality,and efficiency of the proposed compression algorithms.The prototype contains an event collector for collecting multiple types of events in real-time,the proposed data compression strategies,and the result data storage using a high-performance,high compression ratio database.3)Our experiments show that the proposed compression algorithms reach an ideal compression rate and overhead.Without affecting the results of forensic analysis,the system's compression ratio for the number of events is as high as 12.4 to 25.9,and the CPU single-core occupancy rate is 5% when running in real-time.In summary,we propose real-time log data compression algorithms with a high compression rate,low overhead,and generality for multiple types of events.Compare to existing work,it does better for deployments in large-scale cluster environments.