Anti-AntiVirus Technology Research For PE Malicious Code

With the rapid development of information technology,the harm caused by malicious code has been increasing year by year,posing a threat to the information security of individuals,enterprises,and countries.The detection and evasion of malicious code is an ongoing game of confrontation,and research on detection techniques alone cannot keep up with the everchanging state of malicious code.Evasion technology can help malware analysts better understand the latest evasion methods and techniques of malware from the attacker’s perspective,enhance their understanding and analysis capabilities of malicious software,and better respond to constantly changing security threats.Currently,the Windows remains the most widely used operating system in the market,and it is also a major target for malicious code attacks.Therefore,this thesis focuses on research on evasion technology for Windows system PE files,which can help optimize the detection strategy of antivirus software.The main contributions of this thesis are as follows:1.A GAN-LSTM evasion model based on Generative Adversarial Networks and Long Short-Term Memory Recurrent Neural Networks is proposed and implemented.We extract the API call sequence of the malicious software as its behavioral feature for adversarial sample generation,where the generator uses a Seq2 Seq model and the substitute detector uses a Bi LSTM-Attention model to retain the most important context information in the sequence.The sliding window mechanism is used to maximize the retention of the API sequence feature of the malicious software.We design multiple different networks of black box detectors to simulate the machine learning and deep learning detection engines of antivirus software and use the substitute detector to fit the decision boundary of the black box and complete the black box attack.Our solution can reduce the detection rate to below 0.80% with an API attack cost of less than 46% and within 18 queries to the black box.We also verify theattack effect on heterogeneous models,indicating that the attack has transferability to other neural network models.2.A behavior detection bypass model based on hybrid strategies is proposed and implemented.We delay the execution of the malicious code to first explore the target system information,and issue different payloads for further execution of malicious behavior based on a hybrid strategy that matches sandbox rules and malicious code loading methods.Our solution reduces the Virus Total detection rate to around 7% and passes the test on a real machine equipped with enterprise-level antivirus software.3.A multi-dimensional evasion bypass system named Bypass is implemented based on the above methods,which can output malicious software through a PE file,extract the API call sequence of the output file,and generate adversarial samples through the GAN-LSTM model.Referring to the adversarial sequence,we dynamically modify the API call flow of the malicious code using Hook technology and DLL injection technology to generate evasion PE files.The generated samples can evade traditional engine detection of most antivirus software and the generalized detection of machine learning engines for new viruses,thereby extending the evasion time limit.