Research On Malicious Code Generation Countermeasure Based On Deep Reinforcement Learning

JavaScript malicious code detection has become one of the hot issues in the field of Web application security.Currently,the fusion of machine learning or deep learning methods applied to malicious code detection has become a hot research topic.However,it has been shown that malicious code detection models based on machine learning or deep learning are highly vulnerable to adversarial attacks: an attacker generates adversarial samples by obfuscating malicious code,etc.,and performs an adversarial attack on the malicious code detection model so that the detection model produces a specific output that matches the attacker’s expectation.In this thesis,we conduct research on JavaScript malicious code adversarial samples to explore the application of reinforcement learning DQN（Deep Q Network,DQN）in generating adversarial samples in the malicious code domain,and investigate the role of adversarial samples generated based on reinforcement learning DQN in malicious code detection models from both anti-attack and defense aspects.The main work of this thesis is as follows:To address the problems that existing methods for generating adversarial samples of malicious code rely on human control,and it is difficult to maintain the original functionality of the code and low generation efficiency,we propose a reinforcement learning-based model for automated generation of adversarial samples of JavaScript malicious code,DQN-CNN,which can attack the CNN model by automatically adjusting the model’s actions without being informed of the specific internal structure of the CNN model to obtain adversarial samples that bypass the detection of the CNN model.to obtain an adversarial sample that bypasses the detection of the CNN model.The experimental results show that the adversarial samples generated by the DQN-CNN model can successfully bypass the detection of the JavaScript malicious code detection model,in which the recognition rate of the LR detection model is reduced from 95.6% to 1.46%,and the migration success rate of the attacks on machine learning-based malicious code detection models such as RF,DT,SVM,and GBDT exceeds 92.12%.To address the problem that machine learning or deep learning based malicious code detection models are weak against attacks,we propose an adversarial sample-based JavaScript malicious code adversarial defense model ADNet.ADNet consists of an adversarial sample detection module and a malicious code detection module.The samples to be detected are first distinguished as adversarial samples or original samples by the adversarial sample detection module,and then the malicious code detection module distinguishes the original samples as malicious code or benign code.The experimental results show that ADNet has certain defense capability under the adversarial attack based on reinforcement learning DQN,and the success rate of generating adversarial samples by ADNet and DQN forming a new adversarial model DQN-ADNet decreases significantly from 45.7% to 13.6%,which proves that the anti-attack capability of ADNet is significantly improved.Finally,a prototype JavaScript malicious code detection system is designed and implemented to detect the presence of malicious code in web address URLs and to detect whether uploaded files are malicious code.