Malicious Traffic Detection And Adversarial Examples Generation Method Based On Reinforcement Learning

The rapid development of information technology and the Internet is profoundly affecting people's production and life style,and the society is gradually moving towards an information society.According to the 47th”Statistical Report on China's Internet Development Status”^[1],by the end of 2020,the Internet penetration rate of China has reached 70.4%,and people's production and life are closely related to the Internet.However,while enjoying the convenience brought by informatization and intelligentization,the security problems in cyberspace cannot be neglected.The statistics in the”Summary of China's Internet Network Security Situation in2019”^[2]show that about 44.4%of Internet users encountered network security problems in the process of surfing the Internet in 2019,and the network security problems have become more serious.The stability of Cyberspace Security is of great significance,and the security of cyberspace is inseparable from the study of”attack”and”defense”in cyberspace.As a major attack vector in cyberspace,the research on malicious traffic from both”attack”and”defense”has far-reaching significance for safeguarding cyberspace security.From the perspective of defense,although malicious traffic detection technology based on supervised learning methods has shown its su-perior capabilities.However,the features used in supervised learning algorithms are not easy to determine,and it is impractical to exhaust the combination of features and learning algorithms.How to automatically select feature subsets effectively and determine the learning algorithm to build an efficient malicious traffic detector is a key issue that needs to be solved.At the same time,in-depth research on the attack level can also effectively promote the improvement of defense technology.Therefore,considering the insufficient sensitivity of supervised learning algorithms to adversarial samples,it is necessary to study the method of generating malicious traffic adversarial samples from the perspective of attack.How to generate malicious traffic adversarial samples for detector pre-training to deal with potential adversarial attacks and then achieve active defense is also a problem worthy of research.Toward this end,in view of the shortcomings of existing malicious traffic detection tech-nologies and the insufficient robustness of malicious traffic detectors in adversarial scenario,this paper proposes a scheme for constructing malicious traffic detectors based on reinforce-ment learning methods and a method for generating malicious traffic adversarial samples.In this paper,the research on malicious traffic detection and adversarial sample generation can not only improve the performance of malicious traffic detector in non-adversarial scenario,but also improve its robustness in adversarial scenario.The main contributions of this article are as follows:(1)Aiming at the problem that the feature selection and training algorithm are difficult to determine in the process of constructing malicious traffic detectors based on machine learning method,this paper proposes a malicious traffic detector generation scheme DQ-MDCS based on reinforcement learning DQN algorithm.The construction process of the malicious traffic de-tector is formalized as a decision-making problem,and the state space,action space and reward function in the reinforcement learning algorithm are modeled.Through continuous training of the agent,it is possible to select low-redundant traffic features and determine learning algorithm without human intervention,and finally used for the training and construction of malicious traf-fic detector.Based on the Test+dataset of NSL-KDD,the malicious traffic detector constructed by DQ-MDCS can effectively identify malicious traffic,and its detection accuracy rate can reach88%.At the same time,the generated malicious traffic detector can quickly identify malicious traffic,and the volume capacity of the detector is reduced to 56Kb.Compared with malicious traffic detectors based on deep learning method,the detector generated by DQ-MDCS have faster response time and lighter weight,which can adapt to real world scenario.(2)Aiming at the disadvantage that the malicious traffic detector based on supervised learn-ing method is not robust to adversarial sample,this paper proposes a method A3C-MTAG for generating malicious traffic adversarial samples based on the reinforcement learning A3C al-gorithm.This method aims to generate malicious traffic adversarial samples for the training of malicious traffic detectors,improve the stability of the detectors in adversarial scenarios,and realize active defense.By simulating the process of interaction between malicious traffic and malicious traffic detector,the agent is trained to take actions on the original malicious traffic samples and add subtle perturbation to generate malicious traffic adversarial samples.Experi-mental results show that,compared with malicious traffic detectors that have not been trained with adversarial samples,the detection accuracy of malicious traffic detectors after retraining can be increased by up to nearly 50%,which further improves the defense performance.At the same time,based on the corresponding research results,feasible defensive measures for mali-cious traffic detectors based on supervised learning method in adversarial scenario are given.