Research On The Adversarial Attack And Its Countermeasure Of Deep Reinforcement Learning

In recent years,Deep Reinforcement Learning(DRL)keeps developing and has been widely used in various fields.However,many studies indicate that DRL models are vulnerable to adversarial attacks.An attacker can mislead the decisions of DRL models by carefully perturbing the test samples.Therefore,security issues of DRL have become one of the focuses.However,current adversarial attacks and defense methods of DRL models have several drawbacks.Existing attack methods mainly follow the idea of the ones in classification tasks,resulting in attacks only targeting instant rewards without considering the impact on cumulative rewards.Existing defense methods either need to retrain the DRL model,which leads to high computational costs,or both tasks of adversarial sample detection and action correction rely on a single model,which leads to error accumulation and increased difficulty in model training.In order to solve the above problems,this paper has made the following contributions to adversarial attack and defense methods of DRL:(1)We propose an adversarial attack method of DRL with Static Reward Impact Map,in order to deeply explores the security leakage of DRL models.Our attack method first proposes a Static Reward Impact Map to measure the influence of input pixels on the cumulative rewards of DRL models.We use sliding windows to calculate the decrease of the cumulative rewards caused by the perturbation.Also,the correlation among pixels is considered.Then,the pixels with the highest impact value are selected to attack,in order to craft adversarial samples that can effectively reduce the cumulative rewards.The experimental results show that our attack method significantly improves the attack effect under both white-box and black-box attacks.(2)We propose a multi-model based DRL defense method,which improves the robustness of DRL models and lays a foundation for building a safe and reliable DRL system.We split the complex task of defense into two simple subtasks:adversarial sample detection and action correction.First,we propose an adversarial sample detection model based on the Correlation Feature Map.By extracting the correlation between the observations in one state,the change of the correlation caused by the adversarial attack is detected.Secondly,our proposed action correction model maps adversarial samples to clean actions.The experimental results show that our defense method can not only achieve better defense performance under a variety of adversarial attacks,but also greatly reduce the training time of the defense model.