Research On Deep Reinforcement Learning Methods For Dynamic Spectrum Acces

The continuous evolution of the information technology has stimulated the growth of mobile applications and devices,intensifying the competition for spectrum resources and limiting the development and innovation of new applications.To address this issue,the industry has proposed Dynamic Spectrum Access(DSA)as an intelligent spectrum management technology to efficiently utilize limited spectrum resources and support more users and applications.To attain more advanced,secure,and high-quality communication services,as well as ensuring the sustainable development of wireless communication,this article aims to address the issue of spectrum sharing among multiple users and tasks.Specifically,it investigates the application of Deep Reinforcement Learning(DRL)models for achieving secure and efficient spectrum access in various scenarios,such as imperfect spectrum sensing,complex communication services,and intelligent spectrum attacks.While the introduction of DRL technology for Dynamic Spectrum Access(DSA)yields numerous benefits,it also presents some new security concerns that are often overlooked.To guarantee the secure utilization of electromagnetic spectrum,this study analyzes the security issues arising from the use of DRL methods for DSA from an anti-reconnaissance standpoint,offering insights for designing more resilient intelligent DSA methods.The primary research contributions and innovations of this article encompass three aspects:Firstly,a DRL-based distributed multi-user DSA algorithm is proposed to address the spectrum access problem of secondary users in scenarios with imperfect spectrum sensing.Each secondary user acts as an agent and makes a distributed spectrum access decision based solely on its own current and past spectrum sensing results in the presence of sensing errors.The algorithm is improved through mechanisms such as priority experience replay,dueling network architecture,and multi-step updates,enabling secondary users to fully explore the environment and utilize effective historical experience to quickly adapt to the dynamically changing spectrum environment and learn access policies.Simulation results show that the proposed algorithm improves the average access success rate by approximately 20% and reduces the average collision probability between secondary users by approximately 10% compared to existing algorithms.Secondly,a DRL-based multi-group spectrum access algorithm is proposed to address the spectrum access problem of multiple groups in complex communication scenarios.To mitigate the potential chaos arising from dispersed spectrum access of multiple user nodes,the algorithm introduces the first group-based approach to spectrum access.Users are categorized into multiple groups based on the type of communication services being processed,with each group acting as an intelligent agent that dynamically adjusts the group’s spectrum access policy in real-time based on the spectrum sharing situation and the processing status of users within the group.The proposed distributed DRL algorithm is further improved by incorporating a priority experience replay mechanism and a double network structure.Simulation results demonstrate that the proposed algorithm reduces the average processing time of communication services by approximately 17% with a fast convergence rate,outperforming existing algorithms.Thirdly,to improve the robustness of the model in the spectrum intelligent attack scenario,a non-invasive backdoor attack method with low-cost is proposed against DSA-oriented DRL models in cognitive wireless networks.The attacker monitors the wireless channels to select backdoor triggers,and generates backdoor samples into the experience pool of a secondary user’s DRL model.Then,the trigger can be implanted into the DRL model during the training phase.The attacker actively sends signals to activate the triggers in the DRL model during the inference phase,inducing secondary users to take the actions set by the attacker,thereby reducing their success rate of channel access.A series of simulation shows that the proposed backdoor attack method can reduce the attack cost by 20%～30% while achieving above 90%attack success rate,and apply to three different DRL models.