Research On Dynamic Multi-channel Access Method Of Cognitive Radio Based On Reinforcement Learning

Spectrum access is a very important part of cognitive radio technology,which requires access to idle spectrum and utilize it without interfering the use of authorized users according to the results of spectrum sensing.Dynamic channel access can dynamically adjust its own parameters in real time under the condition of time-varying channel occupancy state and access free spectrum resources.As an effective method to solve dynamic system problems,deep reinforcement learning is widely used in the field of wireless communication.Different from traditional methods,it does not need the prior information of the system environment.Deep reinforcement learning can constantly interacts with the system environment to obtain the corresponding rewards and penalties,and learns the dynamic characteristic information of the environment.At the same time,deep reinforcement learning can solve complex state space problems with high performance.The application of deep reinforcement learning in dynamic multi-channel access can enable unauthorized users to track,learn and update the idle channels,and intelligently select the channel detection set,so as to improve the utilization of spectrum and meet the needs of their own communication.In this paper,the problem of single user dynamic multi-channel access is classified as a partially observable Markov decision process,considering the limitations of the sensing ability of unauthorized users and the time-varying spectrum occupied by authorized users.In this paper,twin delayed deep deterministic policy gradient(TD3)is used to solve the problem of single user dynamic multi-channel access bases on this POMDP.Td3 adopts optimization methods such as double critical network and double delay update which can learn better sensing access strategy.Simulation results show that the algorithm is superior to deep q-network(DQN),double deep q-network(DDQN)and deep deterministic policy gradient(DDPG),and improves the access success rate of unauthorized users.In real cognitive radio systems,there are often multiple unauthorized users sharing channel resources.However,in previous work,unauthorized users use channel access method based on independent strategy,that is,they regard other unauthorized users as part of the environment,which violates the assumption of static environment and Markov property.Therefore,the independent decision-making reinforcement learning algorithm can not solve the problem of multi-user dynamic multi-channel access.In this paper,we model a multi-user dynamic multi-channel access problem satisfying the bandwidth constraints of multiple unauthorized users.Then,an algorithm based on multi-agent TD3(MATD3)is proposed,which adopts centralized training and distributed execution of sensing access decision,so as to reduce the non-static impact of the system.Simulation results show that the algorithm outperforms multi-agent DDPG(MADDPG),TD3 and DDQN algorithms with independent strategy,and improves the overall access success rate of unauthorized users in the whole cognitive radio system.