Cross-Layer Design And Optimization Research For Cognitive Radio Networks Based On Reinforcement Learning

With the rapid development of information technology,the number of new communication devices is growing explosively,and the users' demand for communication quality and service experience is also increasing.In order to support more users and achieve higher data transmission rate,the requirement of bandwidth and spectrum occupancy is getting higher for communication services,which results in the serious shortage of spectrum resources.Cognitive Radio(CR)adopts dynamic spectrum access and greatly improves the spectrum utilization through multi-dimensional multiplexing in time domain,space domain and frequency domain,which fundamentally solves the problem of spectrum resource shortage.In order to enhance the robustness and flexibility of the system,the distributed architecture is generally adopted in Cognitive Radio Networks(CRN).Route selection of the network layer is closely related to the spectrum allocation of the Media Access Control(MAC)layer.It brings new challenges to the design of efficient dynamic routing and resource management.To overcome the shortage of prior information and the dynamic change of wireless environment as well as network topology,we focus on the cross-layer design for CRN based on reinforcement learning.It mainly contains the joint routing and resource management in the scenarios where the source nodes and relay nodes are heterogeneous,all of the nodes are homogeneous,expert nodes exist and there exist malicious users,respectively.The main work of this thesis is shown as follows:(1)Considering the lack of network prior information and the poor performance of traditional learning algorithms as the system state space is large,the single-agent deep reinforcement learning is applied to the cross-layer routing design of large-scale CRN in the network scenario where the source nodes and relay nodes are heterogeneous.The joint routing and resource management scheme based on Prioritized Memories Deep Q-Network is proposed.Firstly,the path responsibility rating is introduced.It can transform large action space into large state space and achieve the balance between end-to-end delay and system energy efficiency.Then the Prioritized Memories Deep Q-Network(PM-DQN)is presented,which erases the transitions with lower TD-error in the replay memory periodically.It reduces the memory occupancy and achieves the prioritized experience replay.Furthermore,aiming at the characteristics of the heterogeneous network,a single-agent framework based cross-layer routing protocol is designed and PM-DQN is applied to the joint routing and resource management.Simulation results show that the method effectively solves the cross-layer routing design of large-scale CRN without the prior information,and it achieves lower routing delay and higher energy efficiency while reducing memory occupancy.(2)Aiming at the network scenario where all of the nodes are homogeneous,the multi-agent learning strategy is applied to the joint routing and resource management.Two cross-layer routing protocols based on multi-agent reinforcement learning are proposed.Firstly,the conjectural multi-agent Q-learning based flat routing protocol is proposed.The single-hop responsibility rating is introduced to greatly reduce the action space of cross-layer design problem and achieve a tradeoff between single-hop latency and energy consumption.Then the cross-layer design problem is modeled as a quasi-cooperative stochastic game,and the Equal Reward Timeslots based Conjectural Multi-Agent Q-Learning(ERT-CMAQL)is presented to solve the Nash equilibrium of the game.In the proposed algorithm,experience replay is applied to the update of conjecture belief to break the correlations and enhance the data efficiency.Simulation results show that the scheme outperforms the traditional learning strategies in terms of learning speed,real-time performance and system robustness.When the density of nodes is particularly large in the network,a hierarchical routing protocol based on Energy Consumption Weight(ECW)clustering is proposed.Firstly,the concept of ECW is introduced and the ECW based greedy clustering algorithm is presented.It can minimize the energy consumption of intra-cluster communication.Then the Double Q-learning framework is applied to improve ERT-CMAQL,which is used to optimize routing and resource allocation for inter-cluster communication.Simulation results show that the packet transmission delay and energy consumption of the proposed scheme are much lower than those of the flat routing protocol.(3)In view of the applications which are sensitive to transmission delay and energy consumption,the apprenticeship learning strategy is applied to the cross-layer routing design in the network scenario where expert nodes exist.Two apprenticeship learning based joint routing and resource management schemes are proposed.For the newly generated data source in the network scenario where the source nodes and relay nodes are heterogeneous,a joint routing and resource management scheme based on prioritized memories apprenticeship learning is proposed.Firstly,the reinforcement path responsibility rating is introduced,which regulates transmit power efficiently with multi-level transition mechanism.Then the Prioritized Memories Deep Q-learning from Demonstrations(PM-DQf D)is presented,which periodically erases the low quality self-generated data and outdated expert demonstration data.It can release the memory space and optimize the data structure.Finally,a single-agent framework based cross-layer routing protocol is designed and PM-DQf D is applied to the joint routing and resource management.Simulation results show that the method is superior to the traditional reinforcement learning scheme in terms of learning speed,data transmission quality and network reliability.Aiming at the newly-joined nodes in the network scenario where all of the nodes are homogeneous,a cross-layer routing protocol based on multiple experts apprenticeship learning is proposed.Firstly,the reinforcement single-hop responsibility rating is introduced to enhance the efficiency of power allocation.To avoid the failure of expert node identification for SU's remote location,the adaptive radius Bregman Ball model is introduced.Finally,the Multi-Teacher Deep Q-learning from Demonstrations(MT-DQf D)is proposed to avoid the biased knowledge of a particular teacher.The experiments illustrate that the proposed cross-layer routing protocol reduces the training period,routing latency and system energy consumption compared with the traditional multi-agent reinforcement learning strategy.(4)Considering the complex protocol architecture and the vulnerability to malicious user attacks in CRN,an end-to-end performance based anti-jamming decision algorithm for multi-hop CRN is proposed in view of the overall performance of the network in the scenarios where there exist malicious nodes.Firstly,the anti-jamming strategy takes route selection into account and gives full play to the advantages of robustness in distributed networks.Then the dual-threshold decision mechanism is applied to the reinforcement path responsibility rating,which is used to improve the stability in power allocation.Finally,considering the interference characteristics of nodes in multi-hop networks,the Dueling Deep Q-Network is applied to anti-jamming decision algorithm.Simulation results show that the end-to-end performance of the proposed scheme is better than that of the traditional algorithm in both of the traditional jamming mode and intelligent jamming mode.Furthermore,the robustness and reliability of the network are greatly improved.