Research On IoT Wireless Resource Management Method Based On Reinforcement Learning

The design of resource management schemes in wireless communication networks has always been a popular research for scholars.It has broad prospects in the fifth-generation mobile communication networks(5G),the Internet of Things(IoT),wireless sensor networks,and air-space-ground integrated networks.At present,various smart devices have increasing demands for network resources.However,since the lack of communication network resources,the resource used for transmission,computation,and storage is extremely limited.Hence,wireless communication networks cannot meet the high bandwidth and low latency requirements of massive smart devices and their emerging applications.In the large-scale IoT,traditional resource management technologies can no longer meet the dynamically changing requirements of users,especially massive data processing and latencysensitive tasks.Therefore,it is significant that studying how to perform intelligent and autonomous dynamic resource management in a time-varying wireless communication network for the QoE improvement.Most traditional resource management methods involve centralized cloud computing.This architecture usually has the disadvantages of heavy backhaul link load and long service response latency.As an emerging technology,mobile edge computing(MEC)moves the computing and storage capabilities of cloud computing down to the wireless network's edge,providing users with real-time computation and storage capabilities.With the development of artificial intelligence(AI)technology,many researchers have realized wireless communication network resource management by combining machine learning technology,which greatly improves the autonomy and flexibility of networks.However,at the edge of the wireless communication network that introduces MEC,the traditional wireless resource allocation method is no longer applicable.This is because the traditional wireless resource allocation method does not involve multi-dimensional resources.Therefore,our thesis mainly studies the wireless communication resource management problem in the IoT.It is committed to studying how to integrate the current popular AI algorithm-Reinforcement Learning(RL)into the MEC-assisted IoT to solve above resource allocation issues and further realize intelligent online decision-making.The resource management schemes focus on communication resource allocation and interference management,communication resource allocation and edge caching,computing offloading,and joint resource optimization of communication and computing resources in four scenarios.The resource management scheme mainly ranges from a centralized resource management scheme to a distributed resource management scheme.The optimization problems mainly range from single-objective optimization to multi-objective joint optimization.The basic optimization framework for resource management is constructed based on RL algorithms.This thesis design a series of resource allocation algorithms based on value-based RL(eg.,Q-learning,Deep Q-learning,and Quantum reinforcement learning)and policy-based RL(eg.,Policy Gradient and Asynchronous Advantage Actor-Critic)suitable for wireless communication networks,and a lot of simulations verify the performance of the proposed algorithms.The main research work and innovative results of the thesis are briefly described as follows:1.Research the hierarchical IoT framework based on cognitive ratio technology,abstract the network framework into a three-layer multi-agent system(MAS),involving the layer of secondary users,the layer of primary users,and the layer of base station.Subsequently,two resource allocation solutions are proposed and distributed RL is integrated into the MAS.We propose the intelligent base station control mechanism and resource allocation mechanism.In addition,two different schemes are designed in the resource allocation mechanism:a centralized channel allocation scheme based on Q-learning and a resource distributed allocation scheme based on multi-agent reinforcement learning(MARL).The proposed method relieve the load pressure of communication network and maximize the system capacity effectively.2.For the complex multi-dimensional resource dependency of the information-centric networking and the difference in service requirements between users,the Device-toDevices(D2D)communication technology and MEC are introduced to assist users to efficiently communicate and cache content.First,the communication mode is designed that can cache content according to the needs of users and make a choice for the D2D mode or the cellular communication mode.Then,channel resource allocation and power selection are modeled as a Markov Decision Process(MDP),and a policy-based reinforcement learning method is proposed to learn the resource allocation strategy.The goal of this method is to maximize the reward function,thereby maximizing the spectrum efficiency and the system capacity.Experimental results show that the proposed method avoids interference between users in D2D communication and improves system capacity.3.For the requirements of delay-sensitive and bandwidth-sensitive task in Internet of Vehicles(IoV)and the limited computation resource of vehicles,the joint communication and computation resource allocation problems is studied by edge intelligence in MEC-assisted IoV.Here,the quantum learning algorithm is adopted to solve the transmission,and computation resource allocation problems in the edge-cloud coexistence scenario.And,the joint optimization is modeled as an MDP.According to the requirements of delay-sensitive tasks,it is studied whether to offload tasks at edge nodes for processing or to transfer tasks to the cloud center for computing.Also,quantum reinforcement learning(QRL)is introduced to map classical states and actions to quantum states and actions.The dynamic resource management strategies is learned by quantum collapse rules.Experimental results show that the proposed algorithm can converge quickly and significantly reduce the system latency,energy and cost consumption.4.For the problems of insufficient intelligence and poor security of task offloading and resource allocation schemes,the intelligent and secure resource management scheme is designed in the MEC-assisted Software-defined Cyber-physical Systems(SDNCPS).First,blockchain technology is used to enable data consensus between edge nodes to ensure secure communication.Then,the resource allocation problem is modeled as a MDP,which aims at minimizing the latency of transmission,consensus and calculation.Based on the asynchronous advantage actor-critic(Asynchronous Advantage Actor-Critic,A3C)algorithm,the task offloading and resource allocation scheme is designed for intensive devices.Experimental results show that the proposed algorithm can maximize long-term rewards and realize intelligent resource management while ensuring the secure transmission.5.For the problems of difficulty in capturing network status and lack of intelligence caused by a large number of devices and strong heterogeneity in the IoT,we propose the next-generation IoT paradigm that inspired by artificial intelligence,namely 5G Intelligent Internet of Things(5G I-IoT).In 5G I-IoT,we integrate deep learning,big data processing,reinforcement learning,and other methods into the traditional IoT network.The network is divided into four parts:data sensing area,edge processing area,cloud processing area,and application area.First,a data preprocessing method is proposed in the sensing area,including data integration,data cleaning,and data deredundancy.Secondly,the RL algorithm is used for decision-making and online optimization in the edge processing area.Furthermore,intelligent computing modules and execution modules are designed in the cloud processing area.Experimental results show that the proposed I-IoT network paradigm has better performance than traditional networks and can effectively alleviate the communication network pressure as well as improve the effective utilization of channels.Finally,the thesis presents the intelligent resource management,control framework and key technologies that can be applied to the future B5G/6G networks.