Research On Satellite Communications Resource Allocation Algorithm Based On Reinforcement Learning

Satellite mobile communication system has the advantages of high reliability,large capacity,global seamless coverage communications,etc.Not only has it played an irreplaceable role in military and civil communications,emergency rescue,medical,health and other fields,but also become a significant part of global mobile communications.However,the spectrum and power resources on the satellite communication systems are extremely limited and scarce.Moreover,how to dynamically allocate the resources in a more autonomous,intelligent and flexible way is of great significance to improve the resource utilization and communication performance.Therefore,the dynamic resource allocation technology based on reinforcement learning in the satellite mobile communication system has been studied in this thesis,and the main research result are as follows:Firstly,in order to facilitate the evaluation of the performance of dynamic resource allocation algorithm based on reinforcement learning,a dynamic resource allocation simulation platform for satellite mobile communication system is built and implemented with Matlab.The platform utilizes satellite system modeling and link analysis capabilities of the Satellite Tool Kit(STK)to perform satellite mobile communications channel simulation,as well as link quality parameter calculation function such as satellite antenna gain and free space loss.It can also build a variety of satellite communications scenarios such as uniform or non-uniform distribution of users and communications traffic.In addition,the platform can provide environment status information such as beam user service request and resource usage for resource allocation algorithm,simulate resource allocation process,and then calculate service blocking probability,system capacity and other performance indicators to feed back to the resource allocation algorithm.It not only has better flexibility and scalability,but also provide a guarantee for the performance evaluation of dynamic resource allocation algorithm in the satellite mobile communications environment.Secondly,aiming at the problem of low resource utilization of fixed channel allocation algorithm caused by non-uniform distribution of traffic in multi-beam satellite mobile communication system,a Q learning-based dynamic channel allocation algorithm has been proposed.The algorithm makes the satellite agent sense the channel allocation status and service request from the users in the satellite communications environment and then formulate a channel allocation policy according to the Q table to allocate channel resource for the current service requesting user.The Q value target error is calculated on account of the reward information from the environment,and the multi-step update method is used to optimize and update the Q table.Consequently,optimizing the agent channel allocation strategy and minimizing the system traffic blocking probability can be achieved.The results show that the proposed dynamic channel allocation algorithm based on Q learning has lower traffic blocking probability and higher channel resource utilization.Finally,aiming at the problem of low resource utilization of satellite mobile communication system caused by non-uniform distribution of inter-beam traffic,and further considering the influence of power allocation on system channel capacity,a joint resource allocation and power control algorithm based deep reinforcement learning has been proposed.The algorithm designs a convolutional neural network to extract the feature of the channel allocation state,power allocation state and user service situation in the satellite communications environment and completes the mapping from state to action through the forward transmission of the deep Q network,allocating channel and power resources for the request user.Then the agent calculates the error function,and updates and reverse training the deep Q network by the experience playback technique according to environment reward and the target Q network.Thereby optimizing the channel power allocation strategy and maximizing the satellite system capacity can be achieved.Simulation results show that the proposed dynamic channel allocation algorithm based on deep reinforcement learning can effectively improve the system capacity and the resource utilization in satellite mobile communication systems.