Research On Resource Allocation Based On Reinforcement Learning In Multi-Beam Satellite Communications

High-throughput satellite communication,which utilizes multi-beam and frequency reuse technology to double system capacity,has become an indispensable part of future information and communication network infrastructure,drawing widespread attention from scholars worldwide.However,the growing number of users and the uneven distribution and time-varying nature of services present significant challenges to resource management in multi-beam satellite communication systems.Therefore,researching resource allocation technology for wireless resources such as frequency,power,and time slots in high-throughput satellite communication systems is crucial to improve resource utilization and system throughput performance.In this context,this paper focuses on modeling the resource allocation problem as a multi-armed bandit(MAB)problem in reinforcement learning,and studys dynamic resource allocation technology in multi-beam satellite communication system,with the following main objectives:1.A subcarrier allocation algorithm based on t he MAB method is proposed for multi-beam satellite communication systems.Firstly,a multi-user transmission model is established on the uplink of a multi-beam satellite communication system using Orthogonal frequency division multiple access(OFDMA).Secondly,an optimization problem is formulated with the objective of maximizing user throughput sum rate while satisfying constraints such as maximum transmit power and quality of service.Without requiring known Channel State Information(CSI),the MAB algorithm’s ability to learn arm-selection strategies online in partially unknown environments is employed to solve the optimization problem and obtain a subcarrier allocation scheme.Finally,simulation results demonstrate that the proposed algorithm achieves adaptive subcarrier allocation in multi-beam satellite communication systems,while obtaining user sum rate similar to the greedy algorithm based on perfect CSI.2.A joint subcarrier and transmit power allocation algorithm based on distributed MAB is proposed for multi-beam satellite systems using non-orthogonal multiple access(NOMA)technology.Firstly,multi-colored multiplexing technology is used to eliminate inter-beam interference between adjacent beams.Secondly,under the condition of unknown CSI in the system,a joint optimization problem with the objective of maximizing user sum rate,satisfying quality of service requirements for all users,and limiting the maximum transmit power of each user is established.Next,by m odeling the problem as a multi-armed bandit problem,the algorithm’s cumulative regret upper bound is derived by separately discussing and analyzing the cases of selecting non-optimal arms,leading to the joint allocation algorithm of subcarrier and transmit power.Finally,simulation results validate the effectiveness and superiority of the proposed algorithm.3.A beam hopping time slot resource allocation algorithm based on non-stationary MAB is proposed for multi-beam satellite systems.Firstly,under the condition of unknown traffic distribution,an optimization problem with the objective of minimizing the system’s second-order difference capacity,and constraints on waiting delay and working beam protection spacing is established.Secondly,the satellite is modeled as an intelligent agent,and the remaining number of data packets is used as the reward value to train the intelligent agent’s beam scheduling strategy,thereby obtaining a beam hopping pattern based on non-stationary MAB.Finally,simulation results validate the effectiveness and superiority of the proposed algorithm.