Research On Downlink Beamforming Based On DQN In Millimeter Wave MISO System Under Mobile Scenarios

With the development and application of emerging technologies such as the Internet of Things,artificial intelligence,big data,and cloud computing,the demand for high speed and reliability of mobile communication data transmission is increasing.Therefore,major countries around the world are actively promoting the research and development of the fifth generation(5G)and post-5G wireless communication systems.Millimeter wave(mm Wave)is a key candidate technology for 5G mobile communication due to its high wireless transmission bandwidth,miniaturized large-scale antenna array and high antenna gain.The beamforming technology used in millimeter wave communication can effectively solve the problem of reliable data transmission in mobile environment.However,the traditional beamforming optimization scheme relies on iterative algorithm and convex optimization algorithm,so it has high computational complexity and large delay,and can't meet the real-time service requirements in mobile environment.A large number of studies have shown that using deep Q network(DQN)to solve the beamforming optimization problem has low complexity and time delay.Therefore,this paper studies the downlink beamforming optimization problem of millimeter wave multiple input single output(MISO)system in mobile scenarios based on DQN algorithm.The main research work of this paper is divided into the following three aspects:1.Aiming at the problem that traditional beamforming algorithm has high computational complexity,large delay and can't meet the real-time reliable communication,an optimization scheme of centralized beamforming joint power control based on DQN algorithm is studied and simulated.Firstly,based on the millimeter wave channel model in the mobile scene,we formulate the joint design of beamforming and power control as a non-convex optimization problem of the MISO downlink system to maximize the signal to interference plus noise(SINR);then,by using the SINR and position coordinates that users equipment feed back to the base station,we construct the interaction model between the central control unit which act as an agent and the wireless communication environment,and the future cumulative rewards of binary coding actions(selection of power level and codebook index)are estimated by using the greedy attribute of DQN to solve the optimization problem.Simulation results show that the proposed centralized beamforming scheme based on DQN algorithm can achieve the sum rate capacity of brute force(BF)algorithm with significantly reduced complexity.2.The centralized beamforming optimization scheme needs to collect the global channel state information(CSI)of the dynamic environment through the backhaul link between the base station and the central control unit,which causes a large system transmission overhead and high computational complexity.To solve this problem,this paper proposes and implements a centralized-training distributed-executing(CTDE)beamforming scheme based on the designed information exchange protocol.Firstly,the scheme obtains the global information of dynamic environment based on the information exchange between base stations,and designs the key elements of the reinforcement learning interaction model based on this information;secondly,the central control unit conducts centralized training through the experience values collected from other agents(base stations),and then broadcasts the model parameters to each agent;finally,the agent executes its own actions in a distributed manner based on the local environment information to maximize the cumulative reward.Through the simulation results of the average achievable rate,it's analyzed that the proposed scheme achieves good performance under the condition of significantly reducing the time complexity.3.The CTDE beamforming optimization scheme uses the shared DQN network model to learn the common features of agents,but it is unable to obtain the feature differences between cells,and the communication overhead of backhaul link is large.A distributed beamforming optimization scheme based on information exchange is studied.In this scheme,each base station is regarded as an independent agent,based on the information exchange protocol and DQN interaction model designed in the previous scheme,the interaction with the wireless dynamic network environment is established;then the dynamic characteristics of the environment are learned from the local observation,and the optimal beamforming vector is solved by maximizing the joint reward.The simulation results show that the convergence of this scheme is not as good as that of CTDE beamforming optimization scheme,but the performance is better,and the achievable rate is very close to the FP algorithm which needs to collect global instantaneous CSI measurements under ideal conditions.