Multiagent Reinforcement Learning Based Distributed Beamforming For Wireless Power Transfer

Wireless energy transfer is mainly divided into environmental energy capture,near-field energy transfer and far-field energy transfer.Near-field wireless energy transfer is mainly divided into magnetic coupling mode,electric coupling mode,etc.Far-field wireless energy transfer is mainly based on electromagnetic radiation,and the main research is how to reduce the path attenuation while keeping the power at the transmitting end unchanged,and beamforming is an important technical means to achieve this goal.Distributed beam focussing is an extension of this approach,and a larger virtual antenna array is formed by the cooperative control of multiple antenna arrays.Distributed Beamforming adjusts the spatial distribution of composite waveforms by independently controlling the phase and amplitude of the RF elements of the transmitter antenna array.For energy transfer applications,channel modeling is not feasible,especially for applications where simple electronics are used for energy transfer and the receiver cannot transmit the signals required for channel modeling.Such a black-box control problem for stochastic systems is well suited for reinforcement learning.In turn,the structure of distributed antenna arrays is well suited to be solved by cooperative multi-intelligence.Although there are many beam assignment related algorithms available,and some methods can even solve for approximate analytical solutions,it is predictable that such methods will have limits as the number of transmitting heads continues to increase.The black-box optimization class of methods will most likely break such limits.Although the control matrix obtained from the analytical solution may be inaccurate,can such a method still provide a speedup for reinforcement learning training? In this paper we have tried to investigate this question.We choose to test our idea in the simplest control environment: a controller based on modeling cannot perform the control task if the model is not accurate enough,but such a conventional controller can still provide a speedup for reinforcement learning.Although reinforcement learning has achieved many impressive results in recent years,the unsustainability of its operations may be a hindrance to its development.There are currently many teams studying reinforcement learning from the perspective of reducing operations,and the line of research is multi-intelligent reinforcement learning,where the problem itself is heterogeneous and transformed so as to simplify operations.Firstly,the paper introduces the application of beam fusion technique in wireless energy transfer and the domestic and international research.Next,we introduce the theory and implementation difficulties of beamforming for distributed antenna arrays.In the third chapter we present the main methods of reinforcement learning from the perspective of negative feedback tuning.In the course of studying reinforcement learning,we propose a traditional controller-based acceleration sampling method and validate our method in a single and double pendulum experimental setting,achieving up to 46% training speedup.We believe that our method is a meaningful attempt to open an effective way to combine traditional controllers with RL intelligences,especially for application scenarios with potentially inaccurate traditional controllers like beam fouling.The results of our experiments are presented in the last part of Chapter 3.In Chapter 4 we present the main approaches for collaborative multi-intelligent reinforcement learning.While studying multi-intelligent reinforcement learning,we try to convert a single-intelligent control problem to a multi-intelligent control problem in a simulation environment and compare the data required for training both.Our experiments show that converting a single-intelligent body to a multi-intelligent body can effectively accelerate the training.The experimental results are presented in the last part of Chapter4.We conclude from this experiment that multi-intelligent body reinforcement learning is the best choice to apply to distributed antenna array beam assignment.Chapter 5 shows our simplified experimental environment and the final experimental results for the distributed antenna array beamforming problem.Our experiments show that the multi-intelligent reinforcement learning approach can be effectively combined with the application of distributed antenna arrays for wireless energy transfer to obtain the control matrix through a black-box optimization approach.