Adaptive Optimization Based On Q-Learning In Wireless Energy Harvesting Heterogeneous Networks

To support smart applications such as smart home,a large number of sensors are deployed in femtocell heterogeneous networks.RF(Radio Frequency)-based EH(Energy Harvesting),which can charge sensors via wireless signals,has great potentials to be applied in future femtocell heterogeneous networks.Since RF signals,which carry on the energy and information simultaneously,transmit in the wireless link and compete the resources in the wireless link.The joint optimization problem of information and energy transmitting simultaneously is great important in femtocell heterogeneous network,and at the same time,it also brings in the following challenges:(1)the global model is difficult to establish because of the complicated relationship between the information transmitting and the energy transmitting;(2)too many nodes in the networks,too many system parameters to jointly optimize,and too high difficulty of the joint optimization;(3)it has the huge cost to acquire the channel status information(CSI)in the global network,and it is difficult to get the optimal result real time.Reinforcement learning is an adaptive adjustment strategy,which make the decision based on the feedback from the environment incentive signal via interaction between the agents and environment,which is quite suitable for the optimization in the large-scale networks.Therefore,this thesis studies the optimal design of two typical femtocell heterogeneous wirelesses EH networks based on the reinforcement learning theory,and the specific innovations are as follows:(1)Firstly,wireless powered femtocell heterogeneous network is studied,in which sensor nodes charge themselves by collecting energy from the RF signals transmitted by the access points(APs).In order to maximize the capacity of the femtocell under the constraints of satisfying the requirements of the information transmission rates and the requirements of the energy harvesting capacitors at the sensors,a optimization model is established mathematically.By adjusting the femtocell transmit power,the interference is exploited to charge the sensors while reduced the effect on the information transmission.To solve this problem,an adaptive power control algorithm framework based on Q-learning is designed.For the aim to improving the performance of the algorithm,a piecewise reward function based on distance factor and penalty parameter is designed.In order to more accurate in practice,the nonlinear EH model is considered,which reflects the actual circuit characteristics.By comparing the performance under different reward functions and hyperparameter values,the change rule of network performance behavior is given.The simulation results verify the effectiveness of our proposed algorithm framework and show that the piecewise reward function based on distance factor and penalty parameter has better network performance.Besides,the actual nonlinear EH model can effectively avoid the deviation compared to the traditional ideal linear EH model.(2)The more complex and more universal femtocell heterogeneous network with simultaneous wireless information and power transfer(SWIPT)is studied,in which APs transmit information to information users while charging sensors via RF signals.To this end,the transmit power at the APs and the power division factor at the SWIPT receivers are jointly optimized in order to maximize the total information capacity of the femtocell network under the constraints of satisfying the requirements of the information at the information users and the requirements of the information and energy at the sensors.Aiming at this problem,an adaptive transmit power and power division factor control algorithm framework based on Q-learning is designed.In order to accurate in practice,the nonlinear EH model are also considered in simulation experiment.The simulation results verify the effectiveness of the proposed algorithm framework and give the influence rules of the different reward function parameters on the performance of the femtocell heterogeneous network.The results show that the network performance under the ?-greedy action selection strategy is better than Boltzmann's action selection strategy,the latter is more suitable for scenarios with smaller action scale.