| With the wide application of wireless sensor networks,energy management of wireless sensor nodes has become a critical factor affecting the lifespan of such networks.Traditional wireless sensor nodes are usually battery-powered,and once they are deployed,they often need to work for extended periods of time.However,as battery capacity is limited,battery power will eventually run out.To address this issue,energy harvesting technology has been developed to harvest energy from the surrounding environment,such as the sun,wind,and vibration,and introduce it into wireless sensors.This provides an effective solution for prolonging the lifespan of wireless sensors.In this paper,we refer to this type of wireless sensor that adopts energy harvesting technology as Energy Harvesting Wireless Sensors(EHWS).However,the energy harvested presents dynamic changes over time,either random or periodic,which cannot provide continuous and stable energy for sensor nodes.This results in the risk of energy shortage and death of the nodes.Therefore,this paper focuses on the energy management of EHWS and applies reinforcement learning methods to node energy management.This enables us to realize node energysaving research and adaptive energy management while ensuring node service quality.The main contributions of this paper can be summarized in the following three aspects.1.In response to the problem of energy shortage caused by the dynamic nature of harvested energy in EHWS,this paper proposes a Q-learning-based energy-saving strategy for EHWS mode switching.The energy consumption of wireless sensors mainly includes the energy consumption in the sleep and active states of the nodes.In many experimental studies,the energy consumption caused by state transitions is not considered.However,when the number of state transitions is relatively frequent,the energy consumption caused by state transitions cannot be ignored.Therefore,this paper considers the scenario of EHWS mode switching and proposes the use of the Q-learning algorithm to train nodes to reduce the number of mode transitions,thereby reducing the energy consumption caused by node mode switching and achieving energy-saving goals.Experimental results show that the TQL method proposed in this paper reduces energy consumption by 18.07%compared to the AQL method,while increasing the remaining energy of the nodes by 3.4%.2.In response to the problem of the low energy neutral performance of EHWS caused by the dynamic change of harvested energy,this paper proposes a concept of battery energy neutral threshold to optimize the energy neutrality performance of Energy Harvesting Wireless Sensors(EHWS)and achieve adaptive energy management of the nodes.To adapt to the dynamic characteristics of energy harvesting,a fuzzy Q-learning-based EHWS duty cycle adaptive adjustment algorithm is also presented.The algorithm combines fuzzy inference and reinforcement learning Q-learning and fuzzifies the collected energy to reduce its dynamic nature.As a reinforcement learning state space,it uses the fuzzified energy as input to update the fuzzy rule set according to the reward function,which keeps the battery energy in a neutral state and continuously improves the energy neutrality performance of the node.Compared with AQL and FQL in different battery initial energy scenarios,experimental results show that the proposed Fuzzy_QL algorithm improves node energy neutral performance by 7.43%and 7.92%,battery energy stability performance by 0.11%and 0.32%,and average energy waste is reduced by 2.40%and 5.36%.3.This paper develops a Python-based EHWS energy management visualization system,which displays the process of algorithmic correction during the training process along with the corresponding evaluation metrics.Additionally,the system interface includes functions to initiate and terminate algorithmic training.To prevent an excessive amount of information from displaying in the text box,a data clearing function has also been implemented. |
