Q-learning Based Backup For Energy Harvesting R-Powered Embedded Systems

With the development of Internet of Things(IoTs)such as smart medical care,smart home and smart city,embedded devices have been widely used to influence people's lives.The most of IoT devices are charged by batteries.However,the shortcomings of battery,such as large size and weight,safety,and huge maintenance overhead,cause wide range IoT systems and implanted devices limited.Energy harvesting systems can gain energy from surroundings,like solar,wind and thermal,which are gradually used to replace battery.The energy gained by energy harvesting systems is converted to electric to charge embedded devices,making green,convenient but unstable.Traditional CMOS based processors may be interrupted frequently under unstable power supply since volatile states will be lost after power off.When power resumes,the system needs to reboot and execute program from beginning,causing severe performance degradation.A processor attached with non-volatile memory(NVM),called non-volatile processor(NVP),has been proposed to store volatile data when power failure occurs.After power resumes,these stored data will be copied back into processor,which enables the system to continue executing programs from where being interrupted.NVP can decrease the number of rollbacks and improves forward progress more efficiently.But the procedure of backup/resumption will consume energy which is planned for executing instructions.So,it is important for improving NVP system's performance to optimize backup procedure.Nowadays,researchers are focusing on reducing backup content size and deciding when to back up.But the most of researches need the help of static analysis of program which brings low universalness.This work will study how to dynamically locate checkpoints during program execution with the aim of maximizing forward progress.A reinforcement learning algorithm,Q-learning,has been applied to energy harvesting systems yet,adjusting sensors' power according to input power level.Inspired by this,Q-learning algorithm can be used to learn when to back up dynamically.Energy remained in capacitor,the number of backup content and instruction type are the inputs of Q-learning algorithm and the outputs are backup or not.Reward function is designed for maximizing forward progress.Rewards will be received when forward progress is improved and backup is successful completed.But NVP system will receive punishment when backup fails or executing instructions fails.To find the best policy,many trial-and-error steps have to be performed.This work also develops a gem5 based simulator to evaluate the proposed approach and compare with the most related work.This simulator can simulate backups,resumptions,rollbacks and dormancies of NVP systems.The experimental results show an average of 307.4%and 43.4%improved forward progress compared with traditional instant backup and the most related work,respectively.