Research On Computation Offloading Strategy Based On Deep Reinforcement Learning In Vehicular Networks

With the continuous emergence of novel vehicular applications,the computational resource required by these resource intensive applications shows an explosive growth trend,which poses a severe challenge to the limited on-board computational resource in vehicular networks.Till now,the resource shortage issues in vehicular networks are becoming increasingly prominent.To solve these issues,computation offloading is widely considered as an effective solution.However,it is a very challenging work to perform computation offloading in the complex,diverse and heterogeneous vehicular networks.Due to the inflexibility and the high computational complexity,the traditional optimization algorithms cannot meet the requirements of computation offloading in vehicular networks.Owing to the above reasons,deep reinforcement learning(DRL)is introduced into vehicular networks to perform computation offloading.The uncertainty of resource,the dynamic variability of resource and the high concurrency of multi-type vehicular tasks in vehicular networks can bring huge challenges to computation offloading,such as lack of environmental awareness,environmental instability and high computational complexity.In this way,this dissertation studies how to utilize DRL to solve the computation offloading in vehicular networks.Based on the Beijing Natural Science Foundation project "Research on resource allocation algorithms of Internet of vehicles based on video content understanding driven by dynamic spatio-temporal data"(ID:4202049),in order to solve the computing resource shortage and meet the resource intensive vehicular applications' demand for resource,this dissertation systematically studies computation offloading strategies based on DRL algorithms.More detailed works are listed as follows:This dissertation summarizes and analyzes the related works in the field of vehicular networks,computation offloading,DRL algorithms and DRL enabled computation offloading strategies.Firstly,it summarizes the key technologies and the research status of vehicular networks,and points out that the computational resource shortage issue is the bottleneck of constraining vehicular networks' development.Secondly,the computation offloading's key technologies,research status and challenges are illustrated,and the significance of computation offloading in vehicular networks is clarified.Then,some representative reinforcement learning algorithms and DRL algorithms are introduced in detail.In addition,this dissertation points out that it is feasible and necessary to utilize DRL algorithms to conduct computation offloading in vehicular networks.Finally,the current status and challenges of DRL enabled computation offloading strategies are respectively summarized.Based on these challenges,the scientific issues to be studied in this dissertation are illustrated,and this dissertation's main research line is established.Considering the lack of resource perception ability caused by resource providers' uncertain attituded of sharing resource utilization status in vehicular networks,it can further affect the reliability of computation offloading.To solve it,this dissertation proposes one DRL algorithm enabled resource-awared computation offloading strategy.Firstly,the dissertation analyzes the reasons for uncertainty of resource,which is the resource providers' "negative" attitude towards the sharing of computing resource utilization status.As for the "positive" attitude to sharing historical resource utilization status,one long-short term memory network based resource discovery mechanism is proposed;As for the"negative" attitude to sharing historical resource utilization status,one multi-armed bandit based resource discovery mechanism is proposed.Secondly,one resource-awared deep reinforcement learning algorithm is proposed to realize the computation offloading.The simulation results show that the proposed strategy can achieve better performance in convergence speed and the overall reward.As for the resource instability caused by the dynamically changeable relationship between resource supply and demand in vehicular networks,it can further lead to the deficiency of environment adaptability.To solve it,this dissertation proposes one DRL enabled adaptive computation offloading strategy.Firstly,the dissertation analyzes the dynamic changeability of computational resource in vehicular networks,and classifies the resource status from the perspective of supply and demand of resources.Then,one adaptive optimization target mechanism is proposed,which involves two cases:supply exceeds demand and supply is less than demand.As for the former case,one multi-path computation offloading mechanism is proposed;As for the latter case,one priority based computation offloading mechanism is proposed.Furthermore,one DRL enabled adaptive computation offloading mechanism is proposed.The simulation results show that the proposed strategy can improve the performance of the total reward value,resource utilization and reliability of computation offloading.In view of the computational complexity caused by multi-type vehicular tasks with high concurrency in vehicular networks,it can affect the efficiency of computation offloading.To solve it,this dissertation proposes one DRL enabled distributed computation offloading strategy.Firstly,the dissertation analyzes the relationship among multi-type resource's management and optimization involved by the diverse vehicular tasks.Then,the value decomposition mechanism is introduced into the DRL model to accommodate the concurrent multi-type resource management and optimization.In this way,the DRL model can be decomposed into a combination of multiple simple DRL models,in which each simple DRL model is corresponding to one type resource's management and optimization.Furthermore,one distributed multi-agent DRL algorithm is proposed to conduct the computational resource allocation and spectrum resource allocation.The simulation results show that the proposed strategy can achieve obvious superiority in convergence speed,the performance of computational resource allocation and the performance of communication resource allocation.