Research On Virtual Network Function Placement And Migration Optimization Algorithm Based On Deep Reinforcement Learning

With the diversification of user business requirements and the explosive growth of mobile terminal equipment,the traditional network architecture is obviously not suitable for current network development.In the era of 5G Network,Network Function Virtualization(NFV)technology and Software Define Network(SDN)technology are combined to process each Network business request through a series of Virtualized Network Function(VNF).These VNFs are connected in a specific order to form a Service Function Chain(SFC),to provide services for different businesses.Therefore,this paper focuses on the placement and migration of VNF in virtualized network.The main research work and innovation points of this paper are summarized as follows:1.Aiming at the VNF migration optimization problem caused by the dynamic change of resource demand of service function chain under the NFV/SDN architecture,this paper proposes a VNF migration optimization algorithm based on deep reinforcement learning.Firstly,under the constraints of the underlying CPU,bandwidth resources and SFC end-to-end delay,a stochastic optimization model based on Markov Decision Process(MDP)is established,Secondly,since the state space and action space of this paper are continuous value sets,a VNF intelligent migration algorithm based on deep deterministic policy gradient(DDPG)is proposed to obtain an approximate optimal VNF migration strategy.The simulation results show that the algorithm can achieve the compromise between network energy consumption and SFC end-to-end delay,and improve the resource utilization of the physical network.2.Aiming at the problem of SFC placement optimization caused by the dynamic arrival of network service requests under the NFV/SDN architecture,a VNF placement optimization algorithm based on improved deep reinforcement learning is proposed.Firstly,a stochastic optimization model of MDP is established to jointly optimizes SFC placement cost and delay cost,and is constrained by the delay of SFC,as well as the resources of common server CPU and physical link bandwidth.Secondly,in the process of VNF placement and resource allocation,there are problems such as too large state space,high dimension of action space,and unknown state transition probability.This paper proposes a VNF intelligent placement algorithm based on deep reinforcement learning to obtain an approximately optimal VNF placement strategy and resource allocation strategy.Finally,aiming at the problems of deep reinforcement learning agent's action exploration and utilization through ? greedy strategy,resulting in low learning efficiency and slow convergence speed,this paper proposes a method of action exploration and utilization based on the difference of value function,and further adopts dual experience playback pool to solve the problem of low utilization of empirical samples.Simulation results show that the algorithm can converge quickly,and it can optimize SFC placement cost and SFC end-to-end delay.