Research On Carrier-borne Aircraft Support Scheduling Based On Deep Reinforcement Learning

As the combat weapon of an aircraft carrier,the sortie generation rate of carrier-borne aircraft is an important index to measure the combat capability of an aircraft carrier.Efficient support operations scheduling plan for carrier-borne aircraft is one of the keys to improve the sortie generation rate.Carrier-borne aircraft support operations scheduling plan organizes and arranges support task sequence required by carrier-borne aircraft so that it can efficiently complete the overall support tasks under the constraints of limited time,space,and resources.Based on the analysis of the operation flow,resource requirements,and constraints of carrier-borne aircraft support operations,this paper describes the problem of carrier-borne aircraft support operations scheduling as a resource-constrained project scheduling problem and establishes the corresponding Markov Decision Process model,which includes: the state space of the resource utilization and the unassigned tasks is constructed,and an image form is designed to represent the system state;the tasks are composed of action space,and the actions that do not meet the task execution sequence constraints are excluded by mask operation;the task execution time is regarded as the immediate reward,and the objective is optimized as the cumulative amount of task execution time.To process the state characteristics of the image,a deep neural network composed of three layers of convolution layer and three layers of full connection layer is designed.Its input is the state expressed in the form of an image and the output is image features.Based on the deep neural network,the paper designs the carrier-borne aircraft support operations scheduling based on the Deep Q Network algorithm and the Advantage Actor-Critic algorithm.Among them,the Deep Q Network scheduling algorithm uses Boltzmann action exploration strategy to enhance the exploration ability of the agent,and uses experience replay to improve the learning efficiency;the Advantage Actor-Critic scheduling algorithm considers the adjacent scheduling decision-making actions will have mutual influence and summary multi adjacent actions to improve the learning efficiency.Besides,the generalized advantage function estimation method is introduced to improve the accuracy of the advantage function estimation for the Advantage Actor-Critic scheduling algorithm,and the coordinated training process is designed to control the updating direction of the Actor and Critic public network parameters.This paper designs a discrete event simulation system for carrier-borne aircraft support operations scheduling,and implements the system with Python language.In the solution of the experimental case,which significantly better than other algorithms such as the First-Fit,the Best-Fit,the random selection policy,etc.Besides,the algorithm proposed in this paper improves the balance of resource allocation.