Research On Intelligent Maneuver Decision Generation Of Within Visual Range Air Combat

Intelligent air combat is the key point to realize intelligent military combat,and has become a research hotspot in the military field.As an important part of intelligent air combat,intelligent maneuver decision-making is the research direction of major military powers.Taking the within visual range air combat game as the application background and deep reinforcement learning as the technical method,this dissertation studies the decision-making in aircraft guidance,air combat maneuver and multi-aircraft conflict detection and resolution.The contributions of this dissertation are as follows:1.An agent training method based on reward reshaping and policy reuse is proposed to solve the aircraft guidance problem with dynamic and directional constraints destination in three-dimensional continuous space.The agent can guide an aircraft to reach a moving destination along a specified direction,which meets the basic requirement of intelligent maneuver decision-making for aircraft guidance.Firstly,the problem of aircraft guidance is introduced,and the reinforcement learning training environment and agent are designed.Secondly,continuous action reward function and position reward function are presented,by which the training speed is increased and the performance of the generated trajectory is improved.Simulation results show that in different guidance tasks,the flight trajectory quality of manned aircraft is improved by continuous action reward function,and the agent training speed is improved by position reward function.The agent trained by reinforcement learning method has high computational efficiency,and it can generate a guidance command of manned aircraft and a control command of Unmanned Aerial Vehicle(UAV)in 3ms and 1ms,respectively.Finally,for destinations with different moving patterns,two policy reuse methods based on pre-trained and destination prediction are designed.Using policy reuse method,a new agent can be trained efficiently for a new task based on an existing agent.Training results show that the speed of training a new agent can be improved by using the two algorithms.The pre-trained policy reuse method is better when the similarity between the new task and the original task is high,while the destination prediction policy reuse method is less affected by task similarity and has better stability in different tasks.2.A deep reinforcement learning method for aircraft maneuver in one-on-one within visual range air combat game is proposed to solve the problem of uncertainty strategy of opponent.This method can be used to counter the opponent with intelligent decision-making ability,which solves the problem of insufficient maneuver decision-making for intelligent game with high-level opponent.First,the one-on-one air combat game problem is introduced and regarded as a two-person zero-sum game.Then its discrete value function solution is derived and a minimax deep Q network algorithm is proposed.This algorithm combines reinforcement learning and Markov game together to solve the Nash equilibrium when the opponent strategy is rational,and is verified by simulation.Last,an alternate freeze game framework is presented.Agents are trained in this framework with a deep reinforcement algorithm to deal with non-stationarity.A league system is adopted to avoid the red queen effect in the game where both sides implement adaptive strategies.The simulation results show that the agent trained by this method has excellent performance against the existing algorithms.The key indicators of winning rate and undefeated rate are better than that of its opponent.Moreover,this method has higher computational efficiency and takes 3ms to generate a maneuver instruction.3.A multi-aircraft conflict detection and resolution algorithm based on deep reinforcement learning is proposed,which can be used to effectively reduce the pressure of air traffic controllers and pilots.Firstly,the problem of conflict resolution in airspace is described,and the main operations of conflict resolution are introduced.Secondly,based on the‘Actor-Critic' framework,a conflict resolution agent is designed,which takes the flight plan of each aircraft in the airspace as its input,and takes a position in the airspace,which is the next pass point of the invading aircraft,as its output.Thirdly,a K-control conflict resolution algorithm is presented,which sets the number of control times of conflict resolution scheme,and generates flight trajectory by Dubins shortest path.It can generate conflict resolution scheme within 200 ms and the conflict rate is less than 1%.A multi-agent cooperative scheme generation method is proposed to alleviate the problem that the conflict cannot be completely eliminated.Finally,the multi-aircraft cooperative air combat agent training method based on conflict resolution agent is proposed,and the two-aircraft cooperative air combat simulation is carried out.Simulation results show that this method can avoid aircraft conflict in the early stage of training,which improves the efficiency of air combat agent training.An aircraft guidance and combat game intelligent simulation system is developed to match the simulation system with the air combat intelligent research.First,an open and extensible simulation system is designed based on High Level Architecture(HLA).Second,intelligent interfaces are developed for the training and verification of reinforcement learning agent.Last,the carrier-based aircraft approach agent and one-on-one air combat maneuver agent are trained on the system,and both of them are verified visually.Simulation results show that the high-level decision-making agent can be trained on this system,and the flight trajectory of the aircraft can be displayed intuitively.