Research On Automatic Maintenance Method Of Underwater Vehicle 3D Track Based On Reinforcement Learning

During the underwater mission of the Unmanned underwater vehicle(UUV),due to the influence of wind,waves,currents and other factors,as well as the errors of its own control and execution system,its navigation trajectory cannot be completely consistent with the scheduled route.Therefore,during the voyage,the UUV should be continuously detected and automatically controlled to return to the scheduled course when it deviates seriously from the scheduled course.In this paper,the three-dimensional track keeping problem of UUV in unknown environment is studied.The following research work is mainly carried out:Firstly,the research status of UUV and UUV track keeping technology at home and abroad is introduced,and the motion model of UUV is studied.On the basis of establishing a fixed coordinate system and a motion coordinate system suitable for UUV motion,the kinematics equation of UUV is obtained,and several acting forces on UUV are analyzed,thus the six-degree-of-freedom dynamics equation of UUV is obtained.On the premise of not affecting the kinematic characteristics of the UUV,the six-degree-of-freedom model of the UUV is simplified to a four-degree-of-freedom model according to the research requirements of the paper.Three typical motions are selected to simulate the UUV model,and the accuracy of the UUV model is verified by analyzing the position distance,heading angle,speed and other information.Secondly,the UUV track keeping algorithm is studied.In the research of UUV track keeping field,traditional control methods such as synovial membrane control,backstepping control and PID control with fixed parameters are mostly used in the design of control systems.In order to better cope with the complex and changeable environment and the influence of other interferences,Q learning algorithm in reinforcement learning is integrated with BP neural network.Through analyzing the reinforcement learning problems existing in the track keeping process of UUV,an improved BP neural network-Q learning algorithm is designed,and the Q function is approximated by using the good approximation characteristics of BP neural network,so that it can learn and optimize the UUV in continuous state.Thirdly,the UUV track keeping control system is designed.The control system consists of four controllers and controls the north position,east position,depth and heading respectively.All four controllers are adaptive PID controllers based on BP neural network-Q learning algorithm.Four adaptive PID controllers cooperate to continuously optimize the track keeping effect of UUV.Finally,the adaptive PID controller is designed.Through the design of each module of the adaptive PID controller,the adaptive PID controller can adjust the control parameters in real time and actively learn the optimal PID controller parameters,which well solves the problems that the traditional PID controller cannot automatically adjust the control parameters and has poor adaptability to the environment.The parameter setting effect of the adaptive PID controller and the UUV track keeping effect with and without interference are simulated and verified through typical tracks.