| In recent years,with the development of deep neural networks,there have been many breakthroughs in the field of reinforcement learning,which has led to a lot of interest in Model Based Reinforcement Learning(MBRL).However,at the same time,MBRL also faces many challenges.In order to achieve high performance,many models extract image features through Convolutional Neural Network(CNN)to obtain representation vectors and store large amounts of temporal memory information through Long Short Term Memory(LSTM).However,the representational power of CNNs is limited and the high computational costs of LSTMs are unaffordable when dealing with complex tasks.How to improve the representational power of visual perception models and reduce the computational effort of temporal models has been a hot topic in MBRL.The reinforcement learning environment Carracing-v0,published by Open AI,is extremely challenging due to its complex,continuous action space and is widely used to evaluate the performance of reinforcement learning algorithms,which is used as the target task in this thesis.The main work of the thesis is as follows:Firstly,to address the problem of limited representation capability of traditional CNN,we select the pre-trained model ResNet as the visual perception module.ResNet,after pre-training with a large number of unsupervised datasets from ImageNet,is improved in all evaluation metrics compared to the traditional convolutional neural network.In this thesis,we use logistic regression algorithm confront external datasets and train the visual perception module by multi-task learning algorithm to enhance the representational capability of the model.The final simulation results show that our model improves the recognition accuracy by 1.44%compared to the CNN model trained directly on the target task.Secondly,to address the problem of high computational cost of LSTM,this paper uses Echo state networks(ESN)to replace LSTM as the timing processing module.The hidden layer weights and states of ESN are kept constant throughout the training process,and only the weights of the output layer are updated in each iteration,which is expected to reduce the computational complexity while guaranteeing the stability of its dynamical system.reduce computational complexity and training time.ESN makes the network have rich nonlinear dynamical behaviors by introducing a large number of random connections and nonlinear activation functions.This nonlinear characteristic can help the network capture the complex patterns and dynamic relationships in the Carracing-v0 task,which is conducive to improving the average score of the Carracing-v0 task.However,the hyperparameters of the ESN have a large impact on the model performance.Traditional grid search algorithms seek optimization by enumerating in the parameter space,resulting in excessive time consuming.The search method used in this paper simulates Darwin’s evolutionary theory and is optimized based on Genetic Algorithms(GA),so that the crossover probability and mutation probability change adaptively according to the similarity of individuals in the population in order to accelerate the convergence speed.Simulation experiments show that the optimization time consumed by the improved genetic algorithm in this paper is shortened by 18.75%compared with the genetic algorithm.The ablation experiments show that after replacing the timing module with ESN,the training time is saved by 9.6%compared with LSTM,the computation is reduced,the score in Carracing-v0 is improved by 6.1%compared with LSTM,and the overall performance of the model is improved.Finally,for the problem that artificial neural networks need more parameters to be trained,this paper uses biological neurons to replace part of the artificial neural networks to control some of the actions in Carracingv0.At the same time,the low power consumption of biological neurons is utilized to reduce the energy consumption during model training.The game screen is converted into electrical stimuli and fed into the biological neural network to generate the firing rate sequence,and the firing rate sequence is temporally analyzed by ESN.The final model completed the Carracingv0 task and reduced the number of parameters by 7.69%compared to the artificial neural network. |
