A Study Of Transfer Reinforcement Learning Based On Semantic Selection

Reinforcement learning is a machine learning method for solving sequential problems through interaction between environments and agents.Deep reinforcement learning combines deep learning and reinforcement learning to solve more complex problems in higher dimensions.However,on the one hand,complex environments will result in excessive negative samples and the training overhead of inadequate sampling.On the other hand,as the returns are maximized for a specific target(overfitting),the model performance is greatly degraded by small differences in the environment.A large number of studies have been conducted to address the above issues,among which the representative transfer reinforcement learning generalizes the learned strategies to other domains by reusing the source domain knowledge or pattern features.Existing transfer reinforcement learning methods still suffer from poor cold-start and jumpstart performance in the face of large scale visual disturbance changes.Inspired by inattentional blindness,i.e.,people tend to ignore waking stimuli in the background when their attention is shifted within the visual field,a semantic selection transfer reinforcement learning model is proposed to simulate this phenomenon.The model transfers from the semantic level rather than from the pixel level,selects semantic layers with high reward value correlations,and achieves decoupling of the target domain environment to improve the adaptation of reinforcement learning in the target environment.The main work of the thesis includes the following aspects:(1)To improve jumpstart performance in reinforcement learning when facing visual interference in transfer tasks,we propose a transfer reinforcement learning algorithm based on unsupervised semantic selection(USS).This algorithm innovatively designs a semanticselection-strategy framework(SSS)to simulate the phenomenon of inattentional blindness,which ignores visual elements in the background,thus achieving transfer reinforcement learning.The semantic layer uses unsupervised semantic segmentation to map the state space to the semantic space.The selection layer assigns semantic weights to the target environment guided by rewards and searches for the optimal semantic combination.The strategy layer inherits the source domain strategy network.This method does not rely on prior knowledge.Experimental results show that,compared with related attention mechanism transfer algorithms,the USS algorithm has significant advantages and strong interpretability under complex background interference.However,the transfer effect is constrained by the accuracy and speed of unsupervised semantic selection in decoupling the environment.It is challenging to achieve good transfer results when the visual interference factors and task-related factors have similar colors.(2)To further optimize the USS’s decoupling capability and transfer performance for the environment,we propose a transfer reinforcement learning algorithm based on graph attention networks for semantic selection(GATSS).This algorithm also follows the SSS framework.It first introduces supervised semantic segmentation to enhance the processing of similar visual interferences,resulting in better semantic extraction compared to the unsupervised semantic layer.Next,we propose a method for calculating semantic relevance in reinforcement learning environments,which reflects the impact of semantic distance on reward changes and constructs a target domain semantic relation graph.Finally,for the selection layer,we propose a method that embeds the target domain semantic relation graph into an existing knowledge graph to predict semantic node information,and design a Node2Vec+GAT semantic selector to complete the subsequent transfer reinforcement learning semantic filtering.Experimental results show that compared to the USS and other transfer reinforcement learning algorithms,GATSS significantly improves transfer performance,reduces training cost,and has the capability of continuous transfer.