| Behavior Trees(BTs)driven by expert knowledge are characterized by their hierarchical structure and graphical features,rendering them inherently interpretable.Nevertheless,their conventional design methodology requires manual,costly,and extensive expert knowledge.The utilization of data-driven Reinforcement Learning(RL),particularly Deep Reinforcement Learning(DRL)using neural networks,has achieved impressive results across various domains.Nevertheless,the "black box" nature of the system limited its application in domains that require a high degree of trust.To address these challenges,the thesis applies RL to generate BTs to combine their strengths.In particular,one possible approach to automate and optimize the design of BT is to model it as a sequential decision problem,which can be addressed using RL techniques.In addition,BTs’ hierarchical structure and graphical representation align well with human cognition,facilitating a human-friendly interpretation of the RL "black box" system.The thesis centers on the utilization of RL to produce BTs.Specifically,the thesis investigates two approaches for generating BTs: policy training and policy transformation.In summary,the thesis makes the following contributions:(1)A Q-function based policy training approach for BT generation.Currently,many BT auto-generation approaches that still require the manual design of initial BTs.To tackle this issue,the thesis proposes a novel policy training approach that leverages the Q-function to generate BTs.In particular,the flexible structure of BTs makes it arduous to extract and depict the knowledge generated during the RL training process;two fundamental units consisting of basic nodes of BTs are defined to address this challenge.In addition,the thesis introduces two algorithms for generating BT using Q-Learning and CQI algorithms,respectively.Since Q-functions can establish a direct mapping from states to optimal actions,our algorithms can generate BTs with complete topology during the RL training process without the need for manually designed initial BTs.Our policy training approach can produce BTs that exhibit good performance while maintaining a relatively small BT size,balancing the size and the performance.(2)A policy transformation approach to generate BTs using imitation learning.The thesis proposes a solution to approximate DRL algorithm with deep neural networks using inner explanation models on a global scale.It is achieved by generating BTs using a novel imitation learning based policy transformation approach.First,the thesis proposes a method for calculating the weight of state-action pairs based on the action execution probability function in policy-based DRL algorithms and gives a theoretical derivation.Second,the thesis proposes a resampling rule that utilizes the obtained weights to reduce the size of BTs generated in subsequent steps and enhance generation efficiency.Then,the thesis presents a method to transform resampled stateaction pairs into BTs by utilizing two fundamental units,leveraging the principle of the CART algorithm.Finally,the thesis generates BTs by extracting high-value state-action pairs directly from policy interaction trajectories that effectively approximate DRL models.(3)Evaluation of the proposed approaches based on policy training and policy transformation via a popular benchmark platform.The thesis employs the Open AI Gym as a standard benchmark platform for experimental validation.By adjusting various parameters,multiple BTs with different sizes and performances are generated through the Q-function based policy training approach.The outcomes of the experiments confirm that the proposed approach can generate BTs with high interpretability and good performance.Experiments are also conducted to optimize the size and performance of BTs generated through policy transformation incorporating imitation learning.By adjusting the depth of BTs and the number of fundamental units,results confirm that the proposed approach can generate BTs that closely approximate the performance of the original DRL model with a reduced size.Using the same environments,a comparative analysis between the two approaches above is conducted. |
