The Evolution Of Virtual Creature Based On ERL

As a new study field, evolution of Artificial Life (Alife) is an elementary and a vital project, which researches life system as its object, studies mechanism of life process and application in the range of engineering as its content, expends function in human life as its research target. The interest in study of Alife focuses on the imitation in the characteristic of life system, at the same time, knowledge about the study makes use of top-down and bottom-up method in synthesis, so it brings better adaptability and flexibility to this artificial system. On the other hand, virtual creature is one kinds of experimentative instrument to prove Alife theory as well as a part of virtual environment. In the view of the point of Alife, sensation and cognition of virtual creature are results from spontaneous emergence in computation, and the essence is a process of building mode. Hereby the research on agent in Alife can improve the capability of whole ALife system., the exploration of Alife unfolds theoretic and applicable essentiality in study Artificial Intelligence (AI) System, thereinto the evolution mode of ALife can resolve the issue of machine learning, function optimization in function and signal disposal.Machine learning (ML) is a sort of measure to simulate the ability to study of creature by the computer used by human. It is an important aspect in AI research, and it is meaningful to study and develop AI and robot technology. Reinforcement learning (RL) is a technique in machine learning which is rapidly growing during recent couple of decades. It is a study with weakening supervision, as priori knowledge is not necessary to this means. And it can be studied through trial-and-error method related to the exchange between environment and itself, that has become one of the characteristics of reinforcement learning. As far as reinforcement learning is concerned, it should be a set of one type of problems instead of a set of the type of methods. What an agent will confront with when it prompts its behavior or achieve certain goal by trial-and-error method related to itself and environment is what the problem reinforcement learning must face. Due to Multi-agent system which is an attribute in Alife that the dynamic changes in virtual environment and act of other agents are unknown, the use of reinforcement learning in Alife model is useful to agent's self-learning and self-adaptation in virtual environment.As for Evolutionary Reinforcement Learning (ERL), it is a study combining RL and Computation Intelligence. ERL can work under the study outline of RL, and meanwhile, it makes use of a combination between neural network (NN) and genetic algorithm (GA) to achieve this outline. There are two NNs existed in an agent, they are evaluation neural network and act neural network that get act decision by computing result of evaluation and act with GA betaking optimization in weight from those neural networks. As the balance in RL policy——exploitation in the set of act that is not implemented or continuous exploration about the knowledge captured in the process of study——determines two choices of agent. According to those choices, local optimization will be presented if we use exploitation without exploration, in like manner, although exploration can avoid local optimization and accumulate the speed of study, the function in algorithm will be obstructed if we use exploration without exploitation. ERL Model that has inhered dose not deal with this dilemma very well, in addition, there are some problems about this theory in the field of application such as compression in state-space, assign on credit, etc. In other words, the existing method is the one who is in localization.This paper presents an improved model originating from the Evolutionary Reinforcement Learning, which builds a Multi-agent System and focuses on promotion about the fitness level of agent in a virtual environment. Simultaneously, the reflection of the relationship between study and evolution and how agents deal with the problem of existence and reproduction in the environment will be observed, and this paper builds on an expression to the "emergency" and the intelligence of cybotaxis in Artificial Life.Q-study is a new study method related to machine learning, it is constructed under RL and based on Q-value. As to the model, it combined with ERL including Q-study and the distributed policy under the guide and conform to the study and evolution acted in virtual creature in Alife. Artificial Neural Network (ANN) in this work is used for the study on how to adapt oneself in the virtual environment, and also, to itself, it learns how to evolve. Consequently, it weakens the effect of supervision in the process of study, and agent grounded on study feedback from environment gets better adaptability. In this model, the Evolutionary Algorithms (EA) for reinforcement learning with a method of distributed policy based on Neural Network and the Genetic Algorithm (GA) are introduced in the paper for designing an Actor-Critic model about Artificial Life. The apperception from Actor-Critic model, in this model, is inputted to the Neural Network combined which refers Genetic Algorithm and distributed policy during the computation of NN. The final outputs of act NN are outcomes found on Markov decision chain which is under RL, and it is not a single value but a description on the probability of act. That is, act from an agent is chosen according to the output of act NN itself, and it is helpful to strengthen the convergence effect. Furthermore, as one kind of evolutionary regulation to virtual creature, the Genetic Algorithms make those who own higher fitness keep surviving and reproduce for the continuous of population. The evolution of virtual creature can achieve in view of this, and therefrom, the function in self-adaption is become true.This model is based on Baldwin Effect which proclaims the relationship between agent's study and evolution. Coupled with the co-operation of ANN, Evolutionary Study, Reinforcement Learning and distributed policy, virtual creature reveals the interaction between study and evolution, and expresses that they all are fairly vital to Alife.The experiments related to this filed, at present, refer to the study and evolution of the virtual creature in a single virtual environment. As for this improved model, it not only embodies a faster convergence on algorithm and a better fitness on evolution, but also strengthens the adaptability of agent in non-single environment.By means of concrete achievement and observation to this experiment, a comparison between this experiment and other likenesses is involved at the back of this paper. According to the simulated model assumed and following the advancement of algorithm effect, an agent presents some characteristics of behavior such as free moving, food searching, reproduction, evading, etc. It is obvious that this model is helpful to agent to make a decision of behavior-study, and the same time, the agent can obtain a fine convergence t and fitness in evolution.This project is provided with the potential in the application in engineering and can be applied to other similar systems. It is meaningful to the research of academic and practical study and evolution in virtual creature in Alife as well.