Referring Grounding In Situated Human-robot Dialogue

Using natural language to communicate with robots has become a trend.For ser-vice robots which need to interact with the surrounding environment,there is an un-avoidable problem that is how to connect language to the world.This is a problem of symbol grounding.During human-robot situated dialogue,this problem is inevitable for the robot.Mimicing the behavior of human beings to teach new concepts through referential communication,we use the form of referential communication to ground unknown language symbols with a semantic dictionary.During situated dialogue,the robot and its human partner have different capabilities in perceiving and reasoning about the physical world.Due to lack of a common perception basis,dialogue between a robot and its human partner becomes very difficult.We try to solve this problem from the fol-lowing two aspects:(1)A scene semantic description system is constructed to mediate perceptual dif-ferences to some degree.The referent during situated dialogue is the name of human or object.Two kinds of strategies are used to describe the referent,i.e.,qualitative spatial relations(QSRs)and colors.This system is used to evalute the compatibility between the ROI of an image and the latent descriptions.KeJia equipped with a laser sensor and a depth camera can obtain precise spatial locations.However,as the ambiguity of color description is larger and it is affected by illumination,the acceptability of color descriptions is modeled based on visual similarity in HSV and the confusion matrix of the color classifier.(2)A robust method for context understanding is proposed.The limitations of the robot's perceptual ability have led to a variety of uncertainties,e.g.,mis-detecting ROI,speech misrecognition.A probabilistic model based on discrimination is used for infor-mation filtering and referring grounding.The weight of the descriptions are changed according to the human partner's feedback to robot's grounding result.The description with maximum weight is selected as the restrictive condition of the probabilistic model,thus to achieve the context understanding and the correction of the results of referring grounding.Two sets of experiments were designed to evaluate the performance of our system under the circumstance of a variety of uncertainties existing.The results demonstrate that our system is robust both to perception errors and description errors and the ability of mediating perceptual differences.