Multi-view systems for robot-human feedback and object grasping

In human-robot interaction, computer vision technologies are widely used by robotic systems to analyze the target scene and interpret human commands. Commands to robotic systems can be provided by human input through natural interfaces, such as body gestures, hand motion, etc. Although interactions through natural interfaces are intuitive, the communication may or may not work depending on the operating environment and the robustness of the vision algorithms used. Therefore, some feedback from the robots to the human is necessary to show how the robots understand the human input or the desired behavior. Additionally, feedback from the robots allows human users to respond accordingly to enable further interactions. In this work, we investigate the use of multi-view vision systems that provide such interactive visual feedback to improve human-robot interaction for a specific type of robotic application, namely object grasping.;Two multi-view systems using different visual sensors are developed. The proposed systems detect objects that attract the attention of human observers using a visual attention model, since such objects are likely to be the desired grasping targets. The systems project visual feedbacks for the detected objects using a DLP projector. The projected patterns indicate the possible grasping targets of the system, and also define the commands it can accept. Human operators choose the grasping target by giving a confirmative response according to the projected patterns.;One system consists of a stereo camera and a projector. The trifocal tensor is used to match detected objects and determine feedback patterns. The other system replaces the stereo camera with a RGB-D camera. The 3D structure of the target scene is constructed to decide projected patterns. Both systems allow users to select a grasping target among the detected objects by interacting through the feedback patterns. Then the vision systems can guide a robotic arm to grasp the selected object.;Object grasping is done using the Simultaneous Image/Position Visual Servoing method. An automatic goal pose/image generation method for visual servoing with respect to a projected pattern is proposed. Experimental results are presented to demonstrate how the two systems are used for human-robot-interaction in robot grasping applications.