Online development of cognitive behaviors by a robot: A case study using auditory and visual sensing

Audition and vision are two major sensory modalities for humans to sense and understand the world. Although a significant progress has been made in automatic speech recognition and visual object recognition, the fields still face a tremendous amount of difficulties.; Motivated by the autonomous development process of humans, we are interested in building a robot that automatically develops its auditory and visual cognitive and behavioral skills through real-time interactions with the environment, which typically involves humans. We call such a robot a developmental robot. To resolve the technical challenges for a developmental robot, three basic techniques have been developed and implemented: (1) A fast incremental principal component analysis algorithm, the complementary candid incremental principal component analysis (CCIPCA) algorithm; (2) A customized version of hierarchical discriminant analysis (HDR) for long temporal contexts; (3) A developmental architecture and algorithm that integrate multimodal sensing, action-imposed learning, reinforcement learning, and communicative learning.; Based upon the above three basic techniques, we have designed and implemented a prototype robot that learns cognitive behaviors from simple to complex: (1) Grounded speech learning. The system develops its audition-driven behaviors through physical interactions with the environment. This is the first attempt we know that realizes developmental auditory learning from unsegmented and unlabeled speech streams without using priori knowledge such as handcrafted word models or task-specific heuristic. This emulates a mode in which human children learn. (2) Task transfer. The system applies what has been learned in one context to new contexts (tasks) through verbal instructions from trainers. This is the first system of this kind in an autonomous mental development (AMD) mode. (3) Semantics learning. The system acquires simple semantics through real-time multimodal inputs (vision and audition). This is the first robot with both developmental auditory and developmental visual capabilities to do online learning. It takes advantage of the spatiotemporal continuity of the physical world during learning. Among the above three learning types, the first one uses only the reflexive capability of the proposed architecture, the second one uses an additional priming capability, and the third one uses an additional multimodal integration capability.; The work reported in this thesis serves as a starting point of the on-going research on a highly integrated developmental robot.