Research And Implementation Of Four Cameras Based Multi-touch Technology

The ideal human-machine interaction pattern is to use the cognitive habits and styles, which are naturally formed during people's communications with the nature. Multi-touch technology is a user-centered natural computer interaction method, and refers to a set of interaction techniques, including a touch screen or touchpad, as well as software that recognizes multiple simultaneous contact points. It allows users to interact with a system through the concurrent use of several fingers and permits multiple users to work together through collaborative hand gestures, which reduces the cognitive burdens on users and fully releases potentials of users'interactions, thereby it has become one of the hot spots of human-computer interaction research. It has widespread application prospects in many areas, such as the military or civilian command and decision-making, commercial retail, public information inquiry, information appliances, the mass media as well as the education and so on.Multi-touch system research includes the multi-touch platform construction and its detection, localization and tracking methods of multi contacts, as well as multi touch gesture description method and recognition method. Currently multi-touch platform construction method is one of the most extensive sections, and has emerged in a large number of platforms with successful applications, which contributes to the emergence of multi touch technology. However, most of the existing multi touch platforms are costly, particularly in the construction of large-size platforms, in addition to special requirements on the interactive environment or user interactions, and inconvenient installations, which require modification of the existing display platforms and occupies large spaces, therefore they owns little transplantation. Furthermore most of the detecting, locating and tracking methods are unable to resolve occlusions between contacts, and with low locating accuracies. These issues limit the popularization of multi-touch to some extend.Multi-touch gesture is the bridge between the platform and the user. Current multi-touch applications are limited to a set of predefined gestures, which could not be customized according to application constraints and users'interaction habits, consequently require users to learn, memorize and adapt to specified gestures and limit user's freedoms and potentials, which are separated from user-center natural interaction concept. Furthermore, the multi-touch gesture recognition methods mostly require sophisticated trainings, and any definition changes may easily subvert recognition results of existing gestures, moreover, it is lack of flexibility for users'differences. These problems make it the main bottleneck to promote the widespread of multi-touch application.This dissertation addresses these issues as follows:1) Four collaborative cameras based multi-touch platform construction method.It requires only simple infrared light sources and four wide-angle infrared cameras, and is suitable for construction of various sizes of multi touch platforms, and requires no alterations to existing monitors, on the contrary it can be installed directly in front of existing monitor. It has such characteristics as easily setup, convenient installation, and transplantation, low cost, etc.2) Multiple contacts detection, location and tracking methods.Detecting multiple contacts in sub-pixel based on gray gradient of a projected image for tracking points, and then use the lookup table method or Vanishing Point method or stereo calibration method to build direction lines in each camera, and solve the occlusion by position and space constraints, which can achieve precise location. Finally use an improved nearest neighbor data association method to achieve the stable tracking of contacts. The improvement includes building the weighted bipartite graphs for contacts falling in the estimation of Kalman filter based on location, speed and direction features, and use the KM algorithm to find the maximum matching of the weighted bipartite graphs to build the data association of multiple contacts.3) A semantic and application-oriented interactive multi-touch gestures description method.The description results of multi-touch ontology based on OWL can utilize reasoning tools to identify and guide the resolution of conflicts, which allows the user to customize gestures according to application constraints and their own preferences, and it can easily add, delete, or modify definitions, or derive new gestures from existing results. It is applicable to arbitrary multi-touch platform; therefore it has good extensibility, reusability and semantics.4) Multi-touch gesture recognition method based on shape features and constraints of trajectories.It simplifies the templates building templates, and users'modifications of gesture definitions or new gestures inherited from existing gestures can be directly stored as templates, which requires no complex re-trainings. Moreover, it is robust to different users'operating speed, amplitude differences. In addition, the method is suitable for any complex composition of trajectories for any multi-touch platform.5) Multi-touch system implementation.A four collaborative cameras based multi-touch system is set forth, including a multi-touch platform of 2.4m*1.35m, an application-oriented multi-touch gestures description tool, and a 3-dimensional object manipulation application. A questionnaire for the application is designed and analyzed to build the mostly acceptable gesture sets, and a set of users are invited to testing methods advanced here. Results prove the applicability of these methods.