Research On Mobile Phone And Wearable Devices Based Human Activity Recognition Technologies

Automatic recognition of physical activities, which is commonly referred to as Human Activity Recognition (HAR), has emerged as a key research area in mobile and ubiquitous computing. One goal of activity recognition is to provide information about user’s activity to allow some ubiquitous applications to proactively assist users with their tasks. Most previous researches are based on computer vision technologies which could extract and recognize human activities from images and videos. Although these technologies have been widely researched in the past decades of years, they still have some disadvantages:firstly, due to the dependency of external instruments, computer vision based solutions are limited to the areas where there have been deployed the visual sensors, such as camera; secondly, besides the human activities of interests, many other information could also be extracted from images and videos, which may arise privacy problems; thirdly, image and video processing always require huge, network bandwidth and computational capacity which are hard to be provided by most on-the-shelf devices, especially when the cost is considered.Recent years have seen the fast development of mobile devices (e.g. smartphone, wearable devices) and related sensors (e.g. accelerometer, ECG), and there are also emerging researches of human activities recognition which turn to various sensors e-quipped on mobile devices. These technologies release the dependency of external de-vices, and thus could meet the requirement of "anywhere and anytime" of ubiquitous applications. The related researches could be classified into 3 classes according to the sensors used:1) motion based solutions,2) acoustic based solutions,3) other solutions such as ECG based solutions. As the most widely researched topic among the 3 classes, motion-based human activity recognition on mobile devices is the problem to be stud-ied in this dissertation. Note that, most of the related works solved the human activity recognition problem by using static pattern recognition models or simple temporal prob-abilistic models, all of which ignored an important information about human activities, that is the duration. Besides, in contrasted with image and videos, motion data has also some disadvantage due to the lack of information about environment context, and I ar-gue that it would be better to involve different kinds of sensors to provide informative observations about the context and environment of human activity.To address these problems, different types of mobile devices and sensors used in previous works will first be summarized in this dissertation, and after that, a sensing framework which is consisted of smartphone and wearable device will be introduced which could benefit from both the continuous sensing of wearable device and the plen- tiful sensors and huge computational capacity of smartphone. Based on the framework, four aspects of researches are conducted:1) I introduce semi-Markov models to hu-man activity recognition for the purpose of modeling the duration of different activities, based on which I designed duration-sensitive human activity recognition algorithms; 2) 1 explore how to fuse the data from action sensors and observations about the envi-ronment from other sensors to improve the performance of human activity recognition; 3) I study the problem of human activity prediction, and propose an application about sensors dispatching for saving power; 4) I study the problem of abnormal human ac-tivity detection, and propose a promising application which could automatically detect the aggravated assault to users. The contributions of this work could be summarized as follows:Firstly, I proposes a sensing framework which is consisted of a wearable device and a smartphone, by which the system can benefit from both the continuous sensing of wearable device and the plentiful sensors and huge computational capacity of smart-phone.Secondly, semi-Markov models are introduced to HAR problems to model the du-ration of activity, and to support observations having continuous values in semi-Markov models, GMM is used to fit the distribution of feature vectors of each kind of activi-ty. Besides, two different methods are proposed to combine observations of actions and environments from different kinds of sensors in the semi-Markov models, and their performance are compared by extensive experiments.Thirdly, I study the problem of human activity prediction, which is further classi-fied to prediction of unterminated activity and unstarted activity, and I propose solutions for each kind of the prediction tasks. Based on the prediction of duration of current and next activity, an sensors dispatching application is proposed for the purpose of pow-er saving, which allows the system to turn off the sensors and sleep for a while when current activity is recognized to be disinterested and will last a long time.Finally, I explore the problem of abnormal activity detection, and in addition to those well-known applications of health care and elder care, another application about human security is proposed, which could automatically detect the aggravated assault on users for fast emergency response. To distinguish aggravated assault with activities of daily living, I extract some informative features and design a combinatorial classifica-tion scheme, which is demonstrated to be effective using the data collected by imitating surveillance videos of real assault instances.