Technology And System Research On Activity Sensing And Recognition Based On Mobile Devices

With the development of computer technology,communication technology,mi-croelectronic technology etc,the mobile devices like smart phones and smart watches come forth continuously.The mobile devices often have some capabilities of com-puting,storing and communicating.Besides,they usually integrate the sensors like accelerometers,gyroscopes and magnetometers.Consequently,the mobile devices provide the chance for information collection and data processing anytime anywhere.The development of mobile devices reduces people's dependence of desktop comput-ers and promotes the development of Human-centered Computing.In human-centered computing,activity sensing and recognition is a key technology,it senses people's activities and gets the observation data,and then recognizes people's activities(e.g.,walking,jogging,typing).With the recognized results,it can provide information sup-port and smart services for people.Because mobile devices have the advantages of lightweight,containing multiple sensors,being able to compute and store data,they provide the chance of activity sens-ing and recognition anytime and anywhere for people.It helps people to get rid of the constraints of fixed sensing equipments.Therefore,Activity Sensing and Recognition based on Mobile Devices has aroused people's wide concern.In regard to mobile de-vices,the computing resources and battery life are limited,thus the traditional activity sensing and recognition methods are difficult to work on mobile devices directly.It is necessary to research and design feasible solutions for activity recognition based on mobile devices.In recent years,there is some research work focusing on activity sens-ing and recognition based on mobile devices.However,the existing work often focuses on the recognition of coarse-grained activities,simple and independent activities,etc.Besides,they often have high training cost for activity recognition.Specifically,the ex-isting work has the following limitations.They usually focus on coarse-grained activ-ity recognition,while difficult to realize fine-grained recognition for micro-activities.They often recognize several activities or simple repetitive activities independently,while lacking the consideration of recognizing a series of activities or continuous ac-tivities.Besides,they usually depend on heavy training for activity recognition,while lacking an efficient model to correlate the sensor data with human activities.To solve the above issues and make the recognition approaches work on practical scenarios,this dissertation conducts the research on activity sensing and recognition for fine-grained activities via mobile devices,while considering the relevance of activities.We aim to accurately recognize micro-activities,a series of activities and continuous activi-ties.Besides,we also make the research on establishing the model to correlate the sensor data and human activities,to reduce the training cost of activity recognition.Additionally,considering the properties of mobile devices,we will take the problem of time latency,energy consumption,disturbances into consideration,in order to make the recognition method work on the mobile devices,which have limited resources.The main contributions of this dissertation are concluded as follows:(1)For the problem of sensing and recognizing micro-activities,we propose a recognition method based on target localization.With the recognition approach,we research the problem of text-entry into smartphones.By using a virtual keyboard(e.g.,a piece of paper with keyboard layout),we can input the text into smartphones.We implement the text-entry system and verify the recognition approach.Our system uti-lizes the built-in camera to track the micro-movements of user's fingertips,and then matches the location of a key in the paper keyboard with the location of the fingertip to detect and localize the keystroke.In addition,we adopt the optimizations like adjusting image sizes,focusing on target areas,introducing multiple threads,and changing read-write mode of images to reduce the time latency of keystroke sensing and recognition,to achieve real-time text entry with high accuracy on mobile devices.(2)For the problem of sensing and recognizing a series of activities,we propose a context-aware recognition method.With the recognition approach,we research the problem of the energy-saving during photographing in smartphones.We implement the energy-saving system and verify the recognition approach.Our system utilizes the built-in sensors like accelerometers and gyroscopes,to sense user's activity and deter-mine which state the user is in during photographing.Then,we adopt a corresponding energy-saving strategy to reduce the unnecessary energy consumption.Considering the relationship among the series of activities during photographing,we propose an activity state machine to describe the transfer relationship among activities.Then,we utilize the context information,i.e.,sequential relationship between activities,to rec-ognize the activities progressively.By introducing the context-aware information,we improve the recognition accuracy and the tolerance of recognition error for the series of activities.Then,we can reduce the energy consumption of smartphone during pho-tographing,while guaranteeing the user experience.(3)For the problem of sensing and recognizing continuous activities,we propose a recognition method based on activity contours.With the recognition approach,we research the problem of writing in air with a wrist-worn device.We design and imple-ment the prototype system,and verify the recognition approach.Our system utilizes the built-in sensors like accelerometers,gyroscopes and magnetometers to sense the writ-ing activities in air,and then recognizes an activity as a character.In order to reduce the training cost in activity recognition,we propose a contour-based activity model,which calculates the activity contour in 3D space,to correlate the sensor data with writing activities.For the activity contour,we utilize the methods like spatial transformation,contour segmentation and grammar relationship constraints etc,to recognize it as a character.Besides,we also introduce the Hidden Markov Model(HMM)to further improve the recognition accuracy.By establishing the activity contour model,we can reduce the training cost and improve the scalability of activity recognition methods.