Research On Sensor-based Multimodal Human Activity Recognition Method

Human activity recognition has broad application prospects and potential economic value in the fields of smart home,motion monitoring,game control,smart medical care,and elderly patient monitoring.Therefore,sensor-based activity recognition has been highly concerned by many researchers and has made great progress.However,most of the existing work concentrates on some basic activities(such as walking,running,going upstairs,going downstairs,etc.)or fall detection,and less attention is paid to the analysis and identification of transitional activities contained in continuous human activity data.This thesis comprehensively considers the basic activities and transitional activities,and conducts multimodal activity recognition research from offline and online real-time.The main work of this thesis includes:Firstly,the shortcomings and challenges of existing activity recognition methods are summarized,and key issues and typical methods involved in activity recognition are introduced,such as data acquisition,preprocessing,feature extraction and classifier.Then,experimental analysis is performed on our own collected dataset and public dataset to determine appropriate data preprocessing methods,eigenvalues,segmentation window sizes,and classifiers.Secondly,aiming at the offline recognition problem of multimodal human activity,a method based on time-period activity extraction is proposed to realize accurate recognition of basic activities and transition activities.Firstly,sliding window is used to segment the original sensor data and features are extracted based on window segmentation,and features of transitional activities are resampled.Then,cluster analysis with K-means is used to aggregate activity fragments into time periods.Next,generally realize the classification of basic activities and transition activities according to the shortest duration of basic activities,and deal with the hidden phenomenon of the basic activities.Then,according to the types of two adjacent basic activities,the time period activities between them are processed to determine their activity categories.Finally,the Random Forest classifier is used to accurately identify the basic activities and transition activities.The experimental results on the public dataset show that the proposed method can effectively identify different basic activities and transition activities,and the average recognition accuracy for each activity is higher than 97%.Thirdly,for online real-time recognition of multimodal human activity,this thesis designs and implements a CNN-LSTM based activity recognition model,in which CNN is used to learn local features from raw sensor data,and LSTM is used to extract time dependence from local features and realize the fusion of local features and global features.In this thesis,model training and parameter tuning are carried out respectively for CNN and LSTM.Then,CNN and LSTM are combined to describe the basic activities and transition activities in detail,so as to accurately identify the two activity modes.Experiments on two data sets show that the model designed in this thesis has better recognition rate and higher real-time performance.Finally,based on the above research,an Android-based activity data acquisition and recognition system is designed and implemented,and the CNN-LSTM model is integrated into the Android APP to realize online real-time activity recognition.The system includes Android APP and server system.APP is used for data collection,offline activity analysis,deep learning based real-time recognition,data preprocessing,model building and training.