| With the development of deep learning,sign language recognition and gait prediction have gained rapid development as an important field of human-computer interaction.At present,the public data sets of sign language recognition and gait prediction are basically images or videos,and most of the research methods are based on these images and videos,but the images and videos are easily affected by conditions such as shooting angle and illumination,thereby reducing performance.Image-based gait prediction is less accurate and difficult to use in exoskeleton robot control.In order to overcome the shortcomings of the existing public dataset,this paper proposes a multi-modal dataset based on Kinect,inertial sensor,pressure sensor and AirPods,which is basically independent of shooting angle and illumination compared with image and video-based datasets.At the same time,a deep learning method suitable for multi-modal data sets is proposed,which can effectively identify the collected sign language data sets and predict the gait data sets.This data set has been made public on GitHub.For sign language recognition,this paper builds an acquisition platform based on data gloves,Kinect and AirPods,and collects multi-modal sign language data sets of joint angle,image,bone key points and speech.The data set contains a total of 250 action sequences of 10 sentences multimodal data.Compared with the existing public data sets,the introduction of data gloves as the collection device,the collection receipt is more stable,and the data set has more expressive ability.For the multimodal sign language dataset,the SLRNet network structure is designed by using the convoluteonal neural network in deep learning.It consists of 6 layers of convolutional layer,6 layers of normalized layer and 2 layers of fully connected layer.The input data is timing signal.Rather than common image data.SLRNet's recognition accuracy is as high as 100%,and compared with the manual feature + PCA + SVM based method and LSTM based method,SLRNet has obvious advantages.In view of gait prediction,this paper studies the problem of poor coordination between exoskeleton robots and humans,and builds a data acquisition platform based on sole pressure acquisition equipment and inertial components.It collects flat ground and fast walks,up and down stairs,up and down slopes,and left and right turns.Gait data for 8 gait patterns.Due to the lack of data collection,the HuGaDB public data set was selected for verification of the gait prediction algorithm.Aiming at the characteri-stics of gait data,the time convolutional network(TCN)is improved,and a GPNet network is proposed,which includes 24 layers of one-dimensional convolution layer,24 layers of weight normalization layer and 2 layers of full-coupling layer.The experimental results show that the gait prediction can be effectively performed,and the average prediction error is 2.49%.Compared with the improved BP neural network and LSTM-based methods,the superiority of the GPNet network is demonstrated. |
