Cross-Domain Sign Language Gesture Recognition Using Millimeter-Wave Radar Signals

Millimeter-wave radar is widely used in sign language recognition because of its nonwearable,light-independent,privacy-preserving,and high-precision perceptual properties,which have sparked a lot of interest in the field of sign language gesture detection.This not only breaks down communication barriers in public life for hearing-impaired and speechimpaired people,but also enables intelligent family life interaction.Traditional recognition systems’ performance is poor when faced with environmental change,location change,and user gesture deviation(target domain).When addressing such issues,existing wireless sensing systems require large-scale data collection or deploy additional equipment to extract domain-independent features,which will result in higher data collection,labeling,and computing costs.This thesis investigates how to use a single millimeter wave device to improve recognition accuracy and robustness when the source domain’s training samples are limited,and proposes mm Sign,a mm Wave-based cross-domain sign language gesture recognition system.The contents of this thesis mainly include the following aspects:1)To address the issue of the recognition performance degradation in the new environment,we extract environment-independent features Dynamic Velocity Information(DVI)and Dynamic Range Information(DRI)to achieve fully zero-effort recognition in the cross-environment domain.Furthermore,to address the issue of performance degradation in the new location,we employ the spatiotemporal characteristics of DVI and DRI to construct data in different location domains through data augmentation to replace the training data collected and reduce data collection costs.2)We implement a pseudo-label-based domain adaptation neural network DANet to address the problem of poor recognition performance in the new user domain caused by different users’ gesture deviation.The key insight of DANet is that a model was trained with labeled data and unlabeled data with pseudo-label,enabling DANet to learn the mapping relationship between source domain and target domain,which can weaken the influence of the domain and improve the recognition performance in the cross-user domain.We have implemented mm Sign to perform sign language gesture recognition cross 2environments,5 locations,10 users,and 20 sign language gestures.For in-domain,crossenvironment,cross-location,and cross-user,mm Sign achieves an average accuracy of99.16%,96.5%,91.16%(up to 96.23%),and 90.62%,respectively,which demonstrates the robustness and effectiveness of our system.