| Vision-and-language navigation(VLN)is a complex multimodal task that requires an intelligent body to reach a target location based on visual and verbal cues.This task has received much attention in recent years due to the existence of practical and popular entities in the real world,such as home robots and personal assistants.Although many extensive studies on Transformer-based vision and language have emerged,the difficulty of navigating in real environments and the complexity of human linguistic commands make it still an open problem for intelligence to achieve high accuracy in Vision-andlanguage navigation in invisible environments.Therefore,this thesis is devoted to the study of Vision-and-language navigation methods in unknown environments,focusing on two perspectives: multimodal feature matching and decision network architecture to enhance the perception and decision-making of navigating intelligence.The research in this thesis is specific to the following points.1.A Feature Matching Cross-Modal Object-Aware Transformer for Vision-andLanguage NavigationA multimodal feature matching method is proposed for the problem that it is difficult for existing methods to accurately match and fuse object features in the case of multimodal inputs.The method solves the problem by classifying redundant multimodal data into two levels of object descriptions,visual and linguistic,and constructing a triad of multimodal object references.The description at the linguistic level is used to classify the overlapping objects at the visual level and to provide the necessary alignment information for the image text.At the same time,using the triad information as alignment anchor points and combining it with the attention mechanism,we provide richer feedback information for intelligence to navigate in order to help intelligence to better complete feature matching and fusion.2.A Cross-Modal Object-Aware Transformer for Vision-and-Language NavigationTo address the problem that it is difficult for navigating intelligence to use multimodal information to assign weights and make accurate decisions when dealing with complex features,a composite object and scene feature decision-making method is proposed.The method combines multimodal BERT and attention mask mechanisms to provide object feature decision information and scene feature decision information,while a scene feature vector and multiple object feature vectors are used to represent candidate views,and finally,the attention mechanism is used to combine visual and object decision information to derive an intelligent body navigation decision result.Tests are conducted on extensive datasets R2 R and R4 R by the Mattport3 D simulation platform to verify the validity of the model.3.Visual-verbal navigation system in indoor environmentsA Vision-and-language navigation system in a realistic scenario is designed and implemented for the needs of Vision-and-language navigation in an invisible indoor environment.The system combines the visual language navigation method proposed in this thesis to follow human commands to discriminate the navigation direction of the current indoor image,thus helping users to navigate in real scenarios.Finally,this thesis shows the actual operation of the system to verify its practicality. |
