| By comprehensively utilizing text,image,video,and audio,multi-modal models for representation,alignment,transformation,and generation have drawn more attention from academia and industry.As one of the emerging research fields,Vision-Language multimodal models have a good data foundation and a wide range of downstream applications such as Image Captioning,Text-To-Image Generation,and Text-Image Retrieval.In order to effectively fuse multi-modal features,the dual-stream architecture is a popular paradigm for building Vision-Language multi-modal models since it can decouple the representation process of different data and convert data into feature tensors by pre-computing.Based on the literature review,this thesis summarizes four problems that restrict the efficiency or reasoning ability of Vision-Language multi-modal models,which are time-consuming image tokenization in visual models,slow inference of Transformerbased classifiers in Image-Text Matching,weak entity understanding ability of multimodal models,and weak numerical reasoning ability of multi-modal models.To solve these problems,this thesis proposes effective solutions.Academic contributions are summarized as follows:(1)Image tokenization based on image token co-occurrence and reordered codebook:A visual model can produce image embeddings by image tokenization and trainable vocabulary.Vector Quantization(VQ)is a frequent operator in image tokenization.Vector Quantization needs to search the entire codebook to find the closest quantized value for the query vector.Therefore,time-consuming Vector Quantization harms the efficiency of the visual model in the dual-stream architecture.To this end,we use hyperbolic embedding to capture the co-occurrence information of image tokens for gathering together quantized values of a single image.Furthermore,for the enhanced codebook with hyperbolic embedding,we use Hilbert Curve for coarsegrained reordering of the codebook.Then we can reduce the time consumption of Vector Quantization by block search.The experimental results show that,with the image reconstruction metrics remaining almost lossless,our approach reduces 40%of the time consumption compared with the original VQ.(2)Model acceleration based on shared exit and exit layer prediction:FastBERT based on"Knowledge Distillation" and"Early Exit" is one of the models for accelerating Transformer-based classifiers.In fact,there are other factors that limit FastBERT’s acceleration ability,such as:(1)the teacher classifier and student classifiers are challenging to align because of input gaps;(2)batch size shrinkage and redundant calculation of student classifiers.For the first problem,we utilize Shared Exit to reduce the input gaps between the teacher classifier and student classifiers.Then,the accuracy of the student classifier is improved.For the second problem,a GPU hash table is used to realize Exit Layer Prediction,which avoids batch size shrinkage and the redundant calculation of student classifiers.Experimental results show that our approach outperforms FastBERT and other baselines in accelerating Transformer-based classifiers.(3)Contrastive learning based on external entity information:Entity information is not considered in the training process of IBNS(In-Batch Negative Samples)contrastive learning,which is commonly used by multi-modal models.IBNS-trained multi-modal models have a weak understanding ability of text entity and image entity.To solve this problem,we propose the AGREE approach,which integrates external entity information with IBNS contrastive learning,enhancing the entity understanding ability of multi-modal models.Besides IBNS,AGREE can jointly optimize multiple entity-related loss functions,such as text entity augmentation loss,text entity mask loss,etc.Experimental results show that AGREE achieves better metrics than baseline models on multiple text-image retrieval tasks.Ablation experiments show that every entity-related loss function is effective.A case study shows that AGREE helps multi-mode models to enhance the entity understanding ability.(4)Numerical reasoning based on entity information and math knowledge:Large models like GPT-4 and ChatGPT are changing the human-computer interaction paradigm.Therefore,multi-modal models such as text-image retrieval and text-to-image generation need numerical reasoning ability to understand the complex user inputs.Two problems are limiting the numerical reasoning models:(1)bias introduced by ordinal indexes;(2)insufficient training data.For the first problem,we propose an unsupervised approach named Entity Random Indexing.For the second problem,we augment the mathematical expressions into multiple equivalent transformations and obtain stable numerical reasoning ability by ensemble learning.Experimental results show that our numerical reasoning model is better than the existing sequence-based,tree-based,and pre-training competitors.In this thesis,we propose effective solutions to solve problems that restrict the efficiency or reasoning ability of Vision-Language multi-modal models:(1)We utilize Hyperbolic embedding of image tokens,reordered codebook,and blocking search.With the image reconstruction metrics remaining almost lossless,the time consumption is reduced to about 60%of the original Vector Quantization;(2)We utilize the shared exit and exit layer prediction to outperform FastBERT and other baselines in accelerating Transformerbased classifiers;(3)We integrate external entity information with IBNS contrastive learning.The multi-modal model,with enhanced entity understanding ability,achieves better metrics than baseline models on multiple text-image retrieval tasks;(4)We utilize Entity Random Indexing and mathematical expressions augmentation.The numerical reasoning ability is better than the existing sequence-based,tree-based,and pre-training competitors.The effectiveness,universality,and application value of approaches proposed in this thesis are verified by a large number of experiments. |
