Semantic-Based Multimodal Knowledge Question And Answer Method For Crop Pests And Diseases

Crop diseases are important biological disasters in agricultural production and are one of the major constraints to sustainable agricultural development.The combination of artificial intelligence technology to assist crop pest prevention and control decisions is an emerging cross-cutting hotspot and frontier in this field,and crop pest knowledge quiz systems are one of the main directions.At present,most crop pest knowledge Q&A research is oriented towards textual resources,with less research on multimodal data such as images,videos and audios.With the widespread application of big data technology in the field of crop pests and diseases and the continuous improvement of information storage methods,crop pest and disease data are characterized by multimodality.There are still many challenges in the existing multimodal agricultural knowledge Q&A technology in terms of integrating multimodal semantics and improving matching accuracy.How to cope with the new scientific problems of the new paradigm of multimodal crop pest knowledge quiz,how to effectively parse the image semantics into question text,and how to reduce the semantic gap between multimodal information in agriculture have become hot issues of current research.Regarding the multimodal knowledge of crop pests and diseases,there are three challenges that need to be solved: 1)Restricted by knowledge dissemination channels and the complexity,regional and seasonal characteristics of agricultural production,a complete and easy-to-use knowledge base system has not been formed in the field of crop pests and diseases in China,and the value of multimodal data has not been fully explored;2)Capturing the sentiment tendency of question and answer accurately can effectively improve the question and answer effect,while general text sentiment analysis models lack the analysis and utilization of text characteristics in the field of crop pests and diseases,and therefore perform poorly on agricultural texts;3)Traditional agricultural knowledge Q&A research is oriented towards unimodal and shallow semantic matching,which lacks comprehensive consideration of sentence multi-source information,making it difficult to meet users’ requirements for accurate access to pest and disease identification and control information.Thus,we conduct a multimodal Q&A study on crop pests and diseases from the perspective of multimodal semantic understanding,combined with the characteristics of agricultural data.The main research results of this thesis are as follows:(1)A multimodal knowledge graph construction method for crop pests and diseases incorporating image semantics is proposed.In order to solve the problems of high dimensionality and high noise in the construction of the image entity layer,We have devised a fusion of Artificial Bee Colony and Gradient Boosting Decision Tree algorithm(ABCo DT)to achieve the screening and optimization of the input features.The multimodal knowledge map of crop pests and diseases constructed in this thesis contains 28 crop categories,7028 entities,and 484,932 attribute relationship tables,laying the foundation of the knowledge base for the subsequent knowledge quiz.(2)A method for sentiment analysis of crop pest question sentences with semantic enhancement of domain entities is devised.To address the problem that the semantic information of domain entities is neglected in the sentiment analysis of crop pest question sentences by traditional methods,leading to a highly biased model for the overall sentiment polarity analysis,the method predicts the masked domain entities through the contextual information in the sentences to learn more comprehensive sentiment semantic information.To further improve the effectiveness of sentiment analysis,this thesis constructs a domain sentiment lexicon using a sentiment propensity point mutual information algorithm based on the generic sentiment lexicon.The experimental results show that the accuracy of the proposed entity-level sentiment analysis method combined with sentiment dictionaries improves by 1.57%,1.9% and 2.38% respectively on three datasets including Chn Senti Corp,Online＿shopping＿10＿cats and the self-built dataset in this thesis,compared with the original baseline model.(3)A deep language model based on knowledge graph and deep semantic matching mechanism named KD-BERT was designed.To address the problems of low accuracy and recall due to small sample data volume and inadequate semantic understanding in the existing crop pest knowledge quiz,the knowledge graph is used as an exogenous knowledge to supplement the quiz database,solve the problem of insufficient training samples for some pests and diseases,and adopt a pre-training model called BERT to generate sentence vectors with context and location information,and add Deep Structured Semantic Models(DSSM)on top of BERT to better learn the relationship between question-answer pairs.kd-BERT was used in LCQMC,Anthem dataset set and the crop pest Q&A dataset constructed in this thesis were compared with DSSM and ESIM,and the experimental results showed that its average accuracy,recall and F1 values were higher by 8.68%,2.00% and 5.20% respectively.