Research On Semantic Knowledge Extraction For Domain-Specific Images

With the development of knowledge graph technology,the construction of domain knowledge graph has gradually become a research hotspot.However,in the past,when constructing knowledge graph,people usually pay more attention on the knowledge in the text,while ignoring the visual knowledge in multimedia data such as image,video and so on.Therefore,these knowledge graphs play a limited role in some visual questions and answers.In this paper,the entities and semantic triples obtained from image description text and image vision are regarded as the image's relevant semantic information.Based on semantic information,a multi-modal knowledge graph construction method is proposed,and taking domain data related to human behavior as an example to obtain the image's relevant semantic information for constructing a multimodal semantic knowledge graph.The following research work has been done in the paper.1)A variety of information fusion strategies are designed to fuse the text semantics and relationships extracted from image description text by different information extraction tools.The image description text describes the visual semantic information of the image.OpenIE and Reverb are used to extract the text entities,relationships and text semantic triples in the image description text,and various strategies are designed to fuse the above extracted contents respectively to obtain the best fusion result.Finally,we trained the word vector model using the image description text and the fused text semantic triples.2)A strategy based on triple splicing and filtering is designed to generate visual semantic triples.The trained object recognition model Inception-V4 is used to obtain the image visual entity labels,and the obtained relationship set is used as the source of the relationship between the image visual entities.The visual semantic triple is generated through the strategy of splicing the visual entities and relationships designed in this paper.Finally,the trained word vector model is used to filter and adjust the visual semantic triples.3)According to the semantic triple sets and entity sets between images,different fusion strategies are designed to construct a lightweight multimodal semantic knowledge graph.Based on the trained word vector model,a threshold method is designed to limit the similarity of semantic triples,so as to better obtain the fusion results of text semantic triples set and visual semantic triples of two images,and a strategy based on string comparison is designed to select the common text entity and visual entity labels of two images.Based on the two strategies above,the relations between images are established.In addition,a set of relations for connecting image to image,image to text are defined.Finally,the prototype system is designed to show the method and results of this paper.