Research On Multimedia Semantic Modeling And Applications

The user-centered service mode has become a key weapon in this information service war. Based on users'cognitive psychology and environment, this thesis pays more attention to users'inner information need, ways of needs expressing, and providing users with more abundant, convictive and efficient information service.Multimedia, such as video and image, has become one of main data types in nowadays information systems and services. But, there still exist some difficulties, for example, the variety of data-type, the complexity of extracting and expressing semantics, and the well-known"semantic-gap".The thesis discusses these difficulties that exist in multimedia service from the viewpoint of cognitive psychology, especially multimedia data's semantic extracting, expressing and retrieving. It discusses the existing modes of acquiring multimedia data and extracting semantic content, extends and refines the customary semantic gap. The thesis raises a concept-based multimedia semantic layer-oriented model and retrieval model, and constructs concept set's directed acyclic graph by description logic SHOQ(D).The original contributions of this thesis include the following:Based on characters of information users'cognitive psychology in using multimedia system, a retrieval psychology-behavior model is proposed. It is a user-centered model. It describes the course of users'choosing search engine, analyzing and expressing their information needs, interacting with the search engine, filtering the retrieval results and adjusting the retrieval behavior under the environment's stimulating.Smeulders defines semantic gap as"lack of coincidence between the information that one can extract from the visual data and the interpretation that the same data have for a user in a given situation". This thesis extends this definition and divides semantic gap into some layers, such as gaps between thought and natural language, people and computer, gaps of extracting characters, entity semantic gap and abstract semantic gap etc. This extension and refinement is helpful in finding the sticking point existing in multimedia data analyzing and retrieving. The thesis also discusses the properties of these gaps and introduces their related characters.After analyzing the process of acquiring multimedia data, multimedia semantic difficulties are found originated from their creation mode. In this traditional mode, the data creator and the user are apart. The creator can create and acquire multimedia data in an easy manner, which sacrifices the end user's using these data easily. This paper raises a script-based multimedia-data acquiring method, which means creating multimedia data according to the content of script. The method combines models of objects and rules that the world operates, re-creates or reforms the audiovisual content recorded in the script, and the data's semantic content can be got by analyzing the corresponding script.This paper raises a model of expressing multimedia data's content, which can describe multimedia data's processing, for example, video summarization and object detection. The model describes objects, scenes and events appearing in multimedia data in three dimensions, which are time, space and granularity. User can learn from the model about the story's details and general situations at the same time.A concept-based multimedia semantic representation model is proposed. It designs and extends Concept Hierarchy Net to express concepts, their relationships and distributions. It is also used to define, judge and retrieve some abstract semantic content. For description logic is a formal tool of knowledge representation and reasoning, the thesis adopts description logic to reason, construct and adjust multimedia Concept Hierarchy Net by concept subsumption, which composes a base of multimedia semantic matching and retrieval.The paper introduces and practices the models mentioned above in digital video and image's annotating and retrieving, applied in National Project"Platform of Analyzing Multimedia Semantic Character". The followed is the script-based creating cartoon system, which is to prove the new way of multimedia data creation raised in this paper.In a word, this thesis focuses on multimedia data's semantic content, pays attention on sticking points existing in extracting, expressing and retrieving multimedia semantic content. It raises a top-shaped hierarchical semantic model for multimedia data, realizes and validates these concept-based models by description logic. It explores a new way to fight for semantic difficulties of multimedia data.