Natural language processing of visual language for image storage and retrieval

A natural language processing system is constructed that can interpret text based descriptions of physical objects and create three-dimensional graphic models of the objects. The system design is based on the integration of current theories of human linguistic and visual processing. The system embodies principles for the representation and processing of objects and interpart relationships that permit communication between verbal and vision subsystems. The principles are primary dimensionality, partwise decomposition, and part relationship schematization. The meanings of some shape words such as "width" or "oval" are sensitive to the primary dimensionality of subject. Partwise decomposition simplifies communication by defining differences between complex objects as simple differences in their parts. The meanings of the relations between parts of an object are idealized into a simpler part relationship schematization with few attribute slots.;Domain knowledge and domain independent knowledge of physical part interactions are used to control semantic interpretation and build internal models of the meaning of the text. The domain independent knowledge includes mechanisms for processing selected shape and spatial descriptions. Semantic interpretation of language involves the selection of prototypical shapes from a three-dimensional domain model. Additional linguistic constructs specify modifications to the shape of the prototypes. In a controlled experiment, the system processed descriptions of leaves and acorns from a field guide. Subjects were able to match original text to computer generated images as well as they could match the text to the artist generated images of the objects from the same field guide. This finding is evidence for the descriptive adequacy and computational tractability of this approach to the integration of language and visual processing.