Content based image retrieval using evidence combination

Content based retrieval of multimedia has always been a challenging task. The enormity and the diversity of the multimedia content on the web adds another dimension to this challenge. In this work, we examine ways of combining visual and textual information for content based indexing of multimedia on the web. In particular, we examine different methods of combining evidences due to face detection, Text/HTML analysis and face recognition for identifying person images. We provide experimental evaluation of the following strategies: (i) Face detection on the image followed by Text/HTML analysis of the containing page; (ii) face detection followed by face recognition; (iii) face detection followed by a linear combination of evidences due to text/HTML analysis and face recognition; and (iv) face detection followed by a Dempster-Shafer combination of evidences due to text/HTML analysis and face recognition. These strategies were implemented in an automatic web search agent named Diogenes and compared against some well known web image search engines. The latter includes commercial systems such as AltaVista, Lycos and Ditto, and a research prototype, WebSEEk. We report the results of our experimental retrievals where Diogenes outperformed these search engines for celebrity image queries in terms of average precision. Evidence combination was also used for implementing a fully automatic pseudo-feedback method. An initial set of search results are obtained using face detection and text/HTML analysis alone. These initial set of results are then reevaluated by using a clustering algorithm to form a face database for that person with no user interaction. This face database enables Diogenes to use face recognition to improve upon results obtained by face detection and text/HTML analysis alone.