Visual Entity Linking

Entity Recognition is used in natural language processing for text-based information retrieval and semantic search. Entity recognition can also be useful for image retrieval. Linking image regions and knowledge-base entities can enable computers to better understand images. In this thesis, we propose a novel approach for linking image regions to entities in a knowledge-base. We use a combination of deep neural networks and image segmentation methods for finding bounding-boxes over images and for linking the bounding-boxes to entities in a knowledge-base. With our model, we link regions of images from the Flickr8k dataset to Dbpedia and Freebase entities. We evaluate the correctness of the recognized entities and the precision of identifying the image regions through surveys on Amazon Mechanical Turk.