Visual localization, semantic video segmentation and labeling using satellite maps

In this dissertation, I propose vision-based geo-localization and segmentation methods that make use of semantic and appearance information from satellite images. First, I present a framework for vision-based localization of moving platforms by registering perspective camera images to satellite maps and by employing particle filter tracking techniques. I present different versions of the localization framework for stereo and monocular imagery. Second, I propose a novel computer vision method that uses semantic elements for efficient localization of a given aerial image in a large search area. In this method, I use buildings on the aerial image as semantic elements and make use of building neighborhood structures to obtain accurate localization results, efficiently. For this problem, I perform tests on a very large city building dataset with 300K buildings. Third, I propose a novel framework for semantic segmentation and labeling of videos that propagates semantic information from satellite maps on to globally localized video frames. This method generates accurate labeling of semantic elements without performing any prior learning on the video itself. Finally, in order to understand and extract semantic information from satellite images, I investigate algorithms for semantically labeling satellite images; mainly focusing on labeling buildings, roads, sidewalks, and crosswalks from satellite images. I propose novel techniques to estimate sidewalk paths occluded by trees on satellite images.