Towards joint bottom-up and top-down segmentation of objects in images

Object segmentation and categorization are arguably among the most important and challenging problems in computer vision and image understanding. Segmentation deals with the separation of an image into several regions/objects of interest, while categorization deals with the semantic interpretation of such objects. The fact that objects occur in cluttered scenes and exhibit immense variability in their appearance, shape and pose across natural images, throws up several challenges. The most daunting challenges, probably, are to define object models that account for these variations and to automatically learn such models from the images. Moreover, since natural images contain several objects against cluttered backgrounds, it is not possible to define a unique segmentation for a given image, i.e., segmentation is ill-posed in general. These challenges are the primary reason for the problem of joint segmentation and categorization being largely unsolved today.;Among the existing segmentation algorithms, there are two major genres that try to make the segmentation problem well-posed. Interactive segmentation algorithms require the user to indicate which objects in the image are to be segmented by marking a few pixels as belonging to the object or the background. Such algorithms typically produce the segmentation in a bottom-up fashion by grouping pixels on the basis of their low-level visual features. Any errors in the segmentation can be corrected by additional interaction. On the other hand, category specific segmentation algorithms make the segmentation problem well-posed by assuming prior knowledge about the different possible categories of objects that can occur in an image. This prior knowledge is obtained by learning object models from sample segmentations. Such models drive the segmentation in a top-down manner and can be used in a joint fashion with the bottom-up module.;This thesis makes important contributions to both these genres of algorithms. We first present several frameworks that generalize existing interactive segmentation algorithms and address some of their limitations, such as sensitivity to the location of the user's interaction. We then present a novel formulation for category specific segmentation that helps incorporate Bag of Features based top-down information in most of the existing algorithms that are primarily bottom-up in principle.