Learning-Based Approaches for Pixel-Level Predictio

Images are a rich source of information about our physical world. A fundamental limitation in developing interactive applications that leverage image data has been getting machines to understand what the stream of numbers composing images represents. We study the design of learning-based approaches for understanding images at a pixel level. Our work focuses on addressing the following questions: 1) What representation is most useful for pixel-level reasoning, and how can we obtain these features from image data? 2) How can we design and train deep models for problems where each pixel can have multiple correct interpretations? 3) How can we exploit spatial coherence within adjacent image regions to assist with reasoning about content at the pixel level?;We show that designing pixel-level descriptors by incorporating image-level information (in addition to information from the local neighborhood of a pixel) leads to significant improvements in our ability to estimate depth from a single image.;As it is challenging to learn such pixel-level representations due to a lack of labeled training data, we also study approaches for learning pixel-level representations in unsupervised settings, e.g., colorizing grayscale images and image inpainting.;We propose an architecture targeted at improving the ability of models to predict pixel-level data when there are multiple correct outputs possible for each pixel. We show how to train our proposed architecture to allow for diversity within the output hypothesis space.;Finally, we explore image inpainting as a mechanism for exploiting spatial coherence for improving the performance of patch-based image compression models. Our study reveals that there is a need to design new architectural components for extracting pixel-level information for performing inpainting. We also show that compression performance improves the most when the inpainting model is trained jointly (for an inpainting and compression objective) with a modified learning objective, allowing our model not only to learn how to inpaint effectively but also to discover what to inpaint for bringing about the greatest improvement in compression.