Content-Aware Image Retargeting

Image and texture retargeting is the process of mapping the original image or texture to the target zone. The most common retargeting operation is resizing images to some given dimensions. With the emerging display devices, especially mobile devices are widely used for acquiring, displaying and editing visual media such as images and videos, it's becoming more and more common that adapting the sizes of these media so that they can be shown on different devices. The most simple retargeting algorithm is uniform scaling, however, in most cases it will stretch and distort the important content, such as human faces. On the other hand, the important features will become too small to be recognizable on small screens like cellphones. Therefore, the content-aware image retargeting technique has been widely studied and developed. Content-aware retargeting is a way to protect important content and features during the retargeting process, and avoid visual distortion and stretch at the same time.Different kinds of image editing tasks can be realized with the content-aware retar-geting technique, such as aesthetic optimization of digital photos. As all kinds of digital imaging devices have been invented and widely used, users can easily shoot as many pho-tos as they can. However, most daily users are lack of aesthetic experience, therefore it's a difficult task to touch up and improve these photos automatically.Besides regular planar rectangular area, we can also set the target zone of the retar-geting operation as some 2D manifold surface. The common seen texture-mapping is the process of pasting images to the surfaces of 3D models. But due to the complexity of sur-faces, usually texture-mapping can hardly avoid the texture getting distorted and stretched, thus many techniques have been proposed to synthesize texture directly over the surfaces.In this paper, we have studied on the approaches for image and texture retargeting and developed a series of novel algorithms. Our contributions are summarized as follows.1. Improved algorithms for computing the salience map and detecting salient regions of images. The new salience map reflects the importances of different regions in the image in a better way, thus is more suitable for content-aware image applications. We also present a segments-clustering based line detection algorithm which is more accurate, faster and more practical. The improved salience map and salient region detection and line detection algorithms leverage the study of the problem of content-aware image retargeting.2. Based on quadratic programming, a new image resizing algorithm has been pro-posed. This algorithm uses a convex optimization technique to deform the under-lying quadrilateral mesh, and the resizing energy is optimized globally, therefore minimizes the distortion and stretch during resizing. Furthermore, we extend the framework to completely avoid the underlying mesh getting flipped, encourage the important regions to be enlarged so that they can be recognized even when the image is shown with very small size, and automatically detect and protect the linear features in the images, including grid lines and arbitrary straight lines.3. The content-aware image resizing technique has been used for optimizing photo composition. A content-aware image retargeting while optimizing aesthetic com-position problem is proposed, and we use a search-based method to automatically optimize the aesthetic of the input images. First, the important regions and structures in the images are detected, then they are used to evaluate the basic guidelines for composition, and an aesthetic composition score is proposed. On the other hand, we use a compound operator of cropping and retargeting to change the sizes and relative distances between important regions in the image, thereby change the composition of the images. With the automatic composition optimizing tool, the photographing experience of those daily users who have little aesthetic history can be greatly en-hanced, and even skilled photographers can use it to speed up the process of photo touch-up.4. A new content-aware algorithm is presented to retarget a special kind of image, i.e. texture to surfaces. Based on the repeating structures in the textures, we retarget textures to manifold surfaces by directly synthesizing textures on top of them. We present a new algorithm to resample grid patches from surfaces, and use it to extend both texture optimization and patch-based sampling to 3D surfaces. Among which, texture optimization can generate results with the best quality, while patch-based sampling is faster and it's close to real-time application after further acceleration, and its results are also of good quality. Users can control the orientation and scale variation of the texture by specifying the vector fields, and they can choose to syn-thesize either uniform or progressively variant texture.