Research On Media Processing Pipeline Supporting Non-photoreali-Stic Rendering

Media data is defined as the carrier for digital images,videos,models,animations and interactive contents.Media processing research is based on the theory and technology of capturing,storing,processing and display stage.These stages are connected by the media processing pipelines which are widely applied in the fields such as digital creativity,interactive entertainment,computer simulation,visualization in scientific computing,medical visualization,digital education and publishment.Media processing pipelines are closely related to daily life,for example,every image captured by a camera or every image uploaded on social networks like Baidu or Renren is processed by a specific image processing pipeline which is one of the most important role in media processing pipelines.With the increasing development of technology in the field of digital entertainment,film and television production,online games and teaching,the quality demand on images and videos are getting higher.With the development of media capturing equipments,more and more large and high resolution image libraries,video libraries or 3D models become the main input of media processing pipelines.However,the complex algorithms of the original media processing pipeline can not meet the user’s real-time interactive requirements.With the increase of data sources,highter efficiency needs to be improved.In the other case,in the augmented reality applications,it depends on the computer generated information to set a mixed scene,and inspire users to do some creative work in reality,which requires more artistic expression and appeal rendering results producing by the pipelines supporting multivariate data fusion.This paper presents a framework for media processing pipeline supporting non-photorealistic rendering,and selects a basic problem as the starting point of the research which is the optimization of image pipeline,and then carries on the research on the key stages in the media processing pipeline and the key technology in the area of non-photorealistic rendering.The innovations and contributions of this paper can besummarized as the following aspects:(1)Proposed amedia processing pipeline framework supporting non photorealistic rendering.Based on the characters of non photorealistic rendering technology,this paper proposes amedia processing pipeline framework supporting non-photorealistic rendering.This architecture makes it easy to make video layered fusion and make a variety of artistic effects combined with virtual and real information.It will provide a new solution to the problem of video style rendering in the field of TV production,internet chat,cartoon animation,cartoon effects in online games and so on.(2)Proposed a method for automatically accelerating image pipelines.At eachimageprocessing stage of the media processing pipeline,it works by transforming loops over the image into coarser loops that effectively "skip" certain samples.These missing samples are reconstructed for later stages using a number of different interpolation strategies that are relatively inexpensive to perform compared to the original cost of computing the sample.We also describe a genetic algorithm for automatically exploring the resulting combinatory search space of which loops to perforate,what manner,how much,what reconstruction method to be used.We also present a prototype language that implements image perforation along with several other domain-specific optimization and show results for a number of different image stages and inputs.(3)Proposed an algorithm for video stylization based on PatchTable.In the process of reconstruction,tofind the approximate image patch match in the nearest neighbor distance effectively,it is better to apply PatchTable technology to get some good patch matching seed.Firstly,the initial corresponding seed can be improved by the pre-computed k nearest neighbor.With the help of the natural consistency in the image,it can quickly spread to the surrounding areas.The algorithm can offload as much of the computation as possible to a pre-computation stage that takes modest time,so patch queries can be as efficient as possible,so as to achieve the real-time user experience.This method can be used to solve the real-time problem of the nearest neighbor query for large data sets,and the problem of temporal coherence in the non-photorealistic rendering based on the specific artistic style.(4)Proposed a method to render smooth stylized contours of 3D meshes with temporal coherence.Based on the feature line extraction of 3D model(contour,the normal information of the triangles etc.),this paper analysis the sampling points of feature lines according to temporal coherence,and proposes a new stylization parameter transformation in the non-photorealistic rendering in order to keep the temporal coherence.It works by making full use of the parameters of the new sampling point and transferring them from three-dimensional space to two-dimensional space between the two frames.Thus it is easy for the current frame to keep the style consistency from the previous frame in the subsequent steps.It can void the disturbance and mutation caused by the missing information when three-dimensional topological change during the animation.The method can be applied to a variety of models and generate smooth and coherent style of drawing contour lines,including coarse and non-uniform meshes.This paper is to extend the function of the existing image processing pipeline and propose an automatic optimization method.On the one hand,it makes the pipeline more efficiency,on the other hand,supporting non photorealistic rendering,which can achieve a variety of stylized effects for fusion based on multivariate data,expands its application range.