Extensive operators in lattices of partitions for digital video analysis

An increasing number of video and multimedia applications rely on the possibility of interactively manipulating, distributing, indexing and accessing audio-visual (AV) information based on its content. Conceptually, video analysis can be conducted at least at two different levels. The first one decomposes a video sequence into its parts (the scenes or events that occur in it), i.e., it finds the structure of a video. The second one extracts the objects present in each scene. Both tasks are essentially segmentation problems. This dissertation addresses these two important problems in video analysis within the morphological approach of designing extensive operators in product lattices of partitions and multivalued functions. An extensive operator in such lattice creates a hierarchy of partitions, without introducing new (spatial or temporal) contours. The proposed framework allows for the analysis of fundamental algebraic properties of region merging/classification segmentation algorithms, and for the study of their connections with other classes of morphological operators. The design of the operators relies on a Bayesian formulation, which allows for the incorporation of prior knowledge of the problems, and offers the advantages of a principled methodology.; SVO extraction is the process of segmenting and tracking arbitrary collections of image regions—scene objects—with pixel-wise accuracy, and represents a crucial task for the application of the MPEG-4 standard. In view of the ill-posedness of this problem, a solution consists of the integration of user-assisted definition of SVOs and automatic segmentation and tracking. We conceive SVO extraction as an issue of designing extensive operators on a lattice of spatial partitions. As a result, we propose a framework based on two steps: partition generation and application of extensive operators on the generated partitions. Based on a statistical analysis of the watershed algorithm for image segmentation, we develop a multivalued watershed algorithm that incorporates color and edge information in order to generate fine partitions with preservation of object contours. For object tracking, we propose single-view and multiple-view operators, based on the extraction of spatio-temporal features from one or more scene views, and on the design of Bayes classifiers. Theoretical properties of the proposed operators are established. Experimental results on a standard video database show that our schemes improve the precision of the extracted SVO boundaries compared to traditional techniques, and allow for multiple SVO tracking in static and moving camera scenarios.; The structure of home (consumer) videos is defined by the list of events (clusters) depicted on them. Based on the observation that home filmmakers' behavior induces visual and temporal structure in consumer video content, we propose a framework for video structuring consisting of two steps: the generation of temporal (shot) partitions of the video sequence support, and the application of extensive operators. The proposed operator is a probabilistic hierarchical clustering algorithm which is based on the development of statistical models of visual similarity and temporal duration and adjacency of home video segments, and in the design of a sequential binary Bayes classifier. Extensive experimental results on a large real-life home video database validated the performance of our methodology with respect to several criteria, and demonstrate the effectiveness of the proposed solution.