Deep Learning Based Visual Repeated Pattern Analysis And Application

The visual repetitive pattern refers to a pattern,structure,or object that appears repeatedly in space or time,and forms a certain sequence within the image or video.Specific examples exist in a wide range of human activities and products,windows of buildings,products on shelves or in production workshops,and human actions in videos,etc.Therefore,the research and analysis of repetitive patterns have extremely high academic research value and industrial application value.Today,deep learning has been widely explored in many vision applications,greatly promoting the development of computer vision.However,the research and analysis of visual repetition patterns based on deep learning are still in the early stage of exploration.The reason is the definition of visual repetition patterns is a high-level semantic concept from human recognition,heavily relying on human prior knowledge.Most of the existing convolutional neural network is only designed with the target of the application directly,and the logical reasoning process based on a priori knowledge is ignored,so it is difficult for the existing deep neural network to solve the problem with high-level semantic reasoning,like repetitive pattern analysis.However,considering the powerful feature extraction ability of the convolutional neural network,the analysis and application of repetitive patterns based on deep learning should have great potential.In this paper,to solve the dilemma of deep learning in the analysis and application of visual repetitive patterns from the lack of prior knowledge,we explore how to leverage the human prior knowledge of visual repetitive patterns to the deep learning networks.We aim to the following visual tasks,including counting the repetitive patterns in a video,detecting repetitive and dense objects in an image,and detecting defects hidden in the repetitive pattern.The most innovative works are summarized as follows:1.In this paper,we tailor a context-aware and scale-insensitive framework,to tackle the challenges in repetition counting caused by the unknown and diverse cycle-lengths.Based on the observation on the cycle rule of the repetition motion,we implement a coarse-to-fine cycle refinement method.It avoids the heavy computation of exhaustively searching all the cycle lengths in the video,and,instead,it propagates the coarse prediction for further refinement in a hierarchical manner.We secondly propose a bidirectional cycle length estimation method for a context-aware prediction.It is a regression network that takes two consecutive coarse cycles as input and predicts the locations of the previous and next repetitive cycles.To benefit the training and evaluation of the temporal repetition counting area,we construct a new and largest benchmark,which contains 526 videos with diverse repetitive actions.Extensive experiments show that the proposed network trained on a single dataset outperforms state-of-the-art methods on several benchmarks,indicating that the proposed framework is general enough to capture repetition patterns across domains.2.This paper proposes a dense repeated object detection method based on regional scale regression loss to improve the performance of the algorithm when there are only a small number of annotations that can be used during training.The main contribution of the method is the regional scale regression loss function for dense object detection scenes.The proposed loss function is implemented by adding a regional-scale prediction branch over the basic detection network.By predicting the regional scale of all the targets within the region,the regression loss can be used as extra supervised information to guide the network to learn feature extraction,thereby reducing the overfitting in the scheme of learning with fewer samples.Extensive experiments show the effectiveness of the proposed dense object detection method under the condition that a small number of training samples are available.3.In this work,a de-deformation defect detection network(D4Net)is proposed to detect defects of a non-rigid product with deformation in a given image and its corresponding reference image.The challenges tackled by this network is how to use the repeatability of visual patterns to find defects on the surface during the production monitoring of non-rigid products with repeated visual patterns.The proposed method focuses on differences between high-level semantic features extracted from the deep neural network to emphasize the region of possible defects.In training,a marginal loss is proposed to improve the separability between defects and deformation in images with large patterns.Experimental results show that the D4 Net yields the best performances of 96.9% accuracy and 91.7% F-measure in a real industrial dataset consisting of 67 K images of lace fabric with large patterns from a worldwide top-10 lace fabric manufacturing company.This validates the effectiveness of the proposed method in industrial applications.