A Theory Of Color Coding In The Visual Cortex And Its Applications

Color discrimination and perception are simple tasks for the normal human visual system. However, it is far from the intelligent level of human visual system in terms of the computational coding theory and method, also from the ability of human color perception from the neural basis. Color vision research is becoming the main issues in neuroscience and computer vision fields, which is mainly manifested in the following two aspects. First, from the neuroscience perspective, neurophysiology experiments still stayed in the understanding of retinal cone-signals for a long time, since it is hard to record color neuron behaviors in the visual cortex. Thus hue selectivity, color discrimination and perception from low-level to high-level can not be understood. Second, for computer vision, to extract and represent color features, it is necessary to handle appearance variations of objects/scenes due to the lightness variation, and build a color descriptor, which has the discriminative power to classify object/scene categories. Recently, with the development of the science and technology, new evidence and conclusion of the biological mechanism of color vision have come out, which promotes color vision progress and provides potential ways to solve color vision problems of great significance and value.To address these issues, this thesis focuses on the color transmission and processing in the visual cortex, especially on the spatial and chromatic properties of color neurons based on the recent advance in the neuroscience. We study and analyze the opponency, selectivity, hierarchy of color neurons, and the ability of coding visual stimuli. In particular, we exploit spatial receptive fields (RFs) of color neurons to efficiently encode color surfaces and color edges in natural images. We also extend color coding from low-level to high-level so as to implement color vision systems at different levels and apply it to large-scale data in the real world scenes. Finally, we provide a scientific description, analysis and interpretation for the proposed theory by explicitly relating computational model units to neural data, which enriches cortical color-coding theory. Therefore, the proposed color coding theory successfully mimics the biological mechanisms of human color vision.We propose a theory of color coding in the visual cortex at three levels:algorithm, application, and neuron. The main content of this thesis is organized along two lines.First, exploiting the similarity and specificity of the cortical neuron response space and color information, we propose a new color coding theory in V1area, including two algorithms:Color selective surface coding According to low frequency response and weak oriented selectivity of Single-Opponent (SO) neurons, we propose a new non-separable spatial-chromatic opponent operator based on the spatial coding of oriented linear filters. This method formulates spatial band-pass filtering as weighted linear summation of excitatory/inhibitory and opponent color components that model the SO simple units. Although it is not the first time to use opponent color channels in surface feature extraction, compared to the previous work that model color neurons as un-oriented operator (more like retinal cells), our contributions are three-folds:1) the motivation of color surface coding is to mimic the weak spatial and strong chromatic properties of cortical color neurons. The proposed method represents the spatial electivity of color neurons, which is more intuitive.2) As far as we know, it is the first work to apply nonlinear normalization mechanism to encode the gain control of color neurons that sharpen the color selectivity, and suppress the model unit response. The comparison experiments show that the proposed method can detect more meaningful interest points and salient regions with color cue.Spatial-chromatic selective edge coding Based on the color surface coding, we further propose a color edge coding theory by using spatial filtering. Since the Double-Opponent (DO) neuron signals the color edges, we formulate the detection from surface to edge as spatial filtering the SO simple units. The filters should be consisting of excitatory and inhibitory subunits with center-surround structure so as to build spatial-chromatic opponenct DO simple units that encode color edges. To yield phase-invariance to figure-ground reversal of DO complex units, we compute energy response of a pair of DO simple units with opposite polarity. Although it is not the first time to apply opponent color to spatial filtering, compared to the previous work, our contributions are two folds:1) the proposed DO coding theory mimics the spatial and chromatic properties of DO neurons, which forms spatio-chromatic selectivity and opponency so as to discriminate color edges. In particular, we propose DO complex units invariant to phase by using energy mechanism.2) This theory is built on a serial and recursive color surface coding. In theory, the wiring input of DO units in reasonable because there are large numbers of related neurons in the cortex. Comparing to the exsisting color descriptors, which often filter images in the color space, we can treat the proposed method as a second-order operator to encode color edges, not only to extract abundant local color contrast and texture information, but also indicate functional hierarchy of color neurons, i.e. color surface, color edges, and color contrast invariant edges. More importantly, in theory, the proposed color coding method can be used in any image gradient algorithm.The second part of the thesis exploits the color coding theory in VI, focusing on the feature complexity encoded on the different levels, to extend the coding theory from VI to V4/IT inspired from the bottom-up information processing mechanism in the "what" pathway in the visual cortex. Toward vision tasks with different complexity, we propose color vision systems at different levels. The proposed systems include:V1:contour detection based on the color-texton map the standard contour detection often uses brightness, texture, and color channels based on Gaussian derivatives. However, color cue is abundant in nature images so does texture information. It is well known that V1neurons mainly response to edge/boundary, and the integration of visual cues is combined in a linear fashion. Therefore, based on color coding theory in V1, we propose a color-texton map at multiple scales and orientations to compute color-texture gradient. The visual elements are combined by learning weights in a logistic regression way. The contour is finally determined by the pixel probability. The contribution of proposed system is that we consider the co-variance of color and texture in the natural images and provide a V1neural inspired texture coding framework。V1:scene categorization based on the global color feature Previous neuroscience studies have shown that scene perception only needs to encode global feature of images. It is not necessary to extract features of objects in details and pre-segmentation of scenes. Scene categorization can be seen as the feature coding in V1. In addition, compared with scenes with gray-scale and wrong colors, the accuracy is much higher for the scene categorization with the real colors. Therefore, we propose scene recognition systems encoded by color surfaces and color edges by exploiting the ability of Gabor filters to represent the global features. The contributions of the proposed method is that we provide a general global color scene recognition system which is biologically plausible.V4/IT:color object recognition The previous two color systems are on the basis of the color coding theory in V1. More higher the hierarchy in the "what" pathway, more complex the feature encoded by cortical neurons. The hue tuning and color discrimination with glob position and scale invariance is achieved inV4/IT, which is the neural basis for complex object recognition. Inspired from shape processing in the cortex, we propose feed-forward hierarchical vision systems for color surfaces and color edges by extending wiring computation used for shape cue to color cue from V1to V4/IT. We learn spatio-chromatic selective prototypes from natural images and provide a computational basis for the template matching and object recognition. The contribution of this system is that we improve the hierarchy of color coding theory in the visual cortex and provide a comprehensive appearance modeling for object recognition.In the first two parts of this thesis, we study on the theoretical method and application of color vision. Obviously, cortex is an important brain region to understand color vision. A good theory of color coding and its model laid the foundation for image processing and computer vision. The third important content in this thesis is to unify neural data with the theory, i.e. provide a scientific description, analysis and interpretation for the theory and model at the neural level.Therefore, the third line of this thesis is based on the previous two contents, and explicitly builds the relations between model units and neurons by comparing their responses, which verifies the efficiency and scientificity of the proposed theory. The proposed comparisons include:Comparison of spatio-chromatic selectivity between model units and V1neurons One way to differentiate color neurons is to measure the model units tuning curves over visual elements (e.g. spatial frequency, orientation, phase, and color). We firstly design visual stimuli with color and spatial patterns (e.g. equilluminent color gratings), which are more similar with the electrophysiological experiments, then compute model unit response to different stimuli so as to determine the preferred stimulus pattern for that model unit. Comparing with V1neural data, we verify the ability of model to interpret the spatial and chromatic properties of color neurons. This is a scientific description of color coding theory in V1.Comparison of hue selectivity between model units and V4/IT neurons low-level V1neurons are selective to the specific wavelength, while high-level V4/IT neurons show narrow hue tuning so that they play a key role in color discrimination. To learn prototypes with spatio-chromatic selectivity from natural images, we propose a template matching method based on random sampling with testing color stimuli. Comparing with V4/IT neural data, we verify the ability of model to interpret the hue selectivity of color neurons. This is a scientific description of color coding theory in V4/IT.Comparison of hue map and lightness invariance of color selectivity between model units and V1/V4/IT neurons Since spatial distribution of color-selective neural response to different color stimuli is along the ordering of hue circle, we propose a new solution for the trajectory of model unit responses based on PCA method. The principle components in the lower dimensional space approximate the response trajectory of color neurons. Comparing with the intrinsic optical imaging data, we verify the existence of hue map in the visual cortex. The nearer the distribution of model unit response to preferred color stimuli, the more similar of hue of stimulus. We further verify the lightness invariant to color/hue selectivity by varying the lightness levels of the same hue.