Research On Event Stream Processing Method Of Dynamic Vision Sensor

With advances in semiconductor technology and processes,the performance and imaging quality of optoelectronic imaging devices have been significantly improved in recent decades,and they have been widely used in many fields.However,due to the limitations of device principles and output method,photoelectric imaging devices may encounter problems such as motion blur and underexposure when imaging high-speed,ultra-high-speed moving objects or scenes with high dynamic range.And as the rate increases,generating a large amount of background redundant data while obtaining the target image,which will put a huge burden on the back-end processing and storage system.Dynamic vision sensors are inspired by biological vision.Compared with the current photoelectric imaging device's working mode of fixed-frequency global exposure and outputting grayscale images frame by frame,the dynamic vision sensor is imaging in an asynchronous manner,and the output is a pulse signal representing changes in light intensity,not a grayscale value.This method of imaging and output benefits from that the pixel design and chip architecture of the dynamic vision sensor are closer to the biological retina.When the light intensity on a pixel changes and reaches the threshold,the event pulse is immediately output through the address event representation circuit.When multiple pixels output,the event stream is formed.Dynamic vision sensors have advantages that current photoelectric imaging devices do not have: low latency,high dynamic range,low bandwidth,low power consumption,etc.However,there are some difficulties in the application of event stream: 1.The inherent defects of semiconductors.A more complex pixel structure is more likely to produce noise events.Noise events are generated by factors such as thermal noise and junction leakage current under the condition of constant light intensity.It will increase the amount of data and occupy more bandwidth and computing resources;2.the event stream data form is difficult to process,and there is no mature processing paradigm,the current gray-scale image processing methods are difficult to directly apply to the asynchronous and sparse data form of the event stream.For dynamic vision sensor event stream noise filtering,this article starts from the dynamic vision sensor chip architecture and imaging principles,and analyzes the generation mechanism,visualization method and processing method of event stream.According to the types and characteristics of noise,an event stream denoising method based on event density is proposed.And through the quantification of event relevance,a denoising effect evaluation method based on the probability of real events is proposed.In the object tracking application,a method based on the event correlation index is studied.The main research content of the thesis includes the following points:1.Starting from the chip architecture and imaging principle of the dynamic vision sensor,the mechanism of the event flow generated by the dynamic vision sensor is deeply studied,and the event stream is visualized from different dimensions.The processing methods of event stream is studied,and they are divided into two methods: event-by-event processing and time-slicing event stream processing: the event-by-event processing method combines spatio-temporal neighborhoods to process newly arrived events;the time-slicing event stream processing method batch processes events by quantifying the spatio-temporal correlation of events in a fixed period of time,which will provide a theoretical basis for subsequent event stream processing.2.Propose an event flow noise reduction method based on event density.First,through the 3D visualization of the event stream,combined with the pixel structure and imaging principle of the dynamic vision sensor,the noise events are divided into background activity noise and hot pixel noise,and the characteristics of different types of noise are summarized.The concept of event density is proposed.The noise reduction method is divided into two steps.The first step is to calculate the event density of the temporal and spatial neighborhood of the newly arrived event one by one to filter out the background activity noise with low density.The second step is to filter the hot pixel noise according to the result of the first step.Compared with the current denoise methods through event stream visualization,the noise reduction results of the method in this paper have the least noise events.3.Propose an evaluation method for denoising effect of the event stream.First,based on the temporal and spatial correlation,a real event probability quantification method is proposed,and based on this,the noise in the acquired event stream is distinguished from the real event.Then denoising effect evaluation index is proposed:the number of noise events in the retained event stream(NIR)and the number of true events in the filtered event(RIN).The denoising effect of the event stream is evaluated through these two indicators.The smaller the value of NIR and RIN,the better the noise reduction effect.Compared with the existing evaluation method,this method can evaluate the noise reduction effect without using a fixed image generating device to generate a periodic event stream.And use this method to quantitatively compare and evaluate the above four noise reduction methods,and the results are consistent with the visual comparison results.4.Propose an event stream object tracking method based on event correlation index.This method uses time-slicing event stream processing to calculate the event correlation index of this part of the event,and label the event cluster ID for the events that meet the threshold of the event correlation index,so as to obtain information the target centroid.Furthermore,the centroid method can be used to track the target motion trajectory,and the high time resolution event stream of the motion target can be retained.The experimental results show that this method can effectively find the moving target in the field of view,and track its movement and keep its trajectory.