Studies On Simulated Images Generation And Its Applications On Autonomous Driving

Autonomous driving is the major direction of the development of automotive technology,data-driven approaches based on deep learning are becoming an important part of the perception system for autonomous driving.The image data collected by the onboard camera has been widely used in deep learning model training and object detection,and important progress has been made in the field of automotive autonomous driving,but there are still some problems: on one hand,deep learning based on object detection relies on large amounts of data with semantic labels,which are not only costly but also sometimes dangerous to collect.Moreover,manual labeling of image data is also very cumbersome,which is not only costly but also prone to error.This severely restricts the rapid development of sensing algorithms for autonomous driving.On the other hand,the observers of the images have changed from a human to a deep learning model.One limitation of the deep learning method is lack of interpretability.Therefore,designing an imaging system that can meet the needs of the deep learning model for image data is worth deeply researching.Recently,image simulation technology has increasingly become an effective approach to solve the above problems: computer generated images has the advantages of high efficiency,safety,and automatic pixel-level labeling,which can greatly reduce the cost of data acquisition on road,safety risk,and manual labeling,which can also provide high labeling accuracy;besides that,image simulation can help design and optimize the feature requirements of image data for deep learning algorithms.However,the existing image simulation methods cannot be used to solve the above-mentioned problems: 1)The current image simulation technology still focuses on improving the perception of the human visual system and does not aim to truly express the physical characteristics of the image.Physical attributes can not be quantitatively analyzed and parameters with physical meaning are not provided;2)The detection performance of the deep learning model that uses simulated images as the training set is still significantly different from the model trained with real image data.How to improve the perception system performance which is trained on simulated images is still yet to be solved.With the challenges described above,we carried out corresponding research and achieved the following research results:(1)We proposed a phlyscially based and quantifiable image simulation method.We use ray tracing technology to simulate the propagation process of photons generated from the light source to the camera sensor: the process of the photon interacts with the surface of the object and the transmission of the camera optical system to the surface of the camera sensor,and the process of simulating the imaging process of the photon on the camera sensor includes: photoelectric conversion,color filtering,photon noise,device noise,analog-to-digital conversion,demosaicing and image post-processing.This method uses parameters with meaningful physical units to control the conversion process from photons to images,and the physical accuracy of low-level features is validated through experiments,the effectiveness of the high-level features is verified with a deep learning model.This method can meet both the needs of quantitative analysis and parameterized generation of simulated images for deep learning-based methods.(2)We built and open-sourced a large-scale simulated image generation platform for autonomous driving to meet the needs of high-fidelity scenes and large-scale images generation.The platform uses the traffic flow simulation software SUMO(Simulation of Urban Mobility)developed by the German Aerospace Center to parametrically control the traffic participating objects(e.g.,vehicles,pedestrians,etc.)in the scene,and proposed the SUSO(Simulation of Stationary Objects)method automatically place non-traffic objects(trees,buildings,etc.)in the scene.To meet the requirements of the computing power and storage space for high-quality image generation,this method uses the container orchestration platform Kubernetes to efficiently compute images on a large scale with cloud cluster and uses cloud database flywheel to efficiently store and manage 3D assets.This method provides an efficient way for simulated images generation,parameter configuration,and deep learning-based model training and verification.(3)We proposed an image set quality metrics for object detection task: OD50(Object Distance at 50% average precision—the object distance at an average accuracy of 50%).Traditionally image quality metrics standards for customer photography(MTF50 and SNR)cannot meet the requirements of autonomous driving perception testing,the OD50 adds a third dimension-object distance dimensions(AP,Z)-on the average accuracy(AP),makes it more suitable for the perception test requirements of autonomous driving.Aiming at the problem of lack of interpretability of deep learning algorithms,we use the OD50,and the image generation platform to analyze the key elements of camera imaging including pixel size,color filter array,sensor exposure algorithm,and post-processing algorithm.The impact of the deep learning-based algorithms has been found: The exposure algorithm has a greater impact on the deep learning algorithm,the image post-processing algorithm that is usually more sensitive to humans has no significant impact on the detection accuracy of the deep learning-based algorithm.(4)We improved the generalization ability of the simulated images to the real images we achieved higher detection accuracy in the real scene using a deep learning-based model trained with only simulated images.We studied the domain gap between simulation and real images based on deep learning algorithms,we found that for autonomous driving applications,the key factors that affect the generalization ability of simulated images are training sample diversity,domain gap,scene rendering quality,and camera parameter matching,etc.Finally,we generate the ISETAuto images under the guidance of the key factors of generalization and carried out the model training of the object detection algorithm based on deep learning.We use the real images KITTI and Cityscape commonly used in autonomous driving to test the algorithm.Compared with the state of the art simulated images dataset,the average accuracy of the detection results of the ISETAuto images proposed in this paper has increased by 13% and 40%,respectively,which is similar to the generalization performance between the real images dataset;therefore,the simulated image generation technology proposed in this paper can be used for deep learning model training,reducing the need for a large number of real images for deep learning algorithms,and is of great significance for accelerating the development of autonomous driving perception algorithms.