Fish-eye Images Based Environment Perception Methods In Intelligent Vehicles

In recent years,intelligent vehicle technology has experienced rapid development,and various sensors have been applied in intelligent vehicles.Among them,the fish-eye camera is playing an increasingly critical role in intelligent vehicles due to its large field of view,high semantic information,and low price.However,there are still several critical problems in this research field,including deficiency of seriously distorted images,the difficulty of recognition with seriously distorted features,insufficient mining of hard samples,and low accuracy of the single-task algorithms.In recent years,the development of deep learning technology has made significant progress in intelligent vehicle environment perception algorithms.This work does researches in the intelligent vehicle environment perception based on fish-eye images using deep learning technologies.First,the number of fish-eye images for training is insufficient and the quality is not high.A projective model transformation based fish-eye image generation method is proposed,which converts traditional images into fish-eye images.Building the relationship between the field of view transformation and pixel mapping between the traditional imaging and fish-eye imaging,thereby generating a large amount,high-quality fish-eye training dataset.The experiment is based on the ETH,KITTI,Citypersons,and real fish-eye image datasets,it verifies the reliability of the method.Secondly,the dynamic and static objects have obvious feature distortions on the fish-eye image,which restricts the accuracy of the perceptual tasks such as object detection.An oriented spatial transformer network based distortion rectification method is proposed.Two spatial transformation networks for regions of interest are fused,which effectively rectify the distorted pedestrian features at the feature level.It reduces the pressure on the detector.Based on the experiment on the ETH,KITTI,and real fish-eye image datasets,the accuracy of object detection has been improved.Then,hard example mining is difficult in fish-eye images.An adversarial training based hard example mining method is introduced to efficiently mine hard fish-eye examples.A distortion generation network and a novel loss function are designed,and a three-stage training method is adopted.In the experiment based on the ETH,KITTI,and real fish-eye image dataset,the accuracy of object detection is largely improved using this method.Finally,the traditional single algorithm for single task based method does not fully correlate multi-task information and consumes too much computing resources,making it difficult to meet the actual requirements of intelligent vehicles.A context tensor based multi-task learning method is proposed to build the designable association among different tasks.The relationship between the drivable area,traffic object and lane line detection tasks are build.Combining the context tensor module with the adversarial learning method,the performance of multi-task in fish-eye images is improved obviously.The experiment based on the BDD dataset show that the multi-task framework can improve the accuracy of traffic object detection.In general,this work proposes methods of data augmentation,distortion rectification,hard example mining,and multi-task learning.These methods can effectively expand fish-eye image training data and mine hard examples.They can rectify distorted object feature and multiple perceptual tasks are unified into one deep learning based framework.Therefore,the perception ability of the intelligent vehicle using fish-eye images has been significantly improved.