Application Of Deep Learning Algorithms In Self-driving Vehicle Vision

In recent years,the research on deep learning technology facilitates the development of artificial intelligence in either academic or industrial fields.Deep learning algorithms stem from artificial neural network.They have proved to be effective approaches to apply neural networks with multiple layers in practical applications.Benefited by a great amount of data accumulation such as from Internet,and the great promotion of computing power based on Graphics Processing Unit(GPU),deep learning algorithms have been extended to wider domains like self-driving vehicle.This domain is appealing more and more attentions because self-driving vehicle is expected to be an alternative to bring safer driving,reduce traffic accidents and relieve traffic pression.A self-driving vehicle is a complicated system of which visual perception is considered to be an important component.Visual perception of a self-driving vehicle is functioned to understand its surrounding like road,vehicle and pedestrian etc.With respect to road detection,traditional methods mainly focus on the structured road and straight road.But more complex environment has to be faced when a self-driving vehicle runs under natural situation.In this case,road may have ambiguous roadsides and rugged surface.It may also encounter shadow and crossroad.On the other hand,shallow features are usually applied for object detection and recognition by conventional methods.But shallow features are hard to express the abstract properties inherently owned by objects.It is difficult to deal with a variety of transformations from the same object.And it is also hard to distinguish different objects which share similar features.Additionally,traditional algorithms with high computation complexity are not allowed by the real-time requirement in the practical applications of self-driving vehicle.Compared with conventional hand-designed features,the extracted features from hierarchical networks are characterized of abstract,sparse and invariant properties.In view of these advantages,this dissertation aims at applying deep learning algorithms to the basic vision issues in the applications of self-driving vehicle.Therefore,three aspects are investigated in terms of road detection,car detection and invariant object recognition.Crossroad detection is a challenging problem which a self-driving vehicle has to confront.Considering road direction is a simple but effective form to represent road path,we firstly introduce the concept of road-direction point.We then attempt to predict road-direction point for both single road and crossroad situations.In this dissertation,a real-time road-direction point detection model is proposed based on convolutional neural network(CNN)architecture under complex environment.The predicted road-direction point can serve as a guiding point for a self-driving vehicle to go ahead.In the situation of crossroad,multiple road-direction points can also be detected associated with different road paths which will help this vehicle to make a choice from possible directions.Meanwhile,different types of road surface can be classified by this model for both paved road and unpaved road.This information will be beneficial for a self-driving car to speed up or slow down according to various road conditions.Finally,this road-direction point detection model is deployed on Jetson TX1,which is a platform for deep learning development.And it can reach about 12 FPS for each forward inference on this portable platform so that it is appropriate for real-time road direction estimation.Road-direction point estimation is not enough to safeguard a self-driving vehicle.Correct prediction of road boundaries is the key factor to help this vehicle decide where the passable region is in front of it.Therefore,this dissertation further proposes a real-time multiple road-points detection model based on CNN structure under complex environment from structured road to unstructured road,and from single road to crossroad.Firstly,the concept of road points is introduced to represent road boundary.Seven road points are utilized to represent one single road where road-direction point is used at the distance of road with additional six roadside points averagely allocated along its two road boundaries.For the case of crossroad,multiple batches of road points are available to describe the appearance of the road junction.And each batch is associated to one road path.From the detected road points,each road-direction point can serve as a guiding point to look for a potential road direction in autonomous navigation,even under crossroad situation.Meanwhile,the remaining roadside points can be jointed further to form piece-wise outline of both road boundaries,which can guarantee a self-driving vehicle inside a passable region shaped by this outline.An associated metric method is further proposed by introducing normalized error area between the predicted and ground truth road boundaries.The normalized error area is also minimized as a part of loss function to improve model performance.With this metric definition,model performance is evaluated on different platforms including Jetson TX1 under which the mean speed can reach about 12 FPS.It is demonstrated that this road-points detector can serve as a real-time and useful module in a self-driving vehicle system under various scenarios.And it also offers an efficient and economical solution of road detection in practical applications.Object detection module is anther important component for a self-driving vehicle.Car,pedestrian,and even traffic light which are the common instances in our urban lives need to be detected for environment understanding.In this dissertation,a road-direction point based car detection model with adaptive cropped sub-regions is developed.Information from sub-regions is integrated to detect small objects in the whole image.This model is evaluated on a prepared testset under both urban and off-road environment.Precision and recall curve is utilized for the performance evaluation.Simulation results illustrate that the introduction of fixed or adaptive sub-region can improve the average performance compared with that of the original YOLO model.They can extend the ability of certain detection model to detect smaller objects.Meanwhile,the scheme with adaptive sub-regions can effectively reduce time cost compared with the scheme with fixed sub-regions.On the other hand,this vision based detection model still confronts some challenges like lighting influence.Invariant object recognition is straightforward in the visual path of our brain.We attempt to consider spatial-temporal consistency in object recognition by applying trace learning rule.Therefore,a trace rule based self-organized map(SOM)model is proposed which is built upon a sparse 2-stage deep belief network(DBN)in this dissertation.The combination of SOM and sparse DBN forms a hierarchical network where DBN serves as a V2 features detector while SOM layer learns to extract transformation invariant features guided by trace learning rule during training phase.The performance of the proposed method is evaluated by stimulus specific information(SSI)metric and is compared with classic algorithms.It is demonstrated that trace rule based SOM model can generate more neurons with higher SSI values which is beneficial to convey more discriminative information for further object recognition.From these results,this model can further be developed to learn invariant features online in the applications of self-driving vehicle.