The Design And Implementation Of Automatic Driving Scenario Library Data System

Recently,the automatic driving technology has developed more and more rapidly,but the automatic driving system is still unstable.It needs to build a virtual environment by using a large number of driving scenario library database,so as to pass the thorough test and evaluation.Because of the high cost of data acquisition and maintenance,companies in the industry lack an automatic driving data platform that provides standard data.Therefore,the China Intelligent Connected Vehicle Institute leads some companies in the industry to design and implement the Automatic Driving Scenario Library Data System.The system integrates data preprocessing,storage,labeling and statistical analysis so that recognition algorithms can be trained and tested.Moreover,test cases can be generated to build virtual simulation environment.This thesis mainly analyses the requirements of the data system of the Automatic Driving Scenario Library Data System,and explains the design and implementation of three following functional modules:data validation and submission module,data annotation module and data statistics and analysis module.The system is based on B/S structure and decomposed into front end and back end.And it is designed and implemented hierarchically.Data validation and submission module is responsible for validating the data format,and then preprocessing,combining and storing submitted data.The module is based on Spark to execute data preprocessing and integration tasks.In addition,for characteristics of different data,it integrates many storage components,such as FastDFS,ElasticSearch and HBase,to achieve data distributed storage.Data annotation module is responsible for labelling image,point-cloud and others to fulfill annotation tasks such as object detection and semantic segmentation.For lower human investment,this module uses algorithms for pre-annotation before manual correction.It applies TensorFlow Java API to read the trained PB models,and adopts strategy mode to provide corresponding algorithm to data-annotation tasks.Data statistis and analysis module is responsible for data charts analysis and sentence-inquire function.In the part of charts analysis,real-time data is collected and displayed through the architecture of ELK.And non-real-time data is stored by using HBase,indexed by using ElasticSearch and displayed with the user interface developed by Vue.js framework.Sentence-inquire function is implemented by using Spark SQL to access the data in Hive synchronized from HBase database.At present,the basic functions of the three modules of the system have been developed.The system provides the functions of cleaning,labeling,distributed storage and statistical analysis of autopilot data.It solves the problems in data processing and maintenance of massive autopilot data,and can output standard data to construct virtual simu ation cases.