Design And Implementation Of The Real-time Analysis Engine For Didichuxing's Data

With the rapid development of cloud computing and big data,Hadoop and Spark and other bulk data processing platform also bred.The batch processing system that need to store and then calculate,processes requests with high response delay.However,due to business needs,real-time analysis has been gradually put on the agenda.Although there are now Storm,Samza,Spark Streaming and other streaming computing system,many companies use these systems for specific business scenarios.So,a unified real-time analysis engine is required to unify the processing of various processed or unprocessed streaming data.The real-time analysis engine is a system that is tailored for Didichuxing's data.The system uses Java language development,Kafka,HBase,QDB and RedHat for the development enviroment.First of all,the use case diagram is given by analyzing the system.The second,the overall architecture of the system design is given,and it includes the system structure design and system module partition,among them,the system includes four modules: data collections module,data analysis module,data storage module,data access module.Then,the four modules of the system that includes the structure design,function implemention,flow chart of each module,and design of the database table is designed and introduced.Finally,the whole system test that includes functional test,and performance test and stress test with JMeter is given,in order to verify whether the system functions to meet the needs of the intended purpose.The system can receive data from Storm,Samza,Spark Streaming and other streaming computing systems by Kafka,achieve commonality within the company,but also integrate resources,greatly reducing the duplication of development of human and material resources.