Design And Implementation Of A Big Data Platform In A Railway Station Based On Hadoop

In the past two decades,with the rapid development of science and technology,large amounts of data have been produced in many various fields such as medical care,R&D,sensors,Internet companies and financial institutions.To address the demand of unstructured data and large data sets,big data platforms for data processing have gained more momentum.Compared with traditional data,big data have features of non-structural elements,big data sets and real-time processing.In view of these features above,it is appeared that a systematic architecture can collect,transmit,store and process data based on a big data platform.Using Apache Ambari,we build up and manage a big data platform based on Hadoop,and using the systematic mechanism of the platform realizes collecting,storing,analyzing and managing some station's data.The platform is divided into front-end visual client and back-end server,and the platform(also called the system in this papper)is developed by combining Python with Javascrip,while its Web framework is a lightweight Flask written by using Python.By transmitting,storing,mining data and displaying information,the system is able to perform various functions to monitor the station's security and equipment failure.In this paper,at first,we make a requirement analysis of establishing a big data platform for some railway station and design a systematic architecture for the system accordingly.Then,a prediction algorithm for security problem is developed by using C4.5 decision tree algorithm in order to facilitate management and maintenance.Later,a feasible decision tree is drawn by massive data training and learning,our simulation result shows that the prediction algorithm can achieve our desirable goal.