Research And Implementation Of Data Mining Algorithm For Structural Health Monitoring Based On Hadoop/Spark

In recent years,the rapid development of science and technology has promoted the vigorous development of the field of structural health monitoring.A large number of civil infrastructure,such as long-span bridges,tunnels,high-rise buildings,nuclear power plants,and power transmission towers,have been equipped with structural health monitoring(SHM)systems to monitor their condition and damage.Along with the continuous development of sensor technology and signal transmission technology,the data acquisition ability of SHM system is gradually enhanced,and various types of sensors produce a huge amount of data in the monitoring process.Due to the difference between different sensors in characteristics,acquisition principles,storage format and so on,the SHM system will inevitably be affected by environmental and external loads,which makes the structural health monitoring data have uncertain factors such as noise and data loss.It can be seen that structural health monitoring data has the characteristics of large volume,variety,high velocity and low value density.Traditional offline data processing methods can no longer meet the requirements of data storage,processing,calculation and analysis.Therefore,it is urgent to find new technology and automation tools,which can help us to transform massive monitoring data into useful information and knowledge in real time and efficiently.This paper will use common data mining algorithms and hadoop/spark big data processing platforms to research and implement hadoop/spark-based structural health monitoring data mining algorithms.Through the mining of historical monitoring data and real-time monitoring data,the state of large-scale engineering structure is comprehensively analyzed and the change trend of structure is provided with timely warning.The basic idea of data mining algorithm for structural health monitoring can be applied to bridge health monitoring,as well as to tunnel,dam monitoring,high-rise building monitoring,nuclear power plant,power transmission tower and other structural health monitoring fields,which has a broad application prospect.Firstly,the data preprocessing methods of structural health monitoring are introduced in detail,including data cleaning,data integration and transformation,and data reduction.Secondly,we study the application of common data mining model in structural anomaly recognition,design and implement structure anomaly recognition algorithm based on PCA,structural anomaly recognition algorithm based on GMM and structural anomaly recognition algorithm based on Apriori association rules.Then a Bi-LSTM based structural damage warning algorithm is designed and implemented.Simulation results on a simply supported beam structure show that the algorithm is effective in predicting structural damage.Then a 1 D-CNNs-based structure damage localization algorithm is designed and the accuracy of the algorithm is tested on a benchmark structure.Finally,the hadoop/spark big data platform is built by using docker virtualization technology.Taking the apriori algorithm as an example,the structure health monitoring data mining algorithm is implemented on the platform.By comparing the computing time of the algorithm running on different platforms,the big data platform is proven to have good computing performance for the data mining algorithm.