Anomaly Detection Method Of Atmospheric Visibility Based On LSTM-AE And BO-LightGBM

Atmospheric visibility directly affects human social production activities and various traffic safety issues.Accurately obtaining atmospheric visibility observation data and predicting warnings has important significance and application value.At present,the atmospheric visibility observation system is moving towards automation,intelligence,and informatization,which provides the possibility of providing more accurate observation data and meteorological warning services.However,based on the current automated method for collecting atmospheric visibility observation data,it is easily affected by environmental factors,which leads to a decrease in observation data quality and affects the quality of atmospheric visibility meteorological service products.This thesis conducted research on atmospheric visibility anomaly detection methods,proposed an atmospheric visibility anomaly detection method based on time series prediction and statistical methods to achieve rapid and accurate detection of atmospheric visibility abnormal observation data.Firstly,aiming at the problem of different intensity noise in atmospheric visibility observation data,this thesis used a multi-feature fusion method based on air quality data and atmospheric visibility meteorological data sets to combine atmospheric visibility data with other meteorological observation data for research on data denoising and improvement of data quality methods.It proposed a screening and repair method for atmospheric visibility data based on long short-term memory autoencoder(LSTM-AE).Secondly,aiming at the needs of atmospheric visibility data quality control and warning prediction service products,based on the denoised data,this thesis conducted research on atmospheric visibility anomaly data detection methods and proposed an atmospheric visibility anomaly detection model based on Bayesian optimized light gradient boosting machine(BO-LightGBM).The anomaly detection model is based on time series prediction method and calculates the residual between the predicted value and the observed value as the outlier degree of the detected data.The model can automatically select optimal parameters according to the characteristics of atmospheric visibility data and give the model high accuracy and high generalization ability,which is suitable for rapid training of large-scale data.Finally,aiming at the subjectivity problem of abnormal threshold selection,this thesis proposed a threshold automatic delimitation method based on K-sigma to avoid subjective influence.The method dynamically determines the abnormal threshold based on the mean and standard deviation of the dataset and can adapt to the data characteristics of different observation sites.The proposed atmospheric visibility anomaly detection model was verified by detection effect verification on real observation data provided by Zhangzhou Meteorological Bureau in Fujian Province,and various ablation experiments and multi-model comparison experiments were conducted.The experimental results show that the proposed method is effective,feasible,and has the following advantages: it can effectively reduce the impact of noise on data quality;it can automatically select optimal parameters with high accuracy and high generalization ability;it can dynamically determine the abnormal threshold and adapt to the data characteristics of different observation sites.This provides new ideas for achieving fully automatic and accurate detection of atmospheric visibility anomalies.