Research On Edge Real Time Compression Method For Time Series Data

With the development of 5G,artificial intelligence and big data technology,data intensive industrial applications need to do real-time processing and analysis based on large-scale time series data.Due to the high response delay and network bandwidth requirements,the traditional cloud computing paradigm can not be applied to the industrial scenarios with strict delay requirements.Edge computing paradigm can alleviate the above problems,but the limited storage resources of edge computing nodes can not effectively meet the massive time series data storage requirements of industrial applications.Therefore,it is of great significance to reduce the storage overhead of edge industrial time series data.At present,there are two main problems to reduce the storage cost of edge industrial time series data.Firstly,the time series data should be compressed efficiently while meeting the real-time requirements of industrial applications.The existing solutions have the problems of low compression ratio and long compression time.Secondly,some of the most important time series data in the industrial Internet of things usually need to be stored in a lossless way.This kind of data requires high accuracy and usually needs to be stored in a lossless way.The existing temporal databases store every type of numerical data in a unified format,and the storage efficiency needs to be improved.In order to solve the first problem,this paper proposes a lossy compression algorithm for time series data combining prediction and coding.The compression process of the algorithm consists of three parts: preprocessing,predictive compression and coding compression.In the preprocessing part,the trend correlation between adjacent data segments of time series data is considered,and the influence of trend change of time series data on prediction results is reduced by fusion of data segments with the same trend.The prediction compression part is based on the Holt Exponential Trend model.In this paper,linear fitting is introduced to the Holt Exponential Trend model to improve the prediction accuracy.Meanwhile,the error calculation process of the prediction part is optimized to reduce the overall time complexity of the algorithm.The trend correlation of the time series data processed by the predictive compression part is weak.Considering that the coding compression algorithm is more suitable for compression of such data,the coding compression part is introduced in this algorithm.The LFZip algorithm is used in the coding compression part,which is a typical coding compression algorithm.The theoretical performance analysis and experimental analysis of the proposed algorithm are presented in this paper.The results show that the proposed algorithm has certain advantages in terms of time cost and compression ratio compared with the existing methods.In order to solve the second problem,this paper presents a lossless compression algorithm for time series data based on variable length coding.Considering the problem of bit redundancy in the binary form of small integers and small floating-point numbers,this algorithm adopts a variable-length coding rule proposed in this paper,which eliminates redundant bits by replacing long codes with short codes.The algorithm first encodes each floating-point data or integer data according to coding rules,and then splice the obtained codes,and finally merge multiple data into one data,thus realizing more efficient use of storage space.In this paper,the integrity of the algorithm is proved and the experimental analysis is given.The results show that the algorithm meets the integrity conditions.The experimental results show that the algorithm has certain advantages in terms of time cost and compression ratio compared with the existing schemes.Based on the first algorithm and the Influx DB time-series database,this paper designs and implements a time-series data compression storage and management system for the industrial Internet of Things.The system considers the functional requirements in practical application scenarios and realizes the efficient management of time series data.The experimental results show that compared with the original Influx DB time-series database,the system has significant advantages in the time cost of data writing.