Transformer Based Multivariate Time Series Classification Framework Research

With the rapid development of digital economy,all walks of life have accumulated many multivariate time series data in the process of moving towards the "Internet plus".Therefore,how to process these data has become the key to advance the next generation of business intelligence.Traditional machine learning methods have high requirements for the quality of data.It spends plenty resource on data preprocessing and feature engineering.But the designed methods do not have the generalization ability between different tasks and heterogeneous data.As a result,although the data of different tasks have the same content,the model still needs to design and train each task independently.With the accumulation of data and the continuous iteration of business content,methods and models need to be adjusted on a large scale.In the economic environment of pursuing high-quality development,such high costs are not conducive to business development and iteration.Therefore,the concepts of general model and large model are proposed.A series of large models have achieved very good results in the field of text and image by taking advantage of the weak inductive bias and larger network capacity of Transformer model.However,in terms of multivariate time series data,which is more widely used in industry,Transformer model has only been applied on prediction tasks.Few studies have applied Transformer model to multivariate time series classification tasks.This is because multivariate time series data are varied in forms and contents.Although many multivariate time series data can be easily obtained in most scenarios,there are few data that can be used for training due to many difficulties in manual marking for such high dimension data.In addition,the weak inductive bias priori of Transformer model itself requires large-scale data training to learn the appropriate inductive bias.At present,the application prospect of Transformer model in multivariate time series classification remains unoptimistic by the industry.Therefore,based on the problems of the above,this paper tries to make progress in three perspectives: first,the pre-training task is used to improve the learning results of the model on the data characteristics.In this paper,aiming at the noise problem of the multivariate time series data,a piece reconstruction pre-training task is proposed.It is used to improve the performance of text pre-training task and auto-encoder reconstruction pre-training task on small data sets.Then the temporal convolution Transformer model is proposed.By introducing the priori mechanism of sequence modeling using the temporal convolution structure,the inductive bias of Transformer model is improved.Finally,the idea of supervised contrastive learning is introduced in down-task training phrase,and a supervised contrastive learning module is proposed.Through the multi-task learning form of combination of contrastive learning and supervised learning,the data is reused,which helps the model learn useful features for classification from different angles and improves the performance of the model in classification tasks.On the UEA dataset,compared to the relevant algorithms in recent years,the method proposed in this paper achieved better classification accuracy performance on some tasks;Compared to other Transformer Framework,the method in this article has better classification accuracy performance on most data sets.In the pre-training comparative experiment,the piece reconstruction pre-training task proposed in this paper has better effect,when the complexity and amount of data are in the same condition.Through ablation experiments,this article verifies the impact of the supervised contrastive learning module and the temporal convolution module on downstream classification tasks.The experiment shows that the application of temporal convolution module has the most significant improvement in classification accuracy performance,while the supervised contrastive learning module only improves the classification accuracy performance of the model when there are fewer data categories.On datasets with many categories,the comparison supervised learning module can affect classification results,resulting in a certain degree of decline in classification accuracy.On the oil spill dataset,the temporal convolution transformer model with forecasting pre-training task is compared to recently and representative algorithms on oil over-flow detection task and achieved better recall rate and F1 value.The improvement of temporal convolution module on Transformer model was also verified through ablation experiments on this data.