Research On Multivariate Time Series Classification Algorithm And Application In Lip Reading

Time series usually refers to a set of data formed by continuously taking values for a specific observation point at the same time interval.In many application fields of time series,a classification object may correspond to multiple observation points of di?erent dimensions at the same time,and the continuous data collected by these observation points together constitute a multivariate time series.The current research on univariate time series has been very su cient,but the problem of multivariate time series has not been fully studied.E?ectively extracting features between multiple dimensions in a multivariate time series is one of the di culties in the current time series field.As an application scenario of multivariate time series,Lipreading is also a key research issue in the field of artificial intelligence.Therefore,this paper has carried out research in two di?erent directions on the multivariate time series classification problem,and applied it to the field of lipreading.The main work is as follows:(1)The multivariate time series classification algorithm based on random forest is researched,and a random forest algorithm composed of highly random decision trees is proposed.This method extracts the correlation between multiple dimension sequences by randomly selecting dimensions in the sequence and converting it into an associated time series.The similarity between the randomly selected representative sequence and the sequence to be divided is used as the basis for the branches of the decision tree.After each tree is constructed independently,the weight of each tree is calculated to improve the e?ect of high randomness on the classification accuracy.In addition,the computational complexity of a single decision tree is reduced through random dimension selection and multi-branch decision trees.(2)The multivariate time series classification algorithm based on deep learning is researched,and an algorithm combining the attention mechanism and the recurrent neural network is proposed.This method uses the attention mechanism to assign di?erent weights to the di?erent dimensional sequences of the multivariate time series,so as to obtain more discriminative dimensions,and extract the correlation features within the sequence and between multivariate sequences through the bidirectional recurrent neural network,and finally combine the two network structures up for classification and identification.(3)The application of the above two multivariate time series classification methods in the field of lipreading is carried out,and a multivariate time series representation method for lipreading is proposed.This method obtains the key point coordinate information of the lip through the face key point detection technology,converts the key point coordinates into the angle between the key point and the adjacent point,and combines the height and width of the inner and outer contours to obtain the lipreading feature point sequence.In order to better carry out the research,we have made a sentence-level Chinese lipreading dataset.After that,the two multivariate time series classification methods in(1)and(2)were used to conduct experiments on the public English lipreading dataset and the selfmade Chinese lipreading dataset.Experiments show that the method proposed in this paper e?ectively improves the classification accuracy of multivariate time series data.After being applied to the field of lipreading,it also has application value.