Research On Big Data Of Dimension Reduction And Rapid Processing

With the development of Internet technology, the explosion of information brought vast amounts of data. Although mining these data contains a large amount of information is very attractive, it is not easy to process them. The algorithm to deal with a small amount of data is lack of flexibility when processing the big data. More efficient algorithm is necessary.High-dimensional data usually have very large dimensions and very high number of parameters. It needs high computational efficiency and high storage. Especially when applied to machine learning algorithm to learn too many parameters easily lead to over-fitting with bad generalization. The tensor decomposition using the structural information and low rank property, approximate the original data with only a small amount of data it. By combining the tensor train decomposition and SVM, we introduce a low-rank learning algorithm, which reduce the parameter to be learn, can break the curse of dimensionality.Another problem in the big data processing is massive amounts of data. By extraction feature with convolutional neural network and combining with hash projection algorithm, we achieve a content-based image retrieval system,which greatly improved the performance.Next, in order to further optimize the retrieval system, we introduced the parallelization with OpenMP and CUDA.