Data Collection And Data Completion Based On The Theory Of Sparse Representation

The new technology of data collection and recovery have a rapid development,and have been widely used in environmental monitoring,security monitoring.With the rapid development of information acquisition technique and network technology,the data increase exponentially,which lead to huge challenge for the data.collection.As a.powerful technology for estimating the missing data,the theory of sparse repre-sentation make it possible that recovering the full data by using only a small random sampling.It provides a.new way for efficient.representation and aecurate recovery of data and has been widely used in data collection and recovery.This paper give a intensive study for model and algorithms of weather data ga.th-ering.network monitoring,data recovery in mobile crowd sensing.and recovery of Internet traffic data.based on the sparse representation theory of matrix completion and tensor completion.The main contributions are outlined as follows:(1)A approach of On-Line weather data gathering based on matrix completion is proposed.By analyzing a large set of weather data.collected from 196 sensors in Zhu Zhou.China.,we reveal that weather data have the features of low-rank,temporal stability.and relative rank stability.Taking advantage of these features,we propose an on-line data.gathering scheme based on matrix completion theory to adaptively sample different locations aecording to environmental and weather concditions.To better schedule sampling process while satisfying the required reconstruction accuracy,we propose several novel techniques,including three sample learning principles,an adaptive sampling algorithm based on matrix completion,and a uniform time slot and cross sample model.With these techniques.our scheme can collect,the sensory data at required accuracy while largely reduce the cost for sensing,communication and computation.We have performed extensive simulations based on the real weather data sets and the simulation results validate the efficiency and efficacy of the proposed scheme.(2)We propose a approach of adaptive sampling in network monitoring systems based on matrix completion,and exploit the matrix completion techniques to derive the end-to-end network performance among all node pairs by only measuring a small subset of end-to-end paths.To address the challenge of rank change in the practical system.we propose a sequential and information-based adaptive sampling scheme?along with a novel sampling stopping condition.Our scheme is based only on the data.observed without relying on the reconstruction method or the knowledge on the sparsity of unknown data.We have performed extensive simulations based on real-world trace data,and the results demonstrate that our scheme can significantly reduce the measurement,cost while ensuring high accuracy in obtaining the whole network performance data.(3)We propose a approach of accurately recover the data in mobile crowd sensing based on matrix-reshaping,and exploit the matrix completion techniques to infer the missing environment monitoring data based on a few samples.To address the challenge of poor performance of matrix completion technique when the missing ratio is large,we proposed a novel matrix-reshaping technique,which can change the matrix size by reshaping the matrix from the oblong matrix to a square one for better recovery performance.We have proven the effectiveness of the matrix-reshaping.We have performed comprehensive simulations with real data trace.The simulation results demonstrate that our matrix-reshaping can achieve significantly better performance.(4)A approach of accurate recovery of Internet traffic data based on sequential tensor completion is proposed.Tensor model can make full use of the multiple corre-lation characteristic of data.and overcome the lack of the method based on matrix.To fully exploit hidden spatial-temporal structures of the traffic data.in this paper,we model the traffic data as a 3-way traffic tensor and formulates the traffic data re-covery problem as a low-rank tensor completion problem.To reduce the computation cost,we propose a novel sequential tensor completion algorithm which can efficiently exploit the tensor decomposition result for the previous traffic data to deduce the ten-sor decomposition for the current data.We have done extensive simulations with the real traffic trace.The simulation results demonstrate that our algorithm can achieve significantly better performance even when the data missing ratio is high.(5)A approach of accurate recovery of Internet traffic data based on tensor-CUR decomposition is proposed.we formulate the traffic data recovery problem as a tensor completion problem.In order to design a more effective tensor completion algorithm,we propose a novel method based on tensor-CUR to quickly recover the missing traffic tensor data.In our method,we take advantage of CUR decomposition,where an 3-way tensor is instead decomposed as a product.of three 3-way tensors.Computational experiments demonstrate that the proposed method yield a superior performance over other existing approaches.In summary,this dissertation fully exploits the Intrinsic characteristics of data,and the sparse representation theory to give a.intensive study for model and algorithms of data recovery.