Research On Data Reliability Measurement Method Based On Big Data Processing Technology

With the introduction of the concept of big data,it has rapidly become a hot spot for scholars both at China and abroad.The main research directions include big data collection,big data storage,mining and visualization.However,there is relatively little research on the reliability of big data,and the previous data credible calculation method can not deal with the challenges posed by big data well.Therefore,the research on the new reliable method of big data is an urgent problem to be solved.This thesis focuses on big data processing techniques and big data reliability metrics.First of all,the thesis researched related technologies of big data processing.The main research is the definition,characteristics and computing features of big data,and the related technologies dealing with big data,such as Hadoop platform,Spark distributed computing framework,Hive and HBase database.After that,this thesis researched the credible computing methods and models of data,and focuses on the model of transaction feedback,the trust model based on relationship and the trust model based on interest.In addition,this thesis summarized the advantages and disadvantages of the common calculation models and methods of credibility,and summarized the existing security problems of the current credibility calculation.According to the credibility of the previous research on calculate the existence of the problem,this thesis designed a model and method of trustworthy measurement of big data,the model based on three aspects of the credibility of the data:the entity that evaluates the data,The data source and the related properties of the data itself,the credibility of the data is divided into subjective credibility,global credibility and local credibility of three parts.Subjective credibility is calculated and evaluated through historical interaction records between users and data sources.Global credibility is calculated and evaluated through interaction records released by data sources or generated data.The local credibility is evaluated by the credibility of historical data released by the data source and the similarity of this data.The thesis also validates the algorithm experimentally,the experimental results show that the method of trustworthiness of big data works well.Finally,based on the method of trustworthiness of big data,this thesis designed a trustworthy measurement system for e-commerce data,which can calculate the credit value of sellers and buyers and the trust value between buyers and sellers.Besides,it provides users with credit inquiry service for buyers to provide accurate and reliable decision-making information support.The thesis designed a big data trustworthiness calculation scheme successfully,and applied the scheme to the trusted computing of e-commerce data,which is of great significance to the research of the big data.