Research On Big Data Quality Evaluation Based On MMTD

In recent years,with the rapid development of the Internet,Internet of things,cloud computing technology,data has grown at a explosive speed,and big data has become a hot research at home and abroad.As the big data contains a huge value,so it attracts great concern of the government and enterprises.However,the quality of the data determines the use of big data.Only in high-quality data to get effective and accurate information.However,in the big data environment,the data type is numerous,and the growth rate is amazing and the huge amount of data can not meet the demand of data usage.Therefore,in the high quality data environment for data analysis and decision making,the data quality of big data It is important to carry out effective analysis and evaluation.In this paper,we first introduce the method of medium mathematics and the method of medium truth degree(MMTD).On this basis,the intermediate logic is used to qualitatively analyze and quantitatively analyze the big data data quality dimension.The main work is as follows:(1)We study the canonical representation of structured data,unstructured data and semi-structural data in big data environment.According to the 3V characteristic of big data,the evaluation dimension of data validity in big data environment is analyzed,and the main dimensions of data validity in data environment are given: definition of data integrity,data correctness and data compatibility,The intermediate dimension of data validity is analyzed qualitatively by using the method of intermediary logic,and the measurement model of data validity of big data based on MMTD is established.(2)The information quantity measurement of different data types in big data is studied,and several typical methods of structured,semi-structured and unstructured data are given respectively.(3)On the Hadoop distributed platform,a big data quality evaluation platform based on SSM framework is designed and implemented.The platform uses interface programming and combines other function modules,so it enhances the system scalability.All the functions of the entire system are divided into separated modules,so we only need to modify the corresponding module.This enhances system maintenance and provides evaluation rules set.You can use the defined rules or custom rules on the platform and it increases system usability.On the platform,the rationality and scientificity of the evaluation model proposed in this paper are verified.