| Although the ever-growing data in various fields has increasingly become a critical asset in the information society,low-quality data has greatly reduced the availability of data and led to serious mistakes in business decisions.Deduplication means that all descriptions of the same entity are consistent across single or multiple data sources.High-quality data is of deduplication.To this end,technologies for deduplication of relational data are studied from two levels:attribute value level and record level.Existing attribute-value-level repair methods merely rely on the assumption that the optimal repair solution has the minimum repair cost,ignoring the relationship among attributes.Various types of data constraints for detecting inconsistent attribute values,could not give accurate information on repair values of inconsistent attribute values.Most record-level repair approaches rely on textual similarity,and less on the semantic information implied by attribute values and less on the importance of words and phrases.Different repair algorithms have their own advantages and result in numerous inconsistent pairs.How to utilize the results of existing approaches to resolve inconsistent pairs or to provide repair results that meet quality requirements has not been comprehensively studied.In order to overcome the above-mentioned weaknesses,the problems of repairing inconsistent attribute values,entity resolution,reconciliation of entity resolution and quality control guarantee for the reconciliation of entity resolution are studied in depth.The main contributions are as follows:1.A repair approach for inconsistent attribute values based on the possible world model was proposed to compensate for the limitations of the traditional algorithm based on the minimum repair cost.Traditional repair algorithms only consider repair cost of attribute values,and ignore the relationship among non-functional-dependency and functional-dependency attributes.Most of these repair algorithms assume that the optimal solution has a minimum repair cost,but the solution with the minimum cost is not necessarily optimal.A repair method for inconsistent attribute values based on the possible world model analyzes not only the repair cost,but also the correlation of attribute values,where the correlation refers to utilizing the k nearest neighbors model to estimate the correctness of the candidate repair values.Firstly,each candidate repair solution is modeled as a possible world instance via the possible world model.Secondly,the correctness of the candidate repairing value and the repairing solution are quantified in terms of repair cost and attribute correlation.Finally,the optimal solution is obtained by heuristically searching for the possible world instance with the highest probability.Experimental results show that the proposed approach can effectively repair the inconsistent attribute values,compared with the traditional method.2.An approach for entity resolution based on the multilayer perceptron model was proposed to enhance resolution results via utilizing the semantic information based on the semantic vector space model,the importance of words or phrases and an improved multilayer perceptron model.Considering the semantic information of attribute values and the importance of words or phrases that describe entities,benefit entity resolution,and existing approaches for entity resolution less consider utilizing novel techniques in neural networks,an approach for entity resolution based on the multilayer perceptron model was proposed.Our approach utilizes the semantic vector space model,TF.IDF weighting scheme and the similarity metric to extract three types of features of record pairs(i.e.,semantic similarity,the importance of words or phrases,and textual similarity features)which are the input to the multilayer perceptron model.For entity resolution tasks,matching record pairs and non-matching record pairs are severely imbalanced.To alleviate the impact of such imbalance,a hybrid machine learning model for constructing a relatively balanced training set is proposed,and the improved multilayer perceptron model has the optimization objective function with weight parameters.A construction algorithm for optimal feature combination is proposed to improve the resolution.Experimental results show that the proposed approach can significantly improve the results.3.An incremental k neighbor influence approach and a factor graph approach for reconciling entity resolution were proposed,in order to take full advantages of existing entity resolution algorithms.Although various entity resolution algorithms are proposed,these algorithms have their own advantages when dealing with different datasets.Pre-labeled data for a new entity resolution task,is not directly available(i.e.,labeled data is absent),which makes it difficult to evaluate and select the best one among existing algorithms.However,after executing these algorithms on the same entity resolution task,a large number of inconsistent pairs are generated.The greater difference among selected algorithms,the greater number of selected algorithms,the greater number of inconsistent pairs.According to the matching relationship among records of inconsistent pairs,the reconciliation scenario of entity resolution can be divided into two categories:Clean-Clean and Non-Clean-Clean.To this end,an incremental k neighbor influence reconciliation approach for the Clean-Clean scenario was proposed.This approach does not rely on any labeled data,and reconciles inconsistent pairs according to the constraint of the Clean-Clean scenario and the accumulative entity reconciliation information of inconsistent pairs,which quantifies the influence of the matching status of consistent pairs over inconsistent pairs and the relationship between records of inconsistent pairs.Since the proposed approach assumes that inconsistent pairs belong to the Clean-Clean scenario,in which a record has at most one other record that matches each other,the proposed approach cannot meet the actual scenario requirement in the Non-Clean-Clean scenario.To this end,a reconciliation approach based on factor graph for the Non-Clean-Clean scenario was proposed.For an inconsistent pairp~c,the proposed approach first extracts indicative features that are related to the matching status ofp~c via utilizing textual similarity,kernel density estimation,and graph connectivity,and formalizes these features as factor functions of factor nodes in the factor graph via exponential functions.Then the weight of each factor is estimated based on the maximum likelihood estimation.Finally,whetherp~c is matched or not,is inferred according to the distribution represented by the factor graph.Experimental results show that these proposed approaches outperform individual methods(i.e.,existing resolution algorithms)and baseline methods.4.A quality control approach for the reconciliation of entity resolution was proposed,in order to ensure that the reconciling result satisfied a given precision threshold with the maximum recall.Existing reconciliation methods fail to provide a reconciling result under the quality requirement(e.g.,precision).To this end,the quality control problem for the reconciliation of entity resolution is formalized.Inspired by the feedback of the reinforcement learning,the quality control approach for the reconciliation of entity resolution is proposed.Firstly,the environmental side employs the outlier detection and quality estimation modules,to effectively estimate the matching probability of the inconsistent pair and quality measures of the reconciling result in turn.Secondly,according to the feedback information of the environment,the agent searches for a reconciling solution,that meets the precision requirement and has the highest recall through a binary search algorithm,and updates the reconciling result.Finally,the quality estimation module on the environmental side is optimized using updated consistent result.Such approach iterates until the estimated precision on the environmental side satisfies the given precision threshold.Experimental results show that the proposed approach can effectively solve the quality control problem of the reconciliation of entity resolution,and outperform label-based or probability-based methods. |
