| Entity matching (EM) identifies data instances that refer to the same real-world entity, such as (David Smith, UWMadison) and (D. M. Smith, UWM). This problem has been a long-standing challenge in data management and will be even more critical in the age of data science. Most current EM works focus only on developing matching algorithms. Going forward, we argue that far more efforts should be devoted to building EM systems , in order to significantly advance the field.;We discuss the limitations of current EM systems, then present as a solution Magellan, a new kind of EM systems. Magellan is novel in four important aspects. (1) It provides how-to guides that tell users what to do in each EM scenario, step by step. (2) It provides tools to help users do these steps; the tools seek to cover the entire EM pipeline, not just matching and blocking as current EM systems do. (3) Tools are built on top of the data analysis and Big Data stacks in Python, allowing Magellan to naturally integrate into the Python ecosystem of data science tools, and to borrow a rich set of capabilities in data cleaning, IE, visualization, learning, etc. (4) Magellan provides a powerful scripting environment to facilitate interactive experimentation and quick "patching'' of the system.;While promising, realizing the above novelties raises major challenges. First, it turns out that developing effective how-to guides, even for very simple EM scenarios such as applying supervised learning to match, is already quite difficult and complex. Second, developing tools to support these guides is equally difficult. In particular, current EM work may have dismissed many steps in the EM pipeline as engineering. But here we show that many such steps (e.g., loading the data, sampling and labeling, debugging, etc.) do raise difficult research challenges. Finally, while most current EM systems are stand-alone monoliths, Magellan is designed to be placed within an "ecosystem" and is expected to "play well" with others (e.g., other Python packages). We distinguish this by saying that current EM systems are "closed-world systems" whereas Magellan is an " open-world system", because it relies on many other systems in the eco-system in order to provide the fullest amount of support to the user doing EM. It turns out that building open-world systems raises non-trivial challenges, such as designing the right data structures and managing metadata, among many others.;In this dissertation we discuss how we have addressed these challenges, built, and open sourced Magellan. As far as we can tell, Magellan is the most comprehensive open-source EM system today (August 2018), in terms of the number of features it supports. Magellan has been successfully used in several domain science projects in academia and projects in industry. We describe these "in the wild" experience with Magellan, as well as extensive experiments in controlled settings. Finally, we discuss lessons learned and many possible future research directions. Beside concrete contributions, this dissertation also introduces a new template of research, system development, and education for EM, with many potential impacts. |
