A Deep Learning Approach for Event-Driven Duplicate Detection

Abstract

Clean datasets are paramount in many industrial applications as they are the foundation for the analytic processes and services offered to customers. In this regard, duplicates are a common problem when multiple entries represent the same real-world entity. Duplicate detection is the process of identifying these entries and has been intensively studied in research and practice. However, existing methods hardly take into account the underlying real events that lead to duplicates. We argue that duplicates resulting from such events, e.g., relocation of a person, exhibit specific characteristics and, hence, can hardly be identified by existing approaches, which commonly focus on high similarity with respect to attribute values. To address this issue, we propose an event-driven duplicate detection approach based on deep learning that does not rely on limiting assumptions and can detect duplicates arising from multiple real-world events. Specifically, we combine neural networks with classical string distance metrics to classify duplicate records by learning the subtle differences and similarities between pairs of records. We demonstrate our approach's practical applicability and effectiveness using a real-world scenario by analyzing duplicates of an address book. The evaluation shows that our approach is reliable and useful for decision support, while outperforming existing approaches for duplicate detection.