Data Deduplication with Text Similarity Algorithms: An Application in a Hotels Database

Deduplication is a great need in the world of data that is growing rapidly and uncontrolled. Data deduplication is a technique used to eliminate duplicate copies of data in the database or to identify and describe them with an Id. Finding a hotel with a different name and ID in the records in the database of hotels with name, address, agency, and price information does not allow comparison. Competitor analysis can be done completely, knowing competitor price situations and market follow-up is possible by singularization of all these hotel names. The purpose of this study is to identify identical hotels with a single identification number by matching hotel names and to obtain singular data. Various text similarity algorithms are used for deduplication. In this study, Text Similarity algorithms; Hotels were compared in the database over hotel names. Distance edit based similarity, token based similarity, sequence based similarity algorithms and fuzzy string matching algorithms were examined. As a result of the study, a hybrid model in which Fuzzy String Matching algorithm and Jaro Winkler distance are used together is proposed. Model; In the comparison made over the test data set, it gave 94% Accuracy results in the easy data set and 83% Accuracy in the difficult data set.