Deduplication

RecordManager uses a custom algorithm for deduplication. The algorithm is based on decisions. Deduplication will process, one by one, all records that are marked as needing update. The deduplication process will find a single duplicate for each processed record. If a duplicate is found, it is also marked as processed. When all records are processed, their duplicates will be found incrementally.

Deduplicated records are marked with the same dedup id in the Mongo database. The old method of indexing these records in Solr was to index the dedup id and use Solr's field collapsing to deduplicate the results on the fly. This had some scalability issues, so the new method is to index all the deduplicated records while also creating a merged record for searching. During the Solr query the individual records are filtered out from the results, but the merged records received from Solr are replaced in the result list with individual records according to a defined priority. There is basic built-in support for this in VuFind (see https://vufind.org/wiki/deduplication).

Deduplication Algorithm

This is the flow for finding a duplicate for a record (17 Feb 2014, subject to change):

1. Find candidates for deduplication using ISBNs first, other ID fields (MARC 015, 016 and 024) second and title keys (see below) then as a last resort
2. Discard if the candidate is from the same source or deleted
3. Discard if formats don't match
4. Discard if the candidate has a matching ISBN and a previous ISBN match failed
5. Discard if the candidate has a matching ID field and a previous ID match failed
6. Discard if the candidate has already been deduplicated with a record from the same source as the one being processed
7. Positive match if a common ISBN is found, no further checks performed
8. Positive match if a common ID is found, no further checks performed
9. Discard if both records have ISSN's but none match
10. Discard if publication years don't match
11. Discard if page counts are not within 10 pages
12. Discard if any series ISSN's don't match
13. Discard if any series numberings don't match
14. Discard if the normalized title for either record is empty
15. Discard if levenshtein distance between the titles is equal or more than 10%
16. Discard if authors don't match (see below) or levenshtein distance between them is equal or more than 20%
17. Positive match

String Normalization

The built-in normalization algorithm works best for Finnish material. The following modifications are made to the string to be normalized:

1. Convert accented, diacritical etc. characters to the base character, except ö, ä and å
2. Remove whitespace and all punctuation
3. Convert string to lower case

Title Keys

Keys for title-based search for deduplication candidates are created as follows:

1. Split title to words
2. Take a word starting from the beginning
3. If the word is at least 4 characters long, count it as a word
4. Repeat until there are three words or the resulting string exceeds 25 characters
5. Normalize the string
6. Append normalized first word of author's name

A title key won't be created if the record doesn't have an author.

Author matching

1. Positive match if authors match exactly
2. Discard if either author length is less than 6 characters
3. Positive match if authors match for the length of shorter one
4. Positive match if first words match and at least an initial letter matches

Component Parts

Any separately catalogued component parts (e.g. songs belonging to a CD or articles belonging to a journal) are only marked duplicates if their host records are duplicates and ALL component parts in both host records match.