Googlers Talk about Dupe Detection and How They Chooses Canonical Page

In the latest episode of Search Off the Record Podcast

Some highlights:

1. Google essentially calculates multiple kinds of checksums about textual content of the page and then compare to checksums, because it’s much easier/less resource-intensive to do that than comparing perhaps the actual text content on all duplicate variations. This is set up to catch both duplicates and near-duplicate pages.
2. They put these into a dupe cluster.
3. Then they use about twenty signals to decide which page to pick as canonical from a dupe cluster.