Online news clustering
- Cluster one million articles, as showed below, and save the cluster centroids for all clusters.
- Periodically, for each new article, use canopy clustering to assign it to the cluster whose centroid is closest, based on a very small distance threshold. This ensures that articles on topics that occurred previously are associated with that topic cluster and are shown instantly on the website. These documents are removed from the new document list.
- The leftover documents that aren’t associated with any old clusters form new canopies. These canopies represent new topics that have appeared in the news and that have little or no match with any articles from the past.
- Use the new canopy centroids, cluster the articles that aren’t associated with any of the old clusters, and add these temporary cluster centroids to the centroid list.
- Less frequently, execute the full batch clustering to recluster the entire set of documents. While doing so, use all previous cluster centroids as input to the algorithm so that clustering achieves faster convergence.