K-means clustering algorithm analysis

Algorithm brief description

K-means algorithm principle

We assume that a given data sample X contains n objects $X = \left \{ X_{1},X_{2},X_{3},...,X_{n}\right \}$ , each of which has attributes of m dimensions. The goal of the K-means algorithm is to cluster n objects into designated k clusters based on the similarity between objects. Each object belongs to and only belongs to one cluster with the smallest distance to the cluster center. For the K-means algorithm, you first need to initialize k cluster centers $\left \{ C_{1}, C_{2},C_{3},...,C_{k}, \right \},1<k\leq n$ , and then calculate the Euclidean distance from each object to each cluster center, as shown in the following formula:

$dis(X_{i},C_{i}) = \sqrt{\sum_{t=1}^{m} (X_{it}-C_{jt})^{2}}$

Here $X_{i}$ represents the i-th object $1<i<n$ , $C_{i}$ represents the j-th cluster center $1<j<k$ , $X_{it}$ represents the t-th attribute of the i-th object, $1\leq t\leq m$ , $C_{jt}$ represents the t-th attribute of the j-th cluster center.

Compare the distance between each object and each cluster center in turn, assign the object to the cluster with the nearest cluster center, and obtain k clusters. The $\left \{S_{1},S_{2},S_{3},...,S_{k}\right \}$ kmeans algorithm defines the prototype of the cluster, and the cluster center is the cluster. The mean value of all objects in each dimension is calculated as follows:

$C_{t} = \frac{\sum_{X_{i}\in S_{i}}^{}X_{i}}{\left | S_{l} \right |}$

In the formula, $C_{l}$ represents the l-th cluster center, $1\leq l\leq k$ , represents the number of objects in the $\left | S_{l} \right |$ l-th cluster, represents the i-th object in the l-th cluster, $X_{i}$ $1\leq i\leq \left | S_{i} \right |$

Algorithm implementation process

Randomly set K points in the feature space as the initial clustering center.
For each other point, the distance to K centers is calculated. For unknown points, the nearest cluster center point is selected as the label category.
Then after facing the marked cluster center, recalculate the new center point (average value) of each cluster
If the calculated new center point is the same as the original center point (the center of mass no longer moves), then it is over, otherwise the second step is repeated.

core code

Handwritten implementation of K-means algorithm:

import numpy as np
import random
import matplotlib.pyplot as plt
plt.rcParams['font.sans-serif'] = ['SimHei']
plt.rcParams['axes.unicode_minus'] = False
"""
手写实现Kmeans
"""
data = np.genfromtxt("classes.txt", delimiter='\t')
X = data
K = 5
colors = ['r', 'g', 'b', 'c', 'm', 'y', 'k']
max_iterations = 10000
random.seed(100)
def kmeans(data, K, max_iterations):
    initial_centers = random.sample(list(data), K)
    centers = initial_centers

    for iteration in range(max_iterations):
        clusters = {i: [] for i in range(K)}
        for point in data:
            distances = [np.linalg.norm(point - center) for center in centers]
            cluster_index = np.argmin(distances)
            clusters[cluster_index].append(point)

        new_centers = [np.mean(clusters[i], axis=0) for i in range(K)]
        if np.all(np.array_equal(centers[i], new_centers[i]) for i in range(K)):
            break
        centers = new_centers

    return centers, clusters

final_centers, final_clusters = kmeans(X, K, max_iterations)
for i in range(K):
    cluster = np.array(final_clusters[i])
    plt.scatter(cluster[:, 0], cluster[:, 1], c=colors[i], label=f'簇 {i + 1}')

centers = np.array(final_centers)
plt.scatter(centers[:, 0], centers[:, 1], c='k', marker='x', s=100, label='簇中心')

plt.xlabel('高度')
plt.ylabel('宽度')
plt.legend()
plt.show()

Call the K-means algorithm of the sklearn package:

import numpy as np
from sklearn.cluster import KMeans
import matplotlib.pyplot as plt
plt.rcParams['font.sans-serif'] = ['SimHei']
plt.rcParams['axes.unicode_minus'] = False
"""
调用sklearn库的Kmeans算法
"""
data = np.genfromtxt("classes.txt", delimiter='\t')
X = data
K = 3
num_experiments = 5
colors = ['r', 'g', 'b', 'c', 'm', 'y', 'k']
for i in range(num_experiments):
    kmeans = KMeans(n_clusters=K, init='k-means++', random_state=i)
    kmeans.fit(X)
    print(f"实验 {i + 1} - 初始中心: {kmeans.cluster_centers_}")

kmeans = KMeans(n_clusters=K, init='k-means++', random_state=0)
kmeans.fit(X)

labels = kmeans.labels_

clustered_data = {i: [] for i in range(K)}
for i, label in enumerate(labels):
    clustered_data[label].append(X[i])

for i in range(K):
    cluster = np.array(clustered_data[i])
    plt.scatter(cluster[:, 0], cluster[:, 1], c=colors[i], label=f'簇 {i + 1}')

centers = kmeans.cluster_centers_
plt.scatter(centers[:, 0], centers[:, 1], c='k', marker='x', s=100, label='簇中心')

plt.xlabel('高度')
plt.ylabel('宽度')
plt.legend()
plt.show()

The algorithm process of handwriting K-means:

Randomly select initial centers: Randomly select K data points from the data set as initial centers.
Calculate distance: For each data point, calculate the distance between it and the current K centers.
Assign data points: Assign each data point to the set corresponding to the nearest center.
Update centers: Update the center point of each set to the mean in the set.
Repeat steps 2-4: until centers no longer change, or the maximum number of iterations is reached.
Return centers and clusters: Return the final centers and clusters.

Experimental results and analysis

Use python handwriting to achieve the K-means algorithm effect (assuming K=3):

The effect of using the K-means algorithm in sklearn (assuming K=3):

Use python handwriting to achieve the K-means algorithm effect (assuming K=5):

Here, Python handwriting is used to implement the K-means algorithm, and it is compared with the K-means algorithm in the scikit-learn library. It was found that the effect of the K-means algorithm implemented by handwriting is similar to that of the K-means algorithm in the scikit-learn library, and both can aggregate data points well.

Conclusion and experience

The K-means algorithm is a commonly used clustering algorithm that can be used to group data points into K clusters. In this experiment, we used 600 images from the VOC dataset and annotated the bounding box of each image as a data point. The K-means algorithm is used here to aggregate these data points into K clusters.

classes.txt file: