Optimization: Modeling, Algorithms and Theory (Optimization Modeling - 2)

3.10 K-means clustering

Cluster analysis is a fundamental problem in statistics and has important applications in machine learning, data mining, pattern recognition and image analysis. Clustering is different from classification. In clustering problems, we only know the data points themselves, but not the specific labels of each data point. The task of cluster analysis is to classify some unlabeled data points according to a certain degree of similarity, and then learn their inherent category features from the data points themselves.

Given ppnnin$p-\; dimensional\; space$$n$ data points$a_1,a_2,\cdots ,a_n$, assuming that the similarity between two data points can be measured by their Euclidean distance, our goal is to classify similar points into a class while distinguishing dissimilar points. For simplicity we assume that classes The number of is known, so we might as well record it as $k$ , and the same data point only belongs to one class, so the clustering problem is to find$k$ disjoint non-empty sets$S_1,S_2,\cdots ,S_k$，使得
{ a 1 , a 2 , ⋯   , a n } = S 1 ∪ S 2 ∪ ⋯ ∪ S k \{a_1,a_2,\cdots,a_n\}=S_1\cup{S_2}\cup{\cdots}\cup{S_k} { a1​,a2​,⋯,an​}=S1​∪S2​∪⋯∪Sk​
And the distance between similar points must be close enough. In order to mathematically describe "the distance between similar points is close enough", we define the sum of squared distances within the group as
$$W(S_1,S_2,\cdots ,S_k)=\sum _{i=1}^k\sum _{a\in S_i}||a-c_i|{|}^2\text{\hspace{1em}( 3.10.1 )}$$
here$c_i$Part $The\; center\; of\; the\; i$ -category data point. Note that the question assumes that each category is non-empty.
After defining the clustering criteria, you can recommend an optimized model. We want to find a clustering method that minimizes the sum of squared distances within the group, that is,
min ⁡ S 1 , S 2 , ⋯ , S k ∑ i = 1 k ∑ a ∈ S i ∣ ∣ a − ci ∣ ∣ 2 s . t . { a 1 , a 2 , ⋯ , an } = S 1 ∪ S 2 ∪ ⋯ ∪ S k S j ∩ S j ≠ ∅ , ∀ i ≠ j (3.10.2) \min_{S_1,S_2, \cdots,S_k}\sum_{i=1}^k\sum_{a{\in}S_i}||a-c_i||^2 \\ st {\quad} \{a_1,a_2,\cdots,a_n \}=S_1\cup{S_2}\cup{\cdots}\cup{S_k} \\ S_j \cap S_j {\not=}\varnothing,\forall{i {\not=}j} \tag{3.10. 2}S1​,S2​,⋯,Sk​min​i=1∑k​a∈Si​∑​∣∣a−ci​∣∣2s.t.{ a1​,a2​,⋯,an​}=S1​∪S2​∪⋯∪Sk​Sj​∩Sj​=∅,∀i=j( 3.10.2 )
The independent variable of problem (3.10.2) is the division method of the data point set, which seems difficult to handle. Therefore, it is necessary to write the problem in a familiar form. Next, two matrix expressions of the problem are given. , they are equivalent.

1. Equivalent expression of K-means clustering 1

In the original clustering problem, the sum of squared distances within the group is defined as (3.10.1), that is, $S_i$points in to their center points $c_i$The sum of squares, in fact, select the center point $c_i$It is not necessary to use it as a reference point. We can choose other points $h_i$as a reference to calculate the intra-group distance (in fact, this $h_i$By optimizing the final center point of the representation), the sum of squared distances within the group can be generalized to
$$W(S_1,S_2,\cdots ,S_k,H)=\sum _{i=1}^k\sum _{a\in S_i}||a-h_i|{|}^{2where}$$ H ∈ R k × p H{\in}R^{k \times p
}$H\in R^{k\times p}$ (a point of k classes (dimension p)) and$The\; vector\; of\; row\; i$ is$h_i^T$, in order to express the clustering method $S_1,S_2,\cdots ,S_k$, a natural idea is to use a vector ${\varphi }_i\in R^k$ to represent$a_i$所处的类别
$${({\varphi }_i)}_j=\{\begin{array}{c}1，a_i\in S_j\\ 0，a_i\notin S_j\end{array}$$
The clustering problem is equivalently described as
$$\begin{gathered} \underset{\varphi ,H}{\min }||A-\Phi H|{|}_F^2 \\ s.t.\Phi \in R^{n\times k},\; only\; one\; element\; in\; each\; row\; is1,\; and\; the\; rest\; are0 \\ H\in R^{k\times p}\text{\hspace{1em}( 3.10.3 )} \end{gathered}$$ Φ {\Phi}
here$\Phi$ 's$The\; vector\; of\; row\; i$ is${\varphi }_i^T$

Next, it will be explained that 3.10.3 is equivalent to the original problem 3.10.2. For this purpose, only the reference point set $The\; method\; of\; selecting\; H$ is actually the midpoint of each category. When$When\; Phi$ ,iiThe sum of squared distances within the group of points of type i is$\sum$
$$\sum _{a\in S_i}||a-h_i|{|}^{2According}$$
to the properties of quadratic functions, when$h_i=\frac{1}{n}{\sum }_{a\in S_i}When\; a$ , the sum of squared distances within the group is the smallest,
so$h_i$It will definitely be optimized into iithe center point of class$i$

There are two reasons why we introduce problem (3.10.3):
(1) the form is simple, and the "division method" of the independent variable that is not easy to handle is converted into a matrix
(2) it can be regarded as a matrix decomposition problem, which facilitates us to design algorithms

2. Equivalent expression 2 of K-means clustering

The second equivalent expression of K-means clustering takes advantage of the properties of column orthogonal matrices. This expression is more concise than problem (3.10.3). First define $I_{S_t},1\le t\le k$ is$A\; vector\; in\; n-$ dimensional space where each component takes the value 0 or 1, and
$$I_{S_j}(i)=\{\begin{array}{c}1，a_i\in S_t\\ 0，a_i\notin S_t\end{array}$$
It can be proved that $t$ class$S_t$The sum of the squared distances from each point to its center point can be written as $\frac{1}{2n{\_}_t}Tr(D{I}_{S_t}{I}_{S_t}^T)$，其中$D\in R^{}$The elements of${}^n$${}^{\times }$${}^n$$D_{ij}=||a_i-a_j|{|}^2$ . This shows that$S_t$The sum of squared distances from each point to the center point is equal to $S_t$The sum of the squares of the distances between all pairs of points in is related. Therefore, we transform problem (3.10.2) into
min ⁡ S 1 , S 2 , ⋯ , S k 1 2 T r ( DX ) s . t . X = ∑ t = 1 k 1 nt IS t IS t TS 1 ∪ S 2 ∪ ⋯ ∪ S k = { a 1 , a 2 , ⋯ , an } S i ∩ S j = ∅ , ∀ i ≠ j (3.10.4) \min_{S_1,S_2,\cdots,S_k}\frac{1}{2}Tr(DX) \\ st {\quad}X={\sum}_{t=1}^k\frac{1} {n_t}I_{S_t}I_{S_t}^T \\ S_1{\cup}S_2{\cup}\cdots{\cup}S_k=\{a_1,a_2,\cdots,a_n\} \\ S_i{\ cap}S_j={\varnothing},\forall{i{\not=}j}\tag{3.10.4}S1​,S2​,⋯,Sk​min​21​Tr(DX)s.t.X=∑t=1k​nt​1​ISt​​ISt​T​S1​∪S2​∪⋯∪Sk​={ a1​,a2​,⋯,an​}Si​∩Sj​=∅,∀i=j( 3.10.4 ) Prove XX
for semi-positive definite$X$进行分解$X=Y{Y}^T,Y\in R^{n\times k}$ , we can further obtain the following matrix optimization problem (here$I$ is$n$维向量且分量全为1）
$$\begin{gathered} \underset{Y\in R^{n\times k}}{\min }Tr\left(Y^{T}DY\right) \\ s.t.Y{Y}^{T}I=I, \\ Y{Y}^T=I_k，Y\ge 0\text{\hspace{1em}( 3.10.5 )} \end{gathered}$$
Obtain the solution$Y{Y}^T$ corresponds to the solution of (3.10.4) (to be honest, I don’t understand this part. It is quite simple if Kmeans does it directly)

3.11 Total variation model in image processing

This section may require students with a basic knowledge of image processing to understand it. Anyway, I’m currently confused.

Let’s briefly introduce the image processing model based on total variation (TV), which is defined in the region $Oh\subset R^{}$Functionu ( x , y ) u(x,y)${}^{of\; 2}$$u(x,y)$ , its total variation∣
$$||u|{|}_{TV}={\int }_{}||Du||dx\text{\hspace{1em}( 3.11.1 )}$$
where gradient operator$D$满足：
$$From\_=(\frac{\partial u}{\partial x},\frac{\partial u}{\partial y})$$
here,$||Du||$ can be$l_1$范数，即
$$||Du|{|}_1=|\frac{\partial u}{\partial x}|+|\frac{\partial u}{\partial y}|It$$
is said that the corresponding total variation is anisotropic.
If$l_2$范数，即
$$||Du|{|}_2=\sqrt{(\frac{\partial u}{\partial x}{)}^2+(\frac{\partial u}{\partial y}{)}^2}$$
The corresponding total variation is said to be isotropic.

令$b(x,y)$ is the observed noisy image,$A$ is a linear operator, in the classic$Rudin-Osher-Under\; the\; Fatemi\left(ROF\right)$ model, the image denoising and deblurring problems can be written as
$$\underset{u}{\min }||Au-b|{|}_{L2\_}^2+\lambda ||u|{|}_{TV}\text{\hspace{1em}( 3.11.2 )}$$
Here, the domain is$\Omega$ function$f$ L$L_2$The norm is defined as
$$||f|{|}_{L2}=({\int }_{}{f}^{2}dx{)}^{\frac{1}{2}}$$
If $A$ is the unit operator or blur operator, then the above models correspond to image denoising and deblurring respectively. The first item of the objective function is the data fidelity item, that is, the reconstructed photos must be compatible with the existing collected information. The second term is the regular term, which is used to ensure that the steps of the reconstructed image are sparse, or that the reconstructed image is similar to a piecewise constant function.

The discrete format of the continuous model (3.11.2) is given below. For simplicity, it is assumed that the area $Oh=[0,1]\times [0,1]$ and discretize it as$n\times n$ grid, then the grid point$(\frac{i}{n},\frac{j}{n})$ corresponds to the indicator$(i,j)$ , we set the image$u$ represents the matrix$U\in R^{n\times n}$ , its element$u_{i,j}$Corresponding indicator $(i,j)$ Use forward difference discrete gradient operator$For$ ( DU ) i , j = ( ( D 1 U ) i , j , ( D 2 U ) i , j ) ($(DU{)}_{i,j}=((D_{1}U{)}_{i,j},(D_{2}U{)}_{i,j})$，且有
$$(D_{1}U{)}_{i,j}=\{\begin{array}{c}{u}_{i+1,j}-u_{i,j},i<n\\ 0，i=n\end{array}$$
$$(D_{2}U{)}_{i,j}=\{\begin{array}{c}{u}_{i,j+1}-u_{i,j},j<n\\ 0，j=n\end{array}$$

Here for $i,j=The\; point\; n$ uses the Neumann boundary condition$\frac{\partial u}{\partial n}=0$且有$DU\in R^{n\times n\times 2}$ , then the discrete total variation can be defined as
$$||U|{|}_{TV}=\sum _{i\le i,j\le n}||(DU{)}_{i,j}||\text{\hspace{1em}( 3.11.3 )}$$
where∣$||\cdot ||$ can be$l_1$or $l_2$Norm
For any $U,V\in R^{n\times n\times 2}$ ,let infinitesimal
$$<U,V>=\sum _{1\le i,j\le n,1\le k\le 2}{u}_{i,j,k}{v}_{i,j,k}$$
Then according to the definition, the discrete divergence operator $G$需满足
$<U,GV>=-<DU,V>,\forall U\in R^{n\times n},V\in R^{n\times n\times 2}$
(I’m not sure what the divergence operator does here)
remember$w_{ij}=(w_{i,j,1},w_{i,j,2}{)}^T,W=(w_{ij}{)}_{i,j=1}^n\in R^{n\times n\times 2I}$
have
$$(GW{)}_{ij}=D_{i,j,1}+D_{i,j,2}\text{\hspace{1em}(3.11.4)}$$
其中
$$D_{i,j,1}=\{\begin{array}{c}{w}_{i,j,1}-w_{i-1,j,1},1<i<n\\ w_{i,j,1}，i=1\\ -w_{i,j,1},i=n\end{array}$$
$$D_{i,j,2}=\{\begin{array}{c}{w}_{i,j,1}-w_{i,j-1,1},1<j<n\\ w_{i,j,1}，j=1\\ -w_{i,j,1},j=n\end{array}$$
After applying appropriate discrete format processing, we can obtain the discrete linear operator $A$ and image$B$ (here the notation for the continuous case is used, but$The\; meaning\; of\; A$ is completely different), so the discrete problem
$$\underset{U\in R^{n\times n}}{\min }||AU-B|{|}_F^2+\lambda ||U|{|}_{TV}\text{\hspace{1em}(3.11.5)}$$

In practice, in addition to considering $In\; addition\; to\; the\; ROF$ model, we also consider a deformation of it,$TV-L^1$ model, separate dispersive form
$$\underset{U\in R^{n\times n}}{\min }||AU-B|{|}_1+\lambda ||U|{|}_{TV}\text{\hspace{1em}( 3.11.6 )}$$
One benefit of the above model is that it can better handle non-Gaussian noise situations, such as salt and pepper noise, etc.

3.12 Wavelet model

The wavelet model written in the book is abstract, and many variables are directly mentioned without telling their meaning, so they are skipped here. . I'll take a closer look if I use it in the future.