[Machine Learning] Sparse coding and matrix factorization

---

Dictionary Learning

Dictionary learning is a representation learning method that aims to represent high-dimensional data (such as images, audio, etc.) in a low-dimensional and sparse manner, while trying to retain the key information of the original data. Sparsity means that most of the coefficients are zero and only a few are non-zero. Such a representation can be more efficient, while capturing key information in the data and filtering out noise. Furthermore, sparse representations can also be used for compression, denoising, and other tasks when we have a proper dictionary.

Consider a data point $x$ , we want to pass a "dictionary"$D$ (which is a matrix in which each column is a basis) and a sparse coefficient vector$\alpha$ to approximately represent this data point. Mathematically, we can describe it as$x\approx Da$ .

The reconstruction error is the actual data point $The\; difference\; between\; x$ and the data reconstructed using the dictionary and its corresponding sparse coefficient vector. Mathematically, this error can be expressed as$\Vert x-D\alpha {\Vert }^2$。

Our goal is to find a coefficient vector $\alpha$ to minimize this error, that is:
$$a^{\ast }=\underset{a}{\mathrm{argmin}}\Vert x-D\alpha {\Vert }^2$$

For multiple data points $x_1,x_2,\dots ,x_n$, we can similarly define a global reconstruction error, and the goal is to find a common dictionary $D$ and a sparse representation of each data point. We can stack all the data points into a matrix$X$ , all representations are stacked into a matrix$R$ , and then minimize the error overall:

$$\underset{D\in \mathcal{D},R\in \mathcal{R}}{\mathrm{argmin}}\Vert X-DR{\Vert }_F^2$$

Among them, $D$ and$R$ is a dictionary and a constrained space of representations. For example,$D$ may include column vectors of all unit norms, while$R$ may include all coefficient vectors with sparsity constraints.

$\Vert \cdot {\Vert }_F$is the Frobenius norm, which measures the difference between two matrices. For any matrix $A$ , whose Frobenius norm is defined as:

$$\Vert A{\Vert }_F=\sqrt{\sum _{i=1}^m\sum _{j=1}^n|a_{ij}{|}^2}$$

The goal of dictionary learning is to find an overcomplete dictionary D and a sparse representation R so as to minimize the reconstruction error.

An overcomplete dictionary D means that it has more columns than its rows, that is, the dictionary D contains more atoms (or bases) than the dimensions of the data. This means that it has multiple base elements to choose from to approximate the input data X. The sparsity of R ensures that most elements in R are zero or close to zero. This means that, although D provides many possible base elements, only a few will be activated or used in any particular representation. This not only makes the representation more concise and computationally efficient, but also helps avoid overfitting and makes it more interpretable.

To introduce sparsity, we can modify the optimization problem and add a regularization term:

$$\underset{D\in \mathcal{D},R\in \mathcal{R}}{\mathrm{argmin}}\Vert X-DR{\Vert }_F^2+\lambda \Vert R{\Vert }_p^p$$

First, we define ${\ell }_p$norm. For any vector $\mathbf{v}\in {\mathbb{R}}^n$ , whose${\ell }_p$The norm is defined as:

$\Vert \mathbf{v}{\Vert }_p={\left({\sum }_{i=1}^n|v_i{|}^p\right)}^{\frac{1}{p}}$

这り，$|v_i|$ represents the absolute value of the i-th element in the vector, and p is a positive real number.

As we adjust the value of p, this norm will emphasize different properties of the vector.

${\ell }_0$The norm directly counts the number of non-zero elements, however, using ${\ell }_0$Norm optimization problems are NP-hard. Therefore, in practice, we rarely directly optimize ${\ell }_0$norm. ${\ell }_1$The norm is ${\ell }_0$Optimal convex approximation to the norm and is easier to handle in optimization problems.

When p = 1, we have:

$\Vert v{\Vert }_1={\sum }_{i=1}^n|v_i|$

This is exactly the vector v \mathbf{v}The sum of the absolute values ​​of all elements in$v\; .$Therefore, we call${\ell }_1$The norm is the sum of the absolute values ​​of the vectors.

Consider ${\ell }_1$The normed unit ball is in ${\mathbb{R}}^2$ Performance in space. It is a rhombus with its vertices on the axes. In$R^2$ , the vertices of this rhombus are (1,0), (-1,0), (0,1), and (0,-1).

Now, consider an optimization problem where we wish to minimize some loss function subject to ${\ell }_1$Norm constraints. Assume that the contours of the loss function are elliptical. Our goal is to find the loss function contours with ${\ell }_1$The minimum point where the normed unit spheres intersect.

Since ${\ell }_1$The geometry of the normed unit sphere, whose sharp corners make the first intersection with the loss function contours likely to be at these corners. When the solution lies in $R^2$ at one corner of the rhombus in space, one of the coordinates will be zero, so the solution is sparse.

For higher dimensional spaces $R^n$ , the unit ball will have more corners. These corners lie on coordinate axes or coordinate planes, so solutions at the corners will have zero values ​​in one or more dimensions, resulting in sparse solutions.

Therefore, by adding $\Vert R{\Vert }_1={\sum }_{i,j}|R_{ij}|regularizer$ into our optimization problem, we can encourage the sparsity of R such that it has zero value in many dimensions, where$R_{ij}$Represents the elements of matrix R, located in the i-th row and j-th column.

Regularization and algorithm stability

Stability refers to how much the output of a learning algorithm changes when you make small modifications in the training data (such as changing or removing a sample). Stability is an important indicator of generalization performance because if the algorithm is very sensitive to small changes in the data, then it may be prone to overfitting.

To formalize this concept, consider two training sample sets SSSsum$S$$S^i$ , in which$S^i$ isSSA small variant of$S\; ,\; with\; only\; the\; i-th\; sample\; different.$If the difference between the output produced by an algorithm on these two training sets is small, then we say that the algorithm is stable.

Mathematically, stability can be defined as:

$$|\ell (X,Y,h_S)-\ell (X,Y,h_{S^i})|\le ϵ\left(n\right)$$

Among them, $\ell$ is the loss function,$h_S$sum $h_{S^i}$is the output of the algorithm on the two data sets. If as the number of training samples n increases, the difference $ϵ\left(n\right)$ approaches zero, then the algorithm is uniformly stable.

Generalization error describes the expected error of a model on unseen data. More specifically, it is the difference between the model's average error over the entire data distribution and its error on the training data.

A common bound on generalization error is:

$$R(h_S)-R(h^{\ast })\le 2\underset{h\in H}{\sup }|R(h)-R_S(h)|$$

Among them, $R$ is the expected error over the entire data distribution, and$R_S$is the average error on the training data. $h^{\ast }$ is the best possible model, while$H$ is the hypothesis space of all possible models.

---

Sparse algorithms are not stable

A learning algorithm is said to be stable if slight perturbations in the training data result in small changes in the output of the algorithm, and these changes vanish as the data set grows bigger and bigger.

Algorithmic Stability

We have two different training samples

$S=\{(X_1,Y_1),...,(X_{i-1},Y_{i-1}),(X_i,Y_i),(X_{i+1},Y_{i+1}),...,(X_n,Y_n)\}$

$S^i=\{(X_1,Y_1),...,(X_{i-1},Y_{i-1}),(X_i^{\prime },Y_i^{\prime }),(X_{i+1},Y_{i+1}),...,(X_n,Y_n)\}$

They are different because of only one training example.

An algorithm is uniformly stable if for any example $(X,Y)$

$|\ell (X,Y,h_S)-\ell (X,Y,h_{S^i})\le ϵ(n)|$

Note that $ϵ(n)$ will vanish as n goes to infinity.

Generalisation Error

$R(h_S)-\underset{h\in H}{\min }R(h)=R\left(h_S\right)-R(h^{\ast })=2\underset{h\in H}{\sup }|R(h)-R_S(h)|$

---

Define $\underset{D\in \mathcal{D},R\in \mathcal{R}}{\mathrm{argmin}}\Vert X-DR{\Vert }_F^2$The local minimum $D^{\ast }$ and$R^{\ast }$ , which means we$X\approxeq D^{\ast }{R}^{\ast }$。

But we can also find another matrix pair, such as $D^{\ast }A$sum$A^{-1}{R}^{\ast }$ , they can also be close to$X$（ $A$ is an invertible matrix), because$D^{\ast }{R}^{\ast }=(D^{\ast }A)(A^{-1}{R}^{\ast })$。

So although $D^{\ast }$ and$R^{\ast }$ is the local minimum of the original problem, but we can multiply it by an invertible matrix$A$ and its inverse$A^{-1}$ to find different pairs of matrices that produce the same product, i.e. the same reconstruction error, which is why the problem is non-convex.

Therefore, for the optimization problem of dictionary learning, the alternating optimization method is usually used, that is, fixing one variable and optimizing another variable. This method is also called the coordinate descent method:

1. Initialize dictionary $D$

2. Fixed $D$ , optimize$R$

   Use Lasso or other sparse encoding methods to solve the following problems:

   $$R^{\ast }=\underset{R}{\mathrm{argmin}}\Vert X-DR{\Vert }_F^2+\lambda \Vert R{\Vert }_1$$

   where, $\lambda$ is a regularization parameter that ensuresRRThe sparsity of$R.$

3. Fixed $R$ , optimize$D$

   This step can be complicated because we want to find the DD that minimizes the reconstruction error$D.$ _ A common method is to use gradient-based methods to optimize$D$ , or use other more complex optimization techniques. This question can be written as:

   $$D^{\ast }=\underset{D}{\mathrm{argmin}}\Vert X-DR{\Vert }_F^2$$

   And there may be constraints to ensure that $The\; columns\; of\; D$ are unit normed.

4. Repeat steps 2 and 3 until the value of the objective function changes very little or other stopping criteria are met.

This alternating optimization method can give good results in most practical applications, although it may only find local optimal solutions rather than global optimal solutions. However, due to the non-convex nature of the problem, it is very difficult to find the global optimal solution.

---

K-SVD
$\Vert R_i{\Vert }_0\le k^{\prime }$ , display$R_i$The number of non-zero elements in is less than or equal to k', and $k^{\prime }\ll k$

---

Principal Components Analysis (PCA)

When we only care about the reconstruction error, the goal of dictionary learning becomes finding the best linear combination to represent the data.

In PCA, the dictionary D consists of the first k principal components, and the representation coefficient R is the projection of the data onto these principal components. If we do not enforce the sparsity of R and allow D to consist of the eigenvectors of the covariance matrix of the data, then the reconstruction from PCA is identical to that from dictionary learning. Therefore, PCA can be considered to be a special dictionary learning case that does not consider the sparsity of R, where D is composed of principal components.

The goal of PCA is to find orthonormal bases of the data that maximize the variance of the data. What it produces is a fixed basis set, which means that the representation of each data point is linear and global.

K-means

Without considering the sparsity of the representation R, the goal of dictionary learning is to find the dictionary D and coefficients R in order to approximately reconstruct the data with some vectors in the dictionary.

If we restrict R to be a unit vector of vectors (i.e. only one element is 1 and the rest are 0), then this means that each data point can only be represented by one entry in the dictionary. This can be expressed by the following conditions:

• $\Vert R_i{\Vert }_0=1$，Display$R_i$There is only one non-zero element in .
• $\Vert R_i{\Vert }_1=1$，Display$R_i$The sum of the elements is 1.

In K-means, each data point is associated with a nearest cluster center. This means that the data point $x_i$can be fully represented by its nearest cluster center, while the contribution of other cluster centers is zero. This representation is an extreme form of sparse representation in which there is only one non-zero element (i.e. the nearest cluster center) and the other elements are zero.

When we do not enforce sparsity, the cluster centers of K-means can be viewed as dictionary items in dictionary learning. For a given data point $x_i$, which in K-means is associated with a nearest cluster center, while in dictionary learning it can be viewed as a linear combination of all dictionary terms. However, due to the hard assignment property of K-means, this linear combination becomes very sparse and has only one non-zero element.

Non-negative Matrix Factorisation (NMF)

NMF is a matrix factorization technique where we constrain all elements of the factor matrix to be non-negative. It provides us with a way to clearly interpret and visualize the structure hidden in the data.

When W is treated as a dictionary, NMF can be viewed as a special case of dictionary learning, where both the dictionary and the representation are non-negative. NMF does not directly pursue sparsity, but can increase sparsity constraints through regularization.

Consider the following optimization problem:

$$\underset{D,R}{\mathrm{argmin}}\Vert X-DR{\Vert }_F^2$$

Among them, the constraints are $D\in {\mathbb{R}}_{+}^{d\times \mathbb{k}}\text{, }R\in R_{+}^{\mathbb{k}\times n}$。

We want to find two matrices D and R such that their product is as close as possible to the given matrix X, while satisfying that all elements in D and R are non-negative. This means that all elements of each column of D and R are non-negative. Since each column is non-negative, this means they all lie in the orthogonal quadrant. If we consider the column space, then since every element in the column is non-negative, these column vectors lie in the orthogonal quadrants.

For each column of X, we can think of it as a linear combination of the columns of D (also called dictionary elements or bases), and the coefficients of these linear combinations come from R. When we take two vectors from the orthogonal quadrants and linearly combine them, the elements do not cancel each other out since there are no negative coefficients or elements.

This means they can only capture patterns in the data through additive combinations. This means that decomposition can only "add" features, not "subtract" or negate them. When applied to real-world data such as images, this often means that decomposed features represent discernible parts of the data rather than overall patterns. This part-based representation is often more interpretable, for example, when processing image data, non-negative constraints can make each base ( $D_{:1}$) represents a part or feature of an image rather than a fuzzy combination of the entire image.