Sparse expression in recent years SP, ML, PR, CV field in a hot topic, the article can be described as Putian cover, a feast for the eyes. Course boss needs a course syllabus, I take the opportunity to extend the next, there is something on this in the next three introduce nature. Unfortunately, I was not really a hard working person in most cases, this stuff may be full of bug, the update may be intermittent, do you Tell me please forgive me. By the way, ICCV09 has an associated the Tutorial .
Rumor Bowen number of formulas and its popularity is inversely proportional relationship, a formula can disperse 50% of the readers, I finished writing this number nodded formula (a) after to think about nobody cares. Therefore, before the introduction of the sparse representation, let us apply it in computer vision is to be the next exhibition, attracted by the eye.
The first is the image restoration (before it was posted Obama not remember), restore the right of the results by the left image
Followed by a similar image inpainting
Then the image to blur, blur the top left of the input image, lower right image output clear and estimates of camera movement (in fact PSF), the middle of the iterative process are:
And then the object is detected (bicycle), the left side of the input image, an intermediate for the location probability map, the right side is a detection result
Of course, I personally recommend Yi Ma's sparse face , which is a great effect on the fight against noise, such as noise goes on in the figure below image on the left (you can recognize who is not? This method can!)
且说sparse representation这个概念，早在96-97年的时候就火了一把。最著名的大约要数Nature上的某篇文章，将稀疏性加入least square的regularization，然后得到了具有方向特性图像块（basis）。这样就很好的解释了初级视皮层（V1）的工作机理，即对于线段的方向选择特性。几乎同一时期，著名的LASSO算法也被发表在 J. Royal. Statist. Soc B。Lasso比较好的解决了least square (l2 norm) error + l1 norm regularization的问题。然而，这个时候绝大多数人没有意识到（或者没法解决）这l1 norm和稀疏性之间的联系。其实早在这之前，Osher等人提出的Total Variation （TV）已经包含了l1 norm的概念了，只不过TV原本是连续域上的积分形式。（啥？你不知道Osher…想想Level Set吧）
在进入现代的压缩感知、稀疏表示这一课题前，让我们来首先回顾下这一系列问题的核心，即线性方程组
其中矩阵
，通常而言是满秩的。向量
。现在已知
，求解
。学过线性代数的同学可能都会说：这个不难啊，因为
， 故而这个方程组是欠定的，所以有无穷多组解啊，咱还可以算算基础解系啥的…
但是如果我们希望其解尽可能的稀疏：比如
（即中非零元个数）尽可能的小。那么问题就会变得比较微妙了，下图给出了问题的形象示意。
换言之给定m维空间中一组过完备的基
，如何选择最少个数的基向量，重构给定向量
，其严格定义可以写成
时光之轮播快到2003~2004年，Donoho & Elad做了一个很漂亮的证明，如果矩阵
满足某种条件，具体而言：
那么上文提及的0范数优化问题具有唯一的解。这里的
是个比较诡异（请允许我使用这词）的定义：最小的线性相关的列向量集所含的向量个数（吐槽：明白了么，我做TA的时候就被这个问题问倒了）。本来想在这个概念上唠叨两句，后来发现了Elad的一个talk，清晰明了。
即便是唯一性得到了证明，求解这个问题仍然是NP难的。科研的车轮滚滚向前，转眼到了2006年，传奇性的华裔数学家Terrence Tao登场了，Tao和Donoho的弟子Candes合作证明了在RIP条件下，0范数优化问题与以下1范数优化问题具有相同的解：
其中RIP条件，即存在满足某种条件的（与N相关）常数:
RIP条件是对于矩阵列向量正交性的一种衡量（此处咱就不细说了）。其实早在1993年Mallat就提出过Mutual Coherence对于正交性进行度量，并提出了下文还要提及的matching pursuit方法。
实际上以上的1范数优化问题是一个凸优化，故而必然有唯一解，至此sparse representation的大坑初步成型。总结一下：
1. 如果矩阵满足
，则0范数优化问题有唯一解。
2. 进一步如果矩阵满足RIP条件，则0范数优化问题和1范数优化问题的解一致。
3. 1范数优化问题是凸优化，故其唯一解即为0范数优化问题的唯一解。
进一步可以考虑含噪声情况，即
可以得到相似的结果，有兴趣的同学可以查阅相关文献。理论坑只有大牛能挖，但一般人也能挖挖这个优化算法啊，于是SP、ML、CV邻域里都有做这个优化算法的，这个出招可就真是五花八门了。据我所知，大致可以分为三大流派：
1. 直接优化
一般的方法是greedy algorithm，代表有Matching Pursuit, Orthogonal Matching Pursuit
2. 优化
还记得上面提到的LASSO么，这就是它的模型。
3. 如果已知拉格朗日乘子，优化无约束凸优化问题
解这个的方法现在基本上soft thresholding的方法一统天下，常见的有coordinate descent, Bregman Iteration (又是Osher)等
4. 如果未知拉格朗日乘子，优化
这类方法又叫Homotopy，可以认为是3的扩展。核心出发点是objective function是
的分段线性函数。
除此之外，还有利用p范数逐次逼近0范数的方法等等，此处不再赘述。顺道说一句，稀疏表示在不同的领域连名称都不同，搞信号的管这个叫basis pursuit，搞统计的叫l1 regularization….然后，让我们把话题拉回到Nature的那篇文章：如果我们不知道矩阵，只知道一堆向量
。我们应当如何构造，使得在这一字典（矩阵）下的表示最稀疏？类比以上过程，这个问题被称为Dictionary Learning，可以写成以下优化问题：
This thing may relatively trouble, the most critical is that this is not a convex optimization (optimization variables multiplied). Therefore, the general idea is to block descent: First fixed 
, optimization 
(corresponding to the plurality of independent optimization norm); secondly the calculated fixed, optimization , which is a (possibly with constraints) of the least square problem. So again, until the algorithm converges to a (local) minimum. The method of solution to the problem in fact there are three: efficient sparse coding algorithm NIPS 06; K-SVD tsp 06; Online dictionary learning for sparse coding, ICML 09 & JMLR 10. The first two are the batch method, the latter is online, according to the last individual test methods to be much faster than the first two .... The following is the method I use ICML09 color image from the 1200 to train a set of overcomplete group, having good directional characteristics.
Finally, remember those demo beginning of this article it? INRIA made a sparse representation of matlab toolkit spams , although not open source, but its efficiency (most of the time) is the highest existing toolkit of public (with the bottom of the intel MKL), use this tool kit, a few simple lines matlab code can achieve almost all of the above mentioned demo .... we are interested are welcome to try ^ _ ^
Next Issue: wind through the collaborative filter, Candes in 2008 and dug a new pit of matrix completion. Thus, when a norm of a vector norm extended to trace (trace norm) matrix after ... ..
Reproduced in: https: //my.oschina.net/dake/blog/196653